When Is AI Personalisation Still Helpful—and When Does It Become Surveillance?

Artificial intelligence is increasingly used to help people find the right information, identify a suitable expert or adapt communication to a particular user.

At first glance, this appears to be useful personalisation. The problem begins when the system does more than adapt a response and starts building a profile of the user.

Communication adaptation is not merely a technical feature

When users choose for themselves whether they want concise, detailed or more instructional answers, the choice is relatively clear and controllable. The situation is different when the system infers such a preference from text, behaviour, communication history or other digital traces.

At that point, the system is no longer merely applying a user setting. It is creating a representation of the person. It may infer, for example, that the user:

• needs greater reassurance,
• prefers detailed explanations,
• makes decisions with uncertainty,
• requires more authoritative guidance.

Such information may be useful for providing better support. It may also be sensitive, inaccurate or open to misuse.

A hidden profile changes the relationship between people and organisations

The central question is therefore not merely: Does the personalisation work? A more important question is: Is this form of personalisation justified?

Within organisations, an AI system is often not a neutral tool. It is used in environments characterised by imbalances of power:

• between an employer and an employee,
• a school and a student,
• a service provider and a user,
• an institution and a client.

A user may formally consent to data processing while not genuinely feeling able to refuse. They may fear that declining personalisation will result in poorer support, make them appear difficult or restrict their access to certain benefits offered by the system.

Proportionality of adaptation

One of the key principles is proportionality. When the same result can be achieved through a simpler and less intrusive method, the system should not create a psychological profile.

When users can simply select their preferred response style, there is no need to infer psychological characteristics. If a basic choice between concise, detailed or step-by-step is sufficient, there is no reason to analyse their behaviour over an extended period.

And when the adaptation concerns only the tone of a response, there is no justification for creating a sensitive profile of the person.

Users must be able to see and correct the profile

Another essential condition is the ability to object and seek correction. It is not enough to claim that the system is transparent. Users should be able to discover that an inferred profile is being used and should be able to change, challenge or disable it.

When a system classifies a user as someone who needs simplified instructions, authoritative guidance or greater reassurance, the inference may be wrong. It may reflect a temporary situation, an incorrect interpretation or the demands of a particular task.

Such a profile should not become a permanent label.

Psychological adaptation should therefore not be treated as a truth about the person. It should be understood only as a cautious, temporary and correctable hypothesis about how to support them more effectively in a particular situation.

Where the red line should be drawn

There are situations in which inferred psychological adaptation should not be used at all, particularly when:

• users cannot meaningfully refuse profiling,
• refusal results in a poorer service,
• the profile influences how a person is evaluated,
• it affects employment opportunities or academic grades,
• it influences access to resources or decisions about eligibility and entitlements.

In such cases, it is safer to use a manually selected preference, a simple communication setting or no psychological adaptation at all.

Responsible personalisation requires rules

Several basic conditions are necessary for the responsible use of psychological adaptation:

• a clearly defined purpose,
• proportionality of the adaptation,
• clear and periodically renewed consent,
• limited and temporary inferences,
• transparency,
• the ability to challenge and correct the profile,
• independent human oversight,
• the same basic level of access for users who refuse profiling.

Only under these conditions can AI personalisation be understood as a form of user support rather than hidden surveillance, evaluation or manipulation.

Why this matters

The future of AI systems will not depend only on whether they are accurate, fast or technically advanced. It will also depend on whether people understand what the system infers about them, how those inferences are used and whether they have a meaningful opportunity to object.