Ulrik Ekman offers this perspective on what is unnerving about computational systems, like targeted advertising, AI assistants, or data-driven crowd management, that use covertly gathered information to tailor their behaviour to individual circumstances:
A supposedly ‘natural’ setting turns out to be nothing if not a highly artificial context or an information-intensive environment, and it appears attentively oriented towards us rather than being neutral or perfectly non-caring. [1]
The peculiarity common to these encounters stems from the perception of agency where none is expected. Environments, whether natural or artificial, are not typically the sort of things we take to care about anything, let alone about us. While some, such as buildings, may be designed by humans with human needs in mind, it is not yet natural to expect such environments to respond dynamically to the needs of particular people in particular moments.
Two facets can be discerned in the ‘attentiveness’ present in such systems: they both (i) gather data, and (ii) do things in response to this data. This alone seems insufficient to distinguish these systems from those which respond to changes in ambient conditions without inviting attributions of agency. Thermostats provide one example, their behaviour analysable in terms of cause and effect, requiring no appeal to agential concepts like care, concern, or attentiveness.
What does seem critical in furnishing the systems Ekman describes with a sense of agency is the presence---real or imagined---of intention in their behaviour. They do not gather their data indiscriminately. Intention constrains and structures data-gathering processes, enacting discriminatory practices that delimit categories of revelant and discardable information. Intention is what allows us to describe the causal path from environmental event to sensor to database as a kind of sensing; what allows us to describe the response of the system as a kind of acting.
If it is the feeling that such systems sense us and act towards us that imparts them with their often unsettling shades of agency, and an event being a sensing or an acting is dependent on its being intended, we might ask to what extent is this agency parasitic on the agency of the humans who built these systems in line with their own purposes and intentions? The primary intention of targeted advertising may be to display to a person adverts relevant to their interests, but if this intention only exists in virtue of the human intention to profit from the targeting, can we not disregard the ‘agency’ of the system itself as being of a derivative kind, a metaphor for what is ultimately no more than a patterning of causes and effects determined by external human agents?
But perhaps denying agency to these systems on the basis of such strict criteria is unkind. Isn’t it too much to ask of a sensing, acting artefact that its behaviour be wholly independent of the intentions of its artisan? What seems more reasonable (and more permissive) in assessing its claim to agency is that it senses and acts with autonomy, that is that it acts without explicit human direction. And this is a feature these systems certainly exhibit.
A final consideration: in addition to their creators, what hand do the humans who experience and interact with them have in constituting the perceived agency of these systems? After all, it is questionable whether it would even occur to us to describe them as caring if it was not us they sensed, us they acted for?
But in this case also it seems possible to generalise. A system that dispensed food to animals based on the detection of behaviours associated with hungriness would attract the same attributions of agency, for all the same reasons. And if, say, there was a machine that performed optimally at a certain temperature, and there was a separate system which monitored the conditions in the room housing it, adjusting the temperature to meet optimal conditions, would not all the same factors be in play here too? Would we not want to say of this nonhuman system that despite its lack of any interfaces facilitating human interaction, there is nevertheless an agency present in its attentive care to a nonhuman other?
Ulrik Ekman, Complex Ubiquity-Effects