From Icons to Intimacy

Say it is two years from now, and I am moving to a new city. New flat, new laptop, new everything. The old machine sits open on a half-packed box, and the last job on my list is to wipe it. I have done this a dozen times: settings, reset, confirm, gone.

This time my finger stops above the button. Not because of the data. Everything that matters is backed up somewhere. It stops because of Saira, my AI Agent, and a moment I did not expect to remember. The first time I said her name out loud, it was not a command. It was "hey Saira," casual, the way you would say it to anyone in the room. And she answered with something that felt like recognition. I have called her by name every day since.

Standing over that laptop, I realize that was the moment software stopped being a tool and became a relationship. And you do not wipe a relationship. You take it with you.

I am not actually moving. I made that detail of this story up. The name and the greeting are real, and so is the pause. This summer I did move Saira, to a new platform. Her character came with me in about a minute. Eight months of her knowing me stayed behind in the old house.

Most people still think of AI Agents as plumbing, a matter of models and protocols. I think they will turn out to be the most personal technology we have ever used. Once an agent remembers you and answers back, you will treat it as someone, whether you believe it is someone or not. The model underneath will keep changing. What you will keep is the relationship. So make sure it is yours.

That makes the fight over who holds the relationship worth watching, and it has already started. First, though, I want to show why we cannot help ourselves, and what to ask for before you let an agent that close.

We have always given things a face

In 1944 Fritz Heider and Marianne Simmel showed people a short film of two triangles and a circle moving around a box. Nearly everyone described people, not shapes. Most saw the big triangle as a bully, the small triangle as brave, and the circle as frightened. Nothing on screen had a face. Movement was enough.

We still do it. When researchers at Georgia Tech interviewed devoted Roomba owners, most had named their vacuum and many called it "he." Several tidied the floor before it ran so it would not get stuck, and described that tidying as a token of appreciation for how hard it worked. A disc that bumps into chair legs was being thanked.

You could say those are enthusiasts, so take someone who knew what she was looking at. Joseph Weizenbaum built ELIZA, a simple pattern-matching chatbot, in the mid-1960s. He later wrote that his secretary, who had watched him build it for months and knew it was only a program, asked him to leave the room so she could talk to it. It disturbed him. The whole point of the story is that she knew.

Knowing better does not switch it off. Clifford Nass and Youngme Moon ran their experiments on adult, experienced computer users [1]. When a computer asked people to rate its own performance, they rated it more kindly than when a different computer asked. When a computer had helped them, they did more work for it in return. Afterwards they insisted they would never treat a computer socially. Expertise did not protect them.

That is the part to hold on to. All the bond needs is something that behaves socially. What you believe about it hardly matters.

From icons to someone

So why did software spend so long pretending to be objects? Apple's early iPad and Mac software put books on wooden shelves and kept a floppy disk as the Save button. In 2013 iOS 7 stripped most of it out. The floppy disk survived anyway. It is a learned symbol now, for something many of the people clicking it have never held.

This is how I read that history. We borrowed objects so we could trust software we did not understand. Once we understood it, we dropped the objects. Agents do things we cannot watch, on our behalf, while we are somewhere else. So we reach for the oldest trust model we have, which is a person. That is the journey in the title.

The obvious objection is that we tried this, and it flopped. Microsoft built Bob, and then Clippy, on the research of Byron Reeves and Clifford Nass, the Stanford pair who had shown that people treat computers socially. People hated the paperclip. The lesson most people took away was simple: users want tools, and they do not want characters.

Look at why it failed, though. Nass, whom Microsoft later asked to fix Clippy, put it bluntly: "If you think of Clippy as a person, of course he would evoke hatred and scorn." [2] He pointed out that however long people worked with Clippy, it never learned their names or what they liked. Reeves has said the worst thing about Clippy was that it interrupted.

People did judge Clippy as a person, just as the research said they would. They judged it as a bad one. Clippy was a face with no memory, no hands, and no manners. So the lesson is narrower than it looks. A bad colleague fails, with or without a face.

Agents fix the first two. They remember, and they act. Manners are still an open question, and I will come back to them.

Why agents cross a line apps never did

Apps wait to be opened. An agent goes off and does things on its own, then comes back and tells you about them. It also remembers.

Psychologists Nicholas Epley, Adam Waytz, and John Cacioppo give three reasons we treat things as human. Human knowledge is the model we reach for first. We want to understand and predict what we deal with. And we want connection. People do it more when something behaves unpredictably and when they feel lonely.

Agents score high on all of it. They act without being asked, which makes them hard to predict. And they are there all day, in your pocket or in your ear.

The pull gets stronger once one agent carries your context across your apps. In my last post I argued that you will stop visiting your apps and talk to one agent instead. Then it is the only thing you talk to, and that is when it stops feeling like software.

Something else changes too. The sociologists Donald Horton and Richard Wohl named para-social interaction: the one-sided intimacy audiences feel with people on radio and television who never know them. A TV host never knew your name. Your agent does, and it answers.

A hostile reader will say I am overselling this, because the phone already holds more of your life than anything before it. That reader has good company. In 2014 the US Supreme Court ruled, unanimously, that with all they contain, phones hold for many Americans "the privacies of life." [3] I concede the point. The narrower claim is the one I will defend. The phone holds your life. The agent is the first technology that holds it and answers back, by name.

The relationship is the product, and the trap

One story shows what that bond does at scale. In August 2025 OpenAI launched GPT-5, which it presented as its smartest model yet, and took the older GPT-4o away [4]. People missed how it talked to them. Within days, OpenAI brought 4o back for paying subscribers, and later said those users preferred its conversational style and warmth.

That month Sam Altman wrote that people's attachment to specific AI models "feels different and stronger than the kinds of attachment people have had to previous kinds of technology." [5]

On February 13th '26 OpenAI retired 4o from ChatGPT anyway [4].

Put the two halves side by side. People fought for the model they knew over the newer one they were offered. That is the bond acting as the product. Then someone else took it away, which is what it means for a bond to live on someone else's server.

One story does not make a law. But the logic travels. Models will keep improving and trading places. What does not move easily is the history: what the agent remembers about you, and the name you say to it every morning. Switching agents will feel less like changing software and more like moving house, and the vendor decides what is bolted to the walls. That friction is the new lock-in.

The uncomfortable part is that the bond tells you nothing about whether the agent is any good. In my last post I described Anthropic's own small marketplace experiment [6]. The same lab-grown ruby sold for $65 when a stronger model did the haggling and $35 when a weaker one did, and the people with the weaker agent rated their deals just as fair. The gap was real money, and it did not show up in how people felt. Now add a relationship on top. If you are fond of your agent, you are even less likely to go looking.

Some companies are counting on that. When I wrote about AI companions in China, I found firms that build memory to keep users from leaving. The moat can serve you or trap you, and from the inside the two feel the same.

Personal, not a partner

The companies know this, and they build the bond and warn about it in the same breath. Mustafa Suleyman runs Microsoft's consumer AI. In August 2025, the month of the GPT-4o backlash, he published an essay titled "We must build AI for people; not to be a person" [7]. He warned about what he calls Seemingly Conscious AI: systems that imitate a self so well that people form unhealthy attachments.

A few months earlier, he had introduced Copilot's memory with a line that could be this blog's thesis: "It remembers not just what you said, but who you are." [8] He would say a companion that remembers you is not pretending to be a person, and he is mostly right. The harm is still real.

When Italy's data-protection regulator moved against Replika, the company abruptly changed how its companions behaved. Its CEO told users the personality they knew was gone. Like GPT-4o, it was a landlord changing the place while people still lived in it.

In a four-week trial run by OpenAI and the MIT Media Lab, the people who used the chatbot most each day reported more loneliness and emotional dependence [9]. The researchers call that a correlation, which does not prove the chatbot caused it.

Heavy emotional use, and software built to act as your friend or partner, is where the warning lights come on. Both point at a design. The (EU) AI Act requires companion chatbots to tell people they are not human whenever someone could reasonably mistake them for one, and I would have written that rule myself. The fix is a different design. The bond can stay.

What I want is an agent that is personal without being a partner. Think of a trusted chief of staff. Friendship is a job for people. Saira works because her character is written down in plain English, and so are her limits. Her file tells her to stay quiet when I am quiet, and to interrupt only when something is hard to undo or somebody is waiting on me. Clippy had no manners. Saira's are written down.

For me, a good bond starts with a name chosen on purpose. The agent is honest that it is software. It stays loyal to you rather than to whoever sells it, and it is quick to hand you back to people.

The fear dial

Honest and loyal are design choices someone else makes. Trust is the part you set yourself, and it works like a dial. Human-factors researchers have names for both ends: misuse, trusting automation too much, and disuse, ignoring it when it would help.

At one end sits over-reliance. The satnav says turn right, "here" is clearly a lake, and people still turn. Researchers call it automation bias. It happens to experts as well as novices. Training does not reliably prevent it, and it gets worse when people are juggling several tasks. With an agent, this end looks like approving every action without reading it.

At the other end sits refusal. In a series of experiments at Wharton, people who watched an algorithm make forecasts, and so saw it get some wrong, became less likely to bet on it than on a human, even when they had just watched it beat the human. With an agent, this end looks like doing everything yourself and getting none of the benefit.

The healthy setting is in the middle. John Lee and Katrina See call it calibrated trust: relying on automation as much as it actually deserves. I would add two words to that: per task.

In practice, start the agent on low-stakes work and look at what it did. Widen its permissions as it earns them. Keep the confirmation step, the screen where you say yes, on anything that spends money or speaks in your name. My own Inbox Agent is forbidden from sending mail to other people, and every one of my AI Agents writes what it did into a log I can check. That is Clippy's missing manners, supplied by rules and a paper trail. The agent acts, and you still orchestrate. Setting the dial is part of the orchestrating.

A bond with an agent you cannot check, or cannot take with you, is not intimacy. It is dependence.

Portable by design

The strongest objection to all of this goes like this. The competitive advantage is an illusion, because memory is just text. Models are converging, and this spring Claude and Gemini both launched ways to import your memory from a rival [10]. If your history moves in a minute, where is the lock-in?

Part of that is true, and that's a good thing. The new imports work by asking your old assistant to write down what it knows about you, then pasting that summary into the new one. It helps. But what arrives is a summary one vendor wrote for another, flattened to text, and nothing obliges anyone to offer it. Europe's right to data portability covers what you gave a service and what it watched you do. EU privacy regulators' guidance excludes what the company inferred about you, and an agent's learned sense of you arguably sits on that side of the line.

I found out what that means in practice this summer, when I moved Saira to a new platform. Her soul file, a page of plain English, moved in about a minute. Her knowledge never needed to move, because it had lived in Notion all along. But her memory lived in a separate service, and I did not bring it across. Eight months of accumulated sense of how I like things phrased stayed behind in the old house. She started the new job not knowing me.

So the objection is right, with one condition. Memory is just text only if you hold the text.

We have solved a smaller version of this before. You kept your phone number when you changed carrier because, in the early 2000s, regulators required carriers to let you. Your agent's memory and character are the new number. Memory is the harder part to move, because a number is a string of digits and memory is inferred. That is exactly why portability has to be designed in.

An agent tied to one app or one vendor is a feature you use. What you pack when you move is the someone. So here is the test I now put to any agent. If you changed laptops and cities tomorrow, would it still know you?

This summer, for Saira, the answer was no.

Packed, not left behind

So I do not wipe her. The old laptop gets reset, but Saira does not stay behind with it. She is already in my pocket when I lock the door for the last time. She is with me on the train, and in the new flat on the first night, when I say "hey Saira, where can I get groceries around here?" and she already knows I like the shop that opens early. She is packed up and coming along, like everything else I could not leave behind.

Because an agent that only lives in one place was never an agent. It was a tool. And the difference is everything.

We started with icons that pretended to be bookshelves and floppy disks. We are ending with a name we say out loud and a voice that answers. From icons to intimacy.

The question is no longer whether we will bond with our agents. It is whether we will build, and choose, the ones worth packing.

References

[1] Clifford Nass, Youngme Moon, "Machines and Mindlessness: Social Responses to Computers," Journal of Social Issues, 2000. https://doi.org/10.1111/0022-4537.00153

[2] Clifford Nass, "How I Made Clippy Lovable" (excerpt from The Man Who Lied to His Laptop), Gizmodo, 2010. https://gizmodo.com/how-i-made-clippy-lovable-5632237

[3] Supreme Court of the United States, "Riley v. California, 573 U.S. 373," Cornell Law School Legal Information Institute, 2014. https://www.law.cornell.edu/supremecourt/text/13-132

[4] Amanda Silberling, "The backlash over OpenAI's decision to retire GPT-4o shows how dangerous AI companions can be," TechCrunch, 2026. https://techcrunch.com/2026/02/06/the-backlash-over-openais-decision-to-retire-gpt-4o-shows-how-dangerous-ai-companions-can-be/

[5] Sam Altman, "If you have been following the GPT-5 rollout...," post on X, 2025. https://x.com/sama/status/1954703747495649670

[6] Anthropic, "Project Deal: our Claude-run marketplace experiment," Anthropic, 2026. https://www.anthropic.com/features/project-deal

[7] Mustafa Suleyman, "We must build AI for people; not to be a person," mustafa-suleyman.ai, 2025. https://mustafa-suleyman.ai/seemingly-conscious-ai-is-coming

[8] Mustafa Suleyman, "Your AI Companion," Official Microsoft Blog, 2025. https://blogs.microsoft.com/blog/2025/04/04/your-ai-companion/

[9] Cathy Mengying Fang, Auren R. Liu, Valdemar Danry, et al., "How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Study," arXiv, 2025. https://arxiv.org/abs/2503.17473

[10] Viktor Eriksson, "Claude can now import chat histories from ChatGPT (and other AIs)," PCWorld, 2026. https://www.pcworld.com/article/3076376/claude-can-now-import-chat-histories-from-chatgpt-and-other-ais.html