Business

Designing for Who Watches

When the primary user of an interface is an agent, is watching all that's left for the person? The history of automation suggests that watching is much harder than it looks.

Maria Scarano Week 27 Leer en español
Share Share on LinkedIn Share on X Share on Facebook Share by email
abstract man staring at modern concrete structure
AI assisted/generated image

In 1835, Andrew Ure published The Philosophy of Manufactures, a book that enthusiastically defended the factory system. In one passage he describes quite clearly where industry was heading: the aim was to "reduce the task of his work-people to the exercise of vigilance and dexterity," and to replace skilled labor with "mere overlookers of machines." Ure didn't present this as a loss. He presented it as progress: the machine would do the work, and the person would take care of watching it.

Almost a hundred and fifty years later, in 1983, the psychologist Lisanne Bainbridge published a short paper that became a classic of systems engineering, Ironies of Automation. One of its sentences reads: "The human monitor has been given an impossible task."

Between those two sentences lies, I think, one of the most important questions ahead of us. We are starting to design interfaces whose primary user isn't a person but an agent. The person is left with a layer for reading what the agent did, approving it or stopping it. Once again, the person is left to watch. This piece doesn't try to solve how to design that layer. It tries to order the questions we should be asking first.

The first format written for a machine

There's a precedent that helps me think about what changes when an interface is written to be read by a machine.

On the drawlooms used to weave patterned silk, the weaver handled the shuttle and the rhythm, and an assistant, usually a boy, lifted the right threads by hand at exactly the right moment for the pattern to appear. He was called the draw boy. The work was physical, uncomfortable and very precise. In the early nineteenth century, Joseph Marie Jacquard replaced him with punched cards: each card encoded which threads to lift for one row, and a chain of cards encoded the whole design. Bill de hÓra, who wrote about this as an allegory for AI, sums it up: the skill left the assistant's body and went into the card.

What interests me in that story isn't the replacement, which is well known, but the format. The punched card is perhaps the first document written to be read by a machine before a person. The pattern still existed, but it was no longer visible in anyone's gesture. It lived in a code that the loom read and that the person punching the cards understood. Everyone else watched the cloth appear.

Today we are once again writing documents of that kind: instructions, rules, skills and contexts meant to be read by an agent. And the question Jacquard left us is still open: when the format is written for the machine, what remains visible to the person, and in what language?

What happened to those left watching

Ure made a concrete prediction. The self-acting mule, which Richard Roberts had perfected in 1825, would put most spinners out of work. It didn't quite happen that way. According to economic historians such as William Lazonick, spinners stayed in the mills for decades, partly because their value as supervisors of the process and of their assistants proved hard to replace, and partly because their unions were strong. The craft didn't disappear. It turned into supervision.

That is the pattern Bainbridge studied a century and a half later in industrial process control, and described with a clarity that still surprises. Her argument has several layers. The first is that skills are lost when they aren't used. The operator who has spent years only watching an automated process is no longer the expert they were, and yet they are the one called when something fails, precisely at the moment that demands the most skill. The second is that the knowledge needed to understand an abnormal situation is built by using the system and getting feedback, which is exactly what the operator no longer does. The third is that no one can sustain attention for hours on something that almost never changes. And the fourth, perhaps the most ironic, is that we train people to follow instructions and then put them into the system to provide intelligence.

Bainbridge also left a recommendation worth keeping: "automatic systems should fail obviously." A system that degrades gracefully hides the failure until it can no longer be controlled.

Looking is not seeing

There's an experimental finding that takes the problem a step further. Researchers at the Technical University of Berlin who study complacency toward automation observed people working with a support system that was highly reliable, though not perfect. The expectation was that errors would come from those who stopped checking the information, and some did. But about half of the errors were made by people who had checked everything. They looked at the information without processing it, and several couldn't remember afterward what they had seen.

The same study found something I find even more interesting. What protected people most against error was having seen the system fail before. Among those who had already experienced a failure, about five percent made the mistake; among those who hadn't, around twenty. The authors estimated that it took some twenty positive experiences to erase the effect of a single early failure.

Put another way: visibility doesn't guarantee attention, and the trust a well-functioning system builds is exactly what makes its failure more dangerous. What keeps a person alert isn't seeing more, but having seen that the machine can be wrong.

The layer we are designing now

These ideas come from factories, power plants and cockpits. But they describe quite precisely what is starting to happen with agents.

Obvious, a company that builds agent platforms, describes its proposition this way: a team's judgment and standards become the rules the agents follow. People define the steps, the quality criteria and the checkpoints, approve a plan, and receive the finished work along with the evidence backing it, to review before accepting it. It's a careful design. And it is also, in a different vocabulary, Ure's system: the person defines at the beginning, supervises at the end, and in between the machine works.

A recent study analyzed twenty-two publicly available agent systems and found six recurring interface patterns: visible traces of what the agent thinks, plans and does; mechanisms to interrupt it or take over; a watch mode for sensitive activities; user-defined rules; memory that can be inspected and edited; and sandboxed environments that show their status. Two of the authors' observations stayed with me. One, that traces are only useful if they faithfully reflect what the agent did. The other, that in watch mode, watching doesn't guarantee attention. Bainbridge could have written both.

The questions that remain open

I don't have an answer about how to design this layer, and I'm wary of anyone who claims to. But I think the questions can be put in order.

The first is how visibility happens, and for whom. If information is organized to be read by an agent, the person receives a translation. Who writes that translation? In what language? What gets lost along the way? Jorge Arango defined the purpose of information architecture as increasing agency by making systems more legible. In an architecture designed first for agents, the question becomes legible to whom.

The second is what we leave to the machine and what to the person. Takuma Kakehi observed that automation allocates work according to what the machine can do, never according to what is good for a person to do. Nobody decided that watching was a good use of human time. It's what was left over. Could we allocate it differently, for example according to what a person needs to keep doing in order to keep understanding what they supervise?

The third is how much transparency we take for granted. New development models put agents and their language at the center: rules, instructions and skills written for a machine to follow. That makes them explicit, which is progress. But being written down doesn't make something transparent to a person. And a trace the agent writes about itself: does it count as transparency, or as self-declaration? The authors of the twenty-two-system study propose regulating these patterns, because today they are product decisions a team can remove for convenience or because of a test result. Transparency, for now, isn't a right. It's a design option.

The fourth is whether the person is still the one who triggers actions. In a world where agents start tasks, hand work to each other and talk to other agents, the person may stop being the one who begins and become the one who gets called. There's a precedent that isn't necessarily bad. In the early twentieth century, Sakichi Toyoda designed a loom that stopped itself when a thread broke, so that no one had to watch it all the time. The machine raised its hand and the person came. It's exactly what Bainbridge asked for: that failure be visible. But it also leaves a question open. If the agent calls us when something fails, who decides what counts as failure?

Designing for who watches

When designing the human layer of an agent system, there's a temptation to think of it as a dashboard: show everything that happens, clearly, in real time. The history of automation suggests that isn't enough. A layer that only shows creates spectators. What keeps a person able to judge is practice, the experience of seeing the machine fail, and a real possibility of intervening.

Maybe the question for those of us who design isn't only what to show, but what the person watching needs to be able to do for their watching to still be worth something. Ure imagined attentive watching as the person's natural place in front of the machine. Bainbridge showed that this place, designed that way, is impossible. We are building the new information architectures with very little time to think about that gap. It's worth starting to think about it before the layer we design for watching becomes, once again, someone's impossible task.

LinkedIn

Continue the conversation on LinkedIn

LinkedIn

More to read

Business Why the “Human Premium” and Mentoring Will Define the Organizations of the Future Business When Fine Becomes the Norm Business Who Captures the AI Dividend?

Continue Reading

Explore topics

AI & Society Automation Business Culture Design Human Work +