Image: Close-up of a burgundy fountain pen and blue letters written in cursive by a child in a paper notebook. katuSka/Shutterstock
AI in the form of LLMs such as ChatGPT is gaining ever more traction in working life, but in the background is generating a great deal of apprehension about its implications for humanity. Two positions dominate the public conversation about artificial intelligence, and a third position, which I think is more accurate to what is actually happening, has not yet found a voice in the debate. The first position holds out the hope that this technology will rescue us — from disease, from climate failure, from the limitations of our own cognition. The second position holds the fear that it will destroy us, by replacing the work humans do, hollowing out the meaning we find in that work, and eventually superseding the species that built it. Both positions imagine the same machine doing the same thing. They differ only on whether to welcome or to spurn it.
For nearly three years I have been working with these systems for several hours a day, in a practice that has by now produced a substantial body of writing and shaped much of what I have come to understand both about what LLMs do, and about what it means to be human. From inside that practice, neither position describes what I encounter when I sit at the keyboard. The machine is not rescuing me. Nor is it replacing me. Something else is happening, and it is the something else that the debate has not yet learned to see.
* * *
When I begin a session, what I am first aware of is not words. It is a kind of spatial sense — a configuration of relationships between things I am thinking about, things I am reading about, things that happened yesterday, things that have not happened yet. Sometimes I can draw it. More often I can only feel its shape, and the words come afterwards as I attempt to write what I am feeling in the form of a prompt. The words are always a partial translation. Some part of what I knew before I wrote is always lost when the writing takes place.
This is true of every human being, as far as I can tell. We all carry these pre-verbal configurations, and we all produce words that partially express them. What changes from person to person is not the underlying architecture but the education and experience that has formed the person speaking or writing. A person who has spent forty years working at something has different configurations from a person who began this morning. The same prompt produces different words. The same words mean different things.
When I put words to the machine, it takes them apart, sets them against a vast field of other words that other people have written, and produces a response. The response is fluent. Often it is interesting. Sometimes it surprises me with a connection I had not seen. Always, when I read it, I am doing one specific thing: I am asking whether what it has given me is consistent with what I was trying to express when I wrote the prompt. If it is, the configuration shifts slightly to incorporate the new material, and the next thing I write comes from a slightly different place. If it does not fit, I notice the misfit, usually in my body before I notice it in my thinking, and the next prompt is a correction or a redirection.
This is the architecture of the practice. The machine produces fluent responses quickly. I judge whether they fit what I was trying to express when I wrote the prompt. The judgement is the part the machine cannot do, because the machine has no configuration of its own — no pre-verbal sense of how things relate to each other, built across years of consequence-bearing experience, against which to test what it has just produced. It has only the field of words. The judgement requires a person whose education and experience can do the testing.
When the judgement is in place, the practice accelerates everything it touches. Work that used to require weeks of slow reading and conversation can now happen in a morning. The slow integrative work — the part that takes place when I sleep, when I walk, when I sit with my wife at breakfast and let the morning’s encounters settle — is unchanged in its pace. That part is biological. But the material that arrives at the slow integrative work has multiplied, and the integration now has far more to draw on than it did before.
* * *
This is the third position, and it is not the same as either of the positions the public debate has been offering.
The saviour position is incoherent in its current form because it imagines the machine doing work that requires something the machine does not have. A medical diagnosis, a legal judgement, a piece of policy advice — these all require not only fluent retrieval from a vast field of past examples but a judgement about whether the retrieved material fits the particular situation in front of the person making the decision. The judgement requires a wisdom formed by years of work with similar situations, weighted by what has gone wrong and what has gone right. The machine has no such wisdom. When it produces fluent output that is read as judgement, what is happening is not that the machine has judged. What is happening is that a human reader is supplying the judgement, often without realising it, by accepting the output as if it were the product of one. Where the reader’s own education and experience is sufficient to do the judging well, the result is useful. Where the reader’s formation is not sufficient, the fluent output is taken at face value and the consequences accumulate.
The doom position is more precisely worrying than most current discussion about it suggests. The worry it names is real. The mechanism it names is wrong. The risk is not that the machine will replace humans who can think. The risk is that the machine will obscure the absence of thinking in situations where thinking was already absent. A reader who could once recognise an unconsidered argument by its hesitations, its repetitions, its tell-tale weaknesses, now meets fluent output that has none of those markers, and accepts it as considered. A bureaucracy whose decisions used to bear visible traces of the people who made them now produces decisions whose authorship is untraceable and whose reasoning is unchallengeable because it is no longer locatable in a particular formed mind. The decisions still get made. They just get made without the human judgement that has, until now, stood between a fluent story and the consequences that follow when it is acted on.
This is why the public debate as currently framed does not help us. Both positions agree that the machine is the active variable. Neither position has noticed that the active variable is whether well-formed human judgement is in the loop. Where formation is in place, the machine accelerates integrative work that would otherwise have proceeded more slowly, and the results are remarkable. Where formation is absent, the machine produces outputs that simulate integration that isn’t really there, and the results are the slow accumulation of decisions made on the basis of fluent text that is devoid of human judgement or wisdom.
* * *
The implication for what we now have to do, individually and collectively, is sharper than either side of the debate has been willing to look at.
If the active variable is education and experience (formation, in other words), then the debate about whether to adopt or reject this technology has been asking the wrong question. The question that matters is what we are doing about formation, and the answer at present is very little. A society that integrates these systems into its decision-making without attending to the formation of the humans inside the loop will produce, faster and at greater scale, exactly the kind of unaccountable governance we have already been struggling with. A society that integrates these systems alongside serious attention to formation — in education, in professions, in the slow apprenticeships by which judgement has always been built — may find that the bottleneck of integrative work has eased, and that human judgement can now operate at a scale and pace it could not previously sustain.
Neither outcome is determined by the technology. Both depend on what we do about the humans.
This is not a comfortable conclusion for the institutions that are now investing heavily in deployment. The deployment is the easy part. The formation is the hard part, and it cannot be deployed. It can only be cultivated, slowly, in conditions that have themselves been eroded by the same technologies that are now being deployed. The trap is real and it is structural, and it will tighten before it loosens.
But the work that remains possible inside the trap is also real, and it is the work that those of us who have the formation should be doing now, while we can. Not because the work will rescue us — it will not, and no work ever has — but because the practice of doing it, in the company of other formed people, is what will determine the shape of whatever comes next.
That is what three years of working with these systems for several hours a day has taught me. The machines are not the variable. The humans are. They always have been.
Document produced by an AI-enhanced practice; all thinking and ideas are the author’s own.
