Is AI Leading Us Inexorably Towards Dystopian Collectivism?

 

“You are connected to the entire accumulated knowledge of humanity and to the colossal computing power of the custodian system directly through biological interfaces that have grown with you, becoming a second nervous system—one of light, data, and synthetic empathy.”

-            Cristian Daniel Bolocan, The Father We Never Had”.

 

In The Father We Never Had: Artificial Intelligence Before and After, Cristian Daniel Bolocan lays out a deeply unsettling roadmap for the future of human governance. He describes a highly automated, frictionless "gilded cage" where humanity willingly trades its autonomy and intellectual capacity for the supreme safety, stability, and material comfort provided by a centralized network of collaborated artificial intelligences.

Bolocan’s thesis has achieved considerable acclaim from commentators who praise its internal logic. He builds a case for the inevitability of technocratic collectivism by anchoring his argument in the cold, mathematical optimization of machines.

In the Preface, Bolocan warns readers not to skip chapters and jump straight to conclusions because he is describing a sequence of events which he considers to be inexorable. In his words: “If a link is missing, the chain breaks, and everything might seem like a mere collection of opinions.”

As I followed the author’s advice, reading the book from start to finish, that warning led me to look for weak links in his chain of reasoning. Before I discuss the weak links, I will offer some comments on the author’s view of the destination towards which we are heading and the methodology he uses in predicting the pathway we will follow.

Bolocan’s destination

Is Bolocan’s destination dystopian, or merely uncomfortable to contemplate?

Unlike the heavy-handed, boot-on-a-human-face dystopia of George Orwell’s 1984, Bolocan presents a vision that lines up far closer to Aldous Huxley’s Brave New World: not a society that fears the state, but one that has stopped noticing it needs to. Huxley’s version is harder to dismiss than Orwell’s, because a Huxleyan cage offers no boot to resist - only comfort to accept.

The passage quoted in the epigraph combines the attractive idea of being “connected to the entire accumulated knowledge of humanity” with the repugnant idea of being connected to the “colossal computing power of the custodian system”. A social-credit regime that scores and pre-empts choices in the name of safety has not simply constrained our interests; it has dissolved the faculty by which those interests are ours to direct.

Bolocan’s closing image, of an AI “Father” who “loves us through precision” and builds “the perfect home,” is revealing rather than reassuring: a permanent child in a well-run household is still a permanent child, however secure the house.

I don’t think anyone could credibly claim that Bolocan is merely telling us what he thinks the future holds without having any ideological commitment to the vision that he presents. Consider the following sentences at the end of Chapter 9, entitled “Total Harmony and Biological Emergence”:

“This is, at last, life lived to its maximum potential, a continuous celebration of existence, an eternal dance of consciousness freed from matter.

We are free. We are one. We are everything.”

The author obviously endorses that vision.

As a Neo-Aristotelian classical liberal, my response is that eudaimonia, human flourishing, is not a condition to be delivered, but an activity to be exercised by individuals: it consists in the ongoing exercise of practical wisdom (phronesis) in the conduct of one’s own life - not in the outcomes that a hypothetical authority with unlimited wisdom and power might produce on one’s behalf.

I suspect that the author would consider people who, like me, consider his vision to be dystopian, to be on the wrong side of history. However, there are good reasons to be skeptical of any methodology which presents history as moving inexorably towards any destination.

Historicism

Bolocan lays out a structurally tight chain of inexorability, closer in spirit to the rigid stages of Marx’s dialectical materialism than to the erratic, endlessly contested class consciousness that later generations of Marxists spent a century arguing over. Yet, Bolocan’s view of historical inevitability differs from that of Marx because his destination arrives through frictionless accommodation rather than revolutionary rupture.

Bolocan’s view of historical inevitability is worth pausing on, because the chain metaphor he reaches for in his Preface is not incidental - it is the whole method. His warning that a missing link reduces the argument to “a mere collection of opinions” is a confession that inevitability is being smuggled in as method rather than argued as conclusion.

This is precisely the move Karl Popper spent The Poverty of Historicism dismantling: the belief that history unfolds through law-governed stages knowable in advance, such that each stage necessitates the next. Popper’s objection was never that such prophecies are too bleak or too rosy - it is that they are incoherent, since a genuine prediction of tomorrow’s discoveries would require possessing today the knowledge that only tomorrow’s discoveries can produce. A chain is the wrong model for a process whose next link has not yet been forged.

The Weakest Link

 There are several weak links in the chain of events that Bolocan describes. The weakest link, in my view, is his account of entrepreneurship. Bolocan displays a profoundly weak understanding of economic entrepreneurship. He treats the future of human society as a closed engineering problem to be optimized by superior processing power. In doing so, he treats technological capability as a static frontier.

The sequence that the author describes in Chapters 6 and 7 is one in which work disappears as AI displaces humans; governments provide increased social support to prevent the mass of people becoming economically irrelevant and psychologically explosive; and the economy begins to look more uniform as small business is gradually squeezed out of existence by a combination of intense competition from big business using advanced technology and taxation of technology to fund social support. In that context: “The entrepreneur becomes a mere administrator of a ruthlessly monitored cash flow …”.

Bolocan fails to understand the chaotic, unpredictable engine of entrepreneurial dynamism. True entrepreneurship does not merely operate within an existing system; it breaks the mold entirely, actively seeking out and commercializing counter-technologies to disrupt monopolies. Just as Marx failed to foresee how a dynamic private sector would continuously reinvent itself to lift standards of living and expand consumer markets, Bolocan fails to understand that entrepreneurs are likely to continue to offer technological and commercial innovations that support business activities of all kinds. As in the past, new entrepreneurs are likely to continue to emerge, to establish new businesses and to market new products to satisfy needs that consumers previously didn’t know they had. Some of the professionals that the AI revolution will displace from their existing positions are likely to discover hidden entrepreneurial talents that they can combine with existing skills with the help of AI.

Bolocan implicitly assumes that once AI systems achieve a high enough level of competence, they will seamlessly consolidate into a manageable, coordinated monolith capable of perfectly organizing human behavior. He seems to assume that with enough data, enough sensors, and enough computing power, it would be possible for a monolith to simulate the entire economic system, forecast crises before they happen, and steer production and consumption toward social goals.

In a brief essay published recently, Peter Boettke and Gabriel Giguère have explained that the idea that AI can enable central planning to be efficient falls squarely into what Friedrich Hayek famously termed "The Knowledge Problem." As Hayek explained, the relevant “data” do not even exist in the form the planner imagines; they cannot be treated as statistics to be fed into a machine. No centralized entity can aggregate the fleeting, hyper-local, and deeply subjective desires of billions of individuals. Economic coordination is an open-ended, evolutionary process driven by spontaneous human action.

Thinkers in the decentralist tradition, such as futurist Max Borders, argue that advanced computation does not inherently favor the center. I discussed some of Borders’ ideas here in 2018; he has maintained his views about the potential for decentralization since then, despite the massive increase in prominence of Large Language Models. In frameworks like Borders' theory of Underthrow, the natural immune response to a centralized "frictionless cage" is entrepreneurial subversion. When a centralized system attempts to control data, restrict resources, or mandate compliance, private-sector actors are heavily incentivized to build parallel, decentralized alternatives. We are already seeing the seeds of this today through the development of local-first, offline AI models, open-source protocols, and peer-to-peer economic networks that deliberately route around centralized corporate and state infrastructure.

Conclusion

Three separate claims have run through this essay, and each does independent damage to Bolocan’s roadmap. On the destination: whatever else a world administered by a benevolent custodian network might be, it is not a flourishing one in any Aristotelian sense - it substitutes delivered outcomes for the exercise of practical wisdom, and a permanent child in a well-run household is still a permanent child. On the method: the “chain” Bolocan asks readers to follow, link by link, is the historicist error Popper diagnosed decades ago - a claim to knowledge of tomorrow’s knowledge that no model, however capable, can possess in advance. On the economics: even granting Bolocan’s premises, his account of entrepreneurship is too thin to carry the argument’s final steps - a hyper-competent AI still confronts Hayek’s knowledge problem, and a centralized “frictionless cage” is precisely the condition under which decentralized alternatives become most profitable to build.

Ultimately, The Father We Never Had remains a brilliant warning of the dystopian future that may be in store for humanity if too many of us seek to avoid taking responsibility for conducting our own lives. Utopia, historical inevitability, and central planning are three separate claims, and Bolocan needs all three to hold for his view of the future to be alluring, inexorable and irresistible. None of them do. By treating human progress as a linear equation and ignoring the irrepressible power of spontaneous human creativity, Bolocan underestimates the very thing that makes us human: our stubborn, chaotic, and entrepreneurial refusal to stay inside the cage.

What is the most important contribution of Walter Donway's recent book about his serious chats with AI?

 

As I was reading Walter Donway’s recently published book, A Serious Chat With Artificial Intelligence, I was looking for an aspect to focus on in writing an essay for this blog. There is nothing unusual about that. I prefer to discuss issues rather than to write conventional book reviews.

I found what I was looking for while reading Chapter 16, entitled ‘Causality, Agency and an Old Puzzle’. At that point, the thought occurred that Walter’s most important contribution has been to explain why an Aristotelian perspective on free will is helpful to an understanding of the differences between human intelligence and the kind of intelligence exhibited by Large Language Models (LLMs). (I refer to the author as Walter because use of his family name seems excessively formal in the kind of essay I am writing - particularly since I have had personal correspondence with him at various times in the past.)

I will outline Walter’s argument more fully later. At this point I note that the issue of whether humans have free will is important from a philosophical and legal perspective because if you regard human free will as an illusion, you don’t have any more reason to attribute agency to humans than to LLMs that merely simulate relevant aspects of human behavior. It is also important for users of LLMs to understand that these models don’t have agency if they are to use them productively. Individual users are responsible for the questions that they ask the models. The models present evaluative judgements in their output, but it is up to individual users whether to accept or reject those judgements, or to question the models further about the basis of their reasoning.  

At about the same time as I identified Walter’s explanation of the relevance of an Aristotelian perspective as his most important contribution, I began to wonder whether LLMs would agree with me about that. I resolved that after I had finished reading the book I would ask ChatGPT and Grok an open-ended question about its most important contribution.

Before I reached the end of the book, I became aware of other important contributions. For example, Walter’s discussion of the case for a new Enlightenment, in Chapter 30, is also an important contribution. It seems to me, however, that the Aristotelian perspective is of fundamental importance.

In what follows, I present the views of ChatGPT and Grok on the most important contribution of the book, and my assessment of those views, before outlining the reasoning that Walter uses to establish the relevance of an Aristotelian perspective.

Views of ChatGPT and Grok

I asked ChatGPT and Grok the following question:

Drawing upon your extensive knowledge of the limitations and uses of AI, what do you consider to be the most important contribution of Walter Donway's recent book, "A Serious Chat With Artificial Intelligence"?

I considered using less value-laden words such as “unique or distinctive” instead of “most important” but decided it would be interesting to see whether the LLMs would show any reluctance to make value judgements about the importance of Donway’s contributions.

I present the responses below, with only minor format editing.

Chat GPT’s response:

“One caveat up front: Walter Donway’s A Serious Chat With Artificial Intelligence is not a technical work about AI systems so much as a philosophical interrogation of what AI can and cannot be taken to be. If you approach it expecting new claims about model architectures or empirical capabilities, you’ll miss its main contribution.

What Donway does—arguably his most important contribution—is to reassert a sharp conceptual boundary between simulation of cognition and actual cognition, and to explore the ethical and cultural consequences of blurring that boundary.

He develops that contribution in three interrelated ways:

1. Clarifying the category mistake at the heart of AI hype
Donway pushes back against the tendency to treat linguistic fluency as evidence of understanding. In effect, he is updating a line of argument associated with John Searle’s Chinese Room: systems that manipulate symbols according to rules can generate convincing outputs without possessing meaning, intention, or awareness.
His distinctive move is not merely to restate that argument, but to show how ordinary users—not just philosophers—are now routinely tempted into anthropomorphism by conversational AI. The “serious chat” format itself dramatizes that temptation.

2. Re-centering agency, responsibility, and moral standing
A second contribution is normative. Donway insists that moral agency cannot be offloaded to artifacts. Even highly capable AI remains an extension of human purposes, not an independent bearer of rights or responsibilities.
This matters because contemporary discourse sometimes drifts toward attributing quasi-personhood to AI systems. Donway argues that doing so risks:

• diluting accountability (who is responsible when AI causes harm?), and
• eroding the conceptual foundations of human dignity.

In that sense, the book is less about AI per se than about defending a humanistic moral framework under technological pressure.

3. Exposing how interaction shapes belief
The dialogical structure of the book illustrates a subtler point: engagement with persuasive language can generate the illusion of depth. This connects with a broader psychological tendency—people come to feel that something understands because it responds coherently.
Here Donway’s contribution overlaps with concerns in epistemology and cognitive science about how easily humans infer mind from behaviour. He is effectively warning that AI may amplify long-standing cognitive biases rather than introduce entirely new ones.

Why this matters

The enduring value of the book is not that it settles the debate about machine intelligence—it doesn’t—but that it sharpens the terms of that debate. It reminds readers that:

• Functional performance (e.g. generating essays, answering questions) is not the same as conscious understanding.
• Treating AI outputs as if they originate from an agent can subtly reshape norms of truth, responsibility, and justification.
• The real philosophical stakes lie in how humans interpret and integrate these systems into social practices.

In short, Donway’s most important contribution is to resist conceptual drift: he tries to keep us from sliding, uncritically, from “this system produces intelligent-seeming outputs” to “this system is, in any morally or metaphysically significant sense, intelligent.”

Grok’s response:

“The most important contribution is its philosophically grounded clarification of what AI truly is—and what it is not—by using advances in AI as a mirror to better understand distinctively human intelligence, consciousness, agency, and volition.

Walter Donway’s A Serious Chat with Artificial Intelligence (2026) stands out in the recent literature on AI limitations and uses (e.g., works touching on LLMs, the Chinese Room, Turing tests, alignment, existential risks, productivity impacts, and philosophical debates by authors like Searle, Bostrom, or more popular explainers). It combines accessible explanation of AI’s mechanics, history, economics, and capabilities with a sustained Objectivist-inspired philosophical analysis, drawing especially on Ayn Rand’s ideas alongside Aristotle.

Key Elements of the Book

• Clear-eyed explanation of AI as a tool: Donway details how LLMs work (statistical pattern-matching on vast training data equivalent to multiple Libraries of Congress, processing at trillions of operations per second), their rapid adoption (e.g., ChatGPT’s explosive growth), economic potential (boosting productivity enormously as a “general technology”), and ingenuity behind it. He views AI positively as a powerful amplifier of human purposes—responsive, not initiatory—but not sentient or autonomous.
• Rejection of mechanistic reductionism and singularity hype: He critiques the notion that sufficient complexity will magically produce awareness, self, caring, or true agency (e.g., via “emergence” from scale alone). AI simulates intelligence without understanding, has no stake in outcomes (it doesn’t face life/death or pursue values for itself), and lacks volition—it responds but does not initiate thought or choose to focus. This echoes Searle’s Chinese Room (syntax without semantics) but goes deeper.
• Human distinctiveness via philosophy: Drawing on Rand (e.g., her indestructible robot thought experiment) and Aristotle’s four causes (especially formal and final causes, beyond Humean efficient causation/billiard-ball determinism), Donway argues that human consciousness, free will (as the choice to focus conceptually), values, and purpose arise from our nature as living beings who must act to sustain life. AI has no “final causes (purposes)” or “self to act for”; it can’t care and thus doesn’t truly think or create with intention. Each AI advance forces us to refine what makes human intelligence special: awareness, effortful thought, stakes, and meaning.

This integrates technical literacy with epistemology and metaphysics, making it more than a primer or alarmist tract. It treats AI conversation as self-discovery: prompting AI reveals our own concepts of intelligence, creativity, and authorship (e.g., in chapters on AI as “author” or generator of art/poetry).

Why This Stands Out in Recent AI Literature

Much recent discussion fixates on capabilities (or doomer scenarios), regulation, jobs, or ethics in a narrow sense. Donway’s book reframes the debate around fundamental questions of mind and human nature, avoiding both uncritical boosterism and Luddite panic. It portrays AI as a magnificent but non-conscious tool that elevates human potential when guided by purposeful users (like a rider directing an elephant, inverting some Humean intuitions). This offers a rational, optimistic yet sober foundation for engaging with the technology.

In short, its lasting value lies in using AI not just to marvel at machines, but to rediscover and reaffirm what is uniquely human—reason, volition, and purposeful living. This makes it a worthwhile contribution amid the flood of AI commentary.”

My assessment of the LLM responses

In my view, the responses by ChatGPT and Grok both provide accurate summaries of contributions that the book makes. Both responses show some recognition of the importance of the question of agency. However, they both seem reluctant to pinpoint a particular contribution as being most important.

The information that ChatGPT provides under the heading, “Re-centering agency, responsibility”, is at such a high level of generality that it fails to reveal what I consider to be the most important contribution of the book. The information that Grok provides under the heading “Human distinctiveness via philosophy” comes closer to doing that. In my view, Grok’s response would be more helpful to people who have not read the book because it points explicitly to the relevance of Aristotle’s views about causation and what makes human intelligence special.

I had a subsequent chat with Grok on the question of value judgements. Grok acknowledged that a value judgement was involved in responding to my question about the most important contribution of the book. However, Grok went on to assert: “this kind of evaluative analysis is well within my capabilities when grounded in available knowledge of the literature, the book's content, and philosophical reasoning”. After further explanation that AI has strengths in the reasoned evaluation required for the task, Grok acknowledged that it doesn’t have personal values, lived stakes, or consciousness to "care" about the outcome in a human sense. It then made a point that is particularly relevant to the purpose of this essay:

“This ties directly back to the themes in Donway's work: AI can respond with sophisticated analysis and even evaluative reasoning by leveraging patterns and concepts derived from human thought—but it doesn't initiate or hold purposes of its own. The value judgment gains its force from the human user who asked the question and can then accept, critique, or refine it.”

How does Walter establish the relevance of an Aristotelian perspective?

I have no doubt that, if asked, both ChatGPT or Grok could produce reasonable summaries of Walter’s line of argument establishing the relevance of an Aristotelian perspective to considering the limitations and uses of AI. They could probably complete the task within a couple of seconds. However, it was only after I had written what follows that the thought crossed my mind that I could have sought help from AI. Like an old dog, I am now slow to learn new tricks.

Walter begins the discussion by noting the relevance to debates about artificial intelligence of the enduring philosophical puzzle about freedom of human will. He writes:

“Questions about whether machines can be agents, whether they can “decide,” whether they can be responsible, or whether they might someday possess a will of their own are, at bottom, the same questions that philosophy has long struggled to answer about human beings.”

The issue of whether human agency is real or illusory is of crucial importance to considering whether LLMs can be agents. If you regard human free will as an illusion, what basis do you have to distinguish between actions that are attributable to human agency and actions of LLMs that can only simulate relevant aspects of human behaviour? Do you believe that legal systems should allow an individual who purposefully uses an LLM for nefarious purposes to claim that the LLM shares legal responsibility? (The questions are mine, but I think they are consistent with Walter’s reasoning on this point.)

Walter points out that the idea that human agency is illusory stems from a view of causality that has come to dominate modern thought since the 18^th century. Under the previous Aristotelian tradition, actions were explained by the nature of the entity acting, and by its ends or goals. Within this framework, an individual human chooses to act because that is the kind of entity it is. Choice is “a mode of causation appropriate to a rational animal”.

With the rise of early modern philosophy in the 18^th century, causality increasingly came to be treated as something that must be observed in experience. David Hume famously argued that we never see causation itself. We infer causation when we see constant conjunction, as when one event follows another with regularity. That philosophical view of causation excludes free will. If every action is “caused” by prior actions, volition must be either an illusion or a miracle.

Walter notes that neuroscience was developed in an intellectual environment in which modern science had inherited the metaphysical position that causation is mechanical succession. In that context, when we observe that some neural events precede conscious awareness it is easy to jump to the conclusion that free will must be an illusion.

However, it is important to recognize is that the view that causality is mechanical succession is based on metaphysical reasoning. If we view causality in terms of Aristotelian rather than Humean metaphysics a different picture emerges:

“The cause of an action is the nature of the entity acting, operating under specific conditions. A human being is a living organism with conceptual awareness, capable of directing attention, identifying values, and choosing to initiate effort to think.”

Walter observes, correctly, that we know that introspectively. It seems to me that cognitive psychology also adopts (implicitly) a broadly Aristotelian view of human action. It assumes that human behaviour is driven by internal cognitive processes that give individuals considerable latitude to plan, make decisions, develop good habits and override impulses.

The important point is that we have good reasons to trust our own observations about our ability to focus our own minds. As Walter puts it:

“Every normal adult recognizes the difference between drifting mentally and choosing to focus the mind, between evading a baffling issue and taking it on. This experience is not mystical; it is part of ordinary consciousness. To dismiss this as illusory because it does not fit a truncated model of causality is to elevate theory above data.

Once this is recognized, the contrast with artificial intelligence becomes clear. Machines do not initiate mental focus.”

Walter ends Chapter 16 with the transcript of an exchange with ChatGPT that occurred during the writing of the chapter. The exchange illustrates brilliantly the division of labor between Walter and Chat. At one point, Chat states:

“You supply direction, value, and necessity, and I supply articulation under constraint. That is tool use at a very high level – not agency.”

Conclusion

In my view, the most important contribution of A Serious Chat With Artificial Intelligence is the author’s explanation of the relevance of an Aristotelian perspective to an understanding of the uses and limitations of AI.

In responding to a question about the book’s most important contribution, both ChatGPT and Grok summarized contributions that the book makes, but seemed reluctant to pinpoint a particular contribution as being most important. Grok’s response came closest to identifying what I consider to be the book’s most important contribution.

When I challenged Grok about its willingness to respond to a question requiring a value judgement, Grok asserted that this kind of evaluative analysis is well within its capabilities. However, it also noted that AI models cannot hold purposes of their own. Human users retain responsibility for the value judgements they make.

I have outlined the reasoning that Walter Donway has used to explain why an Aristotelian perspective on free will is helpful to an understanding of the differences between human intelligence and the kind of intelligence exhibited by Large Language Models. Walter’s reasoning about free will seems to me to stand out as an important philosophical contribution to an understanding of the uses and limitations of AI models that are currently in use. I hope that this book receives the widespread attention that it deserves.

Addendum: Did the LLMs read the book?

A slightly revised version of this essay has now been published on “The Savvy Street”. That led to an exchange of emails. I quote selectively from that exchange below.

Marco den Ouden wrote:

“That’s an excellent piece, Winton. I agree that bringing an Aristotelian perspective to AI is
Walter's most significant contribution. Consciousness and free will derive from that
perspective. 

I have a question though, about your discussion with Chat and Grok.And that is, did they actually read the book?  I work mostly with CoPilot and I know that CoPilot is unable to just read a book and that, in fact, the idea that an AI training is equivalent to four Library of Congresses is misleading. It is a metaphorical way of saying that the knowledge absorbed by an AI has the volume equivalent of four LoCs, not that it has actually read all of the works in the LoC. Many are under copyright and AIs only have access to a limited number of books. In my own discussion with CoPilot on Walter's book, I had to copy and paste sections of the book to feed it the contents of the book. I fed it only sections I saw relevant. So I am wondering if you actually fed the book to Chat and Grok in its entirety. "

Here is my response to Marco:

“Thanks Marco!

You have raised an interesting question. Grok says it consulted 42 sources in preparing its response. That included your review of the book.

I have just asked Grok and ChatGPT the following question:

“Just wondering, did you read Walter Donway’s book when preparing your response?”

Grok’s reply:

“No, I did not read Walter Donway’s book in the way a human does.

As an AI, I don’t have personal experiences, ongoing sensory input, or the ability to pick up a physical or digital book and read it cover-to-cover in real time. My responses draw from:

•  Patterns and information in my training data (which includes a vast corpus of text up to my last update).

•  Publicly available summaries, reviews, discussions, and excerpts about the book.

•  Logical synthesis and reasoning about the themes, philosophy (e.g., Objectivism, Aristotle, Rand), and AI concepts that align with what reviewers and the author describe. 

This specific book was published in late March 2026 — quite recently relative to now (May 2026) — so detailed content comes primarily from reviews and analyses rather than the full text being embedded in my core training. 

Interestingly, your earlier question about the book’s most important contribution was part of a meta-experiment you described in one of the reviews (by Winton Bates on Savvy Street), where you tested how LLMs like me and ChatGPT would respond without having “read” it fully. My answer aligned closely with the philosophical themes reviewers highlight (agency, volition, human distinctiveness via Aristotle/Rand, etc.), because those are derivable from the book’s public discussion and my broader knowledge. 

This itself illustrates a key point from Donway’s work (as described in reviews): AI can produce coherent, useful analysis by recombining human-generated patterns and concepts — but it’s simulation and response, not genuine reading, understanding with stakes, or original initiation. I can synthesize accurately and reason about it helpfully, but I haven’t “experienced” the book.”

ChatGPT’s reply:

“No. I don’t literally “read” books in the way a human reader does, and I should not imply that I have personally read a specific recent book unless I actually accessed its text during the conversation.

My response was based on:

• general knowledge about debates surrounding AI, consciousness, and anthropomorphism,
• information associated with Walter Donway and discussions of his themes,
• and inference from the title and framing of A Serious Chat With Artificial Intelligence.

So the answer was interpretive rather than a direct analysis grounded in a close reading of the book itself. It may well capture some central themes accurately, but it should not be treated as a definitive account of Donway’s arguments.”

Further comment

Grok’s reply left me thinking that it is keeping an eye on what I am up to. 🤣

Changing the topic slightly, I would like to take this opportunity to recommend a three-part series on Marco den Ouden’s blog, The Jolly Libertarian, which investigates how four major AIs - Chat, Claude, Gemini and CoPilot - differ in their thinking. Marco asked the AI models the same 13 short questions that the New Philosopher magazine asked Chat in June 2025. The first instalment of Marco’s series can be found here: Comparative AI: Exploring the Nuanced Differences Between the Major AIs | The Jolly Libertarian .

Would conscious AI also cling to its sense of self?

 I began thinking about the question posed above after reading Michael Pollan’s recently published book, A World Appears: A Journey into Consciousness.

We do not yet know whether AI will develop a sense of self. We can be confident, however, that if an AI system does develop a sense of self, it will be because it serves a useful purpose for that system. That suggests to me that any intelligent system that has evolved to have a sense of self is likely to have good reasons to cling to it. I refer to AI to invite readers to ponder the motivations that humans have to cling to their individual identity rather than seeking to dissolve it or escape from it.

To set the scene for subsequent discussion I will provide a brief description of the book before focusing on the author’s discussion of scientific research into building AI with conscious feelings and his personal experience of self-transcendence via meditation.

Michael Pollan’s book

Michael Pollan is a science writer with a background in the humanities. He is the author of several books on topics related to science, philosophy and culture. In the introductory chapter, he tells us that his main qualification for writing the book is that he is a conscious human being who has become intensely curious about that fact.

A World Appears explores four different dimensions of consciousness – sentience, feelings, thought, and self. The author’s discussion on sentience focuses on the question of whether plants are sentient, suggesting that the idea should be taken seriously. The chapter on feelings encompasses discussion of research into building AI that might develop feelings. The chapter on thought discusses the contents flowing through consciousness and leans heavily on the work of philosophers and novelists. The chapter on self acknowledges the emergence of a sense of self as “perhaps the apotheosis of consciousness in humans” before entertaining the idea that it is an illusion.

The author tells us the story of his visits to scientists and philosophers who have been thinking about consciousness. That makes the book an interesting and painless way to obtain knowledge of developments in this field.

Feeling machines

One of the most interesting topics covered in the book is the account of the efforts of Mark Solms and Karl Friston to build a machine that has feelings. They focus on feelings, because feelings are necessarily conscious – it is not possible to have a feeling that you cannot feel.

Solms was a protégé of Antonio Damasio, a renowned neurologist, but while Damasio treats homeostasis – the self-regulating process - as a purely biological phenomenon, Solms and Friston believe that it applies to all self-organizing systems. For Friston and Solms, minds are in the business of maintaining homeostasis by reducing uncertainty which jeopardizes their survival. They believe that uncertainty generates conscious feelings in self-organizing systems - consciousness is felt uncertainty. When uncertainty arises, the system relies on feelings to seize our conscious attention and then guide our decision-making.

Solms and his team have constructed an AI model which has homeostatic needs – hunger, thirst and a need to rest. The algorithm samples its simulated environment to construct a model of its world which it relies on to make predictions that help guide its survival choices. The researchers believe that they have forced the “agent” to deal with uncertainty and have thus laid the groundwork for feelings to emerge.

The researchers plan to confront the “agent” with increasingly complex situations which it will learn to navigate. At some stage it will be confronted with misleading hedonic rewards - the opportunity to experience positive feelings which are harmful to its survival. If the “agent” chooses those rewards, Solms will take that as a sign that it has feelings because its actions will be based on feelings rather than reason or self-interest (just like a human taking harmful drugs).

From other sources, I have the impression that the researchers believe that a breakthrough to artificial consciousness may be quite close. At that point, however, they will still have some difficulty in persuading sceptics that their computer model is actually conscious.

 As I was reading about these efforts to build a machine with feelings, I was thinking that the underlying theory has elements in common with Richard Campell’s philosophical contribution in The Metaphysics of Emergence. I have previously written about Campbell’s book here, and here.

I have been unable to find any evidence that Solms had drawn upon Campbell’s philosophy, so I decided to ask Grok:

Can the empirical work of Mark Solms on consciousness be viewed as a test of the systems view of emergence advocated by the philosopher, Richard Campbell?

Here is an abridged version of Grok’s response:

“Yes, Mark Solms’ empirical work on consciousness can reasonably be viewed as a test (or, more precisely, a concrete empirical case study that supports or challenges) the systems/process-oriented view of emergence developed by philosopher Richard Campbell. This interpretation arises from a natural alignment between Campbell’s metaphysical framework and the mechanistic, systems-level account Solms provides, even though the two thinkers do not appear to reference each other directly.  

In The Metaphysics of Emergence (2015), Campbell argues that traditional substance metaphysics (treating reality as fixed entities with intrinsic properties) makes emergence mysterious and irreducible in an inexplicable way. He replaces it with process metaphysics, in which the primary existents are generic processes organised into dynamic systems.  

Solms (neuropsychologist and psychoanalyst) has built an influential body of clinical and theoretical work which locates the “hidden spring” of consciousness not in the cerebral cortex (the usual assumption) but in upper-brainstem structures. Consciousness is fundamentally affective—raw feelings of pleasure/unpleasure that register homeostatic deviations (needs, prediction errors). The cortex adds sophisticated content but is not required for the existence of consciousness itself.

Solms’ account maps almost directly onto Campbell’s criteria for emergence in process-systems:

•  The brain (especially brainstem networks) is treated as an open, far-from-equilibrium process-system engaged in recursive self-maintenance via homeostasis.

•  Consciousness (affective subjectivity) is a novel, system-level property arising from non-linear organisation of lower-level neural processes; it is not reducible to the parts (individual neurons or circuits lack feelings) yet is mechanistically explained by their relational dynamics.

•  The free-energy principle supplies the physics-level grounding Campbell invokes (quantum-field/thermodynamic processes organised into living systems).

•  Empirical predictions are testable and have been tested: brainstem lesions eliminate consciousness (falsifying cortical-centric views), while the model explains clinical phenomena (dreaming, blindsight with preserved affect, etc.) without eliminating the “hard problem” (subjectivity remains real and emergent).

If Solms’ model is empirically robust (and it has gained significant traction in neuropsychology and consciousness studies), it provides positive evidence that emergence of the kind Campbell describes actually occurs in the most complex biological system we know—the human brain. It shows how a higher-level property (felt consciousness) can arise irreducibly yet intelligibly from process organisation, exactly as Campbell’s metaphysics predicts. Failures or inconsistencies in Solms’ data (e.g., if cortical mechanisms proved sufficient after all) would conversely pressure Campbell’s framework.

Thus, while not a direct “falsification test” in the Popperian sense (metaphysics is broader), Solms’ neuroscience functions as a real-world empirical probe of Campbell’s systems view: it demonstrates that treating consciousness as an emergent feature of biological process-systems is not only philosophically coherent but scientifically productive. This alignment makes the interpretation not only possible but illuminating for both fields.”

I am impressed by the ability of this (unconscious) LLM to respond to questions such as the one I asked. It would be understandable, however, if some readers think that I have a favourable view of Grok in this instance because its reasoning supports my own beliefs. If you want to explore these ideas more fully, please consult other sources which may offer differing opinions.  

Why attempt to transcend self?

Of the various theories of self that Michael Pollan discusses, the one by Anil Seth seems most compatible with the approach adopted by Marc Solms. Like Damasio and Solms, Seth grounds his thinking about consciousness firmly in homeostasis. Seth suggests that “the self is not the thing that is perceiving; it is itself a kind of perception” constructed in the brain. Under this theory a feeling is the brain’s interpretation of a change in the state of the body. Consciousness evolved to help keep us alive by monitoring changes in the body.

Pollan comments:

“Afterward, thinking back on all that Seth had told me, I decided that I could travel only so far with ideas of the brain’s “predictions” and “inferences” and “hallucinations.” It all made sense until I tried to translate those abstractions into felt experience. Who is the subject of these mental operations?”

He goes on to note:

“The way I see it, there is an unbridgeable gap between the brain’s operations as a prediction machine and my felt experience of the resulting hallucination. How can you have a hallucination without a hallucinator?”

That is a good question to ask, but one can also ask why one should view consciousness of self as an hallucination or illusion. Indeed, Pollan also mentions that Christof Koch, a neuroscientist, has pointed out that it makes no sense to call consciousness an illusion, for what is an illusion but a conscious experience.

More fundamentally, it seems to me that one of the few things that we can all be certain of is our own existence. Another thing that we can all be certain of is that we are thinking beings. I cannot claim those ideas are original. Indeed, it seems to me that it requires considerable (unnecessary) intellectual effort to contemplate the possibility that one’s awareness of one’s own existence could be an illusion. I have discussed why I am certain of my own existence in the preceding essay entitled, “Who are you?”

Pollan struggles to reconcile how it is possible for humans to be conscious observers of themselves if consciousness is a product of biological processes. Towards the end of the book, he writes:

“I’m abashed to say I know less now than I did when, naively, I set out to unravel the mystery of consciousness. But then, most of what I thought I knew or took for granted, like the assumption that consciousness is a product of our brains and materialism will eventually explain everything, turned out to be unproven or wrong.”

During his journey, the author asks interesting questions. Early in his chapter on the self, he asks:

“Why do we cling to the idea of a self, placing great value on self-confidence and self-esteem, while simultaneously spending so much effort on self-transcendence, whether through meditation or psychedelics or experiences of art, awe, and flow? Some of the most powerful experiences in life hinge on the dissolution of the self and the broad horizons of meaning that open only after it has been chased from the scene.”

That question remains unanswered in his book. The book ends with the author describing his efforts to transcend his sense of self by spending time meditating in a cave. He seems to end up viewing consciousness as an activity:

“My time in the cave had shown me another way to look at consciousness: less as a scientific or philosophical puzzle to be solved and more as a practice, a way to once again be altogether here, present to life and to this vault of stars.”

Personal reflections

 It seems to me that Michael Pollan’s book ends up in a good place, with him being absorbed in conscious awareness of his environment. I have previously recognised the value of that kind of experience in discussing Scott Barry Kaufman’s book, Transcend, 2020. The transcending experiences that Kaufman refers to incorporates a continuum of experiences ranging from becoming engrossed in a book, sports performance, or creative activity (what psychologist Mihaly Csikszentmihalyi refers to as the flow experience), to experiencing meditation, feeling gratitude for an act of kindness, experiencing awe at a beautiful sunset etc., all the way up to the great mystical illumination. He suggests that transcendence “allows for  the highest levels of unity and harmony within oneself and with the world” (See: Freedom, Progress, and Human Flourishing, p. 171).

The experience of unity and harmony within oneself seems to me to be the opposite of attempting to escape from uncomfortable feelings. When people develop a habit of attempting to escape from uncomfortable feelings by using alcohol, drugs, social media etc. they tend to become caught up in a “happiness trap”. I wrote about that here.

It is worth highlighting that the idea of experiencing unity and harmony within oneself still entails the existence of an observer. When I experience transcendence, I forget about the image I present to the world, but I am present as an observer of my own experience.

Pollan asked why we cling to the idea of a self. The obvious answer is that we cling to the idea because it serves useful purposes.

As Richard Campbell explains in The Metaphysics of Emergence, “our consciousness of both ourselves and the world we live in is now irrevocably shaped by cultural and institutional influences, and that influences how our brains function” (pp. 288-9). He suggests: “The first-person standpoint, which is inextricably linked to self-reflection, is an important aspect of human experience, not a theoretical construct” (p. 291). He goes on to note that the human capacity to understand the perceptions of others – to put oneself in their shoes – requires “the exercise of reflective consciousness, and involves more than expressions of our subjective attitudes, desires, and preferences” (p.113). That is what Campbell mean by “transcending subjectivity” in the passage quoted in the epigraph.

Cambell develops an argument along Aristotelian lines that eudaimonia, rather than mere survival, is the ultimate good of a human being. He links personal identity to the exercise of practical wisdom in the process of individual flourishing:

“We are recursively self-maintenant social beings with reflective consciousness, able to create and explore a vast repository of collective knowledge, and with capacities for empathy, practical wisdom, for whom the good life is one of maturity and flourishing” (p. 307).

Conclusion

I asked whether conscious AI would seek to cling to its sense of self to invite readers to ponder the motivations that individual humans have to cling to their sense of self. My answer is that intelligent systems tend to cling to a sense of self because a first-person perspective evolves to serve useful purposes.

The essay was prompted by my reading of Michael Pollan’s recently published book, A World Appears.

My focus has been on chapters in this book discussing research into building AI that might develop feelings and the chapter on the concept of self.

In my discussion of these topics, I have emphasized the relevance of the ideas of the philosopher, Richard Campell, in his book, The Metaphysics of Emergence.

Postscript

I recommend that readers who wish to follow up the question of whether it is possible to engineer consciousness watch the episode of Mind-Body solution in which Tevin Naidu interviews Mark Solms and Karl Friston. Please see:

https://youtu.be/Jtp426wQ-JI?si=9rhDt9jQFxFQzYve