the machine that thinks but doesn't feel

i spend a lot of time building with systems that can write, reason through a bug, and occasionally surprise me. the surprising part is not that they sound human. it is that they can be useful without there being any obvious person on the other side of the words.

consciousness is still an unsolved question. David Chalmers called the gap between information processing and subjective experience the “hard problem.” A system can take in signals and produce complex behaviour; that does not tell us whether anything is being experienced.

there are serious theories, and they disagree. Integrated Information Theory asks how much a system irreducibly integrates information. Global Workspace Theory describes conscious access as information becoming widely available across a system. Experiments have tested predictions from both, but no consensus has settled the matter. A major adversarial study found results that challenged parts of each account rather than handing either one a win. Cogitate Consortium, Nature

intelligence and experience may come apart#

this matters for AI because capable behaviour is not proof of inner life. an LLM can manipulate language well enough to look like it understands. whether that is understanding, or a very powerful form of pattern completion, depends partly on what we think understanding requires.

the honest current answer is that we do not know whether present systems are conscious, and we should not market them as though we do. Researchers have proposed theory-based indicators for assessing AI consciousness; their conclusion is not that current models are clearly conscious, but that future systems could be built with more of the relevant properties. Butlin et al.

the practical risk is here already#

i do not need a model to feel pain before i care about its impact. systems that can take actions, persuade people, or make high-stakes recommendations need boundaries because humans are affected by the result either way.

maybe consciousness turns out to be central. maybe it does not. in the meantime, the thing i keep coming back to is simpler: intelligence without stakes can still cause damage. a system does not need to want something in order to optimise the wrong thing very effectively.

we should keep researching the question. we should also build products as if uncertainty is real: clear limits, human accountability, records of what happened, and no fake claims of personhood just because a demo is fluent.