The origins and foundations of AI

From the mechanical automata of the eighteenth century to today’s data centres, which devour gigawatts of power, humanity has nursed a constant longing: to create an artificial reflection of its own intelligence. That ambition, which for centuries lived in the realms of myth and science fiction, found its first mathematical and philosophical formulation in the middle of the twentieth century, at the hands of Alan Turing.

In 1950, Turing proposed a thought experiment that would change the philosophical and scientific debate for good: the imitation game, known today the world over as the Turing test. His proposal sidestepped the thorny metaphysical question —can a machine think?— to focus on an observable criterion: can a human judge tell whether they are talking to a person or to a machine over a text channel? If the system’s answers are indistinguishable from a human’s, the machine has passed the test.

Three decades later, in 1980, the philosopher John Searle answered with a famous counterexample: the Chinese room. Picture a person locked in a room who does not know a word of Chinese, but who has a gigantic rulebook in English telling them how to manipulate the Chinese symbols slipped under the door (“if you receive symbol X, reply with symbol Y”). To someone outside, the answers are perfect and the system seems to understand Chinese fluently. Inside the room, however, nobody grasps what the characters mean.

This thought experiment anticipated the great dilemma we still carry today: the yawning gap between syntactic simulation —manipulating symbols effectively— and semantic understanding.

From the human brain to the artificial neuron

Faced with the path of rigid rules, which would end up stalling, the pioneers of cybernetics chose to look at the most efficient computing model there is: the human brain. An organ that, on barely 20 watts —less than a conventional light bulb—, can process emotions, solve equations and compose symphonies. As early as 1943, the neurophysiologist Warren McCulloch and the logician Walter Pitts had described the neuron as a small logical unit that, depending on the signals it receives, either fires or stays at rest. It was the seed of everything that followed.

Anatomy of the biological neuron

The nervous system is made of billions of interconnected cells called neurons: the human brain alone holds some 86 billion. Their basic architecture breaks down into four functional parts:

1. Dendrites: branching structures that act as antennas, receiving electrical and chemical impulses from other neurons.
2. Soma, or cell body: the processing centre, which integrates all the incoming signals. If their sum crosses a certain electrical threshold, the neuron “fires”.
3. Axon: a long extension along which the electrical impulse generated in the soma travels to other cells.
4. Synapse: the point of biochemical connection between the axon of one neuron and the dendrites of another. Its plasticity —the ability to grow stronger or weaker with use— is, as the psychologist Donald Hebb intuited in 1949, the physical basis of learning and memory.

The perceptron: the artificial neuron

In 1957, the psychologist Frank Rosenblatt translated this biological architecture into the language of mathematics by inventing the perceptron, the direct ancestor of today’s neural networks.

Mathematically, an artificial neuron is a function that takes several numerical inputs (x₁, x₂, …, xₙ). Each one is multiplied by a weight (w₁, w₂, …, wₙ), the mathematical equivalent of synaptic strength: a number that says how much that particular input counts towards the final decision.

A correction term called the bias is added to these weighted values, shifting the function so that the neuron finds it harder or easier to fire. The result then goes through an activation function —a mathematical filter, such as a step function or a sigmoid— that sets the neuron’s output.

The analogy with the human neuron is direct:

1. Inputs: the equivalent of the signals arriving through the dendrites.
2. Weights: the strength of the biological synapse. They multiply each input to indicate its importance: some data weigh more than others in the decision.
3. Weighted sum: like the soma, it adds up the products of each input by its weight and adds the bias, which acts as the neuron’s natural tendency to fire or to stay quiet.
4. Activation function: the final trigger. If the result crosses a threshold, the artificial neuron “fires” and passes the signal on to the next layer; if not, it stays inert.

How training works

A single artificial neuron is barely good for solving trivial problems. But when we stack millions —or billions— of them in interconnected layers, we get an artificial neural network.

How does such a network learn? At first, the weights (w) are initialised with random numbers, so the results are nonsense. During training, the system compares its prediction with the right answer through a loss function, which measures how wrong it was. A fundamental algorithm, backpropagation, distributes that error backwards from the last layer, and an optimiser —such as gradient descent— adjusts every weight ever so slightly in the direction that reduces it. Repeat that cycle millions of times over massive amounts of data, and the network learns to recognise complex patterns on its own.

Machine learning versus deep learning

For decades, neural networks were a promise thwarted by a shortage of computing power and by their own mathematical limits: in 1969, Marvin Minsky and Seymour Papert proved that a single-layer perceptron cannot solve even a problem as simple as “exclusive or” (answering “yes” when one of two conditions holds, but not both), and interest in these networks collapsed for years. Meanwhile, traditional computing relied on explicit programming: someone had to write rigid rules (“if A happens, then do B”). Machines excelled on logical assembly lines, but were blind to ambiguous tasks such as recognising a face in a photograph.

The revolution came with machine learning and, later, with its deeper evolution, deep learning, driven by three factors: cheaper graphics processors (GPUs), the internet as an ocean of data, and neural network architectures with many hidden layers. Its symbolic moment came in 2012, when AlexNet, a deep network trained on GPUs, swept the ImageNet image recognition contest.

In deep learning, hand-crafted feature engineering disappears. Feed a deep network millions of images of vehicles without ever explaining what a wheel or a handlebar is, and its first layers will learn to detect elementary patterns (lines, edges, contrasts); the middle layers will combine those lines into geometric shapes (circles, angles), and the final ones will assemble the puzzle to decide whether the object is a car, a bicycle or a lorry. The machine extracts its own abstract representations of the world from the raw data.

Feature	Traditional machine learning	Deep learning (deep networks)
Human involvement	Requires manual feature engineering: an expert has to decide in advance which traits or measurements to look for (for example, edges or textures).	Features are learned autonomously from the raw data, through successive hidden layers.
Data volume	Works well with small or moderate datasets.	Needs massive volumes of data to avoid overfitting and to generalise correctly.
Capacity for abstraction	Limited to the traits the expert provides and to relatively simple relationships between them.	Builds hierarchies of complex concepts: lower layers detect lines; middle layers, shapes; upper layers, faces or whole objects.

The big difference between the two lies in depth: while traditional machine learning works with one or two layers of mathematical transformation, deep learning chains together tens or hundreds of hidden layers, which lets it model strongly non-linear realities such as the human voice, the structure of proteins or natural language.

What is an LLM, really?

When we get to large language models (LLMs), like the ones behind today’s generative AI revolution, the mathematical principle is still the same —deep neural networks tuned through backpropagation—, but applied to the fabric of human communication: text.

The ease with which today’s LLMs write essays, code or hold conversations has spread the false impression that machines reason like people, but nothing could be further from the truth. To understand how they really work, we have to strip language of its mystique and look at its mathematical gears. An LLM’s engine rests on two pillars: the token and stochastic prediction.

What is a token?

A computer does not understand words the way a human does. It does not read letters or syllables: it processes numbers. To make that possible, before processing a text, an LLM breaks it up with a tokenisation algorithm.

A token is not necessarily a whole word: it can be an entire word, a syllable, a suffix or even a single character (in English, a token averages about three quarters of a word, or roughly four characters). For example, a tokeniser might split the phrase “artificial intelligence” into [art], [ificial] and [ intelligence], and give each fragment a unique numerical identifier (say, [433, 16895, 11478]) within a vocabulary that usually runs between 32,000 and 250,000 distinct tokens.

Each token is then projected into a mathematical space of a great many dimensions as an embedding vector, where words with similar meanings or contexts sit close to one another, as on a map. And the relationships between them are preserved: the shift that takes you from “king” to “queen” is very similar to the one that takes you from “prince” to “princess”.

Stochastic prediction

Once the text has become a sequence of numbers and vectors, the heart of the LLM —an architecture based on the attention mechanism, the Transformer— carries out a task that looks simple but has colossal power: predicting the next token.

An LLM is not an encyclopaedic database that looks up a fixed answer in a file. It is an ultra-high-dimensional probabilistic engine. Given an input sequence (a prompt), the neural network applies the statistical regularities it learned from billions of texts and computes a probability distribution over which next token would be most coherent.

If we give it the unfinished sentence “The cat climbed up onto the…”, the neural engine does not “think” about felines, roofs or the physics of gravity. What it does is a massive probabilistic calculation through its synaptic weights, shaped by the billions of texts it analysed during pre-training. The model returns a probability distribution for the next token:

• roof: 49.6%
• tree: 34.3%
• sofa: 10.2%
• …
• shoe: 0.02%

Through a control parameter, the temperature, the system picks one of these tokens at random —strongly favouring the most likely ones, while leaving some room for creative variation— and repeats the process, token by token, recursively.

What is temperature? In language models, temperature is a numerical parameter that controls how random and creative the generated text is. It acts on the probabilities with which the model picks the next token: the lower the temperature, the more deterministic and predictable the answer; the higher, the more creative and varied.

In the same way, if we write “The cat ate the…”, the model will work out that the odds of the next token being “fish” or “food” are very high, while those of “nebula” or “violin” are practically zero. String thousands of these predictions together at high speed and the illusion of a fluid, coherent and creative conversation emerges.

This stochastic fluency, carried out on a colossal scale across billions of parameters, is what so convincingly simulates human cognitive competence, blurring the line between numerical calculation and rational discourse.

References and scientific support

• The paper in which Turing replaced the question “can machines think?” with the imitation game. Turing, A. M. (1950). Computing machinery and intelligence. Mind, 59(236), 433-460. View paper (DOI)
• The Chinese room: manipulating symbols by following rules is not enough to understand what they mean. Searle, J. R. (1980). Minds, brains, and programs. Behavioral and Brain Sciences, 3(3), 417-424. View paper (DOI)
• The first mathematical model of the neuron: a logical unit that fires or not depending on the signals it receives. McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 5(4), 115-133. View paper (DOI)
• At 2% of body weight, the brain takes 20% of the metabolic budget and runs on about 20 watts, less than a laptop. Balasubramanian, V. (2021). Brain power. Proceedings of the National Academy of Sciences, 118(32), e2107022118. View paper (DOI)
• The count that put the number of neurons in the human brain at around 86 billion. Azevedo, F. A. C., Carvalho, L. R. B., Grinberg, L. T., Farfel, J. M., Ferretti, R. E. L., Leite, R. E. P., Jacob Filho, W., Lent, R., & Herculano-Houzel, S. (2009). Equal numbers of neuronal and nonneuronal cells make the human brain an isometrically scaled-up primate brain. The Journal of Comparative Neurology, 513(5), 532-541. View study (DOI)
• The idea that learning means strengthening the connections between neurons that fire together. Hebb, D. O. (1949). The organization of behavior: A neuropsychological theory. Wiley. View book (DOI)
• Rosenblatt’s perceptron: a network that learns to classify by adjusting its weights from examples. Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386-408. View paper (DOI)
• The limit of a single layer: no simple perceptron can learn “exclusive or”. Minsky, M., & Papert, S. (1969). Perceptrons: An introduction to computational geometry. MIT Press. View book (2017 reissue)
• Backpropagation: how to share out the error among the hidden layers so that a multi-layer network can learn. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533-536. View study (DOI)
• AlexNet, the GPU-trained deep network that in 2012 beat its rivals on ImageNet by ten points. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84-90 (originally published at NeurIPS 2012). View study (DOI)
• What each layer of a vision network sees: edges in the first, shapes in the middle, objects in the last. Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding convolutional networks. Computer Vision – ECCV 2014, Lecture Notes in Computer Science, 8689, 818-833. View study (DOI)
• The review by the three pioneers of deep learning: learning representations from data instead of designing them by hand. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444. View paper (DOI)
• Subword tokenisation: breaking rare words into fragments that are common. Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 1715-1725. View study (DOI)
• Analogies in vector space: “king” is to “queen” as “man” is to “woman”. Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic regularities in continuous space word representations. Proceedings of NAACL-HLT 2013, 746-751. View study on ACL Anthology
• The paper that introduced the Transformer, the architecture built on the attention mechanism alone that today’s LLMs rest on. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998-6008. View study on arXiv
• GPT-3: a model trained only to predict the next token that, at scale, solves new tasks from just a few examples. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., … & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901. View study on arXiv
• How each token gets chosen: temperature and other sampling strategies, and why always taking the most likely one produces repetitive text. Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2020). The curious case of neural text degeneration. International Conference on Learning Representations (ICLR). View study on arXiv
• The Chinese room, forty years on: “stochastic parrots” that string linguistic forms together with no access to their meaning. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610-623. View paper (DOI)