In 1948, the founder of information theory, Claude Shannon, proposed modelling language in terms of the probability of the next word in a sentence given the previous words. These types of probabilistic language models were largely derided, most famously by linguist Noam Chomsky: “The notion of ‘probability of a sentence’ is an entirely useless one.”

In 2022, 74 years after Shannon’s proposal, ChatGPT appeared, which caught the attention of the public, with some even suggesting it was a gateway to super-human intelligence. Going from Shannon’s proposal to ChatGPT took so long because the amount of data and computing time used was unimaginable even a few years before.

ChatGPT is a large language model (LLM) learned from a huge corpus of text from the internet. It predicts the probability of the next word given the context: a prompt and the previously generated words.

ChatGPT uses this model to generate language by choosing the next word according to the probabilistic prediction. Think about drawing words from a hat, where the words predicted to have a higher probability have more copies in the hat. ChatGPT produces text that seems intelligent.

There is a lot of controversy about how these tools can help or hinder learning and practising creative writing. As a professor of computer science who has authored hundreds of works on artificial intelligence (AI), including AI textbooks that cover the

© The Conversation