Although large language models, like GPT, have a revolutionary ability to solve a wide range of tasks in an apparently intelligent way, we will in the following assume that they are essentially just machines good at imitating patterns in text based on the training they have received. We will now take a closer look at the challenges this entails when using text-generating tools in scientific work.

Input

First, there are several problematic aspects of the data material used to train large language models. For example, the profile of the corpus will influence the answers the models give. A model primarily trained on American texts will provide answers reflecting American language and culture.¹ Being familiar with the training material can therefore be crucial to assess how suitable the models are for different uses. A challenge here is that the large companies behind the best language models withhold this information.

Furthermore, a serious legal challenge concerns how the large companies use data they do not actually own.² As users of text-generating tools, we should therefore take into account that the data we share with these tools may be used as training data for new models.

The function of the network models

Second, the very way large language models work creates some challenges for use in scientific contexts. As we have seen, these language models are guessing machines. When trained, they do not store the exact information from the training data, but instead encode statistical relationships between words in the corpus. This makes language models good at imitating natural language, while having only an approximate relation to facts. A common user experience is therefore that language models lie or hallucinate. A crucial point here is that such deviations from facts are not due to malfunction in the models, but rather illustrate how they operate. This uncritical mixing of fact and fiction is poorly suited to scientific contexts where the reliability of the content is important.

To some extent, the accuracy level of language models can be improved by the fine-tuning companies do in the final part of training. At the same time, the limitations (so-called "guardrails") added in this part of the process can lead to biases and censorship that, even if unintended, conflict with academic values.

In addition, a challenge to academic integrity is that we do not understand the process from input to output, and that data processing in large language models is therefore described as a black box. It is difficult to reconstruct the steps in this process in a reliable manner, which seems to contradict the principle that scientific knowledge should be reproducible.

Output

Third and finally, there are certain ethical challenges related to the answers language models provide. This is partly due to, as mentioned earlier, how the structure and training of the models influence the answers they give. More generally, one can say that although the answers language models give are unique, they are a mirror image of what we humans have already produced. In this way, language models reinforce the status quo by reproducing and highlighting dominant patterns in what has already been written. This may have a unifying effect in a fragmented media reality, but it can also create ethical challenges by contributing to reinforcing existing inequalities in society. To the extent that science has a role in solving problems we have never faced before, it also seems problematic to rely too much on such a rearview mirror.³

Another important challenge, which we will return to later in the module, is how we should relate to language models’ answers as a source of information. Although it is easy to get the feeling that the answers come from a person, it does not seem reasonable to hold AI tools responsible for the texts they produce. The output cannot therefore be considered statements or scientific work in itself and should not be regarded as sources in a strict sense until a human endorses what is written and is responsible for the content.

Rettberg, J. W. (2025). Fra demokrati til algortimokrati. https://jilltxt.net/fra-demokrati-til-algoritmokrati/Links to an external site..
Mejias, U. A. and Couldry, N. (2024). Data Grab. The New Colonialism of Big Tech and How to Fight Back. The University of Chicago Press.
Vallor, S. (2024) The AI Mirror: How to Reclaim Our Humanity in an Age of Machine Thinking. Oxford University Press.

Social media News media Academic literature Artificial intelligence