Can artificial intelligence help us assess how robust a scientific finding is? This is the question at the heart of the Predicting Replicability Challenge, an international competition organised by the Center for Open Science, in which Federico D’Atri, a PhD student at the University of Trieste, took third place, receiving a prize of $3,375.

The COS competition tested automated systems designed to estimate the likelihood that a scientific finding will be confirmed when a study is independently replicated. In other words, the algorithms are asked to “predict” whether repeating a study with new data and new samples will produce similar results. Replicability is an important factor in assessing the reliability of scientific research, as it helps determine how robust a finding is and whether it can be confirmed using new data.

To generate his predictions, D’Atri developed a system based primarily on a Large Language Model (LLM) specifically adapted and trained for this task. The model analysed the full text of scientific papers, using the information they contained to estimate the likelihood that their findings would be confirmed by an independent replication.

In one of the solutions he submitted, the LLM’s predictions were also combined with those of a more traditional machine-learning model, which used structured information from the studies, including sample size, statistical significance, effect size, year of publication and research discipline.

In the final round of the competition, participants worked on studies from the Replicability Project: Health Behavior, which focuses on health-related behaviours. For each finding, they had to estimate the likelihood that it would be confirmed by an independent replication before learning the actual outcome.

“This topic stems from a long-standing interest of mine in the replicability and reliability of scientific findings, particularly in psychology and the social sciences,” explains D’Atri. “The project gave me the opportunity to explore how artificial intelligence tools can be applied to the analysis of scientific literature”.

Directly replicating a study can require considerable time and resources. This is precisely where systems of this kind could find one of their most interesting future applications. Automated predictions cannot replace actual replications, but they can provide a useful initial indication more quickly and at a lower cost, helping to identify the studies for which further verification may be most valuable.

“The result achieved by PhD student Federico D’Atri once again highlights the central role of curiosity in doctoral training,” says Professor Longo, UniTS Delegate for PhD Programmes. “Curiosity can also lead researchers to explore cross-cutting topics or areas that are not strictly connected with their own research project, as in this case, while playing a fundamental role in shaping their development as researchers”.