Collaborating scientists from several US and Canadian Universities are evaluating how AI (large language models or LLMs in particular) could change the nature of their social science research.
Published this week in the journal Science, Igor Grossmann, professor of psychology at the University of Waterloo and colleagues note that large language models trained on vast amounts of text data are increasingly capable of simulating human-like responses and behaviors. This offers novel opportunities for testing theories and hypotheses about human behavior at great scale and speed.
Data Collection
It has been the tradition in social science studies to rely on a range of methods, including questionnaires, behavioral tests, observational studies, and experiments. A common goal is to obtain a generalized representation of characteristics of individuals, groups, cultures, and their dynamics. With the advent of advanced AI systems, the landscape of data collection in social sciences may shift.
“LLMs might supplant human participants for data collection,” said UPenn psychology professor Philip Tetlock. “In fact, LLMs have already demonstrated their ability to generate realistic survey responses concerning consumer behavior. Large language models will revolutionize human-based forecasting in the next 3 years. It won’t make sense for humans unassisted by AIs to venture probabilistic judgments in serious policy debates. I put an 90% chance on that. Of course, how humans react to all of that is another matter.”
Possible Pitfalls
While opinions on the feasibility of this application of advanced AI systems vary, studies using simulated participants could be used to generate novel hypotheses that could then be confirmed in human populations. However, researchers warn of some of the possible pitfalls in this approach – including the fact that LLMs are often trained to exclude socio-cultural biases that exist for real-life humans. This means that sociologists using AI in this way couldn’t study those biases.
Concerns about data quality, fairness, and equity of access to the powerful AI systems will be substantial. So, the research must ensure that social science LLMs, like all scientific models, are open-source, meaning that their algorithms and ideally data are available to all to scrutinize, test, and modify. Only by maintaining transparency and replicability can studies ensure that AI-assisted social science research can truly contribute to our understanding of human experience.