In a significant study detailing the role of generative AI in higher education, researchers from the École Polytechnique Fédérale de Lausanne (EPFL) have assessed the performance of advanced AI language models on higher education assessments. This large-scale investigation, published in the Proceedings of the National Academy of Sciences, explores the implications of AI tools, such as ChatGPT, on learning outcomes and student assessment methods across various academic disciplines.
Since its remarkable emergence in late 2022, when ChatGPT rapidly attracted over 100 million users within just a month, artificial intelligence has sparked ongoing discussions about its potential effects on a multitude of sectors, including education. The EPFL researchers conducted their study across 50 distinct courses, sampling materials from Bachelor's, Master's, and online programmes in STEM fields. Assistant Professor Antoine Bosselut, who leads the Natural Language Processing Laboratory at EPFL, stated, "We were lucky that a large consortium of EPFL professors, teachers, and teaching assistants helped us collect the largest data set to date of course materials, assessments, and exams to get a diverse array of materials across our degree programs."
The study focused on two advanced AI models, GPT-3.5 and GPT-4, utilising eight different prompting strategies to evaluate their performance. Remarkably, GPT-4 achieved an average correct response rate of 65.8% and could provide accurate answers for at least one prompting strategy to 85.1% of questions. Anna Sotnikova, a scientist at the NLP, remarked, "We were surprised at the results. Nobody expected that the AI assistants would achieve such a high percentage of correct answers across so many courses." She noted the surprising aspect of the results was their attainment using a basic prompt, indicating the accessibility of these powerful tools to users without technical expertise.
The findings raise important questions regarding the potential vulnerabilities associated with AI's integration into education. The researchers posited that while AI can enhance teaching and learning, it may also inadvertently allow students to circumvent traditional learning processes, leading to weaker foundational understanding of concepts that are critical for advanced study. Bosselut warned, "The fear is that if these models are as capable as what we indicate, students that use them might shortcut the process through which they would learn new concepts."
Given these dynamics, the study suggests the need for educational institutions to evolve their assessment methods. Bosselut emphasised modifications in assessment complexity rather than merely increasing question difficulty, advocating for holistic assessments that integrate multiple skills. Pierre Dillenbourg, Vice President for Academic Affairs at EPFL, underlined the pressing challenges for higher education institutions, such as determining the necessary skills for future graduates and how best to provide scaled feedback.
The researchers also anticipate that AI models will continue to improve, further complicating the landscape of education in the future. Notably, the study was concluded before the introduction of specific models designed for mathematics, suggesting that if a similar assessment were conducted today, the results could demonstrate even higher accuracy rates.
As this research stands at the intersection of education and technology, it advocates for a gradual adaptation in pedagogical approaches akin to how calculators were eventually integrated into mathematics instruction. Beatriz Borges, a Ph.D. student involved in the study, highlighted the importance of practical recommendations for stakeholders in education to navigate this transition effectively, ensuring that while students can leverage AI technologies, they continue to develop essential skills.
In summation, the EPFL's comprehensive analysis offers critical insights into the intersection of generative AI and higher education, pointing towards significant shifts in teaching methodologies and assessment practices as educational institutions grapple with the evolving role of technology in learning environments.
Source: Noah Wire Services