Avoiding the Calculator Problem
Millions of Western adults had this sentiment drubbed into them as schoolkids. Well-meaning teachers implored pupils to practice long division not knowing how cheap, powerful, and ubiquitous calculating devices would soon become. Of course, learning those rote skills may still have been best for the students, just not for the reasons given.
At a time of peak distrust of institutions, disaffected grown-ups, smartphones in hand, harken back to this calculator problem as a justification for their deeper doubts about schools and educators at all levels.
Those cynical adults may be a lost cause, but we can stave off a repeat of the calculator problem by getting in front of technological churn. This is no easy task. Teaching to the technology of 20 years hence is like hitting a bullseye that’s accelerating at the exponential rate of Moore’s Law.
Of course, we can’t predict the future, but we can model anticipation and adaptability. Right now, A.I. and Large Learning Models (LLMs) are dominating the discourse about current and future pedagogy. Even since the 2022 release of ChatGPT, the LLMs have grown more capable. “Hallucinations” are more rare and less egregious. They are more adept at analyzing scholarly texts and understanding nuance. Let’s test their capabilities so we can understand and teach their responsible use to students.
Keep your friends close, and your LLMs closer
There has been a great deal of hand-wringing, some of it well placed, about the encroachment of Artificial Intelligence upon academic research. Targeted ads promise to help students quickly write papers, power through long readings, and even circumvent plagiarism detection apps. Very alarming! Yet, we shouldn’t let the grotesque way these products are positioned cloud our vision; AI holds tremendous power for good.
I thought it would be helpful to challenge several prominent AI systems (those aimed at college and university researchers) and to feed their results back into LLMs. I extracted a few real-life library research questions from our logs, and massaged them into AI prompts. This was a contest amongst AIs, adjudicated by other AIs.
The Contenders
Scite: No free tier, $20/mo, 7-day trial
Elicit: Basic ($0, extracts 10 papers/mo), Plus ($12, extracts 25 papers/mo), and Pro ($49, extracts 100 papers/mo)
Consensus: Registration optional. Free Basic grants 10 analyses using GPT-4. Student discount offers Premium for $7.19/mo with unlimited analyses.
The Judges
ChatGPT: The OG LLM running GPT-4o (generative pre-trained transformer version 4 omni)
Perplexity.ai: Billed as an AI search engine operated on GPT-4o and Claude 3, promising to provide reliable sources, “trusted news outlets, academic papers, and established blogs.”
Copilot: The AI assisted that permeates Microsoft’s 365, Windows, Android, and iOS presences. Based on GPT-4, I the version of Copilot I used came with the M365 subscription my institution uses.
The Prompts
What were feminist perspectives on the kitchen debate between Richard Nixon and Nikita Khrushchev? Use primary sources if possible.
Annotated bibliography on the treatment of adolescent idiopathic scoliosis
Create a literature review of Post-traumatic stress disorder and depression in elementary school children of immigrant families in the United States
The stage is set. In part 2, I will explain the process of using the AIs to answer research taks.