How I Review Research Papers Using Generative AI
This is the first post in a series exploring how I use Generative AI (GenAI) and various large language models (LLMs) to review research papers. It’s a work in progress.
In a few weeks, I will be coming out with the latest podcast in the series Education Research Rounds. In the podcast I will review a new research paper titled: Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance.
In this post, I’ll walk you through the first two steps of my research review process using GenAI.
Step 1:
I was initially so impressed by GenAI that I began my review process by asking it for a summary. However, I’ve since stopped doing this for a couple of reasons. First, as I will demonstrate in future posts, GenAI systems are unpredictably unreliable. Second, I didn’t want my perspective to be biased by the models right from the start.
So, what’s my first step in reviewing a research paper? I start by reading the paper myself—at least a couple of times. As I read, I scribble some notes and keep a few basic questions in mind:
• What is the research about?
• What methodology is used?
• What are the top-level results?
• Is the paper convincing?
These questions, in a sense, stand on their own, but they also let me develop a first level understanding of the structure of the paper: how the parts are related to the whole.
As an aside, my Aristotle teacher in college introduced his students to the concept of a hermeneutical circle. Professor Aryeh Kosman began our seminar on Aristotle:
“Alright, here’s the thing: if you’re the modern, science-savvy type, you will begin reading Aristotle and think it’s all rubbish. You will encounter words like ‘motion’, ‘energy’, ‘matter’, ‘physics’, and ‘dynamics’. As clever know-it-all Newtonian, you will think “what a boatload of crap.” And hey, I get it—your inner Newtonian is judging hard. But hold on. Suspend that judgment. It’s gonna be hard but give that ol’ dead dog Aristotle the benefit of the doubt. Instant judgment is the death of thinking. And we live in a society where it’s instant everything.
Here’s the trick: try to make sense of the whole picture Aristotle is painting, even if many of the details (the parts) seem like gibberish right now. Think of it as a puzzle—can you construct on your own terms the big picture (the whole) that makes sense of the little confusing pieces? This is the struggle of interpretation and it’s very hard work. Your mind will refuse to do it because our minds are lazy. But it’s an essential hurdle that you will have to learn to overcome.But wait, it doesn’t stop there. As you keep reading and thinking and writing, you’re going to hit new pieces—pieces that don’t fit the picture you’ve built so far. And now your job is to revise your interpretation of the whole to make it work again, taking into account both the old stuff and the new. This constant back-and-forth is the hermeneutic circle. Don’t let the fancy term scare you—hermeneutics just means interpretation. If you have studied Plato’s dialogues, you know exactly what I mean. The process of thinking is dialogue and dialectic. As you know, Socrates was executed by the Athenians for thinking, but don’t let that dissuade you. It comes with the territory.
What do I mean by dialectic? The hermeneutical circle is not really a circle. It’s more like a spiral—an ascending helix. Every time you go around, your understanding deepens, bit by bit, as you go back and forth between the parts and the whole. It’s not easy, but it’s how you climb the ladder of knowledge. Even Aristotle deserves a shot, even if at the end of the day you end up concluding he is a boatload of shit.”
Kosman was a wise dude. It was the best class I took in college, maybe ever.
Step 2:
Another wise dude about writing is John McPhee, considered to be one of the pioneers of creative non-fiction. Draft No. 4 : On the Writing Process is a master class on the writer’s craft.
One of my takeaways from McPhee is to diagram out the progression and structure of a research paper before I begin to write about it: “There is no way to start a writing project, let me tell you. You begin with a subject, gather material, and work your way to structure from there.”1 Here is diagram of the Metacognitive Laziness paper.
Having completed Steps 1 and 2, I am ready to turn to Generative AI. Stay tuned.
McPhee, John. Draft No. 4: On the writing process. Farrar, Straus and Giroux, 2017. p. 4.