Creating Good Peer Review

My computer is acting funky, at the moment, so I am backing everything up and taking it to the Apple Store this afternoon for a check up. But, since I can’t do any “real” work at the moment while I wait for the back-up to finish, I thought I would do a post on how I teach peer review.

I learned the value of peer review at UConn, where a core feature of our composition courses is the Small Group Tutorial, or SGT, where the instructor would meet with a group of two to four students for an hour, and lead a peer review session. I didn’t like required peer review as an undergraduate, because my own experience was that I didn’t get good feedback; I did much better when I sought out my own readers and exchanged papers with people in different classes, or even at different universities. But SGTs helped me reorient my thinking; I started to think about including peer review not just because it helped the writer, but because it was about training students to do peer review. Rather than expecting students to already know how to give feedback, I was teaching and modeling it for them.

However, the teaching load at most schools really isn’t conducive to hour-long sessions with small groups of students, so at Georgia Tech I’ve been experimenting with how to teach peer review and make it more effective for students. I’m focusing on teaching what peer review is, why we do it, and what it (should) be good for. What I did this semester worked surprisingly well, and a few colleagues here and at other schools asked me for my materials, so I thought I would share what I do. But first, two notes:

1) As I said in my last post, I regularly use Joseph Harris’ book Rewriting in my classes. A lot of the language of this prompt is drawn from his book. Students are already familiar with the terms when we start peer review.

2) For the first round of peer review, I didn’t have students meet together or give comments to each other orally.* Instead, I have them do double-blind peer review, and later I use a discussion of how that worked to set up a discussion of academic research, and how academics use peer review. I use Piazza, which allows me to set up numbered discussion groups as well as anonymous posting. I assign each student a number, and each number to a group, and they post all their papers there. There were at least three students in each group, so that even if one member of a group didn’t give comments, everyone would get something. I also added extra office hours so that if anyone wanted help figuring out what to do with the comments they received, I would be available.

My students get two sets of instructions, one that is built into our class assignment guide, and then a reiteration/extension of those basics, usually in the form of a class lecture/discussion. However, we had a lot of snow days this year, so my students got the second set in an e-mail, which makes it a lot easier for me to share them with you here.

Write an anonymous letter to the authors of the papers you reviewed, including but not limited to the following:

    1. Come to Terms: Explain what you understood as the project of the paper. Start by identifying the thesis, but also give a holistic explanation of what you see the author trying to do.
    2. Forwarding: What works in the draft? How can the author build on and do more of that?
    3. Countering: Where are there places that the logical argument and/or use of evidence could be improved? How could the author acknowledge other points of view or other possibilities? Where do you think, “That’s not what I thought when I read that,” or “Maybe, but what about X?”

Your peers will grade you on the quality and usefulness of your responses, so be sure to put in sufficient effort. Some tips to make your comments more effective:

    • Don’t ever say “this is a good paper.” Instead, identify specific elements that work well.
    • Specifics should be paired with suggestions for improvement. For example: “I really didn’t understand how your quotation of Satan’s appeal to Eve’s beauty in the second paragraph had anything to do with your point that Eve’s behavior was unselfish. Could you find a more appropriate quotation, or explain what you were thinking a bit more?”
    • Quote the paper: “Your thesis in the introduction is that ‘Milton’s God is cold and unfeeling in order to make Satan seem more appealing.’ However, most of your paper actually seems to be about Satan’s perception of God, which isn’t the same thing. Maybe you should rewrite the thesis to better reflect the contents of your paper? I think Satan’s perceptions are more interesting than your original thesis.”

For the in-class or e-mail instructions, I discuss broader issues of why and how we approach peer review. I cover at least the following points.

  1. The anonymity of your drafts and peer review comments is designed to give you freedom in responding: freedom to know that whoever comments on your paper won’t know who you are, freedom to express your confusion about something someone has written, freedom to be critical, etc. You won’t always have this anonymity, but it can be useful when you are first learning how to give feedback.
  2. In addition to thinking in terms of Forwarding and Countering, make sure to think about the rubric that I’ll be using to grade the papers, which is included in the assignment guide. Forwarding and Countering are ways of helping to improve ideas, but you don’t want to forget about the rhetorical elements we’ve been discussing in class.
  3. Your job is not to be “nice.” Your job, as an ethical reader of someone else’s work, is to help them revise their paper into the best it can possibly be. You should be professional, of course, but it is most important that you be honest about what doesn’t work in the paper. One of my favorite fantasy novels, The Curse of Chalion, includes this line, from a teacher (Cazaril) to his students:

“If you desire a man to tell you comfortable lies about your prowess, and so fetter any hope of true excellence, I’m sure you may find one anywhere. Not all prisons are made of iron bars. Some are made of feather beds.”

The point Cazaril makes is that telling someone their not-so-good work is good is easy, but doing so does them a disservice, and may actually hurt them. It binds them, and keeps them from improving. It prevents them from being truly excellent. Even if the paper you are commenting on is better than yours, it doesn’t mean you can’t offer them good feedback–you might still see things in their paper that they can’t. Don’t stop with identifying what is wrong, however; make suggestions for improvement. And even when someone does something that works, try and help them make it even better.

Then, finally, I have students grade the peer review letters they received.

Rank each peer review letter: give a “5” for a letter that was very detailed & helpful; a “4” for one that was somewhat detailed & useful; a “3” for a letter that was useful but not detailed; a “2” for one that was detailed but not useful; a “1” for one that was neither; and “0” for one not submitted.

I was very pleasantly surprised by the quality of comments most of my students produced. Most students did light editing/marginal notes using Word’s reviewing features, and then wrote several paragraphs focusing on the big picture issues. A few students actually full-page letters with comments–in other words, for these 6-8 page papers, they were giving more comments than academic peer reviewers usually do for 20-page articles. And, to be honest, taking into account that these were freshmen papers and freshmen reviewers, the quality of the comments was sometimes higher than the average journal peer review response.

When I asked students about what they liked about this system, a lot of them said they appreciated the anonymity; they felt freer to actually give criticism instead of just saying “good job”. Most also liked that because the letter was the primary form of commenting, it meant the focus was less on editing and more on true revision. But the comment that I found extremely telling came from a particularly bright but somewhat disengaged student, who admitted that in the past, he had ignored peer review comments unless they came from someone he thought was smarter than he was. The anonymous process deprived him of his reliance on perceived authority/intelligence, and forced him to really pay attention and decide based on the merits of the comment. I saw enough nods around the table at that one to think that might be a bigger issue with traditional peer review than I had previously thought, at least with students at Georgia Tech, who are often used to being the smartest among their high school peers.

This is still a work in progress for me. Not all the students bought in; there were a handful who didn’t take the advice of their peers, or who didn’t give good advice. This process also doesn’t really get at the dialectic or collaborative elements that I truly valued when I exchanged work with peers of my own choosing (with long sessions on IM as we figured out how to rewrite paragraphs and reorganize documents). I want my students to get that experience, too.

So: What do you do for peer review that works well? How do you teach good peer review skills? What do you emphasize?


Teaching Difficult Texts in Composition: The Problem of “Study Guides”

Dustin Hannum’s Techstyle post on teaching Moby Dick in a first year writing and communication class hit close to home for me when I read it a few months ago. His articulation of the fears that both students and some teachers have about difficult literary texts in a first-year classroom were very similar to the words I’ve heard about teaching John Milton’s Paradise Lost, which in addition to being nearly as long, symbolic, and difficult as Moby Dick, has the added problem of being seventeenth-century epic poetry. Like Dustin with Moby Dick, I persist in teaching Paradise Lost, alongside its adaptations; I’ve taught pieces of the poem in two different classes at the University of Connecticut, and the whole poem as the centerpiece of two different first-year composition classes, one at UConn and one at Georgia Tech, both with an adaptation component.

One of the objections I’ve heard from other instructors is that when we teach difficult but classic texts, students are more likely to use notes systems like CliffsNotes and SparkNotes, and to plagiarize them in their work. And certainly, I will admit that the downside to teaching a piece of classic literature is that students are far too often tempted to plagiarize. But that has only a little to do with the difficulty of the text; in the past, I’ve found that there are always one or two students willing to admit they look for such notes on any topic or text, no matter how straight forward. I once had a student write in their self-reflections on an essay on the science fiction novel Old Man’s War by John Scalzi (which is very easy reading) that he had found it “difficult and challenging” to write the essay because he hadn’t been able to find any good online discussions of the themes and topics he was writing about. The highlight of the comment was this: “This forced me into a position where I had to develop ALL points and arguments completely on my own. I am more grateful than bitter about this though.” Of course, saying that he was more grateful than bitter implies that the student was still at least a little bit bitter.

When I taught Paradise Lost this spring, during the second week of class another instructor came up to me and confided that she had seen one of my students reading a copy of the Paradise Lost CliffsNotes in the hallway (the student had to be mine, because as far as anyone knows, I’m the only one teaching the poem here). I will admit to being more amused than anything else; I remembered my own first encounter with Paradise Lost in 2001 very distinctly, and it involved just such a study aid. I was 17, and the class was my first substantial encounter with British literature (a function of having skipped most of high school English and attending the local community college where I could fulfill all my requirements with American lit). We had been assigned selections from Paradise Lost out of the Norton Anthology. I read the piece of the text that had been assigned, understood only some of what was going on, read it again, understood only a little more than the previous time, and then promptly went online and found the Spark Notes to help me figure out a little bit more. It helped a lot in terms of giving me the big picture when I couldn’t get past the words and styntax. I could tell its answers were a bit too pat at times, but it was enough for me to start constructing my own readings around and responding to the ideas posed in the notes. I occasionally returned to SparkNotes and other guides when reading a text as an undergrad, but it became more and more rare as I got more and more confident in my ability to read complicated texts. When it comes to Paradise Lost, the trick is to find a way to help students move past their reliance on such notes. And if students can feel confident enough to work with Paradise Lost on their own, they can feel confident about their ability to work with just about any difficult text.

So now, when I teach, I mostly work on the assumption that 75-90% of my students did the actual reading, and that a similar percentage read some sort of guide; obviously, not all who did the reading read the guide, and vice versa. It usually becomes clear to me during class discussion who falls into which category. None of the students are brazen enough to have SparkNotes or CliffsNotes out in class so anyone can see it, even if they used it out in the hall, but the notes do not prep students to engage in close readings of passages on the fly. They are almost universally light on evidence, so when I ask a question about a character and then ask them to point to a particular passage, they usually aren’t ready unless they’ve done the actual reading. So, if a student can make a general statement about a character or the poem, but can’t describe any details about what in the text led them to that conclusion, they invariably relied on the study guide instead of the reading.

But students also inevitably discover that SparkNotes and the equivalent aren’t going to address the big-picture questions I asking in a composition course on Paradise Lost. A lot of what I do is guided by my use of Joseph Harris’ book Rewriting. I became familiar with Harris’ book when teaching at the University of Connecticut, where over the space of a few years, it became a staple of our introduction to writing pedagogy class as well as many instructors’ classrooms. Rewriting emphasizes that all texts, but especially academic texts, are in conversation with one another, and must constantly “rewrite” each other in order to advance the conversation. I assign the book, or pieces of it, alongside the poem at the beginning of the semester.

If you haven’t seen the book, I highly recommend it, but here’s a brief summary of what I find useful: Harris emphasizes four “moves” that all writers make as the essential building blocks of academic writing: coming to terms, forwarding, countering, and taking an approach. Coming to terms might be simply explained as “summarizing,” but with an emphasis on reading a text generously, with the goal of noting both what is useful about an idea and what its limits are. Forwarding is taking an idea (usually one you’ve come to terms with) and building on it. Countering is not simply arguing with an idea, but saying, “yes, but”—that is, generously reading an idea and then identifying its limits, and attempting to solve them. Taking an approach involves a more expansive version of forwarding, as it concerns using the underlying issues of methodology and assumptions of another text, rather than the particulars of expression.

Paradise Lost is a particularly useful text for students to start to learn to identify and use these moves because Book 2’s debate in Pandemonium embodies all of them, but in ways that draw attention to their improper as well as proper use. Belial ungenerously summarizes Moloch’s position before he counters it; Mammon forwards parts of Belial’s argument and counters others, adding his own twist; Beelzebub forwards Satan’s idea of subverting mankind to the rest of the demons (adding his own elements not present in Satan’s speech to Beelzebub), but in failing to acknowledge his source, he misleads the other demons as to their origins—and my students invariably see this as manipulative and unethical, even if they sympathize with Milton’s Satan’s. The general outlines of this reading could be found in something CliffsNotes (which discusses Belial’s and Mammon’s sophistry, and how Satan and Beelzebub defraud their fellow fallen angels), but using Harris adds a layer of complexity by showing that these demons are using rhetorical moves that can be, in other contexts, good. (It fits nicely, in other words, with the theme that evil is the perversion of good, and the notion that the fallen angels still maintain some of what made them angels in the first place.)

In the first few books of Paradise Lost, then, we focus on the moments where Milton has his characters engage in these rhetorical and intellectual moves from Harris. But as we move progressively deeper into the poem, we start to dig into research to see where Milton himself might be coming to terms with an idea, where he plays with forwarding and countering of ideas from his own time period. Was Milton forwarding ideas about Arianism in his poem? Was he countering or forwarding the misogyny of his own period? How was he coming to terms with seventeenth-century scientific debates in representing Adam and Raphael’s discussion of the creation of the universe? How did his epic poem take an approach from his earlier works? (I usually teach Paradise Lost alongside Areopagitica, but students sometimes choose to read On EducationTenure of Kings and Magistrates, or other works on their own.) And then there’s the big question: which of these moves is Milton using in relationship to his most obvious source material, the Bible? (Answer: All of them. The hard part is identifying which ones he’s using at any given moment)

Once they are doing research, students have start making the same moves in their own writing. Are they simply going to come to terms with a source? Can they forward its ideas in a new direction? Do they need to counter something a source says—and what are they to do when two sources disagree with one another? Milton’s poem proves to be fruitful not only in providing content for them to practice their skills, but in providing models for the work itself. They learn, in other words, to “take an approach” from Milton. Of course, Milton can be a bit intimidating as a model—which is one reason why I teach adaptations alongside Milton’s poem, for each adaptation or appropriation has to do some of this work, and in ways that often give students permission to try their own hand at critiquing Milton’s work, or at using the work of others to offer critique.

Difficult texts like Paradise Lost do sometimes scare students into improperly using online notes. I caught two students plagiarizing from them in the first logical outline for the paper. (My comment on Facebook at the time was something like, “The upside to teaching 21st century science fiction instead of epic poetry is that no-one has made a SparkNotes for Old Man’s War.”) But while those two students failed the initial assignment, it was relatively low stakes, and we had good discussions about why they had used the notes. In both cases, it had been because they were afraid of saying something “wrong” about the text, of feeling like they couldn’t come up with their own ideas. I told my students that for this paper they could cite the notes, but they had to do the work of countering and forwarding to make their own argument. What surprised and pleased me was that by the end of the revision process, the material from SparkNotes had completely disappeared, and students had developed their own arguments. One of the papers even ended up being among my favorites for the whole class, as she wrote a compelling paper about the way Milton uses the horrors of hell to discuss what drives (or doesn’t drive) repentance. The notes served as a useful crutch (once we got past the plagiarism issue) to get them started, but by the end, they were standing on their own two feet.