Peer Review, Revisited

I wrote a post two years ago about how I teach peer review. To sum up: I have students write anonymous letters to one another about their work. They have to summarize the project of the paper; identify  the strengths and weaknesses of the argument, evidence, and structure; and offer constructive suggestions for how address the paper’s issues.

This morning, I was asked to do a session on how I teach critique for a presentation to a major donor for our program. I decided it would be a good time to get some student feedback on how well they think my way of teaching peer review works. I just asked them to compare their experience with peer review in my class to any experience they had previously, and explain what they thought worked well (or was difficult) about the peer review in my class.

I thought I would post some of their responses, because they nail exactly why I do peer review the way I do: I want students to understand that giving positive feedback without constructive criticism can hurt, not help, their friends. I want them to realize that feedback on ideas and logic and structure need to come before an emphasis on grammar and mechanics. I want them to understand that they can benefit as much from giving feedback as receiving it.

Here is what ten of my students had to say:

“My experience with peer review in high school was largely negative. I did not feel comfortable having to critique other people’s work, nor did I ever receive feedback on my work that I found particularly helpful. I can easily say that my experience with peer review within the Georgia Tech English program has been completely different. Critical but respectful analysis of other student’s work is encouraged, and almost every peer review letter that I have received this semester has been insightful and constructive. I largely attribute this difference to the way that Dr. Taylor characterized criticism as a vehicle that will improve rather than hinder the quality of an individual’s work. In other words, it is more of a hindrance to provide no criticism than it is to give a thoughtful analysis of another individual’s work.” — J.K.

“Dr. Taylor’s class requires much longer, more detailed peer reviews than I have experienced in the past. In high school, the most extensive review I had to do included writing comments throughout a copy of my partner’s essay and writing a few other general comments at the end. In this class, the review is a full page long letter which we write for and receive from multiple classmates. This provides us with much more detailed commentary to work with, which is more helpful than comments that need to be squeezed into the margins of an essay.” — A.B.

“Peer review in Dr. Taylor’s class is an effective way to provide and receive critique on assignments. In high school, my experience was that when we chose whose papers we reviewed, we ended up choosing our friends and not providing as deep a criticism as is often necessary. Here at Georgia Tech, I’ve found that having our peer review partners chosen for us is beneficial in that I usually receive a partner who I have never met. Not knowing my partner allows me to treat them like a peer, rather than a friend, which in turn allows me to criticize their paper from a more objective standpoint. Personally, I know that I have benefited from this system as I have received a lot of good criticism on my work. I hope that the criticism I have given is just as useful as what I have received.” — J.B.

“Compared to high school, peer review in this course is much more advanced and prominent. Very rarely in high school did I use peer review in class. When we happened to peer review each other, the vast majority of the reviewing was grammatical editing. There wasn’t too much actual reviewing of the writing itself and its argument. However, in this class, peer review mainly focuses on the argument of the writing and its effectiveness in displaying its message. Editing simple grammatical structure and spelling errors are minor when it comes to peer review in this course. As a result, peer reviewing sometimes leads to an entirely different piece of writing if the argument or message is ineffective.” –T.J.

“In high school whenever I had to do peer reviews they were always superficial. We were usually given an outline with blank spaces or specific questions with rankings. For the most part they didn’t help with revision whatsoever. In this class, I like that we’re just told to write a letter. Outlines and rankings don’t always work well because everyone’s project is set up differently. It is more beneficial to write a letter because it can better reflect and critique an individual project for its unique aspects. By doing peer reviews in this letter writing manner, I’ve become a better reader. I’ve gotten better at “coming to terms” with a piece of work and learned to dissect a paper. I’ve also become a better critical reader, not just taking what I read for truth, but considering both the author’s argument and evidence and the things which they fail to consider.” — K.I.

“In this course, the experience with peer review was quite different from that of high school, but was beneficial and taught me a lot about the process of peer reviewing. Previously, students would exchange a hard copy of what was being reviewed and simply wrote notes on the paper. This made it very easy to simply comment on syntax and grammar and not reflect as much on the ideas of the work and its overall organization. With the method of writing a peer review letter, I found I was reflecting less on small grammatical errors. I also benefited a lot more from receiving letters rather than a physical copy that was written all over. It was a little difficult to write the letter because it forced me to stray from the more obvious corrections and think more about the work’s argument and supporting ideas. From this new process I learned the value of incorporating people’s reflections on my ideas and organization into my work rather than corrections on minor syntax errors.” — M.H.

“I found peer review this semester to be extremely effective and helpful. Often times, I’m seeing my work through tinted glasses, and having someone else look at my work helps me make connections that I would never have thought of. Peer review in high school was cursory at best, and we mostly fixed each other’s grammar and syntax. In Dr. Taylor’s class, there is a focus on reviewing the themes and arguments behind our work, and making sure we adequately and logically support them. Having actual analysis and in depth reviews of my work in college have made my artifacts much better by forcing me to rethink my ideas many times.” — S.I.

“I found the peer reviews to be fairly helpful overall. As always, an outside perspective finds flaws and new ideas within your argument that you would never find yourself. I enjoyed the emphasis on reviewing content instead of grammar or syntax, since I can do that myself without issue. I found the more critical reviews the most helpful; they weren’t afraid to call core elements of my arguments into question. If I would change anything, I would change the language in the assignment guide to encourage more criticism and less positive reinforcement.” — J.C.

“My experience with revision in the past was simply a professor handing back my rough draft that dripped with red ink. I would make the changes that were suggested (mostly syntactical and grammatical errors) and then I would hand that back in for a final grade. Now my revision experience is less mechanical, it is an iterative process that smooths my arguments edit by edit like waves flattening the sand on a beach. It consists of edits by peers, Dr. Taylor, and myself all helping the essay evolve and become more concise while turning a simple argument into a multifaceted scholarly one. I have found that the more I revise the better a writer I become and the more likely I am to make a strong paper from the onset.” — D.V.

“Receiving a peer review in the form of a letter has been more thorough than peers reviews I have experienced in the past where edits are just scribbled directly on the page. In this more structured and lengthy form, people tend to reveal their thought process behind the edits they are suggesting, which is often just as if not more helpful than the edits themselves. Also, my editors are more careful to identify where things went well in addition to where I went wrong, so I know where to focus my final draft.” — L.H.

The Academic Job Search: Some Stats

I don’t know that anyone is still reading this blog, but I didn’t want to post this directly to Facebook, because no one still on the job market should have to read this kind of thing if they don’t want to.

Now that I have a job (yay!), I thought it might be a useful exercise to provide some stats about being on the job market. I found it helpful to see this kind of information when I was a grad student, and I think we need to be transparent about exactly what our current system involves–and what it costs.

Years on the Market: 5.

My first year, I was ABD. I defended in October of my second year.

Applications Submitted: 168 (Year 1, 25; Y2, 76; Y3, 8; Y4, 28; Y5, 32)

My first year was a partial search. My second year, I applied for a lot of post-docs, visiting positions, lecturer positions, and composition jobs (as opposed to jobs in my field of Renaissance lit). After I got my 3-year post-doc at Georgia Tech, I did not apply for many visiting or post-doc positions, though I did apply for potentially permanent lecturer positions. I only applied for composition jobs if there was something extremely appealing about them.

First round interviews invitations: 28 (Y1, 3; Y2, 7; Y3, 2; Y4, 9; Y5, 7)

12 of my first round interviews were at MLA. The other 16 were phone or video interviews. In Years 2 and 5 I turned down at least one first round invitation to interview because the invitation came after I had accepted a job offer. In Year 3, I did the same because it came so late in the summer that I couldn’t back out of my obligations to my current school. Two first round interviews were horror shows (one by phone, the other at MLA). All others were completely professional and respectable.

Campus visit invitations: 6 (Y1, 1; Y2, 1; Y3, 0; Y4, 1; Y5, 3)

In Year 5, I  turned down one campus visit invitation because I had already accepted a job offer.

Job offers: 2. (Y1, 0; Y2, 1; Y3, 0; Y4, 0; Y5, 1)

My first job offer was for a three-year post-doc, the second for a tenure track position.

Costs: $4000 (approx.)

This doesn’t count several thousand dollars I shelled out for campus visits; I was fortunate that all of the universities that invited me reimbursed me fully and promptly. Of the five campus visits I went on, only one university made all the arrangements and made it so I did not have to do any major outlay of costs.

I went to MLA every year, though three of those years MLA was in a location where I did not have to pay for a hotel (in Seattle and Vancouver I stayed with family; in Chicago I stayed with a friend). When going to Seattle, Vancouver, Boston, and Austin, the timing or location was such that I did not count the cost of a full round trip ticket as an expense, because I was either already in the city visiting family for the holidays, or was able to make it a stop on my way back from visiting family. When I did stay in a hotel, I split the room with another attendee. Even so, I spent more than $2000 on flight and hotel costs, as well as more than $400 for Interfolio and $550 for MLA membership and registration.

Hours Spent: Countless.

So much of this is idiosyncratic. Even how much you end up having to spend is idiosyncratic, as my own costs show. I easily could have had to spend thousands more if MLA had been in different locations. Success on the job search, so far as I can tell, requires skills, professionalization, training, and experience–but that only gets you in the game. After that, success depends on a combination of all those things with the luck of the fit and luck in timing that is not under a candidate’s control.