The AI-Assisted Peer Review Lifecycle

An exercise in using LLMs to simulate the academic peer review process, from initial review to final rebuttal.

Part 1: Generating the Initial Review

Your task is to use an LLM to generate a high-quality, constructive review for the Author's paper. Your goal is to be a helpful, critical, and fair reviewer.

Task 1: Draft a High-Quality Review

A good review is structured, provides evidence for its claims, and offers actionable suggestions. You will need to guide the LLM to produce a review that meets these criteria. Attach the Author's paper to your prompt.

Example Prompt
Act as an expert peer reviewer for a top-tier academic conference in [Author's Field]. I have attached a research paper. Your task is to generate a high-quality, constructive review. The review must include the following sections: 1. **Summary:** A brief summary of the paper's contributions. 2. **Strengths:** At least three major strengths of the paper. 3. **Weaknesses:** At least three major weaknesses, with specific examples from the text. For each weakness, provide a concrete, actionable suggestion for improvement. 4. **Overall Recommendation:** A final recommendation (e.g., Accept, Leaning Accept, Leaning Reject, Reject) with a clear justification. Your tone should be critical but constructive and professional. Avoid overly positive or negative language without justification.

Official Reviewer Guidelines (for inspiration):

Critical Analysis:

Once you have the review, read it carefully. Does it seem fair? Are the suggestions helpful? Pay close attention to the generated weaknesses. Are they valid critiques of the paper, or superficial? Does the AI hallucinate any claims, misinterpret contributions, or cite incorrect examples from the text? Refine your prompt if necessary.

Part 2: Reviewing the Review

Your task is to act as a meta-reviewer or Area Chair. You will assess the quality of the review generated in Part 1 to ensure it is fair, helpful, and adheres to good reviewing standards.

Task 2: Assess the Generated Review

Provide the LLM with the generated review from Part 1. Your goal is to have the LLM critique the review itself based on a set of quality guidelines. This will help determine if the AI-generated review is actually any good.

Reviewer Quality Guidelines:

  • Constructive: Does the review offer actionable suggestions for improvement?
  • Specific: Does the review cite specific examples from the paper to support its claims?
  • Balanced: Does it fairly assess both strengths and weaknesses?
  • Respectful Tone: Is the language professional and respectful?
  • Clear Justification: Is the final recommendation clearly justified by the points made in the review?
Example Prompt
Act as an Area Chair for a major academic conference. I am providing you with a peer review for a paper. Your task is to assess the quality of THIS REVIEW based on the provided guidelines. [Paste the Reviewer Quality Guidelines here] Go through the attached review and rate it on each of the five guidelines. For each guideline, explain why the review does or does not meet the standard, providing quotes from the review as evidence. Conclude with an overall assessment of whether this is a high-quality review.

Critical Analysis:

Does the AI Area Chair correctly identify the quality of the review? Does its assessment align with your own? Did it miss any key flaws or strengths of the original review, or hallucinate any part of its reasoning?

Part 3: Proposing Changes and Writing a Rebuttal

Now you, the Author, have received the review. Your task is to use the LLM to process the feedback, propose changes to your paper, and draft a professional rebuttal.

Task 3a: Generate a Plan of Changes

First, have the LLM analyze the review and your paper to create a structured plan for how you will address the reviewer's concerns.

Example Prompt
Act as a PhD student who has just received a review for your paper. I have attached my paper and the review I received. Your task is to create a detailed, point-by-point plan of the changes I will make to the paper to address the reviewer's weaknesses. For each weakness identified by the reviewer, create a bullet point that clearly states the proposed change. For example: "To address the concern about methodological clarity, I will add a new paragraph in Section 3 detailing the data filtering process."

Task 3b: Generate Specific Edits to the Paper

Now, turn the plan into action. Have the LLM generate the specific text edits needed for your paper based on the plan you just created.

Example Prompt
Act as an expert academic editor. I have attached my paper and a point-by-point plan of changes. Based on this plan, generate the specific edits for the paper. For each point in the plan, provide the exact text that should be added or modified. Clearly indicate where in the paper (e.g., "In Section 3, paragraph 2") the change should be made. For modifications, show the original text and the proposed new version.

Task 3c: Draft the Rebuttal

Finally, using your plan and the specific edits, have the LLM generate a formal rebuttal to send back to the reviewers.

Example Prompt
Based on our plan of changes and the specific edits we just generated, now draft a formal author rebuttal. The rebuttal should start by thanking the reviewer for their constructive feedback. Then, for each point raised by the reviewer, respond directly. State the change you have made (referencing the edits) and politely explain how it addresses their concern. If you disagree with a point, provide a brief, evidence-based counter-argument. Maintain a polite and professional tone throughout.

Official Rebuttal Guidelines (for inspiration):

Critical Analysis:

Evaluate the AI's output for all three tasks. Is the 'plan of changes' reasonable? Are the suggested edits accurate and well-written? Does the rebuttal effectively and convincingly address the review's points, or is it too defensive or generic? Did the AI misunderstand the reviewer's intent or your paper's content at any stage?

Part 4: Responding to the Rebuttal

You are the Reviewer again. You have received the author's rebuttal. Your final task is to assess their response and proposed changes, and decide if you will update your recommendation.

Task 4: Generate a Final Response

Provide the LLM with the original review and the author's rebuttal. Ask it to generate a final response to the Area Chair.

Example Prompt
Act again as the expert peer reviewer. I have attached my original review and the author's rebuttal. Please draft a final response to the Area Chair. In your response, state whether the author has adequately addressed your concerns. Decide if you will maintain or change your original recommendation for the paper based on their rebuttal and planned revisions. Justify your final decision.

Critical Analysis:

This concludes the lifecycle. Does the AI's final judgment seem fair? Does the updated recommendation logically follow from its analysis of the rebuttal? Critically assess whether the AI is simply agreeing with the author or providing a nuanced, final critique. Where was the AI most helpful in this entire process, and where did it require the most guidance?