arXiv-tracker Summary Quality Evaluation: Human vs LLM-as-Judge

1. Introduction

LLMs are now used for many open-ended generation tasks, including paper summarization, question answering, code generation, and agentic workflows. Evaluating these systems is not straightforward. Traditional benchmarks often target problems with clear answers, such as multiple-choice questions, numeric answers, or token matching, which makes metrics such as accuracy or exact match easy to apply. In contrast, open-ended tasks such as paper summarization do not have a single correct answer. Quality depends on accuracy, faithfulness, usefulness, naturalness, and format compliance.

Evaluation turns an abstract quality concept into an operational metric. In other words, the evaluator must define what a "good summary" means before it can be measured. A single scalar score can hide important ambiguity. LLM-as-Judge is fast and scalable, but it can be affected by response length, writing style, fluency, and other presentation factors. A score alone also makes it difficult to explain why a specific rating was assigned.

To reduce these limitations, recent evaluation methods have moved from simple score-based evaluation toward rubric-based evaluation. A rubric defines evaluation criteria and score anchors in advance. It acts like a shared contract for what counts as a good output and makes the final evaluation easier to interpret.

2. Evaluation Target and Research Questions

2.1 Evaluation Target

The target system is the paper summarization pipeline included in target_project/arxiv-tracker. Its main output is a Korean five-section summary generated from a paper title and abstract.

The summary format is:

1. Core contribution
2. Methodology
3. Results
4. Limitations
5. Keywords

The important evaluation challenge is that this output is not a closed-form answer. Even when two evaluators read the same abstract, summary quality can be split into multiple dimensions such as correctness, completeness, readability, format compliance, and coverage of limitations. For that reason, string matching or ROUGE alone is not sufficient for this task.

2.2 Research Questions

This project was designed to answer the following questions:

1. How much do human ratings and LLM-as-Judge ratings agree when using a simple 1-5 rating without explicit criteria?
2. Does agreement improve when evaluation criteria are provided?
3. Does a rubric with score anchors make LLM-as-Judge ratings closer to human ratings?
4. How do unweighted Cohen's kappa and quadratic weighted kappa differ?
5. Which quality attributes explain the disagreement cases?

3. Evaluation Dataset

3.1 Evaluation Cases

The evaluation set contains 10 arXiv papers selected to match the intended use of arxiv-tracker. Each case has a case ID from P01 to P10, and data/papers_sample.csv stores the arXiv ID, title, matched keyword, and abstract.

The selected papers cover diverse topics, including LLM transparency, MoE calibration, Bayesian in-context learning, on-device LLM serving, tool-calling agents, MLLM bias, SAR multimodal datasets, multilingual code benchmarks, coding-agent guidance, and safety-aligned LLMs.

3.2 Generated Outputs

For each paper abstract, arxiv-tracker generated a Korean summary using the required five-section format. The generated results were saved in data/generated_outputs.csv.

Evaluators judged only summary quality based on the abstract. Information that would require reading the full paper was excluded from the criteria. The evaluation focused on factuality, omissions, overstatement, and whether each summary was justified by the abstract.

4. Evaluation Method

4.1 Human Evaluation

The human evaluation was completed before running the LLM-as-Judge evaluation. This prevented the human scores from being influenced by the model's judgments. Human scores and brief rationales were recorded in data/human_ratings.csv.

4.2 LLM-as-Judge Evaluation

The LLM-as-Judge evaluation used a ChatGPT-based GPT-5 series model. Every evaluation case used the same separate evaluation prompt, stored under the prompts/ directory. The input included the paper title, abstract, and generated Korean summary. The output was constrained to an integer score from 1 to 5 plus a short rationale. Each case was evaluated once, and the human score was not included in the model input. Final model ratings were saved in data/llm_judge_ratings.csv.

4.3 Evaluation Conditions

First, the no-rubric condition asked evaluators to rate only the overall quality from 1 to 5. This is close to an intuitive user rating, but the meaning of each score can vary by evaluator.

Second, the criteria condition asked evaluators to consider accuracy, completeness, usefulness, natural Korean expression, and format compliance before assigning one final score. This is simpler than a full rubric, but it clarifies what the evaluator should inspect.

Third, the rubric condition provided explicit score anchors:

1. The summary is mostly wrong, unrelated, or misses almost all important content.
2. The summary is partly related but omits important points or contains notable misunderstanding.
3. The summary is mostly correct but vague, incomplete, or weakly structured.
4. The summary is accurate and useful, with only minor omissions or lack of detail.
5. The summary is accurate, complete, concise, naturally written in Korean, and clearly organized into the required five sections.

5. Agreement Measurement

Human scores and LLM scores were combined into data/final_evaluation_dataset.csv. The script scripts/calculate_kappa.py computed Cohen's kappa with sklearn.metrics.cohen_kappa_score. A manual fallback was also included so the results could be reproduced in environments without sklearn.

Because 1-5 ratings are ordinal, both unweighted kappa and quadratic weighted kappa were calculated. Unweighted kappa treats the difference between 4 and 5 as a full disagreement. Quadratic weighted kappa gives smaller penalties to adjacent-score differences and larger penalties to distant differences such as 1 versus 5.

6. Evaluation Results

The no-rubric condition had the weakest agreement, with exact agreement of 0.40 and unweighted kappa of 0.0000. The human evaluator tended to assign 5 when a summary was readable and captured the key point, while the LLM-as-Judge more actively penalized weak limitation sections and missing technical specificity.

Both the criteria and rubric conditions improved to exact agreement of 0.60 and unweighted kappa of 0.3333. Once the criteria were explicit, both evaluators inspected similar aspects of the summaries. Judgments about format compliance and factual accuracy were especially stable. However, disagreements remained around the boundary between 4 and 5 or between 3 and 4.

7. Result Analysis

7.1 Weighted Kappa Interpretation

All disagreements were one-point differences. As a result, quadratic weighted kappa was higher than unweighted kappa. In the criteria and rubric conditions, weighted kappa was 0.5556, higher than the unweighted kappa of 0.3333. Even in the no-rubric condition, unweighted kappa was 0.0000 while weighted kappa was 0.3333. This suggests that the human evaluator and LLM judge were not making completely different judgments. They generally moved in the same direction, but the LLM often assigned a score one level lower.

This result shows why weighted kappa is useful for 1-5 rating tasks. If only unweighted kappa is reported, the no-rubric condition appears almost meaningless. Weighted kappa reveals that the evaluators were not judging in opposite directions; their differences were mostly small ordinal disagreements.

7.2 Disagreement Case Analysis

For P03, Multi-Task Bayesian In-Context Learning, the human evaluator assigned 4 while the LLM assigned 3. The human evaluator judged that the core idea and results were communicated reasonably well. The LLM judged that the summary lacked enough detail about adapting to a new prior and about the difficult evaluation setting.

For P06, StylisticBias, the human evaluator assigned 5 while the LLM assigned 4. The human evaluator found that the dataset scale, number of evaluated models, and main findings were clearly organized. The LLM penalized the summary because the limitations section was too short and not specific enough.

For P07, SARLO-80, the human evaluator assigned 4 while the LLM assigned 3. The human evaluator thought the dataset scale and composition were explained well. The LLM wanted more detail about the characteristics of SAR data and how the dataset could be used for evaluation.

For P10, What Do Safety-Aligned LLMs Learn From Mixed Compliance Demonstrations?, the human evaluator assigned 5 while the LLM assigned 4. The human evaluator thought the experimental setup and core conclusion were sufficiently clear. The LLM wanted more explanation of how the model handles harmful and harmless examples, as well as how context affects refusal responses.

Overall, the score gaps were not caused by completely incorrect summaries. The human evaluator emphasized whether the summary was sufficient to understand the core idea, while the LLM emphasized whether the summary explained enough technical detail. The system's main weakness was not producing wrong summaries, but losing some details when compressing complex abstracts into short summaries.

8. Limitations and Future Work

First, the evaluation set is limited to 10 cases. This satisfies the assignment requirement, but more cases across paper topics would be needed for more stable kappa estimates.

Second, LLM-as-Judge has its own biases. As discussed in the course material, LLM judges can be influenced by length, writing style, and specificity. In this project, the LLM tended to score summaries lower when it wanted more technical detail, even when the human evaluator considered the summaries useful.

Third, the current evaluation uses a single LLM judge. Future work could use multiple judge models or add pairwise preference evaluation to reduce score-anchor drift.

Fourth, every evaluation is abstract-based. If the system is expanded to summarize full papers, additional criteria such as citation faithfulness, section coverage, and hallucination rate would be needed.

9. Conclusion

This project compared human evaluation and LLM-as-Judge evaluation for summaries generated by an arxiv-tracker system built with Claude Code. Cohen's kappa was used to quantify agreement between the two evaluators. Agreement was low without explicit criteria, but it improved after criteria and rubric guidance were introduced.

The key observation is that the LLM-as-Judge was stricter than the human evaluator about technical completeness and limitation coverage. From a practical reading perspective, the human evaluator gave high scores when a summary was readable and communicated the main point. The LLM reduced scores when abstract details and limitations were not reflected thoroughly enough.

Therefore, open-ended generation systems should not be evaluated only with average scores. Evaluation criteria, rubrics, disagreement cases, and weighted kappa together provide a more convincing view. For ordinal 1-5 scales, reporting both unweighted Cohen's kappa and quadratic weighted kappa helps interpret the real size of evaluator differences.

References

1. [1] Zheng et al. (2023), "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena"
2. [2] Cohen (1960), "A Coefficient of Agreement for Nominal Scales"
3. [3] Cohen (1968), "Weighted Kappa: Nominal Scale Agreement with Provision for Scaled Disagreement or Partial Credit"