LGAI-EXAONE / Ko-LongRAG

Abstract

The rapid advancement of large language models (LLMs) significantly enhances long-context Retrieval-Augmented Generation (RAG), yet existing benchmarks focus primarily on English. This leaves low-resource languages without comprehensive evaluation frameworks, limiting their progress in retrieval-based tasks. To bridge this gap, we introduce Ko-LongRAG , the first Ko rean long -context RAG benchmark. Unlike conventional benchmarks that depend on external retrievers, Ko-LongRAG adopts a retrieval-free approach designed around Specialized Content Knowledge (SCK), enabling controlled and high-quality QA pair generation without the need for an extensive retrieval infrastructure. Our evaluation shows that o1 model achieves the highest performance among proprietary models, while EXAONE 3.5 leads among open-sourced models. Additionally, various findings confirm Ko-LongRAG as a reliable benchmark for assessing Korean long-context RAG capabilities and highlight its potential for advancing multilingual RAG research.

Dataset Details

• Composition : 600 total examples
  • singledocQA (300): extraction-style QA grounded in a single document
  • multidocQA (300): comparison/bridge reasoning across documents within a domain cluster
• Fields (schema) : id , titles (list[str]), context (str), question (str), answer (str), prompt (str), task (str; "singledocQA" or "multidocQA" )
• Context lengths (approx.) : single ≈ 2,915 tokens; multi ≈ 14,092 tokens
• Unanswerable share : ≈ 16.6%

For construction protocol, prompts, human verification checks, and extended statistics, please refer to the accompanying paper and repository notes. Guidance for dataset cards and their structure follows the Hugging Face documentation. :contentReference[oaicite:5]{index=5}

Usage

from datasets import load_dataset

ds = load_dataset("LGAI-EXAONE/Ko-LongRAG", split="test")
print(ds)
print(ds[0]["task"], ds[0]["question"])

Data Fields

• id — unique identifier (string)
• titles — list of section titles included in the context (list[string])
• context — concatenated long passages (string)
• question — Korean question (string)
• answer — short answer string (or "unanswerable" when appropriate)
• prompt — prompt used during data creation/evaluation (string, optional)
• task — "singledocQA" or "multidocQA"

Benchmark Design (Brief)

• Domain-aware clustering groups documents by topic/keywords to form long contexts suitable for QA.
• Question generation distinguishes extraction-style (single) from cross-document comparison/bridge (multi).
• Quality control uses a human checklist to validate question–answer–context consistency.
• Unanswerable cases are systematically included to assess reliability and calibration under retrieval failure.

License

This dataset is released under CC BY-NC 4.0 (Attribution–NonCommercial 4.0). Please review the Creative Commons BY-NC 4.0 terms before reuse.

Additional Terms (model usage):
This dataset was created using OpenAI GPT-4o . In addition to the license above, the dataset is subject to OpenAI’s Terms of Use and related policies governing use of model outputs. This means the dataset must not be used to develop competing models where such use conflicts with those terms.

This dataset is licensed under CC BY-NC 4.0, and is subject to the Terms of Use of the model (OpenAI GPT-4o) used in its creation.

Citation

@misc{KoLongRAG-2025,
  title  = {Ko-LongRAG: A Korean Long-Context RAG Benchmark Built with a Retrieval-Free Approach},
  author = {Ko-LongRAG Authors},
  year   = {2025},
  note   = {Preprint},
}

Contact

For questions or issues, please open an issue on the dataset repository or contact the maintainers.