Evaluating a retrieval-augmented generation system against real user queries is the gold standard, but collecting that data is slow, expensive, and rarely covers the long tail of failure modes. Synthetic QA generation fills the gap, and recent work has moved decisively from single-prompt generators toward orchestrated multi-agent pipelines that can deliver diversity, grounding, and privacy guarantees in a single run. This article walks through that orchestration layer: the three-specialist-agent framework and its sequential Algorithm 1 workflow, role-based expert deliberation patterns, reasoning model-driven generation pipelines, and label-targeted pipelines that invert the traditional question-first order. The goal is to give engineering teams building a custom evaluation set a concrete blueprint for assembling their own pipeline rather than adopting an off-the-shelf tool.
Monolithic, single-prompt QA generators remain attractive because they are simple to implement and cheap to run. In practice, they tend to fail along three axes that matter for RAG evaluation. First, topical coverage is narrow: one prompt sees only the chunks handed to it, so the resulting questions cluster around whatever the prompt happened to surface, leaving the long tail of retrieval failures invisible. Second, grounding is unreliable: without an explicit verification step, the model can freely hallucinate facts that were never present in the source chunk, producing gold answers that look correct but cannot be matched against retrieved evidence. Third, privacy guarantees are weak: a single prompt rarely enforces structured entity detection and masking, which is a serious problem whenever the underlying corpus contains personally identifiable information (PII), protected health information (PHI), or proprietary business content.
Secondary literature echoes these failure modes. Synthetic QA benchmarks tend to inherit stylistic bias from the generator, oversimplify real-world query patterns, and overrepresent the phrasing style of the model that produced them — issues that erode ecological validity when the set is used to rank retriever or generator architectures.
The framework introduced in arXiv 2508.18929 replaces the monolithic prompt with a sequential orchestration of three specialized agents:
Each agent owns one concern and produces an auditable artifact, which is qualitatively different from a single prompt whose decisions are entangled and opaque.
Multi-agent orchestration is worth the cost in three concrete situations:
Teams building a custom RAG evaluation set in any of these regimes benefit more from a transparent, modular pipeline than from an off-the-shelf single-prompt tool.
The three-specialist-agent framework, introduced in arXiv 2508.18929, decomposes synthetic QA generation for RAG evaluation into three loosely coupled components that execute sequentially under a single orchestration procedure (Algorithm 1 in the paper). Each agent owns a distinct objective — topical coverage, sensitive-data handling, or question-answer synthesis — and emits a structured artifact that the next stage consumes. The authors describe the agents as a unified Python pipeline implemented with LangGraph, but the design treats each agent as an independently swappable module (arXiv 2508.18929).
The pipeline begins with the Diversity Agent, which takes the raw source corpus and a set of clustering hyperparameters as inputs. It computes semantic embeddings over the corpus, groups documents into topical clusters, and then selects representative samples from each cluster. The goal is not uniform sampling but rather breadth: every important region of the embedding space should contribute to the downstream dataset, which mitigates the long-tail blind spots common in naive single-prompt generators. By weighting selection per cluster, the agent enforces that minority topics remain represented in the inputs handed to subsequent stages (arXiv 2508.18929v1).
The cluster representatives flow into the Privacy Agent, which performs entity detection and pseudonymization over personally identifiable information (PII), protected health information (PHI), and personally identifiable workplace information (PWI). Each detected entity is replaced with a consistent surrogate so that downstream prompts never see raw sensitive values. Alongside the rewritten text, the agent emits a structured privacy report describing the categories detected, counts, and confidence levels. This report becomes an auditable artifact that engineering teams can review before any QA is generated, supporting governance reviews such as EU AI Act alignment (arXiv 2508.18929v1).
The privacy-preserved, diversity-weighted inputs reach the QA Curation Agent, which applies advanced prompting techniques — in the reference implementation, GPT-4o — to synthesize ground-truth question-answer pairs. The agent targets the full spectrum from straightforward factual lookups to nuanced, domain-specific questions, ensuring that the resulting set stresses both retrieval and generation. Its output is twofold: the QA pairs themselves, ready to act as ground truth, plus a generation report that summarizes success rates, QA-type distribution, and grounding alignment with the source content (arXiv 2508.18929v1).
A defining choice in the architecture is that each agent is kept modular rather than fused into a single prompt or model call. This separation has three practical consequences:
This modularity is the principal advantage over single-prompt baselines such as direct prompting or evolutionary generators, where diversity, privacy, and utility are tangled together and cannot be evaluated or improved independently (arXiv 2508.18929).
Algorithm 1 formalizes the pipeline as a strictly ordered, three-step interaction. Given a source corpus and a set of clustering hyperparameters, the algorithm produces a synthetic QA dataset enriched with semantic diversity and a privacy guarantee, plus two structured reports for downstream auditing.
The steps proceed as follows:
text-embedding-3-small model with a fixed dimensionality.The flow is intentionally linear: downstream agents see only the outputs of the previous node, which keeps each transformation independently auditable.
All three agents are implemented in Python and orchestrated with the LangGraph framework. Each agent is modeled as a node in a directed state graph, with explicit message passing between nodes carrying both the transformed data and the accompanying report. This state-graph abstraction makes the sequential dependency visible in code and lets engineers insert inspection or branching logic without rewriting the agents themselves.
The model assignments in the reference pipeline are version-sensitive and worth pinning explicitly:
Three engineering decisions recur across the reference implementation and tend to determine whether the pipeline is usable in production:
Taken together, these choices turn Algorithm 1 from a paper-only specification into a reproducible, cost-aware pipeline that engineering teams can adapt to their own corpora.
Discuss-RAG, introduced by Dong et al. (2025), offers a complementary orchestration pattern to the sequential diversity-privacy-QA flow described earlier. Rather than passing data through fixed processing stages, Discuss-RAG instantiates two cooperating agents — a recruiter agent and a summarizer agent — that jointly build context-rich synthetic documents and the QA pairs anchored to them. The recruiter identifies candidate source material and frames it for downstream use, while the summarizer consolidates and grounds the resulting QA content. Because both agents operate with explicit personas, scopes, and interaction contracts, the output quality emerges from the negotiation between roles rather than from a single zero-shot pass (Synthetic QA Datasets for RAG — Emergent Mind).
In a role-conditioned pipeline, every agent receives a system prompt that specifies three things:
These constraints change the prompt landscape substantially. Instead of one monolithic instruction asking the model to "generate diverse private QA pairs," each agent operates on a narrower task with clearer success criteria, which tends to reduce hallucination and improve alignment with the source. The deliberative loop between recruiter and summarizer functions as a lightweight multi-perspective reasoning module, where each role contributes a different angle on the source material (Synthetic QA Datasets for RAG — Emergent Mind).
Dong et al. (2025) report that this role-based deliberation improves contextual realism relative to single-pass prompting, particularly on documents where surface-level QA extraction would miss cross-paragraph dependencies. The recruiter-summarizer exchange approximates an expert discussion, allowing the summarizer to push back on under-specified context and ask the recruiter to surface additional evidence before a QA pair is committed. The authors released Discuss-RAG as open source, including the agent instruction datasets, so teams can reproduce or extend the role assignments for their own corpora (Synthetic QA Datasets for RAG — Emergent Mind).
Discuss-RAG ships with code and instruction data, making it a practical template for teams that want role-based deliberation without building agent harnesses from scratch. However, the broader question of how agents should communicate remains open. Driouich et al. (2025) flag richer inter-agent communication protocols — including standardized messaging layers analogous to the Model Context Protocol — as a key research direction for multi-agent dataset synthesis, noting that current pipelines rely on ad-hoc message formats and limited feedback channels (Synthetic QA Datasets for RAG — Emergent Mind). Teams adopting the Discuss-RAG pattern should therefore plan for iteration on their interaction contracts as more formal protocols mature.
A second paradigm for synthetic QA generation, described in arXiv 2502.15854, replaces the orchestrator-and-specialist design with an end-to-end instruction-tuned reasoning model that produces both the question and the answer in a single pass. The paper evaluates three reasoning-oriented models — DeepSeek-R1, DeepSeek-R1-Distill-Qwen-32B, and Phi-4 — and observes that distilled reasoning variants tend to produce higher-fidelity concept alignment, with the Qwen-32B distill achieving roughly a +14% mean IoU lift over the alternatives on the tested corpora. No single model dominates across all domains, so model choice should track the target corpus rather than be fixed a priori.
What distinguishes this paradigm is the output schema. Rather than emitting a single contiguous answer span anchored to one chunk, the pipeline emits QA pairs whose reference answers span multiple non-contiguous regions of the source document. A reference answer is therefore a list of token spans scattered across the document, not a single block of text. The accompanying reference implementation is open source under the repository aryan-jadon/Synthetic-Data-Generation-and-Evaluation-using-Reasoning-Model.
Discontinuous references force retrieval evaluation to become token-aware rather than chunk-aware. Two new metrics capture this:
The practical consequence is that a retriever returning the right chunk but missing the right tokens inside it is now visibly penalized, and a retriever that needs to stitch together evidence from several distant regions is tested under realistic conditions. Standard Recall@K or Context Precision over coarse chunks cannot distinguish these failure modes.
To validate that the pipeline generalizes beyond shallow generators, the authors run it on three highly technical corpora that share little surface vocabulary:
Each of these domains frustrates naive question generators because the answer cannot be found in one chunk, the entities are low-frequency in pretraining corpora, and correct synthesis requires reasoning across the document. The fact that reasoning-model pipelines emit usable discontinuous references across all three suggests the approach is robust where shallow, single-prompt generators tend to either hallucinate or produce trivially localized questions.
For engineering teams, the practical takeaway is straightforward: when the evaluation target is a corpus where evidence is scattered, discontinuous-span reasoning-model generation paired with Precision Ω and IoU gives a measurably harder and more diagnostic evaluation set than chunk-level synthetic QA.
Standard synthetic QA pipelines ask the model to produce a question first and then to answer it. The result is convenient but unreliable: nothing forces the question to be grounded in any specific statement in the source corpus, and the resulting question-type mix reflects whatever distribution the LLM happens to emit rather than what the evaluation actually needs. Lima et al. (2024) propose a label-targeted, theme-first pipeline that reverses this order and anchors every question to a real extracted claim.
The inversion has two stages.
Two operational benefits follow from this inversion.
Lima et al. (2024) also show that the pattern works with cost-effective generators such as Flan-T5-large fine-tuned with LoRA on balanced templates, so the inversion is not tied to large proprietary models (emergentmind.com).
A strong default is to combine label-targeted generation with the three-agent framework introduced in the previous sections.
The result is a pipeline in which claims are diverse, privacy-safe, and explicitly labeled before any question is written — the strongest available default for assembling a custom RAG evaluation set.
In the three-agent framework from arXiv 2508.18929, the QA Curation Agent does more than emit pairs. It also writes a structured generation report that should be the first artefact a human reviewer reads before the dataset enters evaluation. The report covers three things: the QA type distribution (factual, multi-hop, comparative, domain-specific), the per-type success rates and generation dynamics, and the grounding alignment between each answer and its source chunk. Because the curation agent is implemented on top of GPT-4o for throughput, the report is cheap to produce and stable to parse (arXiv 2508.18929). Treat a low grounding-alignment score or an unexpectedly narrow type distribution as a signal to revisit the Diversity Agent's clustering hyperparameters rather than to silently filter downstream.
Four filters dominate practice, and they compose rather than substitute:
For context-heavy or technical corpora, NLI plus round-trip consistency catches grounding breaks that LLM-as-judge routinely misses. For long-tail open-domain sets, perplexity filtering is the strongest pre-filter for fluency noise.
Multi-agent orchestration roughly triples inference spend compared with single-prompt generation, since each chunk traverses Diversity, Privacy, and QA Curation stages. A pragmatic mitigation is to run the Diversity Agent over a stratified sample (for example, 10–20 percent of clusters) and only feed its representatives into the privacy and curation stages. NVIDIA's NeMo Curator pipeline demonstrates the same principle with an embedding-model-as-a-judge and answerability filter applied selectively (NVIDIA Developer Blog).
Use this rubric when choosing an orchestration shape: