Ragas test-set generation combines a knowledge-graph pipeline with multiple model roles rather than asking one LLM to produce an evaluation set in a single step. A generator proposes questions, critique and filtering improve quality, and embeddings support similarity-dependent transformations and metrics. This article opens that pipeline, explains the evolutionary question types, and distinguishes legacy helper examples from newer API patterns.
When I started pulling the pipeline apart to figure out which model was responsible for what, the whole thing simplified surprisingly fast. Under the hood, Ragas testset generation performs two core operations: it builds a KnowledgeGraph from your documents, and then it uses that enriched graph to produce the final Testset. Everything else — the question mutations, the sampling, the three models themselves — lives inside one of these two stages.
First, Ragas creates a KnowledgeGraph object and adds your documents to it. A graph holding nothing but raw documents is not very useful on its own, so the next step is enrichment through Transformations — components built on the BaseGraphTransformation class. Each transformation reads your documents and attaches extra information to the graph, and that extra information is what gets used later to synthesize test questions.
Two things stand out to me in this design:
default_transforms is the convenience function that assembles the default transformation pipeline. It wraps around two of the three models from my setup: an LLM and an Embedding Model of my choice. This is the exact point where those models enter the picture.BaseGraphTransformation and write your own when your documents need something specific.Once the graph is enriched, the second operation begins. The enriched KnowledgeGraph is used to generate a set of scenarios (built on BaseScenario), and those scenarios in turn produce the final Testset, which follows the documented Testset schema. The way I read this two-step flow, a scenario is a plan for a question — it decides what material from the graph to use and what the question should look like — and the finished question only appears when the scenario is turned into actual testset entries.
One detail that matters for your compute budget: the enriched graph can be saved and reused. The docs refer to the reloaded object as loaded_kg, and you can pass it straight into a TestsetGenerator. That means you can iterate on generation — for example, experimenting with different query distributions — without re-running document loading and transformation enrichment every single time. Since enrichment is where your LLM and embedding model get called across your documents, this kind of reuse is worth a lot.
End to end, the official flow looks like this:
DirectoryLoader from langchain_community, or any LlamaIndex loader. The official tutorial uses the Hugging Face dataset explodinggradients/Sample_Docs_Markdown.LangchainLLMWrapper.generate_with_llama_index_docs instead.The part I appreciate most: both stages can be customized independently. You can tune how the graph is built without touching generation logic, and vice versa — which is exactly what makes the next layer, the evolutionary question machinery, feel approachable rather than magical.
A defining design choice in Ragas test-set generation is that it does not let the LLM simply "write questions." The framework's question creation is grounded in an evolutionary generation paradigm inspired by Evol-Instruct (arXiv:2304.12244). Instead of asking a generator model to dream up an entire test set from scratch, Ragas starts with simple questions and then systematically evolves them into harder, more varied forms based on the documents you provide. That distinction matters more than it might first appear.
The reasoning is twofold, and both parts come straight from how RAG systems fail in the real world.
So instead of trusting the model's instincts, Ragas takes those simple seed questions and deliberately crafts them into questions with specific characteristics:
The payoff is coverage. Because each evolved question stresses a different failure mode, you get a much clearer picture of how the various components inside your pipeline — retriever, generator, reranker — actually behave under pressure, which makes the whole evaluation more robust.
What convinced me this isn't just academic over-engineering is that AWS documents the exact same failure mode in its own RAG evaluation workflow. When question-answer pairs are generated from the same prompt across an entire dataset, the resulting questions appear repetitive and similar in form. They simply do not mimic real end-user behavior — an actual user might, for instance, use abbreviations when asking a question, or phrase things awkwardly, or assume context the system doesn't have.
This is where the evolutionary approach earns its keep:
Looking at it this way, the evolutionary paradigm isn't a clever trick; it's a direct response to a known blind spot. The generator model isn't trusted to be creative — it's given a structure and pushed down specific paths. In the next part, I'll break down what those paths actually look like when Ragas executes them.
So which of the three models actually writes the questions? That's the generator LLM, and its job description is broader than the name suggests: it produces the QA pairs — both the question and the reference answer — from your source documents.
In the documented OpenAI example, the generator is gpt-3.5-turbo-16k, instantiated and wired up like this:
generator_llm = ChatOpenAI(model='gpt-3.5-turbo-16k')
generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)
Two details stand out to me here:
from_langchain expects generator first, critic second, embeddings third. Each slot maps to a distinct role, so mixing up the first two arguments doesn't throw an error — it quietly hands the critic's job to the wrong model.Ragas does not lock you into OpenAI. The wiring pattern stays the same; only the classes change:
ChatOpenAI from langchain-openai.AzureChatOpenAI instances (for example a gpt-4-32k generator paired with a gpt-4o critic), with AzureOpenAIEmbeddings handling embeddings.LangchainLLMWrapper.That last bullet is the one I would underline. The wrapper keeps the pipeline portable — the rest of Ragas never needs to know which provider is behind it.
Here is the part of the black box most people miss. The generator side of the pipeline doesn't stop at the question. In the analogous AWS Bedrock workflow, each evaluation sample is built from two inputs: the generated question and the original source chunk it was derived from. Out of that come two more artifacts:
That extracted text becomes the context component of the evaluation dataset, and it is what makes the test set genuinely usable:
So the generator's real output is not just questions. It is a small, self-contained evaluation record: question, reference answer, and the context that ties them together. What it does not do is judge its own work — that job belongs to the second model, the critic LLM.
If the generator LLM is the creative half of the pipeline, the critic LLM is the editor standing behind it with a red pen. This second model invents nothing — its entire job is to validate the generation process and check the quality of what the generator produced before anything lands in your final test set. The model assignments in the documented example highlight one important design choice.
The docs deliberately pair a cheaper gpt-3.5-turbo-16k generator with gpt-4 as the critic. That asymmetry is not arbitrary — it reflects what each role actually demands:
gpt-4o for exactly the same reason.The takeaway: you can get away with a cheap generator, but a weak critic quietly poisons your test set with questions that look fine but aren't.
The critic's checks follow the critique-agent pattern, and its two main metrics are simple to state:
Based on these checks, questions that are not valid or appropriate for the RAG system get filtered out or modified, either by domain experts or, at scale, by LLM critique agents doing the first pass.
Model separation matters just as much at evaluation time. Using separate LLMs for data generation and RAG evaluation minimizes bias overlap: if the same model — or a very similar one — both generates the synthetic test data and powers the evaluation, the test can be inadvertently optimized for that specific LLM's tendencies, such as its phrasing habits and topic preferences. At that point, your benchmark stops measuring your RAG system and starts measuring the model grading it. A practical split looks like this:
Even with a strong critic and a clean split in place, I would still spot-check 5–10% of the generated triples with a domain expert. Generation errors compound into evaluation errors — a flawed question doesn't just sit in the test set, it produces a flawed verdict about your entire system.
While the generator LLM gets the glory and the critic LLM gets the veto power, the embedding model does the quiet groundwork. In TestsetGenerator.from_langchain(...), it is the third argument sitting next to the two LLMs — OpenAIEmbeddings() in the official quickstart, text-embedding-ada-002 through Azure in the cloud walkthrough. Same slot, same role: whenever Ragas needs to find something — similar chunks, related nodes, usable context — this model is the search engine behind it. This role is easy to overlook, even though it has the longest reach across the pipeline.
Looking at the pipeline, I can separate its job into two responsibilities:
default_transforms(...), and that pipeline consumes the embedding model too. The enrichment steps — anything that needs to judge similarity or relatedness between nodes — run on this same embedding space. So the model shapes the graph before the graph shapes the questions.Here is the catch I want to flag before you move on: this model choice does not retire after generation. It quietly follows you into evaluation, because several core Ragas metrics — answer_relevancy, answer_similarity, and answer_correctness — compute their scores on top of embeddings. The same vector space that decided what went into your test set is now deciding how good your answers look.
That produces two practical consequences for tuning:
So when I look at the three models now, the division of labor is clear: the generator writes, the critic filters, and the embedding model decides what is worth seeing — first while building the graph, then again while judging the results. It is the only lever that touches both ends of the pipeline, which is exactly why I pin it down before tuning anything else.
The conditional riddles and multi-context monsters in a generated set are not random accidents of a confused LLM. They were the direct output of Ragas' evolution taxonomy, which currently ships with five flavors — each one engineered to probe a different weakness in a RAG pipeline. Once you can name the flavor behind a question, you immediately know what that sample is actually testing, and debugging gets far easier.
What strikes me most when I look at this list is how the flavors split the work between the two halves of a RAG pipeline. Simple checks the floor. Reasoning stresses inference. Conditioning adds constraint-handling on top. Multi-context deliberately shifts the pressure onto retrieval, and Conversational changes the interaction contract entirely — the follow-up turn has to make sense in context, which is a very different demand than answering an isolated question.
If you only care about one flavor, make it this one. Multi-hop questions that require cross-chunk reasoning are the most common failure mode in production RAG — and, at the same time, the most underrepresented kind of question in naive synthetic sets.
The failure it catches is subtle. The right chunks exist in your index. Plain top-k retrieval may even surface some of them. But because the answer needs pieces combined from multiple chunks, a reranker that fails to combine them properly returns only part of what the question requires. The LLM then either invents the missing half or gives up — and the system keeps looking fine on simple benchmarks while breaking on real questions.
That is the real value of multi-context samples: they cannot be passed by accident. Either retrieval assembles all the required pieces, or the sample fails loudly. If I were auditing a generated test set for quality, this is the category I would scrutinize first.
One last note from the Ragas roadmap: the range of evolution techniques will be expanded moving forward. Treat today's five flavors as a starting set, not a closed catalog. Practically, this means the question mix is not stable across versions — regenerate a test set after an upgrade and you may see question shapes you have never tuned for. That openness is a feature, not a bug. It reflects the same Evol-Instruct thinking the generator is built on: the definition of a "good test question" is something that should keep evolving.
Tuning the mix comes down to one small Python dictionary. The distributions parameter maps each evolution type — imported from ragas.testset.evolutions — to the fraction of the test set it should occupy, and the documented example looks like this:
from ragas.testset.evolutions import simple, reasoning, multi_context
distributions = {simple: 0.5, multi_context: 0.4, reasoning: 0.1}
That is 50% simple, 40% multi-context, 10% reasoning, and it goes into the generation call as the third argument:
testset = generator.generate_with_langchain_docs(documents, 10, distributions)
The 10 is the number of test samples you want back. LlamaIndex users call generate_with_llamaindex_docs instead — the signature is the same.
A practical Azure example spreads the budget across all four types, with test_size=5:
One rule to check before you run anything: every evolution type you reference in the distributions dictionary must be a type the generator supports. List an unsupported type and the run fails.
The calls above are the classic helper signatures. In the v0.2.x API, the flow changes: the query distribution is configured on the TestsetGenerator itself, before you run the generation step. And if you have no specific requirements, you can skip the tuning entirely and fall back to Ragas' built-in default distribution — a reasonable starting point while you figure out what your corpus can actually support.
Configuring the mix is only half the job. After generation finishes, convert the test set with testset.to_pandas() and analyze the frequency of the question types: how many simple, multi_context, and reasoning questions were actually produced, and whether that frequency matches the distributions you configured. I treat this sanity check as a gate — never run your full evaluation on a test set you have not counted.
The check matters more than it sounds. With test_size=5, a 10% conditional slice does not divide cleanly, so realized counts drift from the configured fractions even when everything works correctly. When the drift is too large, iterate:
Repeat until the composition matches the questions your users actually ask — not the mix you assumed they would ask. That loop is where the machinery stops being a black box: you set the mix, the models build it, and the numbers tell you whether they kept their word.
With the internals now in view, I want to return to the nervousness I described at the start: when people run testset generation as a black box, most synthetic-data regressions trace back to the same five pitfalls. What strikes me is how precisely each one maps onto the components we have just taken apart — the generator LLM, the evolution transforms, and the critic that is supposed to catch mistakes.
Zooming out, these five pitfalls are surface symptoms of four systemic problems. Poorly generated synthetic data introduces:
This is more than intuition. Research on LLM evaluation (Wang et al., 2024) shows that synthetic data quality has a significant impact on evaluation reliability. A weak test set does not merely underperform — it actively misleads.
Two constraints frame everything above. First, the quality of a synthetic dataset is bounded by the accuracy and neutrality of the model that generates it. The critic can filter bad output, but it cannot lift the generator above its own blind spots. Second, the economics: Ragas itself is free and Apache 2.0 licensed, but the model calls used for test generation and scoring still cost real money — and evolutionary generation, with its personas, transforms, and critic verdicts, makes a lot of round-trips.
Opening the black box does not remove these risks. What it does is turn them from mysteries into checklist items — which, to me, is exactly the point.