Argilla's FeedbackDataset is the configurable primitive that has replaced the legacy text-classification, token-classification, and text2text dataset classes, and it is the primary unit engineers interact with when collecting human preference data for DPO or RLHF. The data model rests on four composable entities: Fields define what annotators see, Questions define what they answer, Guidelines standardize how they interpret the task, and Records tie a unit of data to one or more Responses, Suggestions, metadata, and vectors. Understanding how these four primitives compose is more useful than memorizing Python arguments, because the same mental model governs both the Python SDK and the web annotation UI. This article walks through each primitive with the implementation context needed to design a preference dataset that survives a production training pipeline.
A FeedbackDataset is a named, server-side container that holds a collection of records and is fully configured up-front. Once pushed to the Argilla server, the dataset's shape — its fields, questions, and guidelines — is effectively frozen, while the records inside it remain mutable. Annotators can be added, responses updated, suggestions regenerated, and vectors recomputed without redefining the schema. This split between a stable container shape and a flexible payload is the single most important idea behind the data model, because most confusion in the Python SDK comes from mixing up those two layers.
Argilla is migrating to this single class because the legacy task-specific classes (DatasetForTextClassification, DatasetForTokenClassification, and DatasetForText2Text) each assumed a narrow labeling task baked into the schema. The FeedbackDataset removes that assumption: one configurable primitive can serve supervised fine-tuning data, preference data for DPO and PPO, retrieval evaluation data for RAG, and arbitrary multi-task labeling. For engineers, that means a single annotation pipeline can emit datasets for SFT, DPO, and RAG evaluation without forking the schema or maintaining parallel ingestion code. According to the Argilla data model documentation, this consolidation is the direction for Argilla 2.0.
The data model splits cleanly into two layers:
Keeping this distinction clear is what makes the SDK predictable: configuration arguments apply to the schema, while data arguments apply to individual records. As the practical guide to creating datasets notes, you define fields, questions, and guidelines first, then add records.
Argilla also ships predefined templates for common tasks: text classification, summarization, translation, NLI, sentence similarity, QA, SFT, preference modeling, PPO, DPO, and RAG. These templates are convenience wrappers around the same four primitives — they pre-fill sensible field and question configurations but do not introduce new abstractions. The rest of this article focuses on the primitives directly, because understanding them is what lets you adapt a template, debug a custom configuration, or build a new pipeline that survives a production training run.
In Argilla's FeedbackDataset, a Field is the unit an annotator reads. It carries no input widget, no question semantics, and no label — only the data that anchors the annotation task. Argilla 2.x ships three Field classes, each aligned to a data modality.
TextField is the default Field for natural language data. It accepts plain strings or Markdown source. The use_markdown argument (default False) toggles Markdown rendering: when set to True, headings, lists, embedded media, and PDFs are rendered in the UI rather than displayed as raw characters. This is the Field most preference datasets will reach for, since DPO and RLHF prompts and candidate completions are typically textual.
ChatField was introduced to represent conversational data as a list of role/content message pairs (for example, alternating user and assistant turns). Rather than flattening a transcript into a single string, it preserves the turn structure that downstream conversational models expect. This is the right Field whenever the annotator is judging multi-turn dialogue quality, not just single-prompt responses.
ImageField extends the schema to multimodal records. It accepts both remote URLs and Data URLs (base64-encoded payloads), so image corpora can be hosted externally or inlined directly into the record. This unlocks preference work on vision-language models where the prompt is an image or the candidates are generated captions.
Every Field class accepts the same four arguments, so the configuration vocabulary does not change between modalities:
name — the internal identifier used in the SDK and in record payloads.title (optional) — the label rendered in the UI. When omitted, Argilla capitalizes the name for display.required (optional, default True) — whether the Field must be populated for every record. At least one Field in the dataset must be marked required.use_markdown (optional, default False, TextField only) — switches on Markdown rendering.The order in which Fields are appended to the dataset is the order annotators see them in the UI, so layout decisions belong in the schema, not in post-hoc UI tweaks.
A DPO-style record requires one prompt Field followed by N candidate-response Fields in a stable order. The prompt Field should therefore be added first, and each candidate Field (response_a, response_b, ...) appended afterward. A consistent positional mapping between Field index and candidate identity is what lets a RankingQuestion reliably pair its options with the right completions.
A note on older documentation: pre-2.0 Argilla guides sometimes describe "checkboxes and dropdowns" as Field types. Those widgets are Questions in the 2.x model, not Fields. Keeping that distinction clean matters, because mixing the two primitives is the most common source of misconfigured preference datasets.
LabelQuestion presents annotators with a fixed set of options where exactly one selection is allowed. Internally it maps to a one-of enum, which the Argilla UI renders as a rounded affordance to signal mutual exclusivity. MultiLabelQuestion shares the same configuration surface but accepts any subset of options, including the empty set, and renders as a squared affordance. Both accept a labels list (or a dictionary mapping internal keys to display strings) and a required flag that defaults to True. For MultiLabelQuestion, a visible_labels parameter controls how many options are initially expanded in the UI (default 20), with None showing all entries at once (argilla-io.github.io).
RankingQuestion asks annotators to order a list of options. Ties are permitted, and every option must be ranked, which means a complete response cannot be submitted until the list is fully ordered. The values argument is typically the set of Field names that hold candidate responses, so the UI presents the annotator with the actual completions to drag into order rather than abstract labels. This is the canonical primitive for preference collection: a single RankingQuestion over the candidate-response Field values directly produces the chosen/rejected pairs consumed by DPO trainers (docs.v1.argilla.io).
RatingQuestion accepts a list of unique integers constrained to the range [1, 10] (with later SDK releases also accepting0). It is well suited to scoring a single attribute, such as helpfulness or factuality, on a numeric scale. When several RatingQuestions are combined, their averages can be reduced to a binary preference signal. The argilla/ultrafeedback-binarized-preferences dataset follows exactly this pattern: it averages per-attribute ratings across helpfulness, honesty, truthfulness, and instruction following to derive chosen/rejected pairs suitable for DPO (huggingface.co).
SpanQuestion anchors annotation to a specific Field by name (field argument) and lets annotators highlight a substring of that Field and apply a label from a fixed set. An allow_overlapping flag (default False) controls whether two spans on the same selection are permitted. It is a strong fit for rationale annotation, such as highlighting the evidence supporting a rating, but it does not directly produce a preference signal. TextQuestion provides a free-form text area, optionally rendered as Markdown via use_markdown, and is the right tool for rationale write-ups, corrections, or qualitative feedback (argilla-io.github.io).
Every Question carries a required flag that defaults to True, and at least one Question in a FeedbackDataset must be required, otherwise no response can be persisted. The description argument is surfaced as a tooltip in the web UI and should stay short and contextual; the full rubric belongs in the dataset Guidelines rather than being duplicated across tooltips. In practice, the DPO-oriented layout is a single required RankingQuestion over the response Fields, optionally augmented by several required RatingQuestions for finer-grained per-attribute supervision, with TextQuestion and SpanQuestion reserved for optional rationale capture.
Instructions in Argilla live in two deliberately separate places, and treating them as interchangeable is one of the most common design mistakes when configuring a preference dataset.
Dataset-level Guidelines are passed as a guidelines argument when the FeedbackDataset is created in the Python SDK. In the annotation UI they render inside an expandable panel so annotators can reference them without losing context on the record they are judging. On current Argilla releases the Guidelines field supports Markdown, and the document can be edited after deployment from the dataset settings page by users with owner or admin roles — meaning calibration updates do not require re-creating the dataset or interrupting an in-flight annotation queue (Argilla docs – data model; Argilla – create_dataset practical guide).
Per-Question descriptions are passed as a description argument on each Question object. In the UI they surface as a tooltip beside the question label, so they are visible at the moment of answering but are not the primary reference document.
Argilla's own guide recommends always populating dataset-level Guidelines and treating Question descriptions as a short, in-context hint rather than a substitute rubric. Question descriptions can summarize the relevant portion of the Guidelines, but on their own they are rarely enough to align an annotation team (Argilla – dataset how-to guides).
A strong dataset-level Guidelines document typically contains four blocks:
Question, each with at least one positive example and one negative example so annotators can see the boundary of an acceptable answer.RankingQuestion).Discard status that is distinct from Submit, so annotators need to know which records should be marked discarded rather than answered. Without an explicit rule, ambiguous records tend to leak into the training set as low-quality responses.Because Guidelines can be revised in place via the settings UI, teams should plan to iterate on this document during the first batch of annotations, then freeze it once inter-annotator agreement stabilizes.
A Record is the atomic unit of an Argilla FeedbackDataset: a single data point that binds together everything an annotator sees, answers, or that the platform attaches automatically. Conceptually it is a small container with five optional slots — Fields, Responses, Suggestions, metadata, and vectors — and the same abstraction is used by the Python SDK, the web UI, and the push-to-Hub workflow, so anything that holds in one surface holds in the others.
The Fields slot carries the input payload that the annotator reads: typically a prompt, a prompt–response pair, or a chat transcript. Fields are read-only from the annotator's perspective; they describe what is being judged, not the judgment itself.
The Responses slot is where human feedback lives. A Record can carry zero or more Responses, and each Response is attributed to a specific annotator, holds the answer values for the configured Questions, and has a status flag. Three terminal states matter operationally: submitted (finalized, exported to training), draft (work in progress, not yet trusted), and discarded (annotator opted out or marked as unreliable). Because many annotators can respond to the same Record, the dataset naturally stores the redundancy needed for inter-annotator agreement metrics such as Cohen's kappa or Krippendorff's alpha — disagreements are first-class data points, not noise to be averaged away.
The Suggestions slot stores machine-generated weak signals. There is at most one Suggestion per Question, and a Suggestion is structured exactly like a Response so the UI can render it as a pre-filled answer. The most common pattern is to attach a reward-model ranking to a RankingQuestion: the annotator sees the model's ordering and only edits it when the model is wrong. This converts a cold-annotation task into a correction task and dramatically reduces label cost for DPO and RLHF pipelines.
The metadata slot holds typed key-value pairs (for example language, source, difficulty) that are queryable and sortable in the UI, letting teams build focused queues such as "all English records submitted before 2026-09-01, sorted by annotator disagreement."
The vectors slot holds typed embeddings declared on the dataset via VectorField with a required dimensionality. These powers semantic similarity search and the active-learning workflows Argilla supports: a model can flag records far from the training distribution, and annotators can be routed to them through metadata filters.
Because every UI mode — bulk annotation, focus mode, and review — operates on the same Record object, switching view does not change the data model; it only changes how the same Fields, Responses, and Suggestions are presented. This uniformity is what makes a preference dataset designed in Python survive untouched when it moves to the UI for labeling and back to Python for training (Argilla data model).
The four primitives compose into a working DPO recipe through a fixed sequence. Each step maps to a concrete Argilla entity so the resulting dataset can be exported directly into the training format expected by DPO trainers.
Step 1 — Define the prompt Field. A single required TextField carries the prompt that annotators will see. Enabling markdown rendering is recommended because real-world prompts frequently contain code blocks, lists, or links that lose meaning in plain text.
Step 2 — Define N candidate-response Fields. Add one TextField per model being compared, named after the candidate (for example, response_a, response_b, response_c). The Field count is the maximum number of candidates any annotator will rank, so over-provisioning is cheap and under-provisioning forces schema changes later.
Step 3 — Add the primary RankingQuestion. A RankingQuestion whose values list contains the candidate Field names supplies the supervision signal. Per the question contract, all options must be ranked and ties are allowed, so annotators can express genuine uncertainty. This single question is sufficient for binary DPO; additional candidates only improve signal density.
Step 4 — Add per-dimension RatingQuestions. Add RatingQuestion instances for the attributes Argilla has used in published preference data — helpfulness, honesty, truthfulness, and instruction-following — each with values restricted to unique integers in the range [1, 10] (Argilla question types). These per-dimension scores can be aggregated, filtered, or used as tie-breakers, mirroring the construction of argilla/ultrafeedback-binarized-preferences (Hugging Face collection).
Step 5 — Write Guidelines with examples and a discard policy. Guidelines should include two or three example rankings on representative prompts, a definition of ties, and an explicit policy for discarding prompts that are ambiguous, unsafe, or off-task. Without a written discard policy, annotators silently invent their own and inter-annotator agreement collapses.
Step 6 — Pre-load Suggestions. Attach a Suggestion to each Record carrying an initial ranking from a reward model or a pairwise judge LLM. Annotators then edit rather than author from scratch, which raises throughput and reduces variance on easy comparisons.
Step 7 — Export to the DPO contract. Map responses into the standard {prompt, chosen, rejected, score_chosen, score_rejected} schema. The chosen and rejected columns come from the RankingQuestion; the score columns are computed from the RatingQuestions of the corresponding candidate.
Version caveat. This composition is stable on Argilla 2.x. The legacy 1.x FeedbackDataset has a different import path and some Field arguments (notably markdown handling) diverge, so engineers upgrading should re-validate the schema before pushing existing Records through the new export pipeline (Argilla data model).
Several of the primitives described earlier in this article are tied to a specific Argilla release line, and writing portable code means understanding which ones. The notes below flag the version boundaries that matter for a preference-data pipeline.
ChatField and ImageField are 2.x additions. Both field types ship with the FeedbackDataset model and are absent from the legacy 1.x schema, which only supported TextField (plain text and markdown). If a record contains chat turns or image URLs, the target server must be on Argilla 2.x or later; a 1.x server will reject the schema (Argilla changelog).
Vector search endpoints and SDK helpers landed in 2.x. The /api/v1/datasets/:dataset_id/vectors-settings endpoints (create, list, delete, patch) and the matching add_vector_settings, update_vectors_settings, delete_vectors_settings, and vector_settings_by_name methods on FeedbackDataset are part of the 2.x release line. Any code that calls these helpers or relies on the VectorField settings object must target a 2.x server.
FeedbackDataset replaces the task-specific classes. DatasetForTextClassification, DatasetForTokenClassification, and DatasetForText2Text are deprecated in favor of FeedbackDataset and are slated for removal. No automatic conversion utility is shipped, so engineers migrating from 1.x should expect to re-author fields, questions, and guidelines against the new primitives rather than reuse the old configuration.
RatingQuestion upper bound has shifted historically. The current guide documents a 1–10 rating range, but the upper bound has been raised in past versions. Any client that hard-codes a 10-element slider or assumes a fixed cardinality should be re-checked against the target Argilla release before deployment.
Hugging Face Hub auto-suggestion is a starting point, not a schema. When importing from the Hub, Argilla generates an initial layout from the dataset features, including auto-generated RankingQuestions. These suggestions can mismatch the field order of a hand-built schema, so the auto-layout should always be reviewed and adjusted before pushing records.