A multi-gigabyte CSV can paralyze a notebook, yet the same data wrapped in an Arrow file opens in seconds on the same laptop. The difference is not magic; it is how the bytes are stored, mapped, and accessed by the 🤗 Datasets library. This article unpacks the Apache Arrow backend behind Dataset and IterableDataset, explains the zero-copy memory-mapped read path, and shows where each class wins or loses. It is aimed at dataset publishers who decide how bytes are packaged and at engineers who consume those bytes on machines they do not control.
🤗 Datasets takes a different route. As the project README states, the library is "built on an Apache Arrow backend that provides zero-copy memory-mapped storage," and because "datasets are memory-mapped rather than being fully loaded into memory," they "free you from RAM limitations" and let users "work with datasets that are larger than your machine's available memory" (huggingface/datasets). The documentation sharpens the same point: Arrow delivers "zero-copy reads" together with "no memory constraints," giving "optimal speed and efficiency even when processing very large datasets" (Hugging Face Docs).
Memory mapping is an operating-system facility, not a library trick. When a dataset is loaded in non-streaming mode, the Arrow file is registered with mmap() (or an Arrow equivalent), which makes the kernel place the file's bytes directly into the process's virtual address space. No bytes are copied into Python heaps at load time. The process now "sees" the file as if it were a normal in-memory buffer, but the corresponding physical pages only exist on disk.
The second half of the trick is the OS page cache. As user code indexes a row, iterates a batch, or calls .map(), the kernel faults in the 4 KB page(s) that contain the requested offset and serves them from RAM. Pages that are never touched stay on disk; pages touched once may be evicted under memory pressure and re-fetched later. From the user's perspective the dataset feels in-memory, while RAM consumption tracks only the working set rather than the full file.
This behaviour is not free — it depends on two preconditions that the rest of this article unpacks:
Dataset (memory-mapped, random-access) and IterableDataset (lazy, streamable) — and each trades off random access, memory use, and iteration differently. The next sections explain those trade-offs and show when to pick one over the other (huggingface/datasets).
The Arrow file format is what makes zero-copy, memory-mapped access possible in 🤗 Datasets. Arrow IPC files (sometimes called feather files when used as a standalone file format) are columnar, fixed-layout, and aligned, so the operating system can map a contiguous range of bytes into virtual memory and the Arrow libraries can return slices of that mapping as views, rather than parsing and allocating per record.
That layout is fundamentally different from row-oriented formats such as CSV or JSON Lines. In those formats, reaching row N requires scanning every preceding byte, locating field delimiters, and allocating a Python object for each field. Arrow stores each column in its own contiguous buffer with a fixed stride for fixed-width types, so reading row 100,000 of an int64 column is a constant-time pointer offset into a memory-mapped region, with no deserialization step in between.
The on-disk packaging seen on the Hugging Face Hub is not always Arrow directly. When a dataset is uploaded with push_to_hub, the Hub stores it in Parquet by default, because Parquet offers efficient compression, rich typing, and broad support from optimized batched readers (Hub docs: Adding datasets). CSV and JSON Lines are accepted but not recommended for data larger than several gigabytes. After download, the 🤗 Datasets library materializes Arrow IPC files in the local cache directory, and every subsequent read path goes through those memory-mapped files. The library reports itself as "built on an Apache Arrow backend that provides zero-copy memory-mapped storage" precisely so that "datasets are memory-mapped rather than being fully loaded into memory" (huggingface/datasets README). Page faults bring in only the touched blocks, and Python objects are constructed lazily as the user iterates or indexes.
This asymmetry explains the opening observation. A multi-gigabyte CSV cannot open quickly because the parser must walk every byte and allocate per record before any query runs. The same bytes wrapped as Parquet on the Hub and materialized as Arrow locally only pay the cost for rows actually accessed, which is why the dataset appears to open in seconds even when it dwarfs the host's RAM.
One version-sensitive caveat worth flagging: the Arrow IPC layout has been stable for years, but the on-disk alignment guarantees and footer schema can evolve across major format revisions. Engineers and publishers building tooling that depends on a specific block or page size should confirm the current rules in the Apache Arrow specification before hard-coding any alignment constant.
Dataset, ._data, and .nbytesload_dataset('c4', 'en')When the call is made in non-streaming mode, 🤗 Datasets first resolves the dataset script, then downloads the underlying Parquet shards (in the case of C4, one Parquet file per split shard on the Hub) into the local cache directory. Those Parquet files are then materialized into Arrow IPC files on disk; this conversion is what makes subsequent reads cheap, because Arrow's columnar format and fixed-width layout are the substrate that memory mapping can exploit efficiently. The end result is a DatasetDict whose train split is a Dataset whose backing storage is a memory-mapped pyarrow.Table — no full deserialization into Python objects has taken place, and no copy of the rows sits in process RAM yet.
Two attributes make the mapping tangible:
dataset['train']._data returns the underlying pyarrow.Table. Because it is memory-mapped, constructing or printing it is essentially free; the OS has only mapped pages, not read them.dataset['train']._data.nbytes reports the total size of the mapped buffer in bytes — for C4 en this runs into tens of gigabytes. Seeing that number grow without your RSS growing alongside is the clearest signal that mapping, not allocation, is at work.The relevant properties of the Arrow backend — zero-copy reads, fast indexing and slicing, and out-of-core access for data larger than RAM — are documented as the core benefits of the library's design.
Indexing (ds[1000]) and slicing (ds.select(range(100))) do not pull the whole table into memory. Arrow walks its internal record-batch and page offsets, identifies which pages contain the requested row range, and asks the OS to fault those pages in. The values returned are zero-copy views into the mapped buffer; Arrow hands back typed arrays that share memory with the file on disk. Touching new rows means touching new pages; re-touching rows whose pages are still resident is essentially free.
Two practical consequences follow:
This is the contract that makes Dataset a good fit when you can afford the disk footprint and need random access, and that motivates IterableDataset (covered next) when you cannot.
IterableDataset Beats Dataset: Streaming Without a CacheIterableDataset Beats Dataset: Streaming Without a CacheArrow memory mapping has a quiet assumption: there has to be a file on disk to map. When the dataset is larger than the disk, or the user simply cannot wait for a full download before seeing the first example, that assumption breaks. The escape hatch is IterableDataset, produced by calling load_dataset(..., streaming=True). Instead of opening memory-mapped Arrow files, the library fetches shards on demand and yields examples through a Python iterator, never materializing a full pa.Table (Hugging Face Datasets library).
The win is constant RAM and an immediate start. Examples arrive as soon as the first shard is downloaded and parsed, so iteration can begin before the dataset as a whole has crossed the wire. According to the library documentation, streaming mode is now "up to 100x faster with the Xet backend," though the deeper point is architectural: streaming trades random access for bounded memory and zero upfront latency (Hugging Face Datasets library).
What you lose is the interface that depends on knowing the whole table:
len(ds) — the total row count is unknown because shards may not have arrived yet.ds[i] — random access requires an offset map, which streaming cannot maintain cheaply..map() no longer memoizes a full cached table — the cache is a per-shard Arrow cache rather than a single materialized result (Hugging Face Datasets library)..shuffle() becomes a buffer, not a permutation — typically a fixed-size window of recent examples, since a global Fisher–Yates shuffle over an unbounded stream is undefined.This reframe turns packaging into the publisher's lever. If the dataset is too large to cache locally, the only way streaming can be efficient is for the underlying repository to expose many small Parquet shards rather than one monolithic file. Each shard becomes an independent fetch unit, so the iterator can pipeline downloads and amortize seek costs. A single huge Parquet file, by contrast, forces streaming to either pull the whole thing upfront or invent fragile range-request logic. The Arrow backend's zero-copy, out-of-core reads still apply once a shard is in hand (Hugging Face Datasets library), but the granularity of the archive now determines how well streaming behaves on a machine the publisher cannot control.
The reader experience of load_dataset is largely decided long before any consumer runs it — it is shaped by choices made inside the publisher's repo. Four levers matter most: Parquet row-group size, the number of shards, whether a loading script pre-builds Arrow IPC files, and whether the repository is structured for streaming. Get any of them wrong, and even a memory-mapped backend cannot save the first load.
Row-group size for Dataset users. When a consumer opens a repo in non-streaming mode, the Arrow cache is materialized on the consumer's machine, and the page cache is what determines how snappy random access feels. Reasonable row groups — large enough to keep I/O efficient, small enough that the columns a consumer actually uses fit comfortably in the operating system's page cache — make Dataset.__getitem__ and .map() feel near-in-memory. Row groups that are too small inflate metadata and metadata-fetch overhead; row groups that are too large make selective reads waste bandwidth on entire columns.
Sharding for IterableDataset users. Streaming begins as soon as the first shard is fetched, so the right granularity is small: enough shards that no single download dominates the connection, and small enough that the first sample appears within seconds. A single multi-gigabyte Parquet file forces every consumer to wait for the whole blob before iterating, defeating the point of streaming=True.
Pre-built Arrow vs. client-side materialization. A loading script that produces Arrow IPC files on first use effectively pushes the materialization cost onto every consumer individually — each one pays the download-plus-conversion tax once. By contrast, a dataset repository that is already Arrow-ready, or that ships Parquet with sensible metadata, lets the consumer skip that step. Because push_to_hub stores data as Parquet, the Arrow IPC materialization happens client-side after download, which means a slow first load is paid by every consumer rather than amortized by the publisher.
Recommendation summary.
Dataset users: provide an Arrow-ready repo with reasonable row-group sizes so the materialized cache is friendly to the page cache.IterableDataset users: shard data into small Parquet files so streaming can begin on the first shard.In short, the publisher decides whether the consumer hits memory-mapped bliss or stalls on a full download — and the Arrow backend, for all its elegance, cannot rescue a poorly packaged repo.
Memory mapping makes Arrow files look large enough to ignore, but the first access still travels to disk. When the operating system's page cache is empty, every indexed slice triggers real I/O; the cost scales with the speed of the underlying storage. On a slow SATA SSD, a spinning drive, or a network filesystem, the first call into a freshly loaded Dataset is noticeably slower than a fully RAM-resident table would be. Only repeated, localized access patterns benefit from the page cache warming up. If the workload scatters reads across the entire file, the cache helps less than intuition suggests.
Dataset is treated as effectively immutable. Calling .map() does not modify the mapped file in place; it produces a new memory-mapped Arrow file written to the local cache directory. This is the foundation of the library's smart-cache behavior: an identical .map() call — same function, same input hash — reuses the previously written file instead of recomputing ([A]). The trade-off is that a genuinely new transform always pays the cost of an Arrow IPC serialization round-trip, even when the underlying logic is trivial. Engineers iterating quickly on preprocessing should expect disk usage and cache directory size to grow in proportion to the number of distinct transforms they run.
Mapped pages are not reserved RAM. They share the page cache with every other process on the machine, which means heavy concurrent reads can evict pages that the dataset was relying on. A second process that allocates a large array, or a competing mapped dataset, can drive cache churn that reintroduces latency into subsequent reads. On memory-constrained machines or shared hosts, this is a practical ceiling on how many large Dataset objects can stay "hot" at once.
Choose Dataset when random access, slicing, and .map() caching matter and the dataset fits on disk; choose IterableDataset (via load_dataset(..., streaming=True)) when it does not, or when the consumer needs to start processing before a full download finishes ([B]).