Formats: turning a bucket into training examples

A CachedDataset does two things: it figures out what your training examples are, and it serves their bytes fast through the peer cache (local hit → peer hit → S3, with write-back). The first half — how a pile of objects in your bucket becomes a list of training examples — is the job of a Format.

The split of responsibility is deliberate:

Layer	Owns	Knows about
Format (RAMJET)	which object keys make up one example	your storage layout (where images live, where labels live, how they pair)
Your training code	decode bytes, parse labels into targets, model, loss	your task (classification / detection / …)

Format is framework-agnostic: it never fetches bytes, never builds target tensors, never touches the cache. You either use the built-in GenericFormat or write your own subclass — it's one method.

The contract

A Format is the only extension point of CachedDataset. Subclass ramjetio.Format and implement exactly one method:

class Format(ABC):
    def list_examples(self, client) -> List[Dict[str, Optional[str]]]:
        """Ordered list of training examples — a *list of dicts of strings*.

        Each example is a {role: object_key} group: the keys that together
        form one training example. A single-file example is a one-role group
        ({'data': key}); an image+label example is two roles
        ({'image': key, 'label': key}). A role whose value is None is served
        as empty bytes (use it for an optional sibling, e.g. a missing label).

        Order must be deterministic across processes so DDP ranks shard the
        same set of examples.
        """

That's the whole contract. list_examples returns the addresses (object keys); CachedDataset does everything else — it fetches the bytes for every key (local hit → peer hit → S3, with write-back, keyed by the object path), DDP-shards the examples, and hands the caller {role: bytes} for each group.

The client is the data-source client built from your dashboard Data Source (S3Client / LocalClient); it exposes list_files(prefix='', suffix=''). The split (train / val) is the Path Prefix field on the dashboard — it is folded into the client, so the keys you get back are already scoped to it.

The built-in: GenericFormat — each file is one example

ramjetio.CachedDataset(format=ramjetio.GenericFormat(), dataset_name="blobs")

GenericFormat lists every object in the data source and returns one example per file: [{'data': key}, ...]. cached[i] yields {'data': bytes}; your code decodes (cv2, decord, parquet, …). Use it when each file stands alone (one tensor, one record, one video). Option: suffix restricts the scan to one extension (e.g. '.mp4').

⚠️ GenericFormat does not pair files. If an example is split across several objects (an image plus a sidecar label, say), GenericFormat will treat the label as its own example too. Write a custom Format (below).

Writing your own format

Pairing files is a property of your storage layout, so the subclass lives in your training script — you name it, you define it. You only implement list_examples; the cache, DDP sharding and cold→warm behaviour come for free.

Example — pair each image with its sidecar label (images/<stem>.<ext> ↔ labels/<stem>.txt):

import ramjetio

class ImageLabelFormat(ramjetio.Format):
    def list_examples(self, client):
        all_keys = set(client.list_files())
        examples = []
        for key in sorted(all_keys):
            parent, _, name = key.rpartition('/')
            if parent.rpartition('/')[2] != 'images':
                continue                              # skip the label files
            stem = name.rpartition('.')[0] or name
            label = f"{parent[:-len('images')]}labels/{stem}.txt"
            examples.append({'image': key,
                             'label': label if label in all_keys else None})
        return examples

ds = ramjetio.CachedDataset(format=ImageLabelFormat(), dataset_name="my-data")

Then your Dataset reads ds[i]['image'] / ds[i]['label'] as bytes and turns them into a tensor and a target. A worked end-to-end version (YOLO label → class id, DDP) ships in the repo at ramjet-training/train.py.