All pages
Concepts
An organization holds folders and experiments. An experiment has a slug, a
status, configuration (meta), annotations and metrics. A metric is a named
series of numbers at steps, split into partitions by metadata. That is the whole
model.
Organizations
Everything belongs to one organization: folders, experiments, API keys and saved views. People are members with a role. An API key acts as its organization. See organizations and keys.
Folders
Folders nest, and are addressed by path, like /mamba/lr-sweep; / is the
root. Names are unique among siblings and contain no slash. An experiment is in
one folder or at the root. Moving or renaming a folder never changes the
experiments in it, only their path.
Every folder also has an id that never changes. A path follows every rename and
move; the id does not, so use it wherever a reference should keep working when
someone reorganizes: the collector's folder_id, DuckDB's folder_id column,
saved views and app links. The CLI accepts an id wherever it takes a folder
path, and syvain-metrics folder list prints both.
Use one folder per question. The app, the CLI and DuckDB all select whole folders, and a folder selected in the app includes runs added to it later.
Experiments
An experiment is one run. Its slug is its name and identity in the organization: opening an existing slug continues that experiment instead of creating another. The app can give it a display name for reading; the slug stays what the SDK, the CLI and DuckDB use.
| field | what it holds |
|---|---|
slug |
unique name, set when the run opens |
description |
one line of text |
meta |
the configuration: any JSON object |
status |
created, running, done or error |
error |
the message and exception type of a failed run |
| timestamps | created, started, done, last heartbeat, last event |
The collector's run() block sets running on entry and done or error on
exit. A process killed without a chance to report stays running; the time of
its last event tells you when it went quiet.
Metrics and metadata
A metric is a name and a series of points. Each point has a step, a timestamp
and a finite value. Metadata is a small str -> str map on each point that says
which slice of the quantity it measures:
experiment.metric("loss", 2.31, step=100, metadata={"split": "train"})
experiment.metric("loss", 2.47, step=100, metadata={"split": "valid"})Every distinct metadata map is its own partition of the metric, stored and read separately. The catalog lists each metric's name with the metadata keys and values it has, so filters and splits in the app, the CLI and DuckDB use values that exist. Keep metadata to bounded categories: splits, layers, ranks, datasets. See what to log and limits.
Annotations
An annotation is text and a JSON object at a point in time, optionally at a step: a saved checkpoint with its path, an evaluation result, a note about a restart. Use them for anything that is not a number.
Views
A view is a saved setup of the app: which experiments and folders are selected and how every chart is configured. Views belong to the organization, so everyone in it can open them.