Python API client

o41-metrics-api is a small synchronous Python client for the v2 API. Use it from scripts and agents to find and organize folders and experiments, to read one experiment's record, catalog, annotations and series, and to send any organization event the API accepts. For analysis across many runs use DuckDB; to record a training job use the collector.

Which tool

Install and connect

uv add o41-metrics-api

from o41_metrics_api import Metrics

with Metrics() as metrics:
    print(metrics.auth_status().organization)

The package needs CPython 3.14. Metrics(api_key=None, *, host=None, timeout=60.0) resolves credentials like the command line. The key comes from api_key, then O41_METRICS_API_KEY, then the login saved by o41-metrics auth login. A key goes only to the host it is paired with. An argument or environment key goes to host, then O41_METRICS_HOST, then https://metrics.041.io, never to the saved login's host. The saved login's key goes only to the host it was saved for; if host= or O41_METRICS_HOST names another one, the constructor raises MetricsAuthError. With no key anywhere the constructor raises MetricsAuthError. timeout is seconds per request. Metrics is a context manager that closes its HTTP session; close() does the same.

For example, with a saved login for https://custom.example.test, setting O41_METRICS_API_KEY alone sends that key to https://metrics.041.io.

auth_status() checks the key and returns its kind and organization.

Folders and experiments

A folder is its absolute path or its id; / is the root. An experiment is its id or its slug.

mamba = metrics.make_folder("/models/mamba")       # creates missing parents
mamba = metrics.folder("/models/mamba")            # MetricsNotFound when missing
mamba.folders()                                    # direct children
mamba.experiments()                                # direct members
mamba.experiments("run-*", recursive=True)         # slug glob over the subtree
metrics.experiments("lr-sweep-*")                  # slug glob over the organization

exp = metrics.experiment("mamba-run-001")
exp = exp.move("/archive")
mamba = mamba.rename("mamba-v2")
mamba = mamba.move("/")

Folder has id (None for the root), name, path, parent_id and parent. Experiment has id, slug, folder_id, folder_path and folder. Both are immutable values: rename() and move() return a new handle.

Folder lookups and folders() use the folder list, read on first use. experiment() asks for that one experiment by id or slug, and Folder.experiments() reads only that folder's experiments, or its subtree's with recursive=True. Only metrics.experiments(), like the root's recursive listing, reads every experiment, a thousand per request; a glob that starts with literal characters narrows it to that slug prefix. Results are kept until a change made through the client or metrics.refresh(). A folder() lookup that misses reads the folder list again once before raising MetricsNotFound; an experiment() lookup that misses raises it and asks again on its next call. Call metrics.refresh() to see what was created elsewhere.

Reading one experiment

info = exp.info()          # ExperimentInfo
catalog = exp.catalog()    # list[CatalogSeries]
notes = exp.annotations()  # list[Annotation], newest first

for series in exp.series("loss", where={"split": "valid"}):
    print(series.metadata, series.steps[-1], series.values[-1])

series() takes one name, several, or None for every series of the experiment. where keeps partitions whose metadata equals each given value, has keeps partitions that carry each given key, and x="timestamp" orders points by time instead of step. Results follow the order of the names you pass, or name order for None. Each streaming request reads up to 1,000 series; more names make more requests.

A Series has experiment_id, name, metadata and three aligned tuples, steps, timestamps_ms and values: the values at one index are one point, sorted ascending by the x axis. A step is None for points stored without one. series() returns only after the stream completed and its counts matched, so it never returns partial data.

An analysis script

Final and best validation loss for every run of a sweep, with its learning rate from meta, in plain Python:

from o41_metrics_api import Metrics

with Metrics() as metrics:
    rows = []
    for exp in metrics.experiments("lr-sweep-*"):
        info = exp.info()
        if info.status != "done":
            continue
        partitions = exp.series("loss", where={"split": "valid"})
        if len(partitions) != 1 or not partitions[0].values:
            continue  # no validation loss, or split by another key too
        valid = partitions[0]
        best = min(valid.values)
        rows.append((
            exp.slug,
            info.meta.get("lr"),
            valid.steps[-1],
            valid.values[-1],
            best,
            valid.steps[valid.values.index(best)],
        ))

rows.sort(key=lambda row: row[3])
print(f"{'run':<24} {'lr':>9} {'step':>7} {'final':>8} {'best':>8} {'at':>7}")
for slug, lr, step, final, best, best_step in rows:
    print(f"{slug:<24} {lr!s:>9} {step!s:>7} {final:8.4f} {best:8.4f} {best_step!s:>7}")

The script sends two requests per run, one record read and one series stream, plus one folder tree read for the whole script. Past a few dozen runs the same table is one DuckDB query, and cached.

Write session

with metrics.open_experiment(
    "backfill-001", description="Imported run", meta={"seed": 7}, folder="/imports"
) as run:
    run.start()
    run.metric("loss", 0.25, step=1, metadata={"split": "train"})
    run.annotate("imported from wandb", meta={"source": "run/abc123"})
    run.done()

open_experiment(slug, *, description="", meta=None, folder=None) creates or reopens the experiment and returns an ExperimentRun holding its experiment token. Reopening an existing slug replaces its description and meta. An experiment at the root is moved into folder; one already in another folder stays there.

Each call is one acknowledged request. The session refreshes its token five minutes before it expires. A token left idle past its 24 hour expiry cannot be refreshed; open the experiment again. Non-finite metric values are dropped with a warning, while finite points in the same batch are sent. Earlier releases raised ValueError for the whole batch. For real training jobs use the collector, which never blocks the loop.

send()

reply = metrics.send("syvain.v2.query.experiments", {"experimentIds": [exp.id]})

send(event, data) sends one organization event to /api/v2/events and returns the reply's data. It never retries and drops the tree snapshot, since the event may change the tree. Every event and its data are listed in API events.

Errors and retries

Every failure is a MetricsError with code, message, status, retryable and details. code is the API's error code, such as folder_already_exists, or one of the client's http_error, transport_error, invalid_response, not_found and auth_error. MetricsNotFound, also a LookupError, reports a reference that did not resolve. MetricsAuthError reports missing credentials or an invalid host.

Reads (the folder tree, info(), catalog(), annotations(), series(), auth_status()) and metric writes retry transport failures, HTTP 429 and 5xx for up to five minutes, waiting from 0.5 seconds, doubling, up to 10 seconds. A retry waits at least as long as the API's retryAfterMs or Retry-After asks, up to those 10 seconds. A retried metric keeps its original message id, which the API deduplicates, and only the failed points are resent. Every other write fails on the first error, because repeating it can have side effects: folder creation, moves, renames, open_experiment, lifecycle events (a replayed done moves done_at), annotations and send().

Related: DuckDB, the collector, HTTP API, command line.