Metrics
All pages
Docs · Start hereMarkdown

Quickstart

Sign in, log a run from Python and read it back from the shell. Five minutes, three packages: the command line for keys and reading, the collector for logging, and optionally DuckDB for analysis.

1. Get an account and a key

Start from the command line or from the app. Both end with an organization API key in SYVAIN_METRICS_API_KEY, which the collector reads.

From the command line, all of it, over SSH on a GPU box too:

npm install -g syvain-metrics
syvain-metrics auth login --device   # open the link on any device, sign up or sign in, confirm the code
syvain-metrics org create "My Lab"   # when you have no organization yet
export SYVAIN_METRICS_API_KEY=$(syvain-metrics api-key create my-laptop | jq -r .secret)

The browser is the only step outside the terminal, and it can be on another machine. The login also saves a personal key that the command line, the Python API client and DuckDB use without the export. Creating a key needs the admin role, which org create gives you.

From the app: sign up at https://metrics.041.io, create your organization, then create a key under Settings, API keys (/app/settings/api-keys) and export it:

export SYVAIN_METRICS_API_KEY=ak_...

2. Log a run

uv add syvain-metrics-collector
import math

from syvain_metrics_collector import Collector

collector = Collector()  # SYVAIN_METRICS_API_KEY
experiment = collector.experiment(
    slug="quickstart-001",
    meta={"model": "toy", "lr": 0.1},
)

with experiment.run():
    for step in range(1, 1_001):
        loss = 2.0 * math.exp(-step / 200) + 0.1
        experiment.metric("loss", loss, step=step, metadata={"split": "train"})
        if step % 100 == 0:
            experiment.metric("loss", loss + 0.05, step=step, metadata={"split": "valid"})
        if step % 500 == 0:
            experiment.annotation("checkpoint", {"path": f"ckpt/step-{step}.pt"}, step=step)

experiment.flush_or_raise()
print(experiment.url)

The slug is the experiment's name and identity: running the script again reopens the same experiment. run() marks it running, then done, or failed with the exception that ended the block. An annotation is a note at a step, with any JSON attached. Everything is sent in the background; flush_or_raise() waits for the rest and fails the job if a single point was dropped, rejected or not delivered, so a run that finished without an exception has all its data.

3. Read it back

syvain-metrics experiment get quickstart-001
syvain-metrics experiment catalog quickstart-001
syvain-metrics series query quickstart-001 --name loss --filter split=valid
syvain-metrics series render quickstart-001 --name loss -o loss.png

Each command prints JSON lines. catalog lists the metric names and the metadata values they were logged with; series query prints the steps, timestamps and values.

4. Look at it

Open https://metrics.041.io/app. The run is in the folder tree on the right; select it and the loss chart on top shows both splits.

5. Query it

uv add syvain-metrics-duckdb
import duckdb
import syvain_metrics_duckdb

con = duckdb.connect(config={"allow_unsigned_extensions": "true"})
syvain_metrics_duckdb.load(con)
con.sql("ATTACH '' AS m (TYPE syvain_metrics)")
print(con.sql("""
    SELECT metadata->>'split' AS split, min(value) AS best, max(step) AS steps
    FROM m.series
    WHERE experiment_slug = 'quickstart-001' AND metric_name = 'loss'
    GROUP BY ALL
"""))

Next