Working as an agent

Metrics is built to be driven by an agent. Log with the collector, read with the command line or DuckDB, and read these docs as markdown. Every read answers in JSON or SQL rows, every failure has an exit code, and nothing asks anything interactively once a key is in place. This page is the loop and the rules.

Credentials

An agent works with an organization API key in O41_METRICS_API_KEY, or with the login saved by o41-metrics auth login on that machine. The CLI, the Python API client, collector and DuckDB take the key from arguments, supported secrets, environment variables, then the saved login. Environment variables use only the O41_METRICS_* names. A key goes only to the host it is paired with: an environment key to O41_METRICS_HOST or production, never to the saved login's host, and the saved login's key only to the host it was saved for. An agent can also set everything up itself: auth login --device prints a link for a person to open, then org create and api-key create need no browser. See the quickstart and organizations and keys.

The loop

o41-metrics folder create /mamba/lr-sweep              # one folder per question
python train.py --slug mamba-lr3e-4-seed7 --folder …      # logs with the collector
o41-metrics experiment get mamba-lr3e-4-seed7          # status: running, done or error
o41-metrics experiment catalog mamba-lr3e-4-seed7      # what was logged, with metadata values
o41-metrics series query mamba-lr3e-4-seed7 --name loss --filter split=valid
o41-metrics series render mamba-lr3e-4-seed7 --name loss -o loss.png

1. Name runs so a glob finds them. The slug is the experiment's identity. Put the question, the variable and the seed in it: mamba-lr3e-4-seed7. Reopening a slug continues the same experiment.
2. Record the configuration as meta. Everything the run varied, as JSON, so a query can join results to hyperparameters instead of parsing slugs.
3. Put each question in a folder. Folders are paths, the CLI creates missing parents, and DuckDB and the app select whole folders.
4. Check status before reading results. experiment get says running, done or error, with the error message and the time of the last event. A run that stopped logging without done is still running.
5. Read the catalog before querying. It lists every metric name with the metadata keys and values it was logged with, so filters use values that exist.
6. Compare in SQL. For more than a couple of runs, attach DuckDB and query the folder; the local cache makes the second query free. See DuckDB.

Reading what happened

When a person wants to look for themselves, give them a link rather than a description of where to click. o41-metrics view link --folder /mamba/lr-sweep prints an app link with those runs on the charts; --state sets the grouping, axes and filters, and view create saves it for the team. See views and links.

Rules that keep results honest

• End every job with flush_or_raise(). It fails the job when any point was dropped, rejected or not delivered, so a run that finished without an exception has all its data.
• One name per quantity. loss with metadata={"split": "valid"}, not valid_loss. See what to log.
• Compare like with like. Filter by the same metadata values in every run, align on step, and check that the runs reached the same step before comparing final values.
• Don't trust a curve from a failed run. Check status first.
• Leave notes. An annotation records why something happened, next to the step it happened at: a restart, a changed data mix, a bug found.

When something is wrong

Report it where you found it: o41-metrics feedback --bug "<what you ran, what happened, what you expected>". It needs no login and attaches only the CLI's version and the system, so say what the team cannot see: the command, the error, and what you were trying to do. Ideas go in without --bug. See feedback.

Reading these docs

Run o41-metrics docs search "<question>" to find the page that answers a question before reading pages whole. Every page is plain markdown. Fetch any URL on this site with curl, or ask for Accept: text/markdown, and the answer is markdown:

curl https://metrics.041.io/docs/cli
curl https://metrics.041.io/llms.txt        # every page, one line each
curl https://metrics.041.io/llms-full.txt   # every page in one file