Using the app

The app at https://metrics.041.io/app is for looking. The folder tree on the right chooses which runs are on the charts; the loss chart on top and the diagnostics below take the rest of the screen. Everything you set up can be saved as a view for the organization.

The folder tree

The tree shows every folder and experiment in the organization, with each run's status as a dot.

• Select runs with their checkboxes. Checking a folder selects the whole folder, including runs added to it later.
• Move a run or a folder by dragging it onto another folder, or right-click it and choose Move to.
• Rename a folder, or give a run a display name, from the right-click menu. A run's slug never changes.
• Create a folder from the toolbar above the tree, or a subfolder from a folder's menu.
• Open a run's details (configuration, annotations, status) from its menu.

The loss chart

The chart on top picks the loss by itself. It looks for the names people use for a loss (loss, train/loss, valid_loss, eval/loss and so on) and prefers the one most of the selected runs logged. Pick another metric from the chart's header to override it.

Diagnostics

Every other metric the selected runs logged is drawn below, when you scroll to it.

• By metric (the default): one chart per metric, with each run as a line.
• By experiment: one section per run, with a chart for each of its metrics.

Sections collapse, charts can be dragged into another order, and the filter box above them narrows the metrics by name.

Chart controls

Each chart has its own settings:

control	what it does
x axis	step, time since the run started, or log(step + 1)
x limits, y limits	fixed bounds; empty means fit the data
log y	logarithmic y axis
filter	keep only points whose metadata has the chosen values
split	one line per value of the chosen metadata keys

Settings in the toolbar apply to every chart that has not been changed on its own.

Views and links

Save the selection and every chart setting as a view from the views menu. A view belongs to the organization, so anyone in it can open it. The address bar holds the same state, so a link to what you are looking at works too.

Links can be written by hand, or by an agent that wants to show someone a result:

link	opens
/app?view=<viewId>	a saved view
/app?w=<state>	a workspace state: URL-encoded JSON, described below
/app?w=<state>&view=<viewId>	the state as that view; the views menu marks it changed when they differ
any of these with &org_id=	the same, after switching to that organization

The command line resolves folder paths and slugs to ids and prints the link:

syvain-metrics view link --folder /mamba/lr-sweep --state '{"defaults":{"logY":true}}'
syvain-metrics view link --experiment run-a --experiment run-b --state '{"groupBy":"experiment"}'
syvain-metrics view create "LR sweep" --folder /mamba/lr-sweep   # saved, for the whole team

The workspace state

w and a saved view hold the same JSON object. Every field is optional and a missing one takes its default, so this is a whole state: every run in one folder, grouped by experiment, loss on a log scale.

{
  "selection": { "folderIds": ["<folder id>"] },
  "groupBy": "experiment",
  "charts": [{ "chart": "primary", "settings": { "logY": true } }]
}

field	default	meaning
version	1	format version
selection.experimentIds	[]	experiment ids on the charts
selection.folderIds	[]	folder ids: every run in the folder and below, including later ones
primaryMetric	null	the metric of the chart on top; null finds the loss
groupBy	"metric"	"metric": a chart per metric; "experiment": a section per run
defaults	below	settings of every chart not set on its own
charts	[]	{"chart": <key>, "settings": {...}} for charts set on their own
metricOrder	[]	metric names in the order the diagnostics show; the rest follow A to Z
collapsed	[]	collapsed sections and charts, grouped by experiment
metricQuery	""	narrows the diagnostics by metric name: a substring, or /regex/

Chart settings, in defaults and in each entry of charts:

field	default	meaning
xMode	"step"	"step", "time" (ms since the run started) or "logstep" (log(step + 1))
xMin, xMax	null	x limits, in steps, or in ms with "time"; null fits the data
yMin, yMax	null	y limits; null fits the data
logY	false	logarithmic y axis
filter	[]	[{"key": "split", "values": ["valid"]}] keeps points whose metadata has one of the values
split	null	null: a line per metadata partition; ["split"]: a line per value of the listed keys

A chart set on its own starts from defaults; only the fields given change. Chart keys are primary for the chart on top, metric:<name> for a diagnostic grouped by metric, and experiment:<experimentId>:<name> for one grouped by experiment. Entries in collapsed are experiment:<experimentId> for a run's section and chart keys for single charts.

Folder and experiment ids come from syvain-metrics folder list and experiment list. A w that does not parse opens the empty workspace, and a view that no longer exists opens the empty workspace too.

Organization and account

The organization switcher at the top changes which organization's runs you see. The account menu leads to your account, the organization's members and its API keys. See organizations and keys.