Metrics#

Append-only metric recording, and the tidy frame read back from it.

A Metric is created inside a running job and finds that job from the context the orchestrator binds, so a user experiment never has to thread a handle through its own call stack:

def my_experiment(n, f, rounds):
    loss = Metric("loss", dtype=float)
    for step in range(rounds):
        loss.push(step, ...)

Rows are flushed as they are pushed, so a crashed job’s partial output stays readable in its staging directory. See notes/adr-2026-10-02-orchestrator-v2-a-design.md.

class krum.orchestration.metrics.InMemoryTable(columns: Mapping[str, list[Any]])[source]#

Bases: MetricTable

A metric table holding every row in memory, column by column.

This is what collect() builds.

Parameters:: columns – One list of values per column, all of the same length.
Raises:: ValueError – If the columns are not all the same length.

property columns: tuple[str, ...]#: The column names, in order.

rows() → Iterator[dict[str, Any]][source]#: Iterate over the rows, one dict each.

to_dict() → dict[str, list[Any]][source]#: The columns as plain lists, copied rather than read row by row.

class krum.orchestration.metrics.Metric(name: str, dtype: ~typing.Any = <class 'float'>)[source]#

Bases: object

A metric recorded by the running job.

Parameters:

name – The metric’s name, unrestricted; it is percent-encoded on disk.
dtype – The type recorded values are coerced to.

Raises:

NoActiveJob – If no job is being executed in this context.

property name: str#: The metric’s name.

push(step: Any, value: Any, skip_if_exists: bool = False) → bool[source]#

Record one value at one step.

Parameters:

step – The step the value was measured at.
value – The measured value.
skip_if_exists – Leave an already recorded step untouched.

Returns:

Whether a row was written.

class krum.orchestration.metrics.MetricTable[source]#

Bases: ABC

What a metric reads back as: its rows, with their parameters alongside.

Deliberately not a dataframe. A table holds no library of its own, so recording and reading metrics costs no dependency, and the analysis goes to whichever dataframe library the reader prefers:

table = orchestrator.get("loss")
table.columns          # ('step', 'value', 'n', 'f', 'job_key')
table["value"]         # [1.2, 0.7, ...]
len(table)             # the number of rows
for row in table:      # one dict per row
    ...

Only three members carry an implementation — columns, __len__ and rows() — and the rest are derived from them. Holding the rows a different way, streaming them off disk rather than keeping them in memory, is therefore a matter of those three; InMemoryTable is the in-memory one.

to_pandas() and to_csv() are derived too, so a converter the reader writes has exactly the access the shipped ones do:

import polars
polars.DataFrame(table.to_dict())

rows() streams. to_dict() and to_pandas() materialise every row, so prefer rows() for a table that may not fit in memory.

abstract property columns: tuple[str, ...]#: The column names, in order.

abstract rows() → Iterator[dict[str, Any]][source]#: Iterate over the rows, one dict each, in column order.

to_csv(path: str | PathLike[str]) → Path[source]#

Write the table as one CSV file, a row at a time.

Parameters:: path – Where to write it.
Returns:: The path written.

to_dict() → dict[str, list[Any]][source]#

The columns as plain lists.

This is what every converter is built on, the shipped ones included. It materialises every row.

to_pandas() → pandas.DataFrame[source]#

Convert to a pandas.DataFrame, materialising every row.

Returns:: A frame of the same columns, in the same order.
Raises:: ImportError – If pandas is not installed. It is not a requirement of this library; the error says what to do instead.

exception krum.orchestration.metrics.NoActiveJob[source]#

Bases: RuntimeError

Raised when a metric is created outside a running job.

A metric has nowhere to write unless the orchestrator is executing a job, so this is reported rather than silently buffered in memory and lost.

class krum.orchestration.metrics.Sink(path: Path, dtype: Any)[source]#

Bases: object

One metric’s open file within a running job.

close() → None[source]#: Close the underlying file.

property path: Path#: The file this sink writes to.

push(step: Any, value: Any, skip_if_exists: bool = False) → bool[source]#

Append one row, coercing the value to the declared dtype.

Parameters:

step – The step the value was measured at.
value – The measured value.
skip_if_exists – Leave an already recorded step untouched.

Returns:

Whether a row was written.

property rows: int#

How many rows have been written.

Recorded in the manifest when the job finishes, so that the number of rows a metric holds is known without reading the file back.

krum.orchestration.metrics.coercion(dtype: Any) → Callable[[Any], Any][source]#

Resolve a declared dtype to a function coercing one recorded value.

Accepts a Python type such as float or int, and also a torch dtype, which is not callable but reports whether it is floating point.

Parameters:: dtype – The declared metric dtype.
Returns:: A callable coercing a pushed value.

krum.orchestration.metrics.collect(store: JobStore, name: str, keys: Iterable[str] | None = None) → MetricTable[source]#

Gather one metric across completed jobs in a store.

Parameters:

store – The job store to scan.
name – The metric name.
keys – The jobs to read, in the order they should appear. Every completed job in the store is read when this is None, which mixes code versions: a store accumulates one folder per version of an experiment, so two of them can carry the same label and be drawn over each other. Passing the keys of one sweep keeps a reading to that sweep.

Returns:

One tidy table of `[step, value, *params, job_key]`, the parameters
being the sweep dimensions. Hand it to a dataframe library through
:meth:`MetricTable.to_pandas` or :meth:`MetricTable.to_dict` to group
or filter it.

Raises:

KeyError – If none of the jobs read recorded that metric.

krum.orchestration.metrics.concat(tables: Iterable[MetricTable]) → MetricTable[source]#

Stack tables, taking the union of their columns.

Jobs in one sweep need not share a parameter set, so a column absent from a table is filled with None for its rows rather than dropped. Nothing is coerced: an integer parameter stays an integer where it was recorded, which a dataframe would not promise once a column holds a gap.

Parameters:: tables – The tables to stack, in order.
Returns:: One table holding every row.

krum.orchestration.metrics.dtype_name(dtype: Any) → str[source]#: Render a declared dtype for the manifest, readably.

krum.orchestration.metrics.infer_value(field: str) → Any[source]#: Parse one field whose type was not recorded, trying int then float.

krum.orchestration.metrics.iter_metrics(store: JobStore) → Iterator[str][source]#: Iterate over every metric name recorded by a completed job.

krum.orchestration.metrics.parse_value(recorded: str) → Callable[[str], Any][source]#

Resolve a recorded dtype name to a function parsing one CSV field.

The dtype a metric was declared with is kept in the manifest, so a value is parsed back as what it was written as rather than guessed at. A torch dtype is recorded by name, and reads back as the Python type it stands for.

Parameters:: recorded – The dtype name from the manifest, such as float or torch.float32.
Returns:: A callable parsing one field.

krum.orchestration.metrics.read_metric(folder: JobFolder, name: str) → MetricTable | None[source]#

Read one metric from one completed job, with its parameters attached.

Parameters:

folder – The job folder to read.
name – The metric name.

Returns:

A table of `[step, value, *params, job_key]`, or None if this job did
not record that metric.

Raises:

ValueError – If the job’s parameters would shadow the table’s own columns.

krum.orchestration.metrics.reserved(params: Iterable[str]) → list[str][source]#: The parameter names that collide with a metric frame’s own columns.