Orchestration#

Run experiments over parameter ranges, persisting their metrics.

The user writes an experiment as a single run, sweeps it with ordinary Python loops, and reads the results back per metric as a tidy MetricTable, which holds plain Python lists and converts to whichever dataframe library the reader prefers.

Each run is a job, identified by its parameters and by the code it executes (see krum.orchestration.hashing), and owning a folder under the orchestrator’s root (see krum.orchestration.storage). A job is re-executed only when it has no recorded result, when its last attempt failed, when the environment it was recorded in no longer matches, or when one of the functions it called has since changed; otherwise its stored output stands. Orchestrator.plan reports what a sweep would do, and why, without running any of it.

Execution is synchronous and fail-fast: the first failing job stops the sweep, and the remaining jobs stay queued. One process per job, and re-running a job whose dependencies changed, are the next two steps in notes/adr-2026-10-02-orchestrator-v2-a-design.md.

Example:

from krum.orchestration import Metric, Orchestrator
from krum.primitives.aggregators.average import Average
from krum.primitives.aggregators.krum import Krum
from krum.primitives.attacks.alie import ALIEAttack
from krum.primitives.attacks.sign_flip import SignFlipAttack

def my_experiment(n, f, aggregator, attack, seed):
    simulation = ...  # e.g. a krum.simulations class
    loss = Metric("loss", dtype=float)
    for step in range(100):
        simulation.step()
        if step % 10 == 0:
            test_loss, _accuracy = simulation.evaluate()
            loss.push(step, test_loss)

with Orchestrator("byzantine_study") as orch:
    for n, f in [(10, 2), (20, 3)]:
        for aggregator in [Average, Krum]:
            for attack in [ALIEAttack, SignFlipAttack]:
                orch.run(my_experiment, n=n, f=f, aggregator=aggregator, attack=attack, seed=42)

loss = orch.get("loss")  # columns: step, value, n, f, aggregator, attack, seed, job_key
loss["value"]            # one column, as a list
frame = loss.to_pandas() # or: polars.DataFrame(loss.to_dict())
frame.groupby(["aggregator", "attack", "step"])["value"].mean()

Orchestration#

Modules#