MultiKrum#

MultiKrum aggregation rule, multi-gradient averaging.

Reference:: Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. “Machine learning with adversaries: Byzantine tolerant gradient descent.” In Advances in Neural Information Processing Systems 30 (NIPS 2017).

class krum.primitives.aggregators.multikrum.MultiKrum[source]#

Bases: Aggregator

MultiKrum aggregation rule, multi-gradient averaging.

Scores every worker gradient by the sum of squared Euclidean distances to its \(n - f - 2\) closest peers, picks the \(m\) gradients with the smallest scores, and returns their mean. With \(m = 1\) it reduces to Krum.

classmethod aggregate(gradients: Sequence[Tensor] | Tensor, /, out: Tensor | None = None, *, n: int, f: int, m: int | None = None, **specialized: Any) → Tensor[source]#

Aggregate the gradients.

Parameters:

gradients – Sequence of 1-D tensors containing gradients from workers.
out – Optional pre-allocated tensor to write the result into.
n – Total number of workers.
f – Number of Byzantine workers to tolerate. Must satisfy 0 <= f <= (n - 3) // 2.
m – Number of selected gradients to average, with \(1 \le m \le n\). Values above \(n - 2f - 3\) leave the Multi-Krum resilience bound; \(m = n\) recovers Average. If None, defaults to \(n - 2f - 3\).
**specialized – Additional keyword arguments.

Returns:

Aggregated gradient of shape `` (d,)

Raises:

ValueError – If \(n\), \(f\), \(m\), or the gradients count is invalid.

static score(stacked: Tensor, *, n: int, f: int, num_peers: int | None = None, valid_mask: Tensor | None = None) → Tensor[source]#

Score every stacked gradient by its sum of squared distances to its num_peers closest peers.

After torch.sort() on each row, the self-distance is 0 (set via fill_diagonal_()), so column 0 is always the worker itself. Columns \(1\) through num_peers give the num_peers closest other workers. When num_peers is None it defaults to \(n - f - 2\), the Krum neighborhood from Blanchard et al.

The num_peers closest-peers sum approximates how surrounded a gradient is by the (presumed honest) majority; lower scores are better.

When valid_mask is provided, gradients with mask[i] = False are treated as infinitely far from every other gradient (so they cannot win the top-num_peers selection).

Parameters:

stacked – Tensor of shape \((n, d)\) containing the stacked worker gradients.
n – Total number of workers (rows of stacked).
f – Number of Byzantine workers to tolerate.
num_peers – Number of closest peers to consider. Defaults to \(n - f - 2\).
valid_mask – Optional boolean tensor of shape \((n,)`\); False entries are excluded from selection.

Returns:

Tensor of shape :math:` (n,)