Comparing agents

See Comparing agents.

ajax.compare

Compare agents with AdaStop: train seeds in rounds until the evidence decides.

AdaStop (Mathieu, Della Vecchia, Shilova, Centa de Medeiros, Kohler, Maillard and Preux, "AdaStop: adaptive statistical testing for sound comparisons of Deep RL agents", TMLR 2024, arXiv:2306.10882) is a permutation-based group sequential test with a step-down procedure. At each of up to K interims, every agent still in an undecided comparison adds n runs. A comparison stops once its two score distributions are shown to differ. The chance of declaring any difference where there is none (the family-wise error) stays at most alpha. A comparison still undecided after K interims is "equal": not shown to differ, which does not prove the agents equal.

docs/guide/comparing-agents.md lists the differences from the paper and from the authors' adastop package. :func:difference and :func:resolution say what a verdict leaves open: the gap's size, and the smallest gap the runs could detect.

:func:estimate measures one agent instead: it trains seeds until a confidence sequence pins down each chosen metric.

AdaStop

AdaStop(
    names,
    n=5,
    K=5,
    alpha=0.05,
    B=10000,
    comparisons=None,
    seed=0,
)

AdaStop's test (the paper's Algorithm 3), fed one interim at a time.

Parameters:

Name	Type	Description	Default
names	Sequence[str]	the agents.	required
n	int	the runs each agent adds per interim.	5
K	int	the most interims.	5
alpha	float	the family-wise error rate.	0.05
B	int	the Monte Carlo permutations.	10000
comparisons	Optional[Sequence[tuple[str, str]]]	the pairs of names to test; every pair when None.	None
seed	int	draws the permutations, independently of the scores.	0

active property

active

The agents in an undecided comparison: the next interim's runs.

update

update(scores)

Run interim k on the n new scores of every active agent; decisions[(a, b)] becomes "larger" (a's mean is larger), "smaller", or, after the last interim, "equal".

Comparison dataclass

Comparison(decisions, seeds, scores)

What :func:compare decided, and on which runs.

decisions[(a, b)] is "larger", "smaller" or "equal", a against b; seeds[a] and scores[a] are a's runs in training order.

plot

plot(ax=None)

Each agent's scores, one box per agent, the decisions as the title (needs matplotlib).

Estimate dataclass

Estimate(mean, interval, met, scores, seeds)

What :func:estimate measured, and on which runs.

Per metric: mean is the scores' sample mean; interval the (lower, upper) confidence interval when it stopped (adastop's "est" is its midpoint, not the mean); met whether its rule held then; scores one per seed, in training order. seeds are the runs.

final_return

final_return(curves)

Each seed's last evaluation; curves is (seeds, evaluations).

last_third

last_third(curves)

Each seed's mean over the last third of its evaluations (at least one).

auc

auc(curves)

Each seed's area under its evaluation curve, per evaluation: the mean.

compare

compare(
    agents,
    n_timesteps,
    eval_every,
    score=final_return,
    *,
    baseline=None,
    num_episode_test=10,
    seed=0,
    **adastop,
)

Train and score agents in AdaStop's interims until it decides.

At each interim, every agent still in an undecided comparison trains n new seeds in one train call, evaluated every eval_every steps; score turns the (seeds, evaluations) returns into one score per seed. At most n * K runs per agent; the paper advises n * K >= 30.

Parameters:

Name	Type	Description	Default
agents	Mapping[str, Callable[[], Any]]	a factory per name, building the agent to train.	required
n_timesteps	int	each run's training budget.	required
eval_every	int	steps between evaluations.	required
score	Callable[[ndarray], ndarray]	:func:final_return, :func:last_third, :func:auc or another map from evaluation curves to per-seed scores.	final_return
baseline	Optional[str]	compare it with each other agent; every pair when None.	None
num_episode_test	int	episodes per evaluation.	10
seed	int	the permutations' seed and the first run seed. Agents train disjoint seeds, so their runs are independent.	0
**adastop	Any	:class:AdaStop's n, K, alpha and B.	{}

difference

difference(a, b, confidence=0.95, B=10000, seed=0)

mean(a) - mean(b) and its percentile bootstrap interval: B resamples of each sample, with replacement.

resolution

resolution(
    a,
    b,
    n,
    K,
    alpha=0.05,
    power=0.8,
    reps=200,
    B=10000,
    seed=0,
)

The smallest gap between two means that AdaStop, run with n, K and alpha, detects with probability power, for scores spread as a and b: what an "equal" can miss.

A simulation, as the power study of the AdaStop paper (Section 5), from the observed spread rather than a known distribution: each of reps replicates draws n * K scores with replacement from each sample, centred on its mean, and adds the gap to b's. The same draws serve every gap, which a bisection narrows to 1/1000 of the samples' spread; samples without spread give 0. Shifting every score says little about a gap made of a few failing runs among scores at a task's maximum.

half_width

half_width(width)

A rule for :func:estimate: the interval is at most 2 * width wide.

threshold

threshold(value)

A rule for :func:estimate: the interval lies above or below value, so the mean is decided against it.

estimate

estimate(
    agent,
    n_timesteps,
    eval_every,
    metrics,
    *,
    delta=0.05,
    n=8,
    max_seeds=64,
    num_episode_test=10,
    seed=0,
)

Train one agent n seeds per train call until every metric's rule holds or max_seeds have trained.

Each metric's mean gets the hedged betting confidence sequence of Waudby-Smith and Ramdas ("Estimating means of bounded random variables by betting", JRSS-B, Section 4.4): adastop's Bounded_Confseq, from the compare extra. It holds at every seed count at once, so stopping on what the scores show keeps its level. The metrics split delta evenly (a union bound), so all intervals hold together with probability at least 1 - delta. n_timesteps, eval_every and num_episode_test are as in :func:compare.

Parameters:

Name	Type	Description	Default
agent	Callable[[], Any]	builds the agent to train.	required
metrics	Mapping[str, tuple[Callable, tuple[float, float], Callable]]	per name, (score, (low, high), rule). score maps train's evaluations, each (seeds, evaluations), to one value per seed, which low and high must bound for every possible run, not only those seen. rule is :func:half_width, :func:threshold or another test of (lower, upper).	required
delta	float	the chance that any interval misses its mean.	0.05
seed	int	the first run seed; the runs are seed, seed + 1, ....	0

Reproduction data

See Reproduction.

ajax.reproduction

The reference comparisons' published data (docs/reproduction/index.md).

Every run of the reproduction protocol, Ajax's and the reference implementations', as tables: runs has one row per seed and evaluation, batches each batch's speed and manifest, summary each comparison's verdict. Needs the reproduction extra (pip install "ajax-rl[reproduction]").

load

load(table, source, revision=None)

One table as a pandas DataFrame.

Parameters:

Name	Type	Description	Default
table	str	"runs", "batches" or "summary".	required
source	str	a Hugging Face dataset repository, or a local directory holding one (refcmp_publish.py dataset writes it).	required
revision	Optional[str]	the dataset's commit, branch or tag; the latest when None.	None