Skip to content

Ajax vs reference DreamerV3, CartPole-v1

Root: <scratch>/refrun/cmpR; seeds [0, 1, 2]; checkpoints (rows) [2000, 4000, 6000, 8000, 10000, 12000, 14000, 16000, 18000, 20000]; S1 window [16000, 18000, 20000]; S2 at 20000.

Checks

PASS  ref_s0: exit code 0 (ref_s0.DONE)
PASS  ref_s1: exit code 0 (ref_s1.DONE)
PASS  ref_s2: exit code 0 (ref_s2.DONE)
PASS  ajax: exit code 0 (ajax.DONE)
PASS  ref s0: config.yaml has the protocol values
PASS  ref s0: reference code imported from the 29eb964 checkout (['<reference-checkout>/dreamerv3/agent.py', '<reference-checkout>/embodied/__init__.py'])
PASS  ref s0: run_args.json has the protocol values
PASS  ref s0: every evaluation has 10 episodes
PASS  ref s0: episodes length == score, 1 <= length <= 500 (194 episodes)
PASS  ref s0: terminal False only for 500-step episodes
PASS  ref s0: per worker the episodes tile rows 0..row_last with L + 1 rows each
PASS  ref s0: run finished and the flush step ran (timing.json t_end present; flush_episodes 0 == episodes with row_last 1250: 0)
PASS  ref s1: config.yaml has the protocol values
PASS  ref s1: reference code imported from the 29eb964 checkout (['<reference-checkout>/dreamerv3/agent.py', '<reference-checkout>/embodied/__init__.py'])
PASS  ref s1: run_args.json has the protocol values
PASS  ref s1: every evaluation has 10 episodes
PASS  ref s1: episodes length == score, 1 <= length <= 500 (182 episodes)
PASS  ref s1: terminal False only for 500-step episodes
PASS  ref s1: per worker the episodes tile rows 0..row_last with L + 1 rows each
PASS  ref s1: run finished and the flush step ran (timing.json t_end present; flush_episodes 0 == episodes with row_last 1250: 0)
PASS  ref s2: config.yaml has the protocol values
PASS  ref s2: reference code imported from the 29eb964 checkout (['<reference-checkout>/dreamerv3/agent.py', '<reference-checkout>/embodied/__init__.py'])
PASS  ref s2: run_args.json has the protocol values
PASS  ref s2: every evaluation has 10 episodes
PASS  ref s2: episodes length == score, 1 <= length <= 500 (212 episodes)
PASS  ref s2: terminal False only for 500-step episodes
PASS  ref s2: per worker the episodes tile rows 0..row_last with L + 1 rows each
PASS  ref s2: run finished and the flush step ran (timing.json t_end present; flush_episodes 0 == episodes with row_last 1250: 0)
PASS  ajax: config.json has the protocol values
PASS  10 records per seed per side at rows 2000 k
PASS  updates = 1 + floor((N - 1040) / 2) at every N on both sides (9481 at 20000)
PASS  9481 updates at 20000 rows on both sides
PASS  ajax s0: logged S2 95.363 == final buffer mean 95.363 (tol 1e-4)
PASS  ajax s1: logged S2 102.994 == final buffer mean 102.994 (tol 1e-4)
PASS  ajax s2: logged S2 98.059 == final buffer mean 98.059 (tol 1e-4)
PASS  ajax s0: final state rows 20000, n_updates 9481
PASS  ajax s1: final state rows 20000, n_updates 9481
PASS  ajax s2: final state rows 20000, n_updates 9481

Pre-registered rule

  • A = mean over Ajax seeds of S1 = 233.2
  • reference S1 range [min, max] = [241.4, 386.1]
  • LEARNS = False (A >= min reference S1)
  • WITHIN_RANGE = False (min <= A <= max)

Per-seed S1 and S2

seed ref S1 ajax S1 ref S2 ajax S2 (logged) ajax S2 (audit)
0 386.1 255.4 98.0 95.4 95.4
1 246.4 135.0 114.8 103.0 103.0
2 241.4 309.2 106.9 98.1 98.1
mean 291.3 233.2 106.6 98.8

Known differences between the two sides (read before interpreting)

  • D1 / U1, asynchrony (reference only). The reference acts with policy parameters that lag the trained ones by about 2 vector steps (about 16 of 9481 updates; measured on the real reference in refrun/audit_loop), its first training batch is prefetched before the gate opens (11 distinct items of 16 in the audit), and its write-back and metrics are one call late. Ajax acts with the latest parameters. Evaluation uses the latest parameters on both sides.
  • D2 / U2, shared replay-sampler stream (reference only). Stock make_replay passes no seed, so Uniform(seed=0) samples the replay for every reference seed: reference seeds [0, 1, 2] share one index stream (the replay contents differ). The reference S1 [min, max] range therefore comes from runs that are not fully independent. Ajax seeds its sampler from the run seed.
  • Report prefetch stream (stock, D1 / D2). With run.eval_every = -1 the stock report stream (train.py:76-77) is created but never read: when the first replay items appear (about row 1040) its prefetch thread draws 2 batches of 16 (32 samples, popping up to 32 of the earliest online-queue items, plus Uniform-RNG and agent.rng draws) and then blocks for good. A stock run (eval_every 180 s) would draw 16 more every 180 s of wall time. Negligible: training draws 128 samples per vector step against 16 new items.
  • NaN in training-metric windows. The reference Agg skips NaN values; Ajax's accumulator propagates them, so one NaN makes the Ajax window mean NaN. A nan Ajax window against a finite reference window means 'a NaN occurred in that Ajax window', not a divergence of the finite values. Non-finite Ajax window means here: none.
  • Reference metric windows hold updates n0-1 .. n1-1 (one-call lag); Ajax per-episode evaluation returns are not logged (only the mean).

Evaluation curve (mean return of the evaluation episodes)

rows ref s0 ref s1 ref s2 ref mean ajax s0 ajax s1 ajax s2 ajax mean
2000 17.4 25.2 33.4 25.3 21.3 17.8 16.3 18.5
4000 144.8 141.7 196.1 160.9 79.6 140.7 137.1 119.1
6000 202.3 132.6 143.5 159.5 223.5 380.6 338.6 314.2
8000 113.5 88.4 170.3 124.1 87.2 167.9 291.1 182.1
10000 115.3 190.6 90.2 132.0 107.7 433.7 122.7 221.4
12000 192.3 130.6 158.9 160.6 158.5 300.3 65.7 174.8
14000 222.4 136.9 345.6 235.0 153.2 161.9 172.5 162.5
16000 253.5 151.0 180.5 195.0 351.5 151.2 264.9 255.9
18000 500.0 122.2 143.8 255.3 169.5 144.8 455.0 256.4
20000 404.8 466.1 399.8 423.6 245.3 109.0 207.7 187.3

S2 curve (training-episode rolling mean, Ajax definition)

rows ref s0 ref s1 ref s2 ref mean ajax s0 ajax s1 ajax s2 ajax mean
2000 23.1 23.4 21.2 22.5 19.6 21.6 20.5 20.6
4000 25.5 26.9 23.6 25.4 25.0 25.1 25.9 25.4
6000 27.9 33.7 29.3 30.3 29.2 26.7 30.6 28.9
8000 45.7 38.1 42.3 42.1 39.5 28.4 34.3 34.1
10000 57.4 56.9 51.6 55.3 54.9 60.7 58.8 58.1
12000 61.2 69.5 66.6 65.8 66.4 66.8 65.2 66.1
14000 67.4 73.4 66.6 69.2 79.8 73.1 65.2 72.7
16000 73.1 88.1 81.0 80.7 81.0 91.5 92.7 88.4
18000 84.5 100.4 95.6 93.5 89.1 101.9 98.1 96.3
20000 98.0 114.8 106.9 106.6 95.4 103.0 98.1 98.8

Updates per checkpoint

rows expected ref s0 ref s1 ref s2 ajax s0 ajax s1 ajax s2
2000 481 481 481 481 481 481 481
4000 1481 1481 1481 1481 1481 1481 1481
6000 2481 2481 2481 2481 2481 2481 2481
8000 3481 3481 3481 3481 3481 3481 3481
10000 4481 4481 4481 4481 4481 4481 4481
12000 5481 5481 5481 5481 5481 5481 5481
14000 6481 6481 6481 6481 6481 6481 6481
16000 7481 7481 7481 7481 7481 7481 7481
18000 8481 8481 8481 8481 8481 8481 8481
20000 9481 9481 9481 9481 9481 9481 9481

Training metrics, window means (mean over seeds [min, max])

Reference windows hold updates n0-1 .. n1-1 (JAXAgent returns the previous call's metrics); Ajax windows n0 .. n1. Loss names mapped: vector_loss->rec_loss, reward_loss->rew_loss, cont_loss->con_loss, replay_critic_loss->repval_loss.

window ending at 2000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.4052 [0.3851, 0.4279] 0.4263 [0.3865, 0.4514] 1.052
rew_loss 5.2850 [5.2790, 5.2944] 5.2867 [5.2816, 5.2906] 1.000
con_loss 0.3908 [0.3656, 0.4087] 0.3454 [0.3227, 0.3630] 0.884
dyn_loss 2.5956 [2.5333, 2.7120] 2.3860 [2.1729, 2.6847] 0.919
rep_loss 2.5956 [2.5333, 2.7120] 2.3860 [2.1729, 2.6847] 0.919
actor_loss -0.0076 [-0.0090, -0.0068] -0.0091 [-0.0101, -0.0084] 1.195
critic_loss 3.9150 [3.7395, 4.0730] 3.9261 [3.7647, 4.0289] 1.003
repval_loss 10.2157 [10.1804, 10.2589] 10.1743 [10.1446, 10.1998] 0.996
ent/action/mean 0.6931 [0.6930, 0.6931] 0.6931 [0.6930, 0.6931] 1.000
rand/action/mean 0.9999 [0.9998, 1.0000] 0.9999 [0.9998, 1.0000] 1.000
ret/mean 0.0700 [0.0685, 0.0721] 0.0751 [0.0726, 0.0787] 1.072
ret/std 0.1351 [0.1330, 0.1377] 0.1365 [0.1348, 0.1380] 1.011
ret_normed/std 0.1351 [0.1330, 0.1377] 0.1365 [0.1348, 0.1380] 1.011
val/mean 0.0908 [0.0866, 0.0973] 0.0982 [0.0943, 0.1053] 1.082
adv/std 0.1603 [0.1585, 0.1624] 0.1615 [0.1600, 0.1626] 1.007
replay_ret/mean 7.9601 [7.5991, 8.2409] 7.9774 [7.8957, 8.1058] 1.002
weight/mean 0.3624 [0.3470, 0.3762] 0.3639 [0.3505, 0.3720] 1.004
rew/mean 0.0618 [0.0616, 0.0621] 0.0629 [0.0628, 0.0630] 1.018
prior_ent/mean 43.3120 [43.1853, 43.5035] 43.3132 [43.2701, 43.3656] 1.000
post_ent/mean 41.9819 [41.6774, 42.3989] 42.3209 [42.1193, 42.4364] 1.008
td_error 0.1293 [0.1285, 0.1306] 0.1315 [0.1307, 0.1324] 1.017
data_rew/mean 0.9573 [0.9536, 0.9602] 0.9562 [0.9556, 0.9574] 0.999
pred_rew/mean 0.0618 [0.0616, 0.0621] 0.0629 [0.0628, 0.0630] 1.018
activation/embed 0.4287 [0.4196, 0.4388] 0.3867 [0.3754, 0.3931] 0.902
opt_loss 15.9082 [15.6850, 16.1787] 15.6524 [15.2738, 16.0916] 0.984
opt_grad_norm 148.1683 [140.3558, 160.4136] 151.4908 [130.8986, 191.3467] 1.022
opt_update_norm 0.0070 [0.0069, 0.0070] 0.0070 [0.0069, 0.0071] 1.004
opt_param_norm 59.8880 [59.8753, 59.9078] 59.8506 [59.7762, 59.9057] 0.999
opt_grad_steps 240.5000 [240.5000, 240.5000] 241.0000 [241.0000, 241.0000] 1.002
retnorm_scale_derived 1.0000 [1.0000, 1.0000] 1.0000 [1.0000, 1.0000] 1.000
retnorm_scale_exact 1.0000 [1.0000, 1.0000] nan [nan, nan] nan

window ending at 4000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0634 [0.0575, 0.0667] 0.0665 [0.0617, 0.0744] 1.049
rew_loss 2.4989 [2.4825, 2.5198] 2.5104 [2.4896, 2.5285] 1.005
con_loss 0.1502 [0.1413, 0.1554] 0.1535 [0.1522, 0.1548] 1.022
dyn_loss 1.0111 [1.0106, 1.0118] 1.0090 [1.0071, 1.0107] 0.998
rep_loss 1.0111 [1.0106, 1.0118] 1.0090 [1.0071, 1.0107] 0.998
actor_loss 0.1669 [0.1608, 0.1753] 0.1637 [0.1622, 0.1660] 0.981
critic_loss 5.1856 [5.0536, 5.2972] 5.1011 [5.0429, 5.1378] 0.984
repval_loss 7.2307 [7.2091, 7.2731] 7.1795 [7.1172, 7.2350] 0.993
ent/action/mean 0.5135 [0.5056, 0.5178] 0.5087 [0.5033, 0.5177] 0.991
rand/action/mean 0.7408 [0.7294, 0.7470] 0.7339 [0.7261, 0.7469] 0.991
ret/mean 3.3319 [3.2891, 3.3857] 3.3100 [3.2288, 3.3763] 0.993
ret/std 0.8487 [0.7955, 0.8916] 0.8410 [0.8166, 0.8761] 0.991
ret_normed/std 0.3273 [0.3207, 0.3319] 0.3286 [0.3234, 0.3340] 1.004
val/mean 1.8324 [1.7883, 1.8666] 1.8408 [1.8104, 1.8985] 1.005
adv/std 0.3342 [0.3276, 0.3387] 0.3354 [0.3305, 0.3406] 1.004
replay_ret/mean 10.4500 [10.0243, 10.8171] 10.3713 [10.2800, 10.5255] 0.992
weight/mean 0.7315 [0.7136, 0.7444] 0.7276 [0.7259, 0.7303] 0.995
rew/mean 0.3523 [0.3473, 0.3566] 0.3483 [0.3435, 0.3540] 0.989
prior_ent/mean 43.5344 [43.4195, 43.6230] 43.5965 [43.3639, 43.7314] 1.001
post_ent/mean 42.5926 [42.3925, 42.7287] 42.6491 [42.3300, 42.8396] 1.001
td_error 1.5689 [1.5418, 1.5840] 1.5440 [1.4839, 1.5876] 0.984
data_rew/mean 0.9590 [0.9558, 0.9618] 0.9581 [0.9569, 0.9594] 0.999
pred_rew/mean 0.3523 [0.3473, 0.3566] 0.3483 [0.3435, 0.3540] 0.989
activation/embed 0.4307 [0.4203, 0.4379] 0.3910 [0.3847, 0.4007] 0.908
opt_loss 11.3465 [11.1974, 11.4452] 11.2589 [11.1517, 11.3266] 0.992
opt_grad_norm 8.5400 [8.0966, 9.0237] 8.2631 [7.8836, 8.5901] 0.968
opt_update_norm 0.0134 [0.0133, 0.0134] 0.0134 [0.0133, 0.0136] 1.006
opt_param_norm 60.5103 [60.4985, 60.5337] 60.4773 [60.4146, 60.5308] 0.999
opt_grad_steps 980.5000 [980.5000, 980.5000] 981.5000 [981.5000, 981.5000] 1.001
retnorm_scale_derived 2.5915 [2.4807, 2.6863] 2.5589 [2.5252, 2.6228] 0.987
retnorm_scale_exact 6.1737 [5.9708, 6.3542] nan [nan, nan] nan

window ending at 6000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0199 [0.0169, 0.0215] 0.0208 [0.0200, 0.0224] 1.045
rew_loss 0.8086 [0.8033, 0.8130] 0.8057 [0.8010, 0.8129] 0.996
con_loss 0.0788 [0.0735, 0.0824] 0.0835 [0.0811, 0.0856] 1.059
dyn_loss 1.0365 [1.0332, 1.0384] 1.0355 [1.0323, 1.0383] 0.999
rep_loss 1.0365 [1.0332, 1.0384] 1.0355 [1.0323, 1.0383] 0.999
actor_loss 0.1739 [0.1717, 0.1768] 0.1714 [0.1654, 0.1834] 0.986
critic_loss 4.0891 [4.0552, 4.1458] 4.0497 [3.9376, 4.1366] 0.990
repval_loss 6.0414 [6.0160, 6.0878] 5.9931 [5.9467, 6.0310] 0.992
ent/action/mean 0.3004 [0.2937, 0.3087] 0.3053 [0.2918, 0.3142] 1.016
rand/action/mean 0.4333 [0.4238, 0.4454] 0.4404 [0.4210, 0.4533] 1.016
ret/mean 13.5435 [13.3726, 13.6849] 13.5565 [13.3982, 13.6734] 1.001
ret/std 2.8645 [2.7987, 2.9120] 2.9316 [2.7742, 3.1512] 1.023
ret_normed/std 0.3720 [0.3685, 0.3777] 0.3716 [0.3608, 0.3815] 0.999
val/mean 8.6778 [8.5062, 8.7826] 8.7702 [8.6435, 8.9898] 1.011
adv/std 0.3718 [0.3680, 0.3776] 0.3714 [0.3608, 0.3810] 0.999
replay_ret/mean 20.4433 [19.7653, 21.2768] 20.1881 [20.0667, 20.2735] 0.988
weight/mean 0.8648 [0.8557, 0.8766] 0.8603 [0.8481, 0.8728] 0.995
rew/mean 0.9060 [0.9034, 0.9077] 0.9052 [0.9025, 0.9076] 0.999
prior_ent/mean 42.0547 [41.6753, 42.3235] 42.2405 [41.7080, 42.5379] 1.004
post_ent/mean 41.0193 [40.6158, 41.3073] 41.2558 [40.5802, 41.6362] 1.006
td_error 5.0738 [5.0173, 5.1070] 5.0426 [4.9717, 5.1524] 0.994
data_rew/mean 0.9709 [0.9685, 0.9738] 0.9702 [0.9697, 0.9709] 0.999
pred_rew/mean 0.9060 [0.9034, 0.9077] 0.9052 [0.9025, 0.9076] 0.999
activation/embed 0.4296 [0.4194, 0.4476] 0.3946 [0.3870, 0.4094] 0.919
opt_loss 8.1229 [8.0898, 8.1853] 8.0681 [7.9339, 8.1690] 0.993
opt_grad_norm 7.7420 [7.6290, 7.8142] 7.5853 [6.9103, 8.3682] 0.980
opt_update_norm 0.0081 [0.0077, 0.0083] 0.0080 [0.0078, 0.0083] 0.999
opt_param_norm 61.7567 [61.7447, 61.7782] 61.7215 [61.6424, 61.7783] 0.999
opt_grad_steps 1980.5000 [1980.5000, 1980.5000] 1981.5000 [1981.5000, 1981.5000] 1.001
retnorm_scale_derived 7.7000 [7.5955, 7.7946] 7.8831 [7.6883, 8.2595] 1.024
retnorm_scale_exact 8.4072 [8.3665, 8.4748] nan [nan, nan] nan

window ending at 8000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0152 [0.0143, 0.0168] 0.0167 [0.0156, 0.0184] 1.100
rew_loss 0.6922 [0.6910, 0.6928] 0.6917 [0.6906, 0.6929] 0.999
con_loss 0.0555 [0.0533, 0.0568] 0.0572 [0.0545, 0.0598] 1.032
dyn_loss 1.0576 [1.0553, 1.0616] 1.0594 [1.0463, 1.0660] 1.002
rep_loss 1.0576 [1.0553, 1.0616] 1.0594 [1.0463, 1.0660] 1.002
actor_loss 0.1542 [0.1525, 0.1566] 0.1543 [0.1402, 0.1636] 1.000
critic_loss 3.2846 [3.1826, 3.3953] 3.1556 [3.1256, 3.1904] 0.961
repval_loss 5.4217 [5.2938, 5.4997] 5.2699 [5.2644, 5.2800] 0.972
ent/action/mean 0.2649 [0.2605, 0.2690] 0.2699 [0.2443, 0.2833] 1.019
rand/action/mean 0.3822 [0.3759, 0.3881] 0.3893 [0.3524, 0.4087] 1.019
ret/mean 21.5226 [20.6922, 22.0487] 21.9765 [21.7879, 22.1600] 1.021
ret/std 3.4484 [3.4081, 3.4947] 3.6046 [3.4930, 3.7934] 1.045
ret_normed/std 0.4049 [0.3998, 0.4114] 0.4157 [0.4048, 0.4305] 1.027
val/mean 16.2671 [15.5464, 16.7231] 16.7849 [16.6586, 16.9915] 1.032
adv/std 0.3925 [0.3754, 0.4112] 0.3987 [0.3895, 0.4090] 1.016
replay_ret/mean 29.1729 [27.9417, 29.9898] 29.2550 [29.1356, 29.4822] 1.003
weight/mean 0.9055 [0.8967, 0.9117] 0.9030 [0.8970, 0.9090] 0.997
rew/mean 0.9716 [0.9710, 0.9723] 0.9716 [0.9712, 0.9723] 1.000
prior_ent/mean 40.0760 [39.5168, 40.4287] 40.1344 [39.2294, 40.7937] 1.001
post_ent/mean 38.7531 [38.2004, 39.1636] 38.9113 [37.8989, 39.5619] 1.004
td_error 5.5894 [5.5655, 5.6042] 5.5894 [5.5410, 5.6429] 1.000
data_rew/mean 0.9797 [0.9779, 0.9815] 0.9790 [0.9785, 0.9794] 0.999
pred_rew/mean 0.9716 [0.9710, 0.9723] 0.9716 [0.9712, 0.9723] 1.000
activation/embed 0.4199 [0.4071, 0.4368] 0.3917 [0.3849, 0.3996] 0.933
opt_loss 6.9915 [6.8485, 7.1238] 6.8217 [6.8076, 6.8467] 0.976
opt_grad_norm 8.2028 [7.5048, 9.1159] 7.3564 [6.8366, 7.6783] 0.897
opt_update_norm 0.0109 [0.0095, 0.0124] 0.0095 [0.0085, 0.0102] 0.877
opt_param_norm 62.4969 [62.4740, 62.5120] 62.4354 [62.3512, 62.4981] 0.999
opt_grad_steps 2980.5000 [2980.5000, 2980.5000] 2981.5000 [2981.5000, 2981.5000] 1.000
retnorm_scale_derived 8.5164 [8.4942, 8.5304] 8.6691 [8.5662, 8.8116] 1.018
retnorm_scale_exact 8.3140 [8.2107, 8.4424] nan [nan, nan] nan

window ending at 10000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0229 [0.0201, 0.0273] 0.0265 [0.0234, 0.0298] 1.159
rew_loss 0.6712 [0.6702, 0.6719] 0.6720 [0.6714, 0.6731] 1.001
con_loss 0.0546 [0.0502, 0.0590] 0.0546 [0.0514, 0.0567] 1.000
dyn_loss 1.0845 [1.0753, 1.0946] 1.0815 [1.0697, 1.0994] 0.997
rep_loss 1.0845 [1.0753, 1.0946] 1.0815 [1.0697, 1.0994] 0.997
actor_loss 0.1347 [0.1052, 0.1831] 0.1290 [0.1085, 0.1457] 0.958
critic_loss 3.3112 [3.1539, 3.4883] 3.7013 [3.4078, 4.0154] 1.118
repval_loss 5.4033 [5.2477, 5.4933] 5.6943 [5.4822, 5.8800] 1.054
ent/action/mean 0.2793 [0.2549, 0.3247] 0.2763 [0.2313, 0.3025] 0.989
rand/action/mean 0.4029 [0.3677, 0.4684] 0.3986 [0.3336, 0.4364] 0.989
ret/mean 25.1384 [24.5147, 25.6124] 24.2758 [22.3795, 26.3545] 0.966
ret/std 3.9535 [3.6868, 4.2350] 3.7844 [3.5984, 4.0427] 0.957
ret_normed/std 0.3972 [0.3742, 0.4208] 0.3903 [0.3781, 0.4055] 0.983
val/mean 20.0203 [19.5719, 20.4610] 19.3013 [17.6098, 21.1345] 0.964
adv/std 0.3637 [0.3393, 0.3989] 0.3714 [0.3571, 0.3791] 1.021
replay_ret/mean 33.2857 [32.6846, 33.9585] 32.8044 [31.4267, 34.9183] 0.986
weight/mean 0.9080 [0.9022, 0.9158] 0.9059 [0.9014, 0.9115] 0.998
rew/mean 0.9863 [0.9854, 0.9871] 0.9857 [0.9852, 0.9867] 0.999
prior_ent/mean 36.5890 [35.9630, 37.4125] 36.6090 [35.4536, 37.5779] 1.001
post_ent/mean 34.8418 [34.1144, 35.7122] 34.8249 [33.4266, 35.9484] 1.000
td_error 5.4696 [5.2925, 5.6156] 5.3613 [5.1590, 5.5261] 0.980
data_rew/mean 0.9835 [0.9817, 0.9856] 0.9837 [0.9825, 0.9846] 1.000
pred_rew/mean 0.9863 [0.9854, 0.9871] 0.9857 [0.9852, 0.9867] 0.999
activation/embed 0.4037 [0.3864, 0.4141] 0.3787 [0.3582, 0.3926] 0.938
opt_loss 7.0085 [6.8491, 7.2056] 7.4813 [7.1164, 7.8675] 1.067
opt_grad_norm 13.5940 [11.8704, 14.8143] 11.9226 [11.1332, 13.1898] 0.877
opt_update_norm 0.0124 [0.0116, 0.0131] 0.0142 [0.0134, 0.0154] 1.141
opt_param_norm 63.3664 [63.3330, 63.3869] 63.3179 [63.1868, 63.4156] 0.999
opt_grad_steps 3980.5000 [3980.5000, 3980.5000] 3981.5000 [3981.5000, 3981.5000] 1.000
retnorm_scale_derived 10.0034 [8.7622, 11.3176] 9.6929 [9.2910, 9.9689] 0.969
retnorm_scale_exact 12.3124 [9.2841, 14.9662] nan [nan, nan] nan

window ending at 12000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0244 [0.0170, 0.0315] 0.0272 [0.0188, 0.0349] 1.113
rew_loss 0.6636 [0.6628, 0.6647] 0.6638 [0.6631, 0.6645] 1.000
con_loss 0.0522 [0.0496, 0.0568] 0.0523 [0.0483, 0.0556] 1.004
dyn_loss 1.0864 [1.0671, 1.1081] 1.0738 [1.0618, 1.0799] 0.988
rep_loss 1.0864 [1.0671, 1.1081] 1.0738 [1.0618, 1.0799] 0.988
actor_loss 0.0979 [0.0780, 0.1207] 0.1172 [0.1076, 0.1263] 1.198
critic_loss 3.1750 [2.6354, 3.7897] 3.2253 [2.5345, 4.2273] 1.016
repval_loss 5.0965 [4.7109, 5.5176] 5.0243 [4.3801, 5.8478] 0.986
ent/action/mean 0.2777 [0.2609, 0.3062] 0.2571 [0.1895, 0.3057] 0.926
rand/action/mean 0.4007 [0.3764, 0.4417] 0.3708 [0.2734, 0.4410] 0.926
ret/mean 30.9380 [26.9331, 34.9688] 30.6595 [23.0379, 38.3435] 0.991
ret/std 5.0400 [4.3669, 5.5656] 4.2511 [4.1252, 4.3786] 0.843
ret_normed/std 0.3581 [0.3150, 0.4079] 0.4083 [0.3610, 0.4752] 1.140
val/mean 25.7464 [21.7653, 29.9210] 25.4119 [17.8457, 32.6376] 0.987
adv/std 0.3049 [0.2342, 0.3406] 0.3548 [0.3374, 0.3737] 1.164
replay_ret/mean 38.7323 [36.8057, 41.3513] 38.5552 [32.8695, 44.8682] 0.995
weight/mean 0.9201 [0.9158, 0.9286] 0.9197 [0.9149, 0.9270] 1.000
rew/mean 0.9933 [0.9931, 0.9935] 0.9931 [0.9927, 0.9934] 1.000
prior_ent/mean 35.3246 [34.3087, 36.2890] 34.7137 [32.2510, 36.2175] 0.983
post_ent/mean 33.4355 [31.8327, 34.8118] 32.8054 [29.6459, 34.6547] 0.981
td_error 5.7407 [5.6781, 5.7982] 5.6471 [5.4493, 5.8348] 0.984
data_rew/mean 0.9858 [0.9847, 0.9877] 0.9856 [0.9846, 0.9862] 1.000
pred_rew/mean 0.9933 [0.9931, 0.9935] 0.9931 [0.9927, 0.9934] 1.000
activation/embed 0.3982 [0.3814, 0.4120] 0.3709 [0.3564, 0.3859] 0.931
opt_loss 6.7370 [6.0472, 7.5038] 6.7744 [5.8750, 8.0396] 1.006
opt_grad_norm 13.6046 [10.9870, 16.4320] 12.7893 [9.7042, 15.6921] 0.940
opt_update_norm 0.0105 [0.0098, 0.0113] 0.0104 [0.0093, 0.0116] 0.994
opt_param_norm 64.2743 [64.2112, 64.3360] 64.2119 [64.0337, 64.4228] 0.999
opt_grad_steps 4980.5000 [4980.5000, 4980.5000] 4981.5000 [4981.5000, 4981.5000] 1.000
retnorm_scale_derived 14.1795 [12.4272, 16.4680] 10.5446 [8.9427, 11.4266] 0.744
retnorm_scale_exact 17.6708 [13.4947, 25.6072] nan [nan, nan] nan

window ending at 14000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0205 [0.0172, 0.0237] 0.0237 [0.0178, 0.0310] 1.154
rew_loss 0.6609 [0.6600, 0.6620] 0.6610 [0.6604, 0.6618] 1.000
con_loss 0.0491 [0.0447, 0.0517] 0.0511 [0.0496, 0.0531] 1.041
dyn_loss 1.0887 [1.0852, 1.0923] 1.0761 [1.0632, 1.0934] 0.988
rep_loss 1.0887 [1.0852, 1.0923] 1.0761 [1.0632, 1.0934] 0.988
actor_loss 0.0477 [0.0303, 0.0667] 0.0963 [0.0670, 0.1350] 2.020
critic_loss 2.4061 [2.2681, 2.6215] 2.5873 [2.1323, 3.0222] 1.075
repval_loss 4.3971 [4.2212, 4.6315] 4.6848 [4.4561, 4.9088] 1.065
ent/action/mean 0.2116 [0.1988, 0.2190] 0.2261 [0.1609, 0.3132] 1.069
rand/action/mean 0.3053 [0.2869, 0.3159] 0.3262 [0.2321, 0.4519] 1.069
ret/mean 41.8206 [38.7728, 45.1628] 40.2884 [29.9165, 51.2458] 0.963
ret/std 7.6580 [5.8207, 10.2459] 4.6620 [4.2197, 5.1478] 0.609
ret_normed/std 0.3348 [0.3224, 0.3497] 0.4197 [0.3500, 0.5138] 1.254
val/mean 36.4418 [33.1599, 40.0833] 34.8919 [25.0991, 45.3586] 0.957
adv/std 0.1811 [0.1458, 0.2191] 0.3776 [0.2809, 0.4885] 2.085
replay_ret/mean 48.2353 [47.0644, 50.0910] 46.8301 [39.2461, 55.5177] 0.971
weight/mean 0.9304 [0.9206, 0.9406] 0.9285 [0.9220, 0.9342] 0.998
rew/mean 0.9961 [0.9960, 0.9962] 0.9959 [0.9958, 0.9961] 1.000
prior_ent/mean 35.5475 [34.8418, 36.5772] 34.1293 [31.0195, 36.1003] 0.960
post_ent/mean 33.9853 [33.0962, 35.0949] 32.3941 [28.7272, 34.6670] 0.953
td_error 5.6699 [5.5841, 5.8241] 5.7423 [5.3718, 6.0332] 1.013
data_rew/mean 0.9872 [0.9859, 0.9890] 0.9871 [0.9862, 0.9880] 1.000
pred_rew/mean 0.9961 [0.9960, 0.9962] 0.9959 [0.9958, 0.9961] 1.000
activation/embed 0.3973 [0.3781, 0.4086] 0.3703 [0.3624, 0.3744] 0.932
opt_loss 5.7011 [5.4912, 6.0018] 6.0087 [5.4870, 6.5522] 1.054
opt_grad_norm 11.6176 [10.4261, 12.5588] 12.0010 [10.4005, 14.9788] 1.033
opt_update_norm 0.0093 [0.0089, 0.0100] 0.0097 [0.0094, 0.0102] 1.043
opt_param_norm 65.2772 [65.1939, 65.3511] 65.0772 [64.9691, 65.1967] 0.997
opt_grad_steps 5980.5000 [5980.5000, 5980.5000] 5981.5000 [5981.5000, 5981.5000] 1.000
retnorm_scale_derived 23.0706 [16.6470, 31.7807] 11.4566 [8.9883, 14.7058] 0.497
retnorm_scale_exact 27.9082 [19.5583, 38.6512] nan [nan, nan] nan

window ending at 16000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0231 [0.0182, 0.0329] 0.0204 [0.0180, 0.0230] 0.883
rew_loss 0.6603 [0.6597, 0.6613] 0.6601 [0.6593, 0.6608] 1.000
con_loss 0.0479 [0.0470, 0.0492] 0.0481 [0.0440, 0.0515] 1.003
dyn_loss 1.0922 [1.0854, 1.0961] 1.0779 [1.0603, 1.0909] 0.987
rep_loss 1.0922 [1.0854, 1.0961] 1.0779 [1.0603, 1.0909] 0.987
actor_loss 0.0337 [0.0145, 0.0557] 0.0616 [0.0305, 0.1179] 1.831
critic_loss 2.2684 [1.9555, 2.7569] 2.2814 [1.9674, 2.7524] 1.006
repval_loss 4.1718 [3.8318, 4.6589] 4.3620 [4.0749, 4.8708] 1.046
ent/action/mean 0.1939 [0.1501, 0.2250] 0.2047 [0.1557, 0.2639] 1.056
rand/action/mean 0.2797 [0.2165, 0.3247] 0.2953 [0.2246, 0.3808] 1.056
ret/mean 51.6292 [42.3982, 59.8716] 50.5572 [38.7047, 59.6857] 0.979
ret/std 10.3261 [6.2004, 14.8583] 7.5479 [5.0012, 10.4897] 0.731
ret_normed/std 0.3197 [0.3139, 0.3235] 0.4153 [0.3442, 0.4719] 1.299
val/mean 46.0847 [36.6766, 54.3872] 45.3704 [33.7401, 54.3738] 0.984
adv/std 0.1589 [0.1189, 0.2152] 0.3108 [0.1734, 0.4285] 1.956
replay_ret/mean 57.2003 [50.8359, 63.3076] 55.9818 [47.6822, 63.4585] 0.979
weight/mean 0.9365 [0.9313, 0.9423] 0.9327 [0.9279, 0.9385] 0.996
rew/mean 0.9975 [0.9974, 0.9977] 0.9974 [0.9972, 0.9976] 1.000
prior_ent/mean 35.2954 [34.1937, 36.7061] 33.6363 [29.7786, 35.7795] 0.953
post_ent/mean 33.7274 [32.3779, 35.2964] 32.0935 [28.0064, 34.4508] 0.952
td_error 5.8187 [5.6493, 5.9452] 5.7775 [5.7349, 5.8477] 0.993
data_rew/mean 0.9890 [0.9880, 0.9905] 0.9886 [0.9873, 0.9896] 1.000
pred_rew/mean 0.9975 [0.9974, 0.9977] 0.9974 [0.9972, 0.9976] 1.000
activation/embed 0.3990 [0.3762, 0.4109] 0.3724 [0.3550, 0.3823] 0.933
opt_loss 5.4864 [5.0390, 6.1562] 5.5659 [5.1753, 6.2500] 1.015
opt_grad_norm 13.1151 [9.9690, 18.0323] 12.5358 [8.7030, 14.9776] 0.956
opt_update_norm 0.0096 [0.0092, 0.0100] 0.0095 [0.0090, 0.0100] 0.991
opt_param_norm 66.2621 [66.1968, 66.3103] 65.9276 [65.7215, 66.1035] 0.995
opt_grad_steps 6980.5000 [6980.5000, 6980.5000] 6981.5000 [6981.5000, 6981.5000] 1.000
retnorm_scale_derived 32.1720 [19.7513, 45.9352] 19.2378 [10.5976, 30.4747] 0.598
retnorm_scale_exact 35.5267 [19.0161, 52.6481] nan [nan, nan] nan

window ending at 18000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0218 [0.0196, 0.0239] 0.0221 [0.0200, 0.0238] 1.013
rew_loss 0.6597 [0.6591, 0.6604] 0.6596 [0.6592, 0.6602] 1.000
con_loss 0.0483 [0.0457, 0.0496] 0.0486 [0.0464, 0.0508] 1.007
dyn_loss 1.1036 [1.0831, 1.1229] 1.1092 [1.0904, 1.1258] 1.005
rep_loss 1.1036 [1.0831, 1.1229] 1.1092 [1.0904, 1.1258] 1.005
actor_loss 0.0196 [0.0095, 0.0279] 0.0266 [0.0123, 0.0517] 1.357
critic_loss 1.9494 [1.7754, 2.1543] 2.0182 [1.8168, 2.3879] 1.035
repval_loss 3.8488 [3.6533, 4.1994] 3.9668 [3.6655, 4.3211] 1.031
ent/action/mean 0.1529 [0.1315, 0.1732] 0.1682 [0.1375, 0.2269] 1.100
rand/action/mean 0.2207 [0.1897, 0.2499] 0.2426 [0.1983, 0.3273] 1.100
ret/mean 63.8287 [54.5444, 70.5577] 62.3731 [53.0972, 67.1152] 0.977
ret/std 12.8770 [8.2869, 18.7773] 11.6901 [8.2058, 15.4517] 0.908
ret_normed/std 0.3290 [0.3270, 0.3305] 0.3495 [0.3392, 0.3679] 1.062
val/mean 58.5918 [49.2876, 65.2099] 57.1575 [47.4458, 62.0653] 0.976
adv/std 0.1353 [0.1041, 0.1628] 0.1740 [0.1200, 0.2375] 1.286
replay_ret/mean 67.4472 [61.0886, 72.1202] 65.9090 [59.3234, 69.5998] 0.977
weight/mean 0.9383 [0.9337, 0.9423] 0.9360 [0.9332, 0.9375] 0.998
rew/mean 0.9983 [0.9982, 0.9984] 0.9983 [0.9982, 0.9983] 1.000
prior_ent/mean 35.0959 [34.3216, 36.6028] 33.8745 [31.0730, 35.5787] 0.965
post_ent/mean 33.6875 [32.8737, 35.2232] 32.4714 [29.5583, 34.2969] 0.964
td_error 5.6640 [5.4834, 5.9300] 5.8062 [5.5673, 6.2404] 1.025
data_rew/mean 0.9895 [0.9886, 0.9906] 0.9894 [0.9887, 0.9901] 1.000
pred_rew/mean 0.9983 [0.9982, 0.9984] 0.9983 [0.9982, 0.9983] 1.000
activation/embed 0.4046 [0.3866, 0.4174] 0.3815 [0.3639, 0.3932] 0.943
opt_loss 5.0673 [4.8687, 5.3614] 5.1852 [4.8606, 5.7047] 1.023
opt_grad_norm 11.5554 [9.9091, 13.6849] 11.4340 [9.3927, 14.6013] 0.989
opt_update_norm 0.0103 [0.0091, 0.0110] 0.0108 [0.0103, 0.0112] 1.058
opt_param_norm 67.0005 [66.8052, 67.1753] 66.5527 [66.2861, 66.9864] 0.993
opt_grad_steps 7980.5000 [7980.5000, 7980.5000] 7981.5000 [7981.5000, 7981.5000] 1.000
retnorm_scale_derived 39.1047 [25.3414, 56.9720] 33.7420 [22.3058, 45.2708] 0.863
retnorm_scale_exact 41.3496 [28.8141, 55.6319] nan [nan, nan] nan

window ending at 20000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0232 [0.0201, 0.0277] 0.0220 [0.0162, 0.0266] 0.948
rew_loss 0.6598 [0.6593, 0.6604] 0.6597 [0.6593, 0.6604] 1.000
con_loss 0.0475 [0.0449, 0.0503] 0.0469 [0.0433, 0.0490] 0.986
dyn_loss 1.1045 [1.0906, 1.1158] 1.1153 [1.1090, 1.1270] 1.010
rep_loss 1.1045 [1.0906, 1.1158] 1.1153 [1.1090, 1.1270] 1.010
actor_loss 0.0117 [0.0084, 0.0155] 0.0145 [0.0095, 0.0236] 1.248
critic_loss 1.8011 [1.6800, 1.9658] 2.0076 [1.6704, 2.4572] 1.115
repval_loss 3.5680 [3.3225, 3.8618] 3.7386 [3.4468, 4.2277] 1.048
ent/action/mean 0.1418 [0.1309, 0.1602] 0.1410 [0.1252, 0.1577] 0.994
rand/action/mean 0.2045 [0.1888, 0.2311] 0.2034 [0.1806, 0.2275] 0.994
ret/mean 74.6112 [62.3665, 81.9878] 71.0503 [61.6171, 77.2963] 0.952
ret/std 16.7543 [13.0293, 21.1412] 15.2843 [10.9527, 17.5260] 0.912
ret_normed/std 0.3359 [0.3322, 0.3380] 0.3384 [0.3211, 0.3609] 1.008
val/mean 69.7412 [57.7651, 77.0393] 65.8506 [55.5910, 72.6388] 0.944
adv/std 0.1128 [0.0983, 0.1234] 0.1314 [0.1047, 0.1728] 1.165
replay_ret/mean 77.1496 [68.1577, 82.2093] 73.7937 [67.6484, 78.1194] 0.957
weight/mean 0.9409 [0.9377, 0.9430] 0.9390 [0.9362, 0.9440] 0.998
rew/mean 0.9988 [0.9988, 0.9989] 0.9988 [0.9987, 0.9988] 1.000
prior_ent/mean 34.6428 [33.6872, 36.0757] 35.6842 [34.2921, 36.5994] 1.030
post_ent/mean 33.2464 [32.2440, 34.7589] 34.4299 [33.1111, 35.3526] 1.036
td_error 5.5760 [5.2719, 5.8094] 5.9261 [5.3512, 6.7275] 1.063
data_rew/mean 0.9905 [0.9898, 0.9914] 0.9902 [0.9896, 0.9913] 1.000
pred_rew/mean 0.9988 [0.9988, 0.9989] 0.9988 [0.9987, 0.9988] 1.000
activation/embed 0.4100 [0.3957, 0.4246] 0.3851 [0.3709, 0.3931] 0.939
opt_loss 4.8287 [4.7015, 5.0667] 5.0992 [4.6848, 5.6898] 1.056
opt_grad_norm 11.0210 [9.4156, 12.2078] 10.6333 [9.6133, 11.3099] 0.965
opt_update_norm 0.0110 [0.0104, 0.0117] 0.0106 [0.0092, 0.0114] 0.958
opt_param_norm 67.6027 [67.3803, 67.8515] 67.0351 [66.6661, 67.6384] 0.992
opt_grad_steps 8980.5000 [8980.5000, 8980.5000] 8981.5000 [8981.5000, 8981.5000] 1.000
retnorm_scale_derived 49.8165 [39.2173, 62.5471] 44.9429 [34.1078, 52.5793] 0.902
retnorm_scale_exact 55.5971 [44.9350, 65.0703] 47.7236 [31.8915, 58.2991] 0.858

Training metrics per seed (ref / ajax)

seed 0

metric 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
rec_loss 0.402 / 0.451 0.067 / 0.074 0.021 / 0.020 0.015 / 0.016 0.020 / 0.026 0.025 / 0.028 0.021 / 0.022 0.018 / 0.018 0.020 / 0.022 0.020 / 0.027
rew_loss 5.294 / 5.291 2.520 / 2.513 0.813 / 0.803 0.693 / 0.691 0.672 / 0.671 0.663 / 0.663 0.661 / 0.660 0.660 / 0.659 0.660 / 0.659 0.660 / 0.659
con_loss 0.409 / 0.323 0.154 / 0.155 0.082 / 0.081 0.056 / 0.055 0.059 / 0.056 0.057 / 0.056 0.051 / 0.053 0.049 / 0.051 0.050 / 0.051 0.047 / 0.049
dyn_loss 2.533 / 2.685 1.011 / 1.011 1.038 / 1.032 1.055 / 1.046 1.075 / 1.070 1.108 / 1.062 1.092 / 1.063 1.096 / 1.060 1.123 / 1.090 1.116 / 1.109
rep_loss 2.533 / 2.685 1.011 / 1.011 1.038 / 1.032 1.055 / 1.046 1.075 / 1.070 1.108 / 1.062 1.092 / 1.063 1.096 / 1.060 1.123 / 1.090 1.116 / 1.109
actor_loss -0.009 / -0.009 0.175 / 0.166 0.173 / 0.183 0.154 / 0.164 0.105 / 0.133 0.078 / 0.126 0.046 / 0.067 0.031 / 0.031 0.021 / 0.012 0.011 / 0.011
critic_loss 3.740 / 4.029 5.206 / 5.123 4.055 / 4.137 3.183 / 3.190 3.291 / 3.681 3.100 / 2.914 2.329 / 2.607 2.093 / 2.124 1.775 / 1.817 1.680 / 1.895
repval_loss 10.208 / 10.178 7.210 / 7.186 6.020 / 6.031 5.294 / 5.265 5.469 / 5.721 5.061 / 4.845 4.339 / 4.689 4.025 / 4.075 3.694 / 3.666 3.520 / 3.541
ent/action/mean 0.693 / 0.693 0.517 / 0.505 0.299 / 0.310 0.261 / 0.282 0.258 / 0.295 0.266 / 0.306 0.199 / 0.204 0.207 / 0.194 0.173 / 0.140 0.131 / 0.140
rand/action/mean 1.000 / 1.000 0.746 / 0.729 0.431 / 0.447 0.376 / 0.407 0.373 / 0.426 0.384 / 0.441 0.287 / 0.295 0.298 / 0.281 0.250 / 0.202 0.189 / 0.202
ret/mean 0.072 / 0.073 3.321 / 3.325 13.573 / 13.673 22.049 / 21.982 25.612 / 24.093 30.912 / 30.597 41.526 / 39.703 52.618 / 53.281 66.384 / 66.907 79.479 / 74.238
ret/std 0.133 / 0.138 0.796 / 0.830 2.799 / 2.774 3.442 / 3.493 4.235 / 3.712 5.187 / 4.379 6.907 / 5.148 9.920 / 10.490 11.567 / 15.452 16.092 / 17.526
ret_normed/std 0.133 / 0.138 0.321 / 0.328 0.368 / 0.361 0.404 / 0.405 0.374 / 0.378 0.315 / 0.389 0.332 / 0.350 0.322 / 0.344 0.330 / 0.341 0.337 / 0.333
val/mean 0.097 / 0.094 1.842 / 1.810 8.745 / 8.677 16.723 / 16.705 20.461 / 19.160 25.553 / 25.752 36.082 / 34.218 47.190 / 47.997 61.278 / 61.961 74.419 / 69.322
adv/std 0.159 / 0.163 0.328 / 0.335 0.368 / 0.361 0.375 / 0.398 0.339 / 0.379 0.234 / 0.353 0.178 / 0.281 0.143 / 0.173 0.139 / 0.120 0.117 / 0.105
replay_ret/mean 8.040 / 7.896 10.508 / 10.280 20.288 / 20.274 29.587 / 29.147 33.214 / 32.068 38.040 / 37.928 47.550 / 45.727 57.457 / 56.805 69.133 / 68.804 81.082 / 75.613
weight/mean 0.347 / 0.372 0.736 / 0.730 0.862 / 0.873 0.908 / 0.909 0.902 / 0.905 0.916 / 0.915 0.930 / 0.929 0.936 / 0.928 0.939 / 0.933 0.943 / 0.936
rew/mean 0.062 / 0.063 0.347 / 0.347 0.903 / 0.905 0.972 / 0.971 0.985 / 0.985 0.993 / 0.993 0.996 / 0.996 0.997 / 0.997 0.998 / 0.998 0.999 / 0.999
prior_ent/mean 43.247 / 43.304 43.561 / 43.694 41.675 / 42.538 39.517 / 40.380 36.392 / 36.795 35.376 / 35.673 35.224 / 35.268 34.986 / 35.780 34.363 / 35.579 34.165 / 36.161
post_ent/mean 42.399 / 42.119 42.657 / 42.778 40.616 / 41.636 38.200 / 39.273 34.699 / 35.100 33.662 / 34.115 33.765 / 33.788 33.508 / 34.451 32.966 / 34.297 32.736 / 34.826
td_error 0.129 / 0.132 1.542 / 1.588 5.017 / 5.152 5.598 / 5.643 5.501 / 5.399 5.678 / 5.449 5.601 / 5.822 5.649 / 5.848 5.483 / 5.611 5.647 / 5.700
data_rew/mean 0.958 / 0.956 0.959 / 0.957 0.970 / 0.970 0.980 / 0.978 0.983 / 0.983 0.985 / 0.985 0.987 / 0.986 0.988 / 0.987 0.989 / 0.989 0.990 / 0.990
pred_rew/mean 0.062 / 0.063 0.347 / 0.347 0.903 / 0.905 0.972 / 0.971 0.985 / 0.985 0.993 / 0.993 0.996 / 0.996 0.997 / 0.997 0.998 / 0.998 0.999 / 0.999
activation/embed 0.420 / 0.393 0.420 / 0.388 0.419 / 0.387 0.416 / 0.391 0.411 / 0.385 0.401 / 0.370 0.409 / 0.374 0.411 / 0.380 0.417 / 0.388 0.425 / 0.391
opt_loss 15.685 / 16.092 11.397 / 11.298 8.094 / 8.169 6.849 / 6.847 6.971 / 7.460 6.660 / 6.409 5.610 / 5.987 5.264 / 5.273 4.869 / 4.861 4.701 / 4.923
opt_grad_norm 160.414 / 130.899 8.500 / 7.884 7.783 / 6.910 7.505 / 6.837 11.870 / 11.445 16.432 / 12.972 10.426 / 10.624 9.969 / 8.703 9.909 / 9.393 9.416 / 10.977
opt_update_norm 0.007 / 0.007 0.013 / 0.013 0.008 / 0.008 0.009 / 0.010 0.013 / 0.014 0.010 / 0.010 0.009 / 0.010 0.009 / 0.009 0.011 / 0.011 0.012 / 0.011
opt_param_norm 59.881 / 59.906 60.499 / 60.531 61.745 / 61.778 62.474 / 62.498 63.333 / 63.416 64.211 / 64.179 65.194 / 65.066 66.279 / 65.958 67.021 / 66.386 67.576 / 66.666
opt_grad_steps 240.500 / 241.000 980.500 / 981.500 1980.500 / 1981.500 2980.500 / 2981.500 3980.500 / 3981.500 4980.500 / 4981.500 5980.500 / 5981.500 6980.500 / 6981.500 7980.500 / 7981.500 8980.500 / 8981.500
retnorm_scale_derived 1.000 / 1.000 2.481 / 2.529 7.596 / 7.688 8.530 / 8.630 11.318 / 9.819 16.468 / 11.265 20.784 / 14.706 30.829 / 30.475 35.001 / 45.271 47.685 / 52.579
retnorm_scale_exact 1.000 / nan 5.971 / nan 8.367 / nan 8.442 / nan 14.966 / nan 13.911 / nan 25.515 / nan 34.916 / nan 39.603 / nan 56.786 / 58.299

seed 1

metric 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
rec_loss 0.385 / 0.441 0.066 / 0.062 0.017 / 0.020 0.014 / 0.018 0.027 / 0.023 0.031 / 0.019 0.024 / 0.018 0.033 / 0.020 0.022 / 0.020 0.022 / 0.023
rew_loss 5.279 / 5.282 2.483 / 2.490 0.809 / 0.801 0.693 / 0.691 0.672 / 0.671 0.665 / 0.664 0.662 / 0.661 0.661 / 0.660 0.660 / 0.660 0.660 / 0.659
con_loss 0.398 / 0.363 0.141 / 0.152 0.074 / 0.084 0.053 / 0.057 0.050 / 0.051 0.050 / 0.048 0.045 / 0.050 0.047 / 0.049 0.046 / 0.049 0.045 / 0.048
dyn_loss 2.541 / 2.173 1.011 / 1.007 1.038 / 1.038 1.062 / 1.066 1.095 / 1.099 1.084 / 1.080 1.085 / 1.093 1.095 / 1.091 1.083 / 1.111 1.091 / 1.127
rep_loss 2.541 / 2.173 1.011 / 1.007 1.038 / 1.038 1.062 / 1.066 1.095 / 1.099 1.084 / 1.080 1.085 / 1.093 1.095 / 1.091 1.083 / 1.111 1.091 / 1.127
actor_loss -0.007 / -0.010 0.165 / 0.162 0.172 / 0.165 0.157 / 0.159 0.116 / 0.109 0.121 / 0.108 0.067 / 0.087 0.056 / 0.036 0.028 / 0.016 0.016 / 0.009
critic_loss 3.932 / 3.765 5.297 / 5.043 4.146 / 3.938 3.276 / 3.126 3.488 / 3.408 3.790 / 2.534 2.622 / 2.132 2.757 / 1.967 2.154 / 1.850 1.966 / 1.670
repval_loss 10.259 / 10.145 7.273 / 7.117 6.088 / 5.947 5.472 / 5.280 5.493 / 5.482 5.518 / 4.380 4.632 / 4.456 4.659 / 4.140 4.199 / 3.914 3.862 / 3.447
ent/action/mean 0.693 / 0.693 0.506 / 0.503 0.294 / 0.314 0.265 / 0.283 0.255 / 0.231 0.306 / 0.189 0.217 / 0.161 0.225 / 0.156 0.154 / 0.137 0.160 / 0.125
rand/action/mean 1.000 / 1.000 0.729 / 0.726 0.424 / 0.453 0.383 / 0.409 0.368 / 0.334 0.442 / 0.273 0.313 / 0.232 0.325 / 0.225 0.222 / 0.198 0.231 / 0.181
ret/mean 0.069 / 0.079 3.386 / 3.376 13.685 / 13.598 21.827 / 22.160 24.515 / 26.354 26.933 / 38.343 38.773 / 51.246 42.398 / 59.686 54.544 / 67.115 62.366 / 77.296
ret/std 0.135 / 0.135 0.892 / 0.876 2.883 / 3.151 3.408 / 3.793 3.939 / 4.043 4.367 / 4.250 5.821 / 4.618 6.200 / 7.153 8.287 / 11.413 13.029 / 17.374
ret_normed/std 0.135 / 0.135 0.332 / 0.334 0.370 / 0.382 0.400 / 0.431 0.397 / 0.406 0.351 / 0.475 0.350 / 0.514 0.314 / 0.430 0.327 / 0.339 0.332 / 0.361
val/mean 0.088 / 0.105 1.867 / 1.898 8.783 / 8.990 16.532 / 16.992 19.572 / 21.134 21.765 / 32.638 33.160 / 45.359 36.677 / 54.374 49.288 / 62.065 57.765 / 72.639
adv/std 0.160 / 0.160 0.339 / 0.341 0.370 / 0.381 0.391 / 0.390 0.353 / 0.357 0.341 / 0.374 0.219 / 0.488 0.215 / 0.331 0.163 / 0.165 0.123 / 0.117
replay_ret/mean 8.241 / 7.931 10.817 / 10.308 21.277 / 20.067 29.990 / 29.482 33.959 / 34.918 36.806 / 44.868 47.064 / 55.518 50.836 / 63.459 61.089 / 69.600 68.158 / 78.119
weight/mean 0.364 / 0.350 0.744 / 0.726 0.877 / 0.848 0.912 / 0.897 0.916 / 0.911 0.929 / 0.927 0.941 / 0.934 0.942 / 0.932 0.942 / 0.938 0.942 / 0.937
rew/mean 0.062 / 0.063 0.357 / 0.354 0.908 / 0.908 0.972 / 0.972 0.987 / 0.987 0.993 / 0.993 0.996 / 0.996 0.998 / 0.998 0.998 / 0.998 0.999 / 0.999
prior_ent/mean 43.503 / 43.270 43.623 / 43.364 42.165 / 41.708 40.429 / 39.229 37.412 / 37.578 34.309 / 36.217 34.842 / 36.100 34.194 / 35.351 34.322 / 34.972 33.687 / 36.599
post_ent/mean 41.869 / 42.436 42.729 / 42.330 41.135 / 40.580 39.164 / 37.899 35.712 / 35.948 31.833 / 34.655 33.096 / 34.667 32.378 / 33.823 32.874 / 33.559 32.244 / 35.353
td_error 0.129 / 0.131 1.584 / 1.560 5.097 / 5.004 5.604 / 5.584 5.292 / 5.526 5.746 / 5.835 5.824 / 6.033 5.945 / 5.735 5.578 / 5.567 5.272 / 5.351
data_rew/mean 0.960 / 0.956 0.962 / 0.958 0.974 / 0.970 0.982 / 0.979 0.986 / 0.985 0.988 / 0.986 0.989 / 0.987 0.991 / 0.989 0.991 / 0.989 0.991 / 0.990
pred_rew/mean 0.062 / 0.063 0.357 / 0.354 0.908 / 0.908 0.972 / 0.972 0.987 / 0.987 0.993 / 0.993 0.996 / 0.996 0.998 / 0.998 0.998 / 0.998 0.999 / 0.999
activation/embed 0.439 / 0.375 0.434 / 0.385 0.422 / 0.388 0.407 / 0.385 0.386 / 0.358 0.381 / 0.356 0.378 / 0.362 0.376 / 0.355 0.387 / 0.364 0.396 / 0.371
opt_loss 15.861 / 15.274 11.445 / 11.152 8.185 / 7.934 7.002 / 6.808 7.206 / 7.116 7.504 / 5.875 6.002 / 5.487 6.156 / 5.175 5.361 / 4.990 5.067 / 4.685
opt_grad_norm 143.735 / 191.347 8.097 / 8.590 7.814 / 8.368 7.988 / 7.554 14.814 / 11.133 13.395 / 9.704 12.559 / 10.401 18.032 / 14.978 11.072 / 10.308 11.439 / 9.613
opt_update_norm 0.007 / 0.007 0.013 / 0.014 0.008 / 0.008 0.011 / 0.010 0.013 / 0.013 0.011 / 0.009 0.009 / 0.010 0.010 / 0.010 0.009 / 0.011 0.010 / 0.011
opt_param_norm 59.908 / 59.776 60.534 / 60.415 61.778 / 61.642 62.512 / 62.351 63.379 / 63.187 64.336 / 64.034 65.287 / 64.969 66.310 / 65.721 67.175 / 66.286 67.852 / 66.801
opt_grad_steps 240.500 / 241.000 980.500 / 981.500 1980.500 / 1981.500 2980.500 / 2981.500 3980.500 / 3981.500 4980.500 / 4981.500 5980.500 / 5981.500 6980.500 / 6981.500 7980.500 / 7981.500 8980.500 / 8981.500
retnorm_scale_derived 1.000 / 1.000 2.686 / 2.623 7.795 / 8.260 8.525 / 8.812 9.930 / 9.969 12.427 / 8.943 16.647 / 8.988 19.751 / 16.641 25.341 / 33.650 39.217 / 48.142
retnorm_scale_exact 1.000 / nan 6.354 / nan 8.475 / nan 8.211 / nan 12.687 / nan 13.495 / nan 19.558 / nan 19.016 / nan 28.814 / nan 44.935 / 52.980

seed 2

metric 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
rec_loss 0.428 / 0.387 0.058 / 0.063 0.021 / 0.022 0.017 / 0.016 0.021 / 0.030 0.017 / 0.035 0.017 / 0.031 0.018 / 0.023 0.024 / 0.024 0.028 / 0.016
rew_loss 5.282 / 5.288 2.494 / 2.528 0.803 / 0.813 0.691 / 0.693 0.670 / 0.673 0.663 / 0.664 0.660 / 0.662 0.660 / 0.661 0.659 / 0.660 0.659 / 0.660
con_loss 0.366 / 0.351 0.155 / 0.153 0.081 / 0.086 0.057 / 0.060 0.055 / 0.057 0.050 / 0.053 0.052 / 0.051 0.047 / 0.044 0.050 / 0.046 0.050 / 0.043
dyn_loss 2.712 / 2.300 1.012 / 1.009 1.033 / 1.036 1.056 / 1.066 1.084 / 1.075 1.067 / 1.080 1.089 / 1.072 1.085 / 1.082 1.105 / 1.126 1.107 / 1.110
rep_loss 2.712 / 2.300 1.012 / 1.009 1.033 / 1.036 1.056 / 1.066 1.084 / 1.075 1.067 / 1.080 1.089 / 1.072 1.085 / 1.082 1.105 / 1.126 1.107 / 1.110
actor_loss -0.007 / -0.008 0.161 / 0.163 0.177 / 0.165 0.152 / 0.140 0.183 / 0.146 0.095 / 0.118 0.030 / 0.135 0.015 / 0.118 0.010 / 0.052 0.008 / 0.024
critic_loss 4.073 / 3.985 5.054 / 5.138 4.066 / 4.075 3.395 / 3.151 3.154 / 4.015 2.635 / 4.227 2.268 / 3.022 1.956 / 2.752 1.918 / 2.388 1.758 / 2.457
repval_loss 10.180 / 10.200 7.209 / 7.235 6.016 / 6.002 5.500 / 5.264 5.248 / 5.880 4.711 / 5.848 4.221 / 4.909 3.832 / 4.871 3.653 / 4.321 3.323 / 4.228
ent/action/mean 0.693 / 0.693 0.518 / 0.518 0.309 / 0.292 0.269 / 0.244 0.325 / 0.302 0.261 / 0.276 0.219 / 0.313 0.150 / 0.264 0.131 / 0.227 0.134 / 0.158
rand/action/mean 1.000 / 1.000 0.747 / 0.747 0.445 / 0.421 0.388 / 0.352 0.468 / 0.436 0.376 / 0.398 0.316 / 0.452 0.217 / 0.381 0.190 / 0.327 0.194 / 0.228
ret/mean 0.068 / 0.074 3.289 / 3.229 13.373 / 13.398 20.692 / 21.788 25.288 / 22.379 34.969 / 23.038 45.163 / 29.917 59.872 / 38.705 70.558 / 53.097 81.988 / 61.617
ret/std 0.138 / 0.137 0.859 / 0.817 2.912 / 2.869 3.495 / 3.527 3.687 / 3.598 5.566 / 4.125 10.246 / 4.220 14.858 / 5.001 18.777 / 8.206 21.141 / 10.953
ret_normed/std 0.138 / 0.137 0.329 / 0.323 0.378 / 0.373 0.411 / 0.412 0.421 / 0.387 0.408 / 0.361 0.322 / 0.395 0.323 / 0.472 0.330 / 0.368 0.338 / 0.321
val/mean 0.087 / 0.095 1.788 / 1.814 8.506 / 8.644 15.546 / 16.659 20.028 / 17.610 29.921 / 17.846 40.083 / 25.099 54.387 / 33.740 65.210 / 47.446 77.039 / 55.591
adv/std 0.162 / 0.162 0.336 / 0.330 0.378 / 0.372 0.411 / 0.409 0.399 / 0.378 0.340 / 0.337 0.146 / 0.364 0.119 / 0.428 0.104 / 0.237 0.098 / 0.173
replay_ret/mean 7.599 / 8.106 10.024 / 10.526 19.765 / 20.224 27.942 / 29.136 32.685 / 31.427 41.351 / 32.870 50.091 / 39.246 63.308 / 47.682 72.120 / 59.323 82.209 / 67.648
weight/mean 0.376 / 0.369 0.714 / 0.726 0.856 / 0.860 0.897 / 0.903 0.906 / 0.901 0.916 / 0.917 0.921 / 0.922 0.931 / 0.939 0.934 / 0.937 0.938 / 0.944
rew/mean 0.062 / 0.063 0.353 / 0.343 0.907 / 0.903 0.971 / 0.971 0.987 / 0.985 0.993 / 0.993 0.996 / 0.996 0.998 / 0.998 0.998 / 0.998 0.999 / 0.999
prior_ent/mean 43.185 / 43.366 43.420 / 43.731 42.323 / 42.476 40.283 / 40.794 35.963 / 35.454 36.289 / 32.251 36.577 / 31.019 36.706 / 29.779 36.603 / 31.073 36.076 / 34.292
post_ent/mean 41.677 / 42.407 42.392 / 42.840 41.307 / 41.551 38.895 / 39.562 34.114 / 33.427 34.812 / 29.646 35.095 / 28.727 35.296 / 28.006 35.223 / 29.558 34.759 / 33.111
td_error 0.131 / 0.131 1.581 / 1.484 5.107 / 4.972 5.566 / 5.541 5.616 / 5.159 5.798 / 5.657 5.584 / 5.372 5.862 / 5.750 5.930 / 6.240 5.809 / 6.727
data_rew/mean 0.954 / 0.957 0.956 / 0.959 0.968 / 0.971 0.978 / 0.979 0.982 / 0.984 0.985 / 0.986 0.986 / 0.988 0.988 / 0.990 0.989 / 0.990 0.990 / 0.991
pred_rew/mean 0.062 / 0.063 0.353 / 0.343 0.907 / 0.903 0.971 / 0.971 0.987 / 0.985 0.993 / 0.993 0.996 / 0.996 0.998 / 0.998 0.998 / 0.998 0.999 / 0.999
activation/embed 0.428 / 0.392 0.438 / 0.401 0.448 / 0.409 0.437 / 0.400 0.414 / 0.393 0.412 / 0.386 0.405 / 0.374 0.410 / 0.382 0.410 / 0.393 0.410 / 0.393
opt_loss 16.179 / 15.592 11.197 / 11.327 8.090 / 8.101 7.124 / 6.811 6.849 / 7.868 6.047 / 8.040 5.491 / 6.552 5.039 / 6.250 4.972 / 5.705 4.718 / 5.690
opt_grad_norm 140.356 / 132.227 9.024 / 8.316 7.629 / 7.477 9.116 / 7.678 14.097 / 13.190 10.987 / 15.692 11.868 / 14.979 11.344 / 13.927 13.685 / 14.601 12.208 / 11.310
opt_update_norm 0.007 / 0.007 0.013 / 0.013 0.008 / 0.008 0.012 / 0.008 0.012 / 0.015 0.010 / 0.012 0.010 / 0.009 0.010 / 0.010 0.011 / 0.010 0.011 / 0.009
opt_param_norm 59.875 / 59.870 60.499 / 60.487 61.747 / 61.744 62.505 / 62.457 63.387 / 63.351 64.276 / 64.423 65.351 / 65.197 66.197 / 66.104 66.805 / 66.986 67.380 / 67.638
opt_grad_steps 240.500 / 241.000 980.500 / 981.500 1980.500 / 1981.500 2980.500 / 2981.500 3980.500 / 3981.500 4980.500 / 4981.500 5980.500 / 5981.500 6980.500 / 6981.500 7980.500 / 7981.500 8980.500 / 8981.500
retnorm_scale_derived 1.000 / 1.000 2.608 / 2.525 7.710 / 7.702 8.494 / 8.566 8.762 / 9.291 13.643 / 11.427 31.781 / 10.676 45.935 / 10.598 56.972 / 22.306 62.547 / 34.108
retnorm_scale_exact 1.000 / nan 6.196 / nan 8.380 / nan 8.289 / nan 9.284 / nan 25.607 / nan 38.651 / nan 52.648 / nan 55.632 / nan 65.070 / 31.892

Reference evaluation returns per episode

  • s0 rows 2000: [18.0, 13.0, 14.0, 10.0, 31.0, 21.0, 11.0, 17.0, 24.0, 15.0]
  • s0 rows 4000: [307.0, 195.0, 85.0, 145.0, 200.0, 74.0, 168.0, 100.0, 98.0, 76.0]
  • s0 rows 6000: [188.0, 149.0, 168.0, 263.0, 174.0, 181.0, 205.0, 206.0, 196.0, 293.0]
  • s0 rows 8000: [110.0, 106.0, 111.0, 110.0, 95.0, 104.0, 127.0, 132.0, 121.0, 119.0]
  • s0 rows 10000: [110.0, 125.0, 130.0, 115.0, 123.0, 113.0, 129.0, 107.0, 115.0, 86.0]
  • s0 rows 12000: [177.0, 178.0, 177.0, 211.0, 160.0, 215.0, 219.0, 199.0, 177.0, 210.0]
  • s0 rows 14000: [210.0, 209.0, 200.0, 228.0, 225.0, 197.0, 240.0, 217.0, 208.0, 290.0]
  • s0 rows 16000: [299.0, 249.0, 305.0, 277.0, 286.0, 260.0, 211.0, 239.0, 239.0, 170.0]
  • s0 rows 18000: [500.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0]
  • s0 rows 20000: [449.0, 389.0, 500.0, 409.0, 432.0, 403.0, 422.0, 370.0, 375.0, 299.0]
  • s1 rows 2000: [14.0, 47.0, 38.0, 16.0, 33.0, 11.0, 30.0, 15.0, 17.0, 31.0]
  • s1 rows 4000: [141.0, 49.0, 89.0, 462.0, 134.0, 75.0, 147.0, 92.0, 127.0, 101.0]
  • s1 rows 6000: [159.0, 167.0, 162.0, 89.0, 123.0, 144.0, 120.0, 101.0, 146.0, 115.0]
  • s1 rows 8000: [86.0, 83.0, 88.0, 101.0, 81.0, 88.0, 84.0, 88.0, 91.0, 94.0]
  • s1 rows 10000: [210.0, 170.0, 180.0, 233.0, 171.0, 189.0, 198.0, 208.0, 179.0, 168.0]
  • s1 rows 12000: [128.0, 125.0, 127.0, 137.0, 133.0, 133.0, 137.0, 127.0, 124.0, 135.0]
  • s1 rows 14000: [140.0, 164.0, 138.0, 124.0, 138.0, 129.0, 125.0, 128.0, 146.0, 137.0]
  • s1 rows 16000: [156.0, 140.0, 149.0, 166.0, 144.0, 138.0, 161.0, 149.0, 146.0, 161.0]
  • s1 rows 18000: [123.0, 136.0, 124.0, 121.0, 121.0, 120.0, 114.0, 118.0, 127.0, 118.0]
  • s1 rows 20000: [434.0, 227.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0]
  • s2 rows 2000: [17.0, 44.0, 37.0, 17.0, 42.0, 46.0, 35.0, 20.0, 42.0, 34.0]
  • s2 rows 4000: [208.0, 176.0, 195.0, 263.0, 120.0, 192.0, 252.0, 180.0, 198.0, 177.0]
  • s2 rows 6000: [147.0, 143.0, 160.0, 93.0, 139.0, 141.0, 131.0, 179.0, 127.0, 175.0]
  • s2 rows 8000: [125.0, 124.0, 154.0, 311.0, 193.0, 155.0, 169.0, 150.0, 142.0, 180.0]
  • s2 rows 10000: [106.0, 111.0, 108.0, 100.0, 50.0, 63.0, 101.0, 109.0, 42.0, 112.0]
  • s2 rows 12000: [156.0, 153.0, 162.0, 215.0, 182.0, 154.0, 174.0, 116.0, 168.0, 109.0]
  • s2 rows 14000: [356.0, 464.0, 348.0, 298.0, 290.0, 375.0, 318.0, 298.0, 289.0, 420.0]
  • s2 rows 16000: [180.0, 166.0, 181.0, 211.0, 184.0, 169.0, 186.0, 178.0, 179.0, 171.0]
  • s2 rows 18000: [147.0, 145.0, 144.0, 142.0, 154.0, 122.0, 148.0, 133.0, 146.0, 157.0]
  • s2 rows 20000: [463.0, 367.0, 436.0, 325.0, 390.0, 405.0, 356.0, 500.0, 347.0, 409.0]