Skip to content

Ajax vs reference DreamerV3, CartPole-v1

Root: <scratch>/refrun/full; seeds [0, 1, 2]; checkpoints (rows) [2000, 4000, 6000, 8000, 10000, 12000, 14000, 16000, 18000, 20000]; S1 window [16000, 18000, 20000]; S2 at 20000.

Checks

PASS  ref_s0: exit code 0 (ref_s0.DONE)
PASS  ref_s1: exit code 0 (ref_s1.DONE)
PASS  ref_s2: exit code 0 (ref_s2.DONE)
PASS  ajax: exit code 0 (ajax.DONE)
PASS  ref s0: config.yaml has the protocol values
PASS  ref s0: reference code imported from the 29eb964 checkout (['<reference-checkout>/dreamerv3/agent.py', '<reference-checkout>/embodied/__init__.py'])
PASS  ref s0: run_args.json has the protocol values
PASS  ref s0: every evaluation has 10 episodes
PASS  ref s0: episodes length == score, 1 <= length <= 500 (194 episodes)
PASS  ref s0: terminal False only for 500-step episodes
PASS  ref s0: per worker the episodes tile rows 0..row_last with L + 1 rows each
PASS  ref s0: run finished and the flush step ran (timing.json t_end present; flush_episodes 0 == episodes with row_last 1250: 0)
PASS  ref s1: config.yaml has the protocol values
PASS  ref s1: reference code imported from the 29eb964 checkout (['<reference-checkout>/dreamerv3/agent.py', '<reference-checkout>/embodied/__init__.py'])
PASS  ref s1: run_args.json has the protocol values
PASS  ref s1: every evaluation has 10 episodes
PASS  ref s1: episodes length == score, 1 <= length <= 500 (182 episodes)
PASS  ref s1: terminal False only for 500-step episodes
PASS  ref s1: per worker the episodes tile rows 0..row_last with L + 1 rows each
PASS  ref s1: run finished and the flush step ran (timing.json t_end present; flush_episodes 0 == episodes with row_last 1250: 0)
PASS  ref s2: config.yaml has the protocol values
PASS  ref s2: reference code imported from the 29eb964 checkout (['<reference-checkout>/dreamerv3/agent.py', '<reference-checkout>/embodied/__init__.py'])
PASS  ref s2: run_args.json has the protocol values
PASS  ref s2: every evaluation has 10 episodes
PASS  ref s2: episodes length == score, 1 <= length <= 500 (212 episodes)
PASS  ref s2: terminal False only for 500-step episodes
PASS  ref s2: per worker the episodes tile rows 0..row_last with L + 1 rows each
PASS  ref s2: run finished and the flush step ran (timing.json t_end present; flush_episodes 0 == episodes with row_last 1250: 0)
PASS  ajax: config.json has the protocol values
PASS  10 records per seed per side at rows 2000 k
PASS  updates = 1 + floor((N - 1040) / 2) at every N on both sides (9481 at 20000)
PASS  9481 updates at 20000 rows on both sides
PASS  ajax s0: logged S2 108.519 == final buffer mean 108.519 (tol 1e-4)
PASS  ajax s1: logged S2 107.013 == final buffer mean 107.012 (tol 1e-4)
PASS  ajax s2: logged S2 107.378 == final buffer mean 107.378 (tol 1e-4)
PASS  ajax s0: final state rows 20000, n_updates 9481
PASS  ajax s1: final state rows 20000, n_updates 9481
PASS  ajax s2: final state rows 20000, n_updates 9481

Pre-registered rule

  • A = mean over Ajax seeds of S1 = 204.1
  • reference S1 range [min, max] = [241.4, 386.1]
  • LEARNS = False (A >= min reference S1)
  • WITHIN_RANGE = False (min <= A <= max)

Per-seed S1 and S2

seed ref S1 ajax S1 ref S2 ajax S2 (logged) ajax S2 (audit)
0 386.1 180.6 98.0 108.5 108.5
1 246.4 143.5 114.8 107.0 107.0
2 241.4 288.3 106.9 107.4 107.4
mean 291.3 204.1 106.6 107.6

Known differences between the two sides (read before interpreting)

  • D1 / U1, asynchrony (reference only). The reference acts with policy parameters that lag the trained ones by about 2 vector steps (about 16 of 9481 updates; measured on the real reference in refrun/audit_loop), its first training batch is prefetched before the gate opens (11 distinct items of 16 in the audit), and its write-back and metrics are one call late. Ajax acts with the latest parameters. Evaluation uses the latest parameters on both sides.
  • D2 / U2, shared replay-sampler stream (reference only). Stock make_replay passes no seed, so Uniform(seed=0) samples the replay for every reference seed: reference seeds [0, 1, 2] share one index stream (the replay contents differ). The reference S1 [min, max] range therefore comes from runs that are not fully independent. Ajax seeds its sampler from the run seed.
  • Report prefetch stream (stock, D1 / D2). With run.eval_every = -1 the stock report stream (train.py:76-77) is created but never read: when the first replay items appear (about row 1040) its prefetch thread draws 2 batches of 16 (32 samples, popping up to 32 of the earliest online-queue items, plus Uniform-RNG and agent.rng draws) and then blocks for good. A stock run (eval_every 180 s) would draw 16 more every 180 s of wall time. Negligible: training draws 128 samples per vector step against 16 new items.
  • NaN in training-metric windows. The reference Agg skips NaN values; Ajax's accumulator propagates them, so one NaN makes the Ajax window mean NaN. A nan Ajax window against a finite reference window means 'a NaN occurred in that Ajax window', not a divergence of the finite values. Non-finite Ajax window means here: none.
  • Reference metric windows hold updates n0-1 .. n1-1 (one-call lag); Ajax per-episode evaluation returns are not logged (only the mean).

Evaluation curve (mean return of the evaluation episodes)

rows ref s0 ref s1 ref s2 ref mean ajax s0 ajax s1 ajax s2 ajax mean
2000 17.4 25.2 33.4 25.3 22.4 19.4 15.9 19.2
4000 144.8 141.7 196.1 160.9 102.3 189.3 60.9 117.5
6000 202.3 132.6 143.5 159.5 480.5 230.5 476.4 395.8
8000 113.5 88.4 170.3 124.1 142.6 322.4 336.5 267.2
10000 115.3 190.6 90.2 132.0 163.9 500.0 149.1 271.0
12000 192.3 130.6 158.9 160.6 147.4 118.4 389.5 218.4
14000 222.4 136.9 345.6 235.0 224.7 174.1 246.5 215.1
16000 253.5 151.0 180.5 195.0 290.0 84.7 212.9 195.9
18000 500.0 122.2 143.8 255.3 108.6 167.5 151.9 142.7
20000 404.8 466.1 399.8 423.6 143.3 178.3 500.0 273.9

S2 curve (training-episode rolling mean, Ajax definition)

rows ref s0 ref s1 ref s2 ref mean ajax s0 ajax s1 ajax s2 ajax mean
2000 23.1 23.4 21.2 22.5 19.5 21.4 20.5 20.5
4000 25.5 26.9 23.6 25.4 24.1 23.9 26.4 24.8
6000 27.9 33.7 29.3 30.3 29.7 29.9 29.3 29.7
8000 45.7 38.1 42.3 42.1 36.0 31.5 31.5 33.0
10000 57.4 56.9 51.6 55.3 52.7 36.2 46.6 45.2
12000 61.2 69.5 66.6 65.8 62.2 50.2 69.1 60.5
14000 67.4 73.4 66.6 69.2 69.4 71.7 79.5 73.5
16000 73.1 88.1 81.0 80.7 78.1 95.0 89.7 87.6
18000 84.5 100.4 95.6 93.5 89.1 96.2 92.4 92.6
20000 98.0 114.8 106.9 106.6 108.5 107.0 107.4 107.6

Updates per checkpoint

rows expected ref s0 ref s1 ref s2 ajax s0 ajax s1 ajax s2
2000 481 481 481 481 481 481 481
4000 1481 1481 1481 1481 1481 1481 1481
6000 2481 2481 2481 2481 2481 2481 2481
8000 3481 3481 3481 3481 3481 3481 3481
10000 4481 4481 4481 4481 4481 4481 4481
12000 5481 5481 5481 5481 5481 5481 5481
14000 6481 6481 6481 6481 6481 6481 6481
16000 7481 7481 7481 7481 7481 7481 7481
18000 8481 8481 8481 8481 8481 8481 8481
20000 9481 9481 9481 9481 9481 9481 9481

Training metrics, window means (mean over seeds [min, max])

Reference windows hold updates n0-1 .. n1-1 (JAXAgent returns the previous call's metrics); Ajax windows n0 .. n1. Loss names mapped: vector_loss->rec_loss, reward_loss->rew_loss, cont_loss->con_loss, replay_critic_loss->repval_loss.

window ending at 2000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.4052 [0.3851, 0.4279] 0.4264 [0.3866, 0.4518] 1.052
rew_loss 5.2850 [5.2790, 5.2944] 5.2867 [5.2816, 5.2906] 1.000
con_loss 0.3908 [0.3656, 0.4087] 0.3454 [0.3227, 0.3630] 0.884
dyn_loss 2.5956 [2.5333, 2.7120] 2.3860 [2.1729, 2.6847] 0.919
rep_loss 2.5956 [2.5333, 2.7120] 2.3860 [2.1729, 2.6847] 0.919
actor_loss -0.0076 [-0.0090, -0.0068] -0.0096 [-0.0106, -0.0087] 1.256
critic_loss 3.9150 [3.7395, 4.0730] 3.9265 [3.7747, 4.0231] 1.003
repval_loss 10.2157 [10.1804, 10.2589] 10.1612 [10.1355, 10.1844] 0.995
ent/action/mean 0.6931 [0.6930, 0.6931] 0.6921 [0.6909, 0.6928] 0.999
rand/action/mean 0.9999 [0.9998, 1.0000] 0.9985 [0.9967, 0.9994] 0.999
ret/mean 0.0700 [0.0685, 0.0721] 0.0758 [0.0730, 0.0793] 1.083
ret/std 0.1351 [0.1330, 0.1377] 0.0099 [0.0091, 0.0109] 0.073
ret_normed/std 0.1351 [0.1330, 0.1377] 0.0099 [0.0091, 0.0109] 0.073
val/mean 0.0908 [0.0866, 0.0973] 0.1004 [0.0965, 0.1075] 1.107
adv/std 0.1603 [0.1585, 0.1624] 0.0101 [0.0095, 0.0110] 0.063
replay_ret/mean 7.9601 [7.5991, 8.2409] 7.9780 [7.8962, 8.1061] 1.002
weight/mean 0.3624 [0.3470, 0.3762] 0.3642 [0.3512, 0.3721] 1.005
rew/mean 0.0618 [0.0616, 0.0621] 0.0624 [0.0623, 0.0625] 1.009
prior_ent/mean 43.3120 [43.1853, 43.5035] 43.3131 [43.2701, 43.3656] 1.000
post_ent/mean 41.9819 [41.6774, 42.3989] 42.3208 [42.1189, 42.4362] 1.008
td_error 0.1293 [0.1285, 0.1306] 0.0246 [0.0223, 0.0281] 0.191
data_rew/mean 0.9573 [0.9536, 0.9602] 0.9562 [0.9556, 0.9574] 0.999
pred_rew/mean 0.0618 [0.0616, 0.0621] 0.0624 [0.0623, 0.0625] 1.009
activation/embed 0.4287 [0.4196, 0.4388] 0.3867 [0.3754, 0.3931] 0.902
opt_loss 15.9082 [15.6850, 16.1787] 15.6484 [15.2804, 16.0810] 0.984
opt_grad_norm 148.1683 [140.3558, 160.4136] 152.1094 [131.6112, 191.8375] 1.027
opt_update_norm 0.0070 [0.0069, 0.0070] 0.0072 [0.0072, 0.0073] 1.036
opt_param_norm 59.8880 [59.8753, 59.9078] 59.8509 [59.7766, 59.9060] 0.999
opt_grad_steps 240.5000 [240.5000, 240.5000] 241.0000 [241.0000, 241.0000] 1.002
retnorm_scale_derived 1.0000 [1.0000, 1.0000] 1.0000 [1.0000, 1.0000] 1.000
retnorm_scale_exact 1.0000 [1.0000, 1.0000] nan [nan, nan] nan

window ending at 4000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0634 [0.0575, 0.0667] 0.0653 [0.0605, 0.0736] 1.030
rew_loss 2.4989 [2.4825, 2.5198] 2.5103 [2.4903, 2.5275] 1.005
con_loss 0.1502 [0.1413, 0.1554] 0.1529 [0.1510, 0.1554] 1.018
dyn_loss 1.0111 [1.0106, 1.0118] 1.0094 [1.0086, 1.0104] 0.998
rep_loss 1.0111 [1.0106, 1.0118] 1.0094 [1.0086, 1.0104] 0.998
actor_loss 0.1669 [0.1608, 0.1753] 0.1488 [0.1455, 0.1520] 0.891
critic_loss 5.1856 [5.0536, 5.2972] 5.5446 [5.4760, 5.6279] 1.069
repval_loss 7.2307 [7.2091, 7.2731] 7.4848 [7.4255, 7.5486] 1.035
ent/action/mean 0.5135 [0.5056, 0.5178] 0.4565 [0.4485, 0.4611] 0.889
rand/action/mean 0.7408 [0.7294, 0.7470] 0.6586 [0.6470, 0.6652] 0.889
ret/mean 3.3319 [3.2891, 3.3857] 3.2713 [3.2168, 3.3266] 0.982
ret/std 0.8487 [0.7955, 0.8916] 0.7844 [0.7489, 0.8179] 0.924
ret_normed/std 0.3273 [0.3207, 0.3319] 0.2796 [0.2674, 0.2863] 0.854
val/mean 1.8324 [1.7883, 1.8666] 1.7864 [1.7566, 1.8313] 0.975
adv/std 0.3342 [0.3276, 0.3387] 0.2796 [0.2674, 0.2862] 0.837
replay_ret/mean 10.4500 [10.0243, 10.8171] 10.3835 [10.2506, 10.5989] 0.994
weight/mean 0.7315 [0.7136, 0.7444] 0.7314 [0.7292, 0.7344] 1.000
rew/mean 0.3523 [0.3473, 0.3566] 0.3483 [0.3438, 0.3538] 0.989
prior_ent/mean 43.5344 [43.4195, 43.6230] 43.5896 [43.3493, 43.7277] 1.001
post_ent/mean 42.5926 [42.3925, 42.7287] 42.6449 [42.3132, 42.8424] 1.001
td_error 1.5689 [1.5418, 1.5840] 1.5174 [1.4662, 1.5504] 0.967
data_rew/mean 0.9590 [0.9558, 0.9618] 0.9581 [0.9569, 0.9601] 0.999
pred_rew/mean 0.3523 [0.3473, 0.3566] 0.3483 [0.3438, 0.3538] 0.989
activation/embed 0.4307 [0.4203, 0.4379] 0.3911 [0.3842, 0.4010] 0.908
opt_loss 11.3465 [11.1974, 11.4452] 11.7777 [11.6617, 11.8918] 1.038
opt_grad_norm 8.5400 [8.0966, 9.0237] 8.6889 [8.3670, 9.0024] 1.017
opt_update_norm 0.0134 [0.0133, 0.0134] 0.0140 [0.0138, 0.0143] 1.048
opt_param_norm 60.5103 [60.4985, 60.5337] 60.5191 [60.4579, 60.5654] 1.000
opt_grad_steps 980.5000 [980.5000, 980.5000] 981.5000 [981.5000, 981.5000] 1.001
retnorm_scale_derived 2.5915 [2.4807, 2.6863] 2.8050 [2.7469, 2.8676] 1.082
retnorm_scale_exact 6.1737 [5.9708, 6.3542] nan [nan, nan] nan

window ending at 6000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0199 [0.0169, 0.0215] 0.0209 [0.0192, 0.0237] 1.051
rew_loss 0.8086 [0.8033, 0.8130] 0.8058 [0.8017, 0.8111] 0.997
con_loss 0.0788 [0.0735, 0.0824] 0.0826 [0.0821, 0.0836] 1.048
dyn_loss 1.0365 [1.0332, 1.0384] 1.0360 [1.0341, 1.0391] 0.999
rep_loss 1.0365 [1.0332, 1.0384] 1.0360 [1.0341, 1.0391] 0.999
actor_loss 0.1739 [0.1717, 0.1768] 0.1502 [0.1460, 0.1554] 0.864
critic_loss 4.0891 [4.0552, 4.1458] 4.0536 [3.9570, 4.1150] 0.991
repval_loss 6.0414 [6.0160, 6.0878] 5.9926 [5.9322, 6.0432] 0.992
ent/action/mean 0.3004 [0.2937, 0.3087] 0.2658 [0.2575, 0.2701] 0.885
rand/action/mean 0.4333 [0.4238, 0.4454] 0.3835 [0.3715, 0.3896] 0.885
ret/mean 13.5435 [13.3726, 13.6849] 13.5813 [13.5365, 13.6476] 1.003
ret/std 2.8645 [2.7987, 2.9120] 2.9491 [2.8584, 3.1098] 1.030
ret_normed/std 0.3720 [0.3685, 0.3777] 0.3781 [0.3715, 0.3878] 1.016
val/mean 8.6778 [8.5062, 8.7826] 8.8127 [8.7020, 9.0187] 1.016
adv/std 0.3718 [0.3680, 0.3776] 0.3778 [0.3714, 0.3874] 1.016
replay_ret/mean 20.4433 [19.7653, 21.2768] 20.2949 [20.0715, 20.6341] 0.993
weight/mean 0.8648 [0.8557, 0.8766] 0.8592 [0.8488, 0.8670] 0.993
rew/mean 0.9060 [0.9034, 0.9077] 0.9053 [0.9033, 0.9076] 0.999
prior_ent/mean 42.0547 [41.6753, 42.3235] 42.1852 [41.5852, 42.4880] 1.003
post_ent/mean 41.0193 [40.6158, 41.3073] 41.2104 [40.4710, 41.5911] 1.005
td_error 5.0738 [5.0173, 5.1070] 5.0347 [5.0007, 5.0780] 0.992
data_rew/mean 0.9709 [0.9685, 0.9738] 0.9705 [0.9696, 0.9720] 1.000
pred_rew/mean 0.9060 [0.9034, 0.9077] 0.9053 [0.9033, 0.9076] 0.999
activation/embed 0.4296 [0.4194, 0.4476] 0.3935 [0.3854, 0.4088] 0.916
opt_loss 8.1229 [8.0898, 8.1853] 8.0505 [7.9333, 8.1283] 0.991
opt_grad_norm 7.7420 [7.6290, 7.8142] 7.5068 [7.0433, 8.0157] 0.970
opt_update_norm 0.0081 [0.0077, 0.0083] 0.0079 [0.0078, 0.0081] 0.986
opt_param_norm 61.7567 [61.7447, 61.7782] 61.8299 [61.7560, 61.8739] 1.001
opt_grad_steps 1980.5000 [1980.5000, 1980.5000] 1981.5000 [1981.5000, 1981.5000] 1.001
retnorm_scale_derived 7.7000 [7.5955, 7.7946] 7.7976 [7.6791, 8.0191] 1.013
retnorm_scale_exact 8.4072 [8.3665, 8.4748] nan [nan, nan] nan

window ending at 8000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0152 [0.0143, 0.0168] 0.0160 [0.0150, 0.0173] 1.053
rew_loss 0.6922 [0.6910, 0.6928] 0.6915 [0.6900, 0.6932] 0.999
con_loss 0.0555 [0.0533, 0.0568] 0.0570 [0.0558, 0.0579] 1.028
dyn_loss 1.0576 [1.0553, 1.0616] 1.0566 [1.0445, 1.0652] 0.999
rep_loss 1.0576 [1.0553, 1.0616] 1.0566 [1.0445, 1.0652] 0.999
actor_loss 0.1542 [0.1525, 0.1566] 0.1323 [0.1263, 0.1356] 0.858
critic_loss 3.2846 [3.1826, 3.3953] 3.1401 [3.0905, 3.1805] 0.956
repval_loss 5.4217 [5.2938, 5.4997] 5.2863 [5.2558, 5.3350] 0.975
ent/action/mean 0.2649 [0.2605, 0.2690] 0.2287 [0.2160, 0.2368] 0.863
rand/action/mean 0.3822 [0.3759, 0.3881] 0.3299 [0.3117, 0.3416] 0.863
ret/mean 21.5226 [20.6922, 22.0487] 22.2497 [22.1054, 22.4649] 1.034
ret/std 3.4484 [3.4081, 3.4947] 3.6700 [3.5701, 3.8527] 1.064
ret_normed/std 0.4049 [0.3998, 0.4114] 0.4270 [0.4138, 0.4478] 1.055
val/mean 16.2671 [15.5464, 16.7231] 17.0731 [16.8829, 17.3766] 1.050
adv/std 0.3925 [0.3754, 0.4112] 0.4119 [0.4082, 0.4174] 1.050
replay_ret/mean 29.1729 [27.9417, 29.9898] 29.4273 [28.8639, 30.0003] 1.009
weight/mean 0.9055 [0.8967, 0.9117] 0.9034 [0.8957, 0.9111] 0.998
rew/mean 0.9716 [0.9710, 0.9723] 0.9718 [0.9712, 0.9725] 1.000
prior_ent/mean 40.0760 [39.5168, 40.4287] 40.0529 [38.7624, 40.8279] 0.999
post_ent/mean 38.7531 [38.2004, 39.1636] 38.8630 [37.4371, 39.6257] 1.003
td_error 5.5894 [5.5655, 5.6042] 5.6239 [5.5903, 5.6449] 1.006
data_rew/mean 0.9797 [0.9779, 0.9815] 0.9789 [0.9778, 0.9804] 0.999
pred_rew/mean 0.9716 [0.9710, 0.9723] 0.9718 [0.9712, 0.9725] 1.000
activation/embed 0.4199 [0.4071, 0.4368] 0.3901 [0.3807, 0.4054] 0.929
opt_loss 6.9915 [6.8485, 7.1238] 6.7850 [6.7356, 6.8454] 0.970
opt_grad_norm 8.2028 [7.5048, 9.1159] 7.1955 [6.3448, 7.6734] 0.877
opt_update_norm 0.0109 [0.0095, 0.0124] 0.0087 [0.0078, 0.0093] 0.799
opt_param_norm 62.4969 [62.4740, 62.5120] 62.5264 [62.4456, 62.5727] 1.000
opt_grad_steps 2980.5000 [2980.5000, 2980.5000] 2981.5000 [2981.5000, 2981.5000] 1.000
retnorm_scale_derived 8.5164 [8.4942, 8.5304] 8.5938 [8.5509, 8.6268] 1.009
retnorm_scale_exact 8.3140 [8.2107, 8.4424] nan [nan, nan] nan

window ending at 10000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0229 [0.0201, 0.0273] 0.0171 [0.0166, 0.0177] 0.750
rew_loss 0.6712 [0.6702, 0.6719] 0.6717 [0.6710, 0.6731] 1.001
con_loss 0.0546 [0.0502, 0.0590] 0.0519 [0.0497, 0.0541] 0.951
dyn_loss 1.0845 [1.0753, 1.0946] 1.0810 [1.0521, 1.0963] 0.997
rep_loss 1.0845 [1.0753, 1.0946] 1.0810 [1.0521, 1.0963] 0.997
actor_loss 0.1347 [0.1052, 0.1831] 0.1308 [0.1115, 0.1567] 0.971
critic_loss 3.3112 [3.1539, 3.4883] 2.8739 [2.8098, 2.9514] 0.868
repval_loss 5.4033 [5.2477, 5.4933] 5.1610 [4.7928, 5.3489] 0.955
ent/action/mean 0.2793 [0.2549, 0.3247] 0.2360 [0.2174, 0.2701] 0.845
rand/action/mean 0.4029 [0.3677, 0.4684] 0.3405 [0.3136, 0.3897] 0.845
ret/mean 25.1384 [24.5147, 25.6124] 29.5013 [29.0529, 29.9280] 1.174
ret/std 3.9535 [3.6868, 4.2350] 3.8739 [3.7988, 3.9985] 0.980
ret_normed/std 0.3972 [0.3742, 0.4208] 0.4364 [0.4338, 0.4380] 1.099
val/mean 20.0203 [19.5719, 20.4610] 24.1270 [23.6146, 24.5681] 1.205
adv/std 0.3637 [0.3393, 0.3989] 0.3971 [0.3931, 0.4013] 1.092
replay_ret/mean 33.2857 [32.6846, 33.9585] 36.9148 [36.5909, 37.1556] 1.109
weight/mean 0.9080 [0.9022, 0.9158] 0.9154 [0.9110, 0.9193] 1.008
rew/mean 0.9863 [0.9854, 0.9871] 0.9862 [0.9858, 0.9866] 1.000
prior_ent/mean 36.5890 [35.9630, 37.4125] 37.8955 [37.3968, 38.5125] 1.036
post_ent/mean 34.8418 [34.1144, 35.7122] 36.5290 [35.8578, 37.3137] 1.048
td_error 5.4696 [5.2925, 5.6156] 5.6590 [5.6107, 5.6847] 1.035
data_rew/mean 0.9835 [0.9817, 0.9856] 0.9839 [0.9827, 0.9850] 1.000
pred_rew/mean 0.9863 [0.9854, 0.9871] 0.9862 [0.9858, 0.9866] 1.000
activation/embed 0.4037 [0.3864, 0.4141] 0.3796 [0.3616, 0.3945] 0.940
opt_loss 7.0085 [6.8491, 7.2056] 6.4828 [6.3038, 6.6267] 0.925
opt_grad_norm 13.5940 [11.8704, 14.8143] 9.3156 [8.1343, 10.5228] 0.685
opt_update_norm 0.0124 [0.0116, 0.0131] 0.0119 [0.0103, 0.0128] 0.956
opt_param_norm 63.3664 [63.3330, 63.3869] 63.2954 [63.2447, 63.3218] 0.999
opt_grad_steps 3980.5000 [3980.5000, 3980.5000] 3981.5000 [3981.5000, 3981.5000] 1.000
retnorm_scale_derived 10.0034 [8.7622, 11.3176] 8.8773 [8.6723, 9.1441] 0.887
retnorm_scale_exact 12.3124 [9.2841, 14.9662] nan [nan, nan] nan

window ending at 12000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0244 [0.0170, 0.0315] 0.0205 [0.0174, 0.0236] 0.839
rew_loss 0.6636 [0.6628, 0.6647] 0.6643 [0.6636, 0.6650] 1.001
con_loss 0.0522 [0.0496, 0.0568] 0.0504 [0.0468, 0.0537] 0.967
dyn_loss 1.0864 [1.0671, 1.1081] 1.0869 [1.0753, 1.0955] 1.001
rep_loss 1.0864 [1.0671, 1.1081] 1.0869 [1.0753, 1.0955] 1.001
actor_loss 0.0979 [0.0780, 0.1207] 0.1095 [0.0929, 0.1333] 1.119
critic_loss 3.1750 [2.6354, 3.7897] 2.5396 [2.5048, 2.5626] 0.800
repval_loss 5.0965 [4.7109, 5.5176] 4.6666 [4.4581, 4.8484] 0.916
ent/action/mean 0.2777 [0.2609, 0.3062] 0.2133 [0.2047, 0.2283] 0.768
rand/action/mean 0.4007 [0.3764, 0.4417] 0.3077 [0.2954, 0.3294] 0.768
ret/mean 30.9380 [26.9331, 34.9688] 38.4030 [37.6805, 39.5081] 1.241
ret/std 5.0400 [4.3669, 5.5656] 4.5209 [4.2877, 4.8910] 0.897
ret_normed/std 0.3581 [0.3150, 0.4079] 0.4537 [0.4116, 0.4962] 1.267
val/mean 25.7464 [21.7653, 29.9210] 32.8359 [32.1808, 33.7802] 1.275
adv/std 0.3049 [0.2342, 0.3406] 0.3840 [0.3068, 0.4295] 1.259
replay_ret/mean 38.7323 [36.8057, 41.3513] 45.0535 [43.5066, 46.5038] 1.163
weight/mean 0.9201 [0.9158, 0.9286] 0.9257 [0.9203, 0.9286] 1.006
rew/mean 0.9933 [0.9931, 0.9935] 0.9930 [0.9927, 0.9934] 1.000
prior_ent/mean 35.3246 [34.3087, 36.2890] 36.5117 [35.7166, 37.3028] 1.034
post_ent/mean 33.4355 [31.8327, 34.8118] 35.0096 [34.1714, 35.8994] 1.047
td_error 5.7407 [5.6781, 5.7982] 5.8297 [5.7252, 5.8845] 1.015
data_rew/mean 0.9858 [0.9847, 0.9877] 0.9864 [0.9851, 0.9870] 1.001
pred_rew/mean 0.9933 [0.9931, 0.9935] 0.9930 [0.9927, 0.9934] 1.000
activation/embed 0.3982 [0.3814, 0.4120] 0.3798 [0.3629, 0.3928] 0.954
opt_loss 6.7370 [6.0472, 7.5038] 5.9800 [5.9199, 6.0433] 0.888
opt_grad_norm 13.6046 [10.9870, 16.4320] 10.2577 [9.8210, 10.5847] 0.754
opt_update_norm 0.0105 [0.0098, 0.0113] 0.0107 [0.0101, 0.0116] 1.028
opt_param_norm 64.2743 [64.2112, 64.3360] 64.1683 [64.0784, 64.2904] 0.998
opt_grad_steps 4980.5000 [4980.5000, 4980.5000] 4981.5000 [4981.5000, 4981.5000] 1.000
retnorm_scale_derived 14.1795 [12.4272, 16.4680] 10.0659 [8.6411, 11.8821] 0.710
retnorm_scale_exact 17.6708 [13.4947, 25.6072] nan [nan, nan] nan

window ending at 14000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0205 [0.0172, 0.0237] 0.0213 [0.0174, 0.0252] 1.038
rew_loss 0.6609 [0.6600, 0.6620] 0.6616 [0.6608, 0.6622] 1.001
con_loss 0.0491 [0.0447, 0.0517] 0.0486 [0.0442, 0.0529] 0.990
dyn_loss 1.0887 [1.0852, 1.0923] 1.0866 [1.0816, 1.0947] 0.998
rep_loss 1.0887 [1.0852, 1.0923] 1.0866 [1.0816, 1.0947] 0.998
actor_loss 0.0477 [0.0303, 0.0667] 0.0665 [0.0307, 0.0865] 1.395
critic_loss 2.4061 [2.2681, 2.6215] 2.3058 [2.1927, 2.4777] 0.958
repval_loss 4.3971 [4.2212, 4.6315] 4.5631 [4.3203, 4.8157] 1.038
ent/action/mean 0.2116 [0.1988, 0.2190] 0.1988 [0.1681, 0.2465] 0.939
rand/action/mean 0.3053 [0.2869, 0.3159] 0.2867 [0.2425, 0.3557] 0.939
ret/mean 41.8206 [38.7728, 45.1628] 46.7378 [45.5936, 48.1135] 1.118
ret/std 7.6580 [5.8207, 10.2459] 6.3234 [5.1351, 7.9023] 0.826
ret_normed/std 0.3348 [0.3224, 0.3497] 0.4222 [0.3209, 0.5266] 1.261
val/mean 36.4418 [33.1599, 40.0833] 41.3328 [39.9229, 42.7495] 1.134
adv/std 0.1811 [0.1458, 0.2191] 0.3155 [0.1984, 0.4527] 1.742
replay_ret/mean 48.2353 [47.0644, 50.0910] 52.7677 [51.5549, 54.4375] 1.094
weight/mean 0.9304 [0.9206, 0.9406] 0.9299 [0.9266, 0.9319] 0.999
rew/mean 0.9961 [0.9960, 0.9962] 0.9960 [0.9959, 0.9962] 1.000
prior_ent/mean 35.5475 [34.8418, 36.5772] 35.3672 [34.4461, 36.2464] 0.995
post_ent/mean 33.9853 [33.0962, 35.0949] 33.8216 [32.7594, 34.7699] 0.995
td_error 5.6699 [5.5841, 5.8241] 5.7563 [5.5178, 6.0344] 1.015
data_rew/mean 0.9872 [0.9859, 0.9890] 0.9880 [0.9868, 0.9887] 1.001
pred_rew/mean 0.9961 [0.9960, 0.9962] 0.9960 [0.9959, 0.9962] 1.000
activation/embed 0.3973 [0.3781, 0.4086] 0.3784 [0.3632, 0.3878] 0.953
opt_loss 5.7011 [5.4912, 6.0018] 5.6680 [5.5184, 5.8712] 0.994
opt_grad_norm 11.6176 [10.4261, 12.5588] 12.2113 [10.4470, 14.5420] 1.051
opt_update_norm 0.0093 [0.0089, 0.0100] 0.0104 [0.0098, 0.0111] 1.114
opt_param_norm 65.2772 [65.1939, 65.3511] 65.1407 [65.0182, 65.2818] 0.998
opt_grad_steps 5980.5000 [5980.5000, 5980.5000] 5981.5000 [5981.5000, 5981.5000] 1.000
retnorm_scale_derived 23.0706 [16.6470, 31.7807] 16.1778 [9.7505, 24.6291] 0.701
retnorm_scale_exact 27.9082 [19.5583, 38.6512] nan [nan, nan] nan

window ending at 16000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0231 [0.0182, 0.0329] 0.0193 [0.0147, 0.0234] 0.832
rew_loss 0.6603 [0.6597, 0.6613] 0.6604 [0.6598, 0.6607] 1.000
con_loss 0.0479 [0.0470, 0.0492] 0.0480 [0.0436, 0.0512] 1.002
dyn_loss 1.0922 [1.0854, 1.0961] 1.1045 [1.0989, 1.1104] 1.011
rep_loss 1.0922 [1.0854, 1.0961] 1.1045 [1.0989, 1.1104] 1.011
actor_loss 0.0337 [0.0145, 0.0557] 0.0329 [0.0197, 0.0519] 0.976
critic_loss 2.2684 [1.9555, 2.7569] 1.9897 [1.9344, 2.0729] 0.877
repval_loss 4.1718 [3.8318, 4.6589] 4.1513 [3.9327, 4.4051] 0.995
ent/action/mean 0.1939 [0.1501, 0.2250] 0.1500 [0.1350, 0.1608] 0.774
rand/action/mean 0.2797 [0.2165, 0.3247] 0.2164 [0.1948, 0.2320] 0.774
ret/mean 51.6292 [42.3982, 59.8716] 57.5719 [56.7033, 58.0750] 1.115
ret/std 10.3261 [6.2004, 14.8583] 8.6063 [5.1580, 11.6254] 0.833
ret_normed/std 0.3197 [0.3139, 0.3235] 0.3863 [0.3310, 0.4908] 1.208
val/mean 46.0847 [36.6766, 54.3872] 52.2728 [51.3622, 52.8784] 1.134
adv/std 0.1589 [0.1189, 0.2152] 0.2629 [0.1277, 0.4912] 1.654
replay_ret/mean 57.2003 [50.8359, 63.3076] 62.4652 [60.7286, 63.8527] 1.092
weight/mean 0.9365 [0.9313, 0.9423] 0.9370 [0.9314, 0.9445] 1.000
rew/mean 0.9975 [0.9974, 0.9977] 0.9975 [0.9975, 0.9976] 1.000
prior_ent/mean 35.2954 [34.1937, 36.7061] 35.2595 [33.9141, 36.9597] 0.999
post_ent/mean 33.7274 [32.3779, 35.2964] 33.7996 [32.2987, 35.6512] 1.002
td_error 5.8187 [5.6493, 5.9452] 5.7009 [5.6041, 5.8640] 0.980
data_rew/mean 0.9890 [0.9880, 0.9905] 0.9890 [0.9882, 0.9895] 1.000
pred_rew/mean 0.9975 [0.9974, 0.9977] 0.9975 [0.9975, 0.9976] 1.000
activation/embed 0.3990 [0.3762, 0.4109] 0.3807 [0.3690, 0.3924] 0.954
opt_loss 5.4864 [5.0390, 6.1562] 5.2105 [5.1074, 5.3846] 0.950
opt_grad_norm 13.1151 [9.9690, 18.0323] 11.5522 [10.4987, 12.6389] 0.881
opt_update_norm 0.0096 [0.0092, 0.0100] 0.0101 [0.0097, 0.0104] 1.053
opt_param_norm 66.2621 [66.1968, 66.3103] 66.0416 [65.8934, 66.3352] 0.997
opt_grad_steps 6980.5000 [6980.5000, 6980.5000] 6981.5000 [6981.5000, 6981.5000] 1.000
retnorm_scale_derived 32.1720 [19.7513, 45.9352] 24.1433 [10.5097, 35.1233] 0.750
retnorm_scale_exact 35.5267 [19.0161, 52.6481] nan [nan, nan] nan

window ending at 18000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0218 [0.0196, 0.0239] 0.0184 [0.0129, 0.0248] 0.843
rew_loss 0.6597 [0.6591, 0.6604] 0.6598 [0.6596, 0.6602] 1.000
con_loss 0.0483 [0.0457, 0.0496] 0.0470 [0.0444, 0.0489] 0.973
dyn_loss 1.1036 [1.0831, 1.1229] 1.1099 [1.0992, 1.1192] 1.006
rep_loss 1.1036 [1.0831, 1.1229] 1.1099 [1.0992, 1.1192] 1.006
actor_loss 0.0196 [0.0095, 0.0279] 0.0216 [0.0104, 0.0379] 1.105
critic_loss 1.9494 [1.7754, 2.1543] 1.8503 [1.8148, 1.9143] 0.949
repval_loss 3.8488 [3.6533, 4.1994] 3.8770 [3.5701, 4.0893] 1.007
ent/action/mean 0.1529 [0.1315, 0.1732] 0.1352 [0.1307, 0.1406] 0.884
rand/action/mean 0.2207 [0.1897, 0.2499] 0.1950 [0.1886, 0.2028] 0.884
ret/mean 63.8287 [54.5444, 70.5577] 67.7322 [65.7089, 70.0568] 1.061
ret/std 12.8770 [8.2869, 18.7773] 10.9500 [6.2882, 16.1412] 0.850
ret_normed/std 0.3290 [0.3270, 0.3305] 0.3525 [0.3257, 0.4029] 1.071
val/mean 58.5918 [49.2876, 65.2099] 62.6696 [60.6629, 65.1633] 1.070
adv/std 0.1353 [0.1041, 0.1628] 0.1969 [0.1050, 0.3257] 1.455
replay_ret/mean 67.4472 [61.0886, 72.1202] 71.1455 [69.0065, 73.5685] 1.055
weight/mean 0.9383 [0.9337, 0.9423] 0.9385 [0.9361, 0.9398] 1.000
rew/mean 0.9983 [0.9982, 0.9984] 0.9983 [0.9982, 0.9984] 1.000
prior_ent/mean 35.0959 [34.3216, 36.6028] 34.8135 [34.4458, 35.4919] 0.992
post_ent/mean 33.6875 [32.8737, 35.2232] 33.4034 [33.0307, 34.1167] 0.992
td_error 5.6640 [5.4834, 5.9300] 5.6056 [5.5147, 5.6763] 0.990
data_rew/mean 0.9895 [0.9886, 0.9906] 0.9897 [0.9893, 0.9902] 1.000
pred_rew/mean 0.9983 [0.9982, 0.9984] 0.9983 [0.9982, 0.9984] 1.000
activation/embed 0.4046 [0.3866, 0.4174] 0.3842 [0.3720, 0.3979] 0.949
opt_loss 5.0673 [4.8687, 5.3614] 4.9811 [4.8552, 5.0781] 0.983
opt_grad_norm 11.5554 [9.9091, 13.6849] 9.8298 [9.2521, 10.2911] 0.851
opt_update_norm 0.0103 [0.0091, 0.0110] 0.0104 [0.0094, 0.0110] 1.012
opt_param_norm 67.0005 [66.8052, 67.1753] 66.7709 [66.4631, 67.2166] 0.997
opt_grad_steps 7980.5000 [7980.5000, 7980.5000] 7981.5000 [7981.5000, 7981.5000] 1.000
retnorm_scale_derived 39.1047 [25.3414, 56.9720] 32.2216 [15.6079, 49.0654] 0.824
retnorm_scale_exact 41.3496 [28.8141, 55.6319] nan [nan, nan] nan

window ending at 20000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0232 [0.0201, 0.0277] 0.0186 [0.0144, 0.0230] 0.802
rew_loss 0.6598 [0.6593, 0.6604] 0.6598 [0.6595, 0.6604] 1.000
con_loss 0.0475 [0.0449, 0.0503] 0.0463 [0.0449, 0.0487] 0.973
dyn_loss 1.1045 [1.0906, 1.1158] 1.1120 [1.0854, 1.1319] 1.007
rep_loss 1.1045 [1.0906, 1.1158] 1.1120 [1.0854, 1.1319] 1.007
actor_loss 0.0117 [0.0084, 0.0155] 0.0157 [0.0088, 0.0214] 1.346
critic_loss 1.8011 [1.6800, 1.9658] 1.8680 [1.7326, 2.0645] 1.037
repval_loss 3.5680 [3.3225, 3.8618] 3.6394 [3.2858, 3.8443] 1.020
ent/action/mean 0.1418 [0.1309, 0.1602] 0.1442 [0.1354, 0.1585] 1.017
rand/action/mean 0.2045 [0.1888, 0.2311] 0.2080 [0.1954, 0.2287] 1.017
ret/mean 74.6112 [62.3665, 81.9878] 74.0936 [68.1780, 78.8971] 0.993
ret/std 16.7543 [13.0293, 21.1412] 14.6161 [10.2626, 20.1408] 0.872
ret_normed/std 0.3359 [0.3322, 0.3380] 0.3495 [0.3330, 0.3785] 1.041
val/mean 69.7412 [57.7651, 77.0393] 69.0034 [63.0574, 74.0279] 0.989
adv/std 0.1128 [0.0983, 0.1234] 0.1565 [0.0918, 0.2213] 1.388
replay_ret/mean 77.1496 [68.1577, 82.2093] 76.9562 [71.4557, 81.9675] 0.997
weight/mean 0.9409 [0.9377, 0.9430] 0.9403 [0.9376, 0.9437] 0.999
rew/mean 0.9988 [0.9988, 0.9989] 0.9988 [0.9987, 0.9988] 1.000
prior_ent/mean 34.6428 [33.6872, 36.0757] 34.5989 [32.9537, 35.6120] 0.999
post_ent/mean 33.2464 [32.2440, 34.7589] 33.0877 [31.0357, 34.3483] 0.995
td_error 5.5760 [5.2719, 5.8094] 5.8345 [5.6067, 6.0531] 1.046
data_rew/mean 0.9905 [0.9898, 0.9914] 0.9904 [0.9899, 0.9914] 1.000
pred_rew/mean 0.9988 [0.9988, 0.9989] 0.9988 [0.9987, 0.9988] 1.000
activation/embed 0.4100 [0.3957, 0.4246] 0.3886 [0.3753, 0.4055] 0.948
opt_loss 4.8287 [4.7015, 5.0667] 4.9235 [4.7665, 5.1797] 1.020
opt_grad_norm 11.0210 [9.4156, 12.2078] 11.2106 [9.7753, 13.1750] 1.017
opt_update_norm 0.0110 [0.0104, 0.0117] 0.0104 [0.0101, 0.0107] 0.940
opt_param_norm 67.6027 [67.3803, 67.8515] 67.3097 [66.8619, 67.8456] 0.996
opt_grad_steps 8980.5000 [8980.5000, 8980.5000] 8981.5000 [8981.5000, 8981.5000] 1.000
retnorm_scale_derived 49.8165 [39.2173, 62.5471] 42.4144 [27.1152, 59.7552] 0.851
retnorm_scale_exact 55.5971 [44.9350, 65.0703] 52.6593 [42.5683, 69.0615] 0.947

Training metrics per seed (ref / ajax)

seed 0

metric 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
rec_loss 0.402 / 0.452 0.067 / 0.074 0.021 / 0.019 0.015 / 0.015 0.020 / 0.017 0.025 / 0.024 0.021 / 0.025 0.018 / 0.023 0.020 / 0.025 0.020 / 0.023
rew_loss 5.294 / 5.291 2.520 / 2.513 0.813 / 0.805 0.693 / 0.691 0.672 / 0.671 0.663 / 0.664 0.661 / 0.661 0.660 / 0.660 0.660 / 0.660 0.660 / 0.659
con_loss 0.409 / 0.323 0.154 / 0.155 0.082 / 0.084 0.056 / 0.058 0.059 / 0.054 0.057 / 0.054 0.051 / 0.053 0.049 / 0.051 0.050 / 0.049 0.047 / 0.049
dyn_loss 2.533 / 2.685 1.011 / 1.010 1.038 / 1.034 1.055 / 1.044 1.075 / 1.052 1.108 / 1.075 1.092 / 1.095 1.096 / 1.099 1.123 / 1.111 1.116 / 1.119
rep_loss 2.533 / 2.685 1.011 / 1.010 1.038 / 1.034 1.055 / 1.044 1.075 / 1.052 1.108 / 1.075 1.092 / 1.095 1.096 / 1.099 1.123 / 1.111 1.116 / 1.119
actor_loss -0.009 / -0.009 0.175 / 0.152 0.173 / 0.155 0.154 / 0.135 0.105 / 0.157 0.078 / 0.102 0.046 / 0.087 0.031 / 0.027 0.021 / 0.017 0.011 / 0.017
critic_loss 3.740 / 4.023 5.206 / 5.530 4.055 / 4.089 3.183 / 3.149 3.291 / 2.810 3.100 / 2.563 2.329 / 2.193 2.093 / 1.934 1.775 / 1.914 1.680 / 2.065
repval_loss 10.208 / 10.164 7.210 / 7.480 6.020 / 6.002 5.294 / 5.256 5.469 / 4.793 5.061 / 4.848 4.339 / 4.320 4.025 / 4.116 3.694 / 3.971 3.520 / 3.788
ent/action/mean 0.693 / 0.693 0.517 / 0.460 0.299 / 0.270 0.261 / 0.233 0.258 / 0.270 0.266 / 0.205 0.199 / 0.247 0.207 / 0.161 0.173 / 0.134 0.131 / 0.159
rand/action/mean 1.000 / 0.999 0.746 / 0.663 0.431 / 0.389 0.376 / 0.337 0.373 / 0.390 0.384 / 0.295 0.287 / 0.356 0.298 / 0.232 0.250 / 0.194 0.189 / 0.229
ret/mean 0.072 / 0.073 3.321 / 3.271 13.573 / 13.560 22.049 / 22.105 25.612 / 29.928 30.912 / 37.680 41.526 / 46.506 52.618 / 56.703 66.384 / 65.709 79.479 / 68.178
ret/std 0.133 / 0.010 0.796 / 0.786 2.799 / 2.879 3.442 / 3.587 4.235 / 3.799 5.187 / 4.384 6.907 / 5.933 9.920 / 9.035 11.567 / 10.421 16.092 / 13.445
ret_normed/std 0.133 / 0.010 0.321 / 0.286 0.368 / 0.375 0.404 / 0.420 0.374 / 0.438 0.315 / 0.453 0.332 / 0.419 0.322 / 0.337 0.330 / 0.326 0.337 / 0.333
val/mean 0.097 / 0.097 1.842 / 1.757 8.745 / 8.702 16.723 / 16.883 20.461 / 24.568 25.553 / 32.181 36.082 / 41.326 47.190 / 51.362 61.278 / 60.663 74.419 / 63.057
adv/std 0.159 / 0.010 0.328 / 0.286 0.368 / 0.375 0.375 / 0.408 0.339 / 0.397 0.234 / 0.429 0.178 / 0.295 0.143 / 0.170 0.139 / 0.160 0.117 / 0.157
replay_ret/mean 8.040 / 7.896 10.508 / 10.251 20.288 / 20.072 29.587 / 28.864 33.214 / 36.591 38.040 / 43.507 47.550 / 51.555 57.457 / 60.729 69.133 / 69.007 81.082 / 71.456
weight/mean 0.347 / 0.372 0.736 / 0.731 0.862 / 0.862 0.908 / 0.903 0.902 / 0.916 0.916 / 0.920 0.930 / 0.927 0.936 / 0.931 0.939 / 0.936 0.943 / 0.938
rew/mean 0.062 / 0.062 0.347 / 0.347 0.903 / 0.905 0.972 / 0.971 0.985 / 0.986 0.993 / 0.993 0.996 / 0.996 0.997 / 0.997 0.998 / 0.998 0.999 / 0.999
prior_ent/mean 43.247 / 43.304 43.561 / 43.692 41.675 / 42.488 39.517 / 40.569 36.392 / 38.512 35.376 / 36.516 35.224 / 35.409 34.986 / 34.905 34.363 / 34.503 34.165 / 32.954
post_ent/mean 42.399 / 42.119 42.657 / 42.779 40.616 / 41.591 38.200 / 39.526 34.699 / 37.314 33.662 / 34.958 33.765 / 33.936 33.508 / 33.449 32.966 / 33.031 32.736 / 31.036
td_error 0.129 / 0.024 1.542 / 1.550 5.017 / 5.078 5.598 / 5.636 5.501 / 5.682 5.678 / 5.879 5.601 / 5.518 5.649 / 5.635 5.483 / 5.626 5.647 / 6.053
data_rew/mean 0.958 / 0.956 0.959 / 0.957 0.970 / 0.970 0.980 / 0.978 0.983 / 0.983 0.985 / 0.985 0.987 / 0.987 0.988 / 0.988 0.989 / 0.989 0.990 / 0.990
pred_rew/mean 0.062 / 0.062 0.347 / 0.347 0.903 / 0.905 0.972 / 0.971 0.985 / 0.986 0.993 / 0.993 0.996 / 0.996 0.997 / 0.997 0.998 / 0.998 0.999 / 0.999
activation/embed 0.420 / 0.393 0.420 / 0.388 0.419 / 0.386 0.416 / 0.384 0.411 / 0.383 0.401 / 0.384 0.409 / 0.384 0.411 / 0.381 0.417 / 0.382 0.425 / 0.385
opt_loss 15.685 / 16.081 11.397 / 11.779 8.094 / 8.090 6.849 / 6.774 6.971 / 6.304 6.660 / 6.043 5.610 / 5.518 5.264 / 5.139 4.869 / 5.078 4.701 / 5.180
opt_grad_norm 160.414 / 131.611 8.500 / 8.367 7.783 / 7.043 7.505 / 6.345 11.870 / 8.134 16.432 / 10.585 10.426 / 10.447 9.969 / 11.519 9.909 / 10.291 9.416 / 13.175
opt_update_norm 0.007 / 0.007 0.013 / 0.014 0.008 / 0.008 0.009 / 0.009 0.013 / 0.010 0.010 / 0.012 0.009 / 0.010 0.009 / 0.010 0.011 / 0.011 0.012 / 0.010
opt_param_norm 59.881 / 59.906 60.499 / 60.565 61.745 / 61.874 62.474 / 62.573 63.333 / 63.322 64.211 / 64.290 65.194 / 65.282 66.279 / 66.335 67.021 / 67.217 67.576 / 67.846
opt_grad_steps 240.500 / 241.000 980.500 / 981.500 1980.500 / 1981.500 2980.500 / 2981.500 3980.500 / 3981.500 4980.500 / 4981.500 5980.500 / 5981.500 6980.500 / 6981.500 7980.500 / 7981.500 8980.500 / 8981.500
retnorm_scale_derived 1.000 / 1.000 2.481 / 2.747 7.596 / 7.679 8.530 / 8.551 11.318 / 8.672 16.468 / 9.675 20.784 / 14.154 30.829 / 26.797 35.001 / 31.992 47.685 / 40.373
retnorm_scale_exact 1.000 / nan 5.971 / nan 8.367 / nan 8.442 / nan 14.966 / nan 13.911 / nan 25.515 / nan 34.916 / nan 39.603 / nan 56.786 / 46.348

seed 1

metric 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
rec_loss 0.385 / 0.441 0.066 / 0.061 0.017 / 0.024 0.014 / 0.016 0.027 / 0.017 0.031 / 0.021 0.024 / 0.021 0.033 / 0.020 0.022 / 0.017 0.022 / 0.018
rew_loss 5.279 / 5.282 2.483 / 2.490 0.809 / 0.802 0.693 / 0.690 0.672 / 0.671 0.665 / 0.664 0.662 / 0.662 0.661 / 0.661 0.660 / 0.660 0.660 / 0.659
con_loss 0.398 / 0.363 0.141 / 0.152 0.074 / 0.082 0.053 / 0.057 0.050 / 0.052 0.050 / 0.047 0.045 / 0.044 0.047 / 0.044 0.046 / 0.044 0.045 / 0.045
dyn_loss 2.541 / 2.173 1.011 / 1.009 1.038 / 1.039 1.062 / 1.060 1.095 / 1.095 1.084 / 1.090 1.085 / 1.084 1.095 / 1.104 1.083 / 1.099 1.091 / 1.085
rep_loss 2.541 / 2.173 1.011 / 1.009 1.038 / 1.039 1.062 / 1.060 1.095 / 1.095 1.084 / 1.090 1.085 / 1.084 1.095 / 1.104 1.083 / 1.099 1.091 / 1.085
actor_loss -0.007 / -0.011 0.165 / 0.146 0.172 / 0.146 0.157 / 0.136 0.116 / 0.112 0.121 / 0.133 0.067 / 0.082 0.056 / 0.052 0.028 / 0.038 0.016 / 0.021
critic_loss 3.932 / 3.775 5.297 / 5.476 4.146 / 3.957 3.276 / 3.090 3.488 / 2.860 3.790 / 2.505 2.622 / 2.247 2.757 / 2.073 2.154 / 1.815 1.966 / 1.733
repval_loss 10.259 / 10.136 7.273 / 7.425 6.088 / 5.932 5.472 / 5.268 5.493 / 5.341 5.518 / 4.693 4.632 / 4.553 4.659 / 4.405 4.199 / 4.089 3.862 / 3.844
ent/action/mean 0.693 / 0.691 0.506 / 0.448 0.294 / 0.270 0.265 / 0.237 0.255 / 0.217 0.306 / 0.228 0.217 / 0.182 0.225 / 0.135 0.154 / 0.141 0.160 / 0.135
rand/action/mean 1.000 / 0.997 0.729 / 0.647 0.424 / 0.390 0.383 / 0.342 0.368 / 0.314 0.442 / 0.329 0.313 / 0.262 0.325 / 0.195 0.222 / 0.203 0.231 / 0.195
ret/mean 0.069 / 0.079 3.386 / 3.327 13.685 / 13.648 21.827 / 22.465 24.515 / 29.523 26.933 / 39.508 38.773 / 48.114 42.398 / 57.937 54.544 / 67.431 62.366 / 75.206
ret/std 0.135 / 0.011 0.892 / 0.818 2.883 / 3.110 3.408 / 3.853 3.939 / 3.998 4.367 / 4.288 5.821 / 5.135 6.200 / 5.158 8.287 / 6.288 13.029 / 10.263
ret_normed/std 0.135 / 0.011 0.332 / 0.285 0.370 / 0.388 0.400 / 0.448 0.397 / 0.437 0.351 / 0.496 0.350 / 0.527 0.314 / 0.491 0.327 / 0.403 0.332 / 0.378
val/mean 0.088 / 0.107 1.867 / 1.831 8.783 / 9.019 16.532 / 17.377 19.572 / 24.198 21.765 / 33.780 33.160 / 42.749 36.677 / 52.578 49.288 / 62.183 57.765 / 69.925
adv/std 0.160 / 0.011 0.339 / 0.285 0.370 / 0.387 0.391 / 0.417 0.353 / 0.401 0.341 / 0.416 0.219 / 0.453 0.215 / 0.491 0.163 / 0.326 0.123 / 0.221
replay_ret/mean 8.241 / 7.932 10.817 / 10.301 21.277 / 20.179 29.990 / 29.418 33.959 / 36.998 36.806 / 46.504 47.064 / 54.438 50.836 / 63.853 61.089 / 70.862 68.158 / 77.445
weight/mean 0.364 / 0.351 0.744 / 0.729 0.877 / 0.849 0.912 / 0.896 0.916 / 0.911 0.929 / 0.929 0.941 / 0.932 0.942 / 0.945 0.942 / 0.940 0.942 / 0.940
rew/mean 0.062 / 0.062 0.357 / 0.354 0.908 / 0.908 0.972 / 0.973 0.987 / 0.987 0.993 / 0.993 0.996 / 0.996 0.998 / 0.998 0.998 / 0.998 0.999 / 0.999
prior_ent/mean 43.503 / 43.270 43.623 / 43.349 42.165 / 41.585 40.429 / 38.762 37.412 / 37.777 34.309 / 37.303 34.842 / 36.246 34.194 / 36.960 34.322 / 35.492 33.687 / 35.231
post_ent/mean 41.869 / 42.436 42.729 / 42.313 41.135 / 40.471 39.164 / 37.437 35.712 / 36.415 31.833 / 35.899 33.096 / 34.770 32.378 / 35.651 32.874 / 34.117 32.244 / 33.879
td_error 0.129 / 0.028 1.584 / 1.535 5.097 / 5.001 5.604 / 5.590 5.292 / 5.611 5.746 / 5.884 5.824 / 5.717 5.945 / 5.864 5.578 / 5.676 5.272 / 5.844
data_rew/mean 0.960 / 0.956 0.962 / 0.957 0.974 / 0.970 0.982 / 0.979 0.986 / 0.984 0.988 / 0.987 0.989 / 0.988 0.991 / 0.990 0.991 / 0.989 0.991 / 0.990
pred_rew/mean 0.062 / 0.062 0.357 / 0.354 0.908 / 0.908 0.972 / 0.973 0.987 / 0.987 0.993 / 0.993 0.996 / 0.996 0.998 / 0.998 0.998 / 0.998 0.999 / 0.999
activation/embed 0.439 / 0.375 0.434 / 0.384 0.422 / 0.385 0.407 / 0.381 0.386 / 0.362 0.381 / 0.363 0.378 / 0.363 0.376 / 0.369 0.387 / 0.372 0.396 / 0.375
opt_loss 15.861 / 15.280 11.445 / 11.662 8.185 / 7.933 7.002 / 6.736 7.206 / 6.518 7.504 / 5.977 6.002 / 5.614 6.156 / 5.385 5.361 / 5.010 5.067 / 4.824
opt_grad_norm 143.735 / 191.838 8.097 / 9.002 7.814 / 8.016 7.988 / 7.568 14.814 / 9.290 13.395 / 10.368 12.559 / 14.542 18.032 / 12.639 11.072 / 9.946 11.439 / 9.775
opt_update_norm 0.007 / 0.007 0.013 / 0.014 0.008 / 0.008 0.011 / 0.009 0.013 / 0.013 0.011 / 0.010 0.009 / 0.010 0.010 / 0.010 0.009 / 0.009 0.010 / 0.010
opt_param_norm 59.908 / 59.777 60.534 / 60.458 61.778 / 61.756 62.512 / 62.446 63.379 / 63.245 64.336 / 64.078 65.287 / 65.018 66.310 / 65.893 67.175 / 66.633 67.852 / 67.222
opt_grad_steps 240.500 / 241.000 980.500 / 981.500 1980.500 / 1981.500 2980.500 / 2981.500 3980.500 / 3981.500 4980.500 / 4981.500 5980.500 / 5981.500 6980.500 / 6981.500 7980.500 / 7981.500 8980.500 / 8981.500
retnorm_scale_derived 1.000 / 1.000 2.686 / 2.868 7.795 / 8.019 8.525 / 8.604 9.930 / 9.144 12.427 / 8.641 16.647 / 9.751 19.751 / 10.510 25.341 / 15.608 39.217 / 27.115
retnorm_scale_exact 1.000 / nan 6.354 / nan 8.475 / nan 8.211 / nan 12.687 / nan 13.495 / nan 19.558 / nan 19.016 / nan 28.814 / nan 44.935 / 42.568

seed 2

metric 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
rec_loss 0.428 / 0.387 0.058 / 0.062 0.021 / 0.020 0.017 / 0.017 0.021 / 0.018 0.017 / 0.017 0.017 / 0.017 0.018 / 0.015 0.024 / 0.013 0.028 / 0.014
rew_loss 5.282 / 5.288 2.494 / 2.528 0.803 / 0.811 0.691 / 0.693 0.670 / 0.673 0.663 / 0.665 0.660 / 0.662 0.660 / 0.661 0.659 / 0.660 0.659 / 0.660
con_loss 0.366 / 0.351 0.155 / 0.151 0.081 / 0.082 0.057 / 0.056 0.055 / 0.050 0.050 / 0.051 0.052 / 0.049 0.047 / 0.049 0.050 / 0.048 0.050 / 0.045
dyn_loss 2.712 / 2.300 1.012 / 1.009 1.033 / 1.035 1.056 / 1.065 1.084 / 1.096 1.067 / 1.095 1.089 / 1.082 1.085 / 1.110 1.105 / 1.119 1.107 / 1.132
rep_loss 2.712 / 2.300 1.012 / 1.009 1.033 / 1.035 1.056 / 1.065 1.084 / 1.096 1.067 / 1.095 1.089 / 1.082 1.085 / 1.110 1.105 / 1.119 1.107 / 1.132
actor_loss -0.007 / -0.009 0.161 / 0.149 0.177 / 0.149 0.152 / 0.126 0.183 / 0.124 0.095 / 0.093 0.030 / 0.031 0.015 / 0.020 0.010 / 0.010 0.008 / 0.009
critic_loss 4.073 / 3.982 5.054 / 5.628 4.066 / 4.115 3.395 / 3.180 3.154 / 2.951 2.635 / 2.551 2.268 / 2.478 1.956 / 1.962 1.918 / 1.822 1.758 / 1.807
repval_loss 10.180 / 10.184 7.209 / 7.549 6.016 / 6.043 5.500 / 5.335 5.248 / 5.349 4.711 / 4.458 4.221 / 4.816 3.832 / 3.933 3.653 / 3.570 3.323 / 3.286
ent/action/mean 0.693 / 0.693 0.518 / 0.461 0.309 / 0.258 0.269 / 0.216 0.325 / 0.221 0.261 / 0.207 0.219 / 0.168 0.150 / 0.154 0.131 / 0.131 0.134 / 0.139
rand/action/mean 1.000 / 0.999 0.747 / 0.665 0.445 / 0.372 0.388 / 0.312 0.468 / 0.318 0.376 / 0.298 0.316 / 0.243 0.217 / 0.222 0.190 / 0.189 0.194 / 0.200
ret/mean 0.068 / 0.075 3.289 / 3.217 13.373 / 13.536 20.692 / 22.179 25.288 / 29.053 34.969 / 38.020 45.163 / 45.594 59.872 / 58.075 70.558 / 70.057 81.988 / 78.897
ret/std 0.138 / 0.009 0.859 / 0.749 2.912 / 2.858 3.495 / 3.570 3.687 / 3.824 5.566 / 4.891 10.246 / 7.902 14.858 / 11.625 18.777 / 16.141 21.141 / 20.141
ret_normed/std 0.138 / 0.009 0.329 / 0.267 0.378 / 0.371 0.411 / 0.414 0.421 / 0.434 0.408 / 0.412 0.322 / 0.321 0.323 / 0.331 0.330 / 0.329 0.338 / 0.337
val/mean 0.087 / 0.097 1.788 / 1.771 8.506 / 8.717 15.546 / 16.960 20.028 / 23.615 29.921 / 32.547 40.083 / 39.923 54.387 / 52.878 65.210 / 65.163 77.039 / 74.028
adv/std 0.162 / 0.009 0.336 / 0.267 0.378 / 0.371 0.411 / 0.410 0.399 / 0.393 0.340 / 0.307 0.146 / 0.198 0.119 / 0.128 0.104 / 0.105 0.098 / 0.092
replay_ret/mean 7.599 / 8.106 10.024 / 10.599 19.765 / 20.634 27.942 / 30.000 32.685 / 37.156 41.351 / 45.150 50.091 / 52.311 63.308 / 62.814 72.120 / 73.569 82.209 / 81.967
weight/mean 0.376 / 0.369 0.714 / 0.734 0.856 / 0.867 0.897 / 0.911 0.906 / 0.919 0.916 / 0.928 0.921 / 0.931 0.931 / 0.935 0.934 / 0.940 0.938 / 0.944
rew/mean 0.062 / 0.062 0.353 / 0.344 0.907 / 0.903 0.971 / 0.972 0.987 / 0.986 0.993 / 0.993 0.996 / 0.996 0.998 / 0.998 0.998 / 0.998 0.999 / 0.999
prior_ent/mean 43.185 / 43.366 43.420 / 43.728 42.323 / 42.482 40.283 / 40.828 35.963 / 37.397 36.289 / 35.717 36.577 / 34.446 36.706 / 33.914 36.603 / 34.446 36.076 / 35.612
post_ent/mean 41.677 / 42.407 42.392 / 42.842 41.307 / 41.569 38.895 / 39.626 34.114 / 35.858 34.812 / 34.171 35.095 / 32.759 35.296 / 32.299 35.223 / 33.063 34.759 / 34.348
td_error 0.131 / 0.022 1.581 / 1.466 5.107 / 5.025 5.566 / 5.645 5.616 / 5.685 5.798 / 5.725 5.584 / 6.034 5.862 / 5.604 5.930 / 5.515 5.809 / 5.607
data_rew/mean 0.954 / 0.957 0.956 / 0.960 0.968 / 0.972 0.978 / 0.980 0.982 / 0.985 0.985 / 0.987 0.986 / 0.989 0.988 / 0.989 0.989 / 0.990 0.990 / 0.991
pred_rew/mean 0.062 / 0.062 0.353 / 0.344 0.907 / 0.903 0.971 / 0.972 0.987 / 0.986 0.993 / 0.993 0.996 / 0.996 0.998 / 0.998 0.998 / 0.998 0.999 / 0.999
activation/embed 0.428 / 0.392 0.438 / 0.401 0.448 / 0.409 0.437 / 0.405 0.414 / 0.395 0.412 / 0.393 0.405 / 0.388 0.410 / 0.392 0.410 / 0.398 0.410 / 0.406
opt_loss 16.179 / 15.584 11.197 / 11.892 8.090 / 8.128 7.124 / 6.845 6.849 / 6.627 6.047 / 5.920 5.491 / 5.871 5.039 / 5.107 4.972 / 4.855 4.718 / 4.767
opt_grad_norm 140.356 / 132.879 9.024 / 8.697 7.629 / 7.461 9.116 / 7.673 14.097 / 10.523 10.987 / 9.821 11.868 / 11.645 11.344 / 10.499 13.685 / 9.252 12.208 / 10.682
opt_update_norm 0.007 / 0.007 0.013 / 0.014 0.008 / 0.008 0.012 / 0.008 0.012 / 0.013 0.010 / 0.011 0.010 / 0.011 0.010 / 0.010 0.011 / 0.011 0.011 / 0.011
opt_param_norm 59.875 / 59.870 60.499 / 60.534 61.747 / 61.860 62.505 / 62.561 63.387 / 63.320 64.276 / 64.136 65.351 / 65.122 66.197 / 65.896 66.805 / 66.463 67.380 / 66.862
opt_grad_steps 240.500 / 241.000 980.500 / 981.500 1980.500 / 1981.500 2980.500 / 2981.500 3980.500 / 3981.500 4980.500 / 4981.500 5980.500 / 5981.500 6980.500 / 6981.500 7980.500 / 7981.500 8980.500 / 8981.500
retnorm_scale_derived 1.000 / 1.000 2.608 / 2.800 7.710 / 7.694 8.494 / 8.627 8.762 / 8.816 13.643 / 11.882 31.781 / 24.629 45.935 / 35.123 56.972 / 49.065 62.547 / 59.755
retnorm_scale_exact 1.000 / nan 6.196 / nan 8.380 / nan 8.289 / nan 9.284 / nan 25.607 / nan 38.651 / nan 52.648 / nan 55.632 / nan 65.070 / 69.062

Reference evaluation returns per episode

  • s0 rows 2000: [18.0, 13.0, 14.0, 10.0, 31.0, 21.0, 11.0, 17.0, 24.0, 15.0]
  • s0 rows 4000: [307.0, 195.0, 85.0, 145.0, 200.0, 74.0, 168.0, 100.0, 98.0, 76.0]
  • s0 rows 6000: [188.0, 149.0, 168.0, 263.0, 174.0, 181.0, 205.0, 206.0, 196.0, 293.0]
  • s0 rows 8000: [110.0, 106.0, 111.0, 110.0, 95.0, 104.0, 127.0, 132.0, 121.0, 119.0]
  • s0 rows 10000: [110.0, 125.0, 130.0, 115.0, 123.0, 113.0, 129.0, 107.0, 115.0, 86.0]
  • s0 rows 12000: [177.0, 178.0, 177.0, 211.0, 160.0, 215.0, 219.0, 199.0, 177.0, 210.0]
  • s0 rows 14000: [210.0, 209.0, 200.0, 228.0, 225.0, 197.0, 240.0, 217.0, 208.0, 290.0]
  • s0 rows 16000: [299.0, 249.0, 305.0, 277.0, 286.0, 260.0, 211.0, 239.0, 239.0, 170.0]
  • s0 rows 18000: [500.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0]
  • s0 rows 20000: [449.0, 389.0, 500.0, 409.0, 432.0, 403.0, 422.0, 370.0, 375.0, 299.0]
  • s1 rows 2000: [14.0, 47.0, 38.0, 16.0, 33.0, 11.0, 30.0, 15.0, 17.0, 31.0]
  • s1 rows 4000: [141.0, 49.0, 89.0, 462.0, 134.0, 75.0, 147.0, 92.0, 127.0, 101.0]
  • s1 rows 6000: [159.0, 167.0, 162.0, 89.0, 123.0, 144.0, 120.0, 101.0, 146.0, 115.0]
  • s1 rows 8000: [86.0, 83.0, 88.0, 101.0, 81.0, 88.0, 84.0, 88.0, 91.0, 94.0]
  • s1 rows 10000: [210.0, 170.0, 180.0, 233.0, 171.0, 189.0, 198.0, 208.0, 179.0, 168.0]
  • s1 rows 12000: [128.0, 125.0, 127.0, 137.0, 133.0, 133.0, 137.0, 127.0, 124.0, 135.0]
  • s1 rows 14000: [140.0, 164.0, 138.0, 124.0, 138.0, 129.0, 125.0, 128.0, 146.0, 137.0]
  • s1 rows 16000: [156.0, 140.0, 149.0, 166.0, 144.0, 138.0, 161.0, 149.0, 146.0, 161.0]
  • s1 rows 18000: [123.0, 136.0, 124.0, 121.0, 121.0, 120.0, 114.0, 118.0, 127.0, 118.0]
  • s1 rows 20000: [434.0, 227.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0, 500.0]
  • s2 rows 2000: [17.0, 44.0, 37.0, 17.0, 42.0, 46.0, 35.0, 20.0, 42.0, 34.0]
  • s2 rows 4000: [208.0, 176.0, 195.0, 263.0, 120.0, 192.0, 252.0, 180.0, 198.0, 177.0]
  • s2 rows 6000: [147.0, 143.0, 160.0, 93.0, 139.0, 141.0, 131.0, 179.0, 127.0, 175.0]
  • s2 rows 8000: [125.0, 124.0, 154.0, 311.0, 193.0, 155.0, 169.0, 150.0, 142.0, 180.0]
  • s2 rows 10000: [106.0, 111.0, 108.0, 100.0, 50.0, 63.0, 101.0, 109.0, 42.0, 112.0]
  • s2 rows 12000: [156.0, 153.0, 162.0, 215.0, 182.0, 154.0, 174.0, 116.0, 168.0, 109.0]
  • s2 rows 14000: [356.0, 464.0, 348.0, 298.0, 290.0, 375.0, 318.0, 298.0, 289.0, 420.0]
  • s2 rows 16000: [180.0, 166.0, 181.0, 211.0, 184.0, 169.0, 186.0, 178.0, 179.0, 171.0]
  • s2 rows 18000: [147.0, 145.0, 144.0, 142.0, 154.0, 122.0, 148.0, 133.0, 146.0, 157.0]
  • s2 rows 20000: [463.0, 367.0, 436.0, 325.0, 390.0, 405.0, 356.0, 500.0, 347.0, 409.0]