Skip to content

Ajax vs reference DreamerV3, CartPole-v1

Root: <scratch>/refrun/cmpE; seeds [0, 1, 2]; checkpoints (rows) [2000, 4000, 6000, 8000, 10000, 12000, 14000, 16000, 18000, 20000]; S1 window [16000, 18000, 20000]; S2 at 20000.

Checks

PASS  ref_s0: exit code 0 (ref_s0.DONE)
PASS  ref_s1: exit code 0 (ref_s1.DONE)
PASS  ref_s2: exit code 0 (ref_s2.DONE)
PASS  ajax: exit code 0 (ajax.DONE)
PASS  ref s0: config.yaml has the protocol values
PASS  ref s0: reference code imported from the 29eb964 checkout (['<reference-checkout>/dreamerv3/agent.py', '<reference-checkout>/embodied/__init__.py'])
PASS  ref s0: run_args.json has the protocol values
PASS  ref s0: every evaluation has 10 episodes
PASS  ref s0: episodes length == score, 1 <= length <= 500 (197 episodes)
PASS  ref s0: terminal False only for 500-step episodes
PASS  ref s0: per worker the episodes tile rows 0..row_last with L + 1 rows each
PASS  ref s0: run finished and the flush step ran (timing.json t_end present; flush_episodes 0 == episodes with row_last 1250: 0)
PASS  ref s1: config.yaml has the protocol values
PASS  ref s1: reference code imported from the 29eb964 checkout (['<reference-checkout>/dreamerv3/agent.py', '<reference-checkout>/embodied/__init__.py'])
PASS  ref s1: run_args.json has the protocol values
PASS  ref s1: every evaluation has 10 episodes
PASS  ref s1: episodes length == score, 1 <= length <= 500 (186 episodes)
PASS  ref s1: terminal False only for 500-step episodes
PASS  ref s1: per worker the episodes tile rows 0..row_last with L + 1 rows each
PASS  ref s1: run finished and the flush step ran (timing.json t_end present; flush_episodes 0 == episodes with row_last 1250: 0)
PASS  ref s2: config.yaml has the protocol values
PASS  ref s2: reference code imported from the 29eb964 checkout (['<reference-checkout>/dreamerv3/agent.py', '<reference-checkout>/embodied/__init__.py'])
PASS  ref s2: run_args.json has the protocol values
PASS  ref s2: every evaluation has 10 episodes
PASS  ref s2: episodes length == score, 1 <= length <= 500 (212 episodes)
PASS  ref s2: terminal False only for 500-step episodes
PASS  ref s2: per worker the episodes tile rows 0..row_last with L + 1 rows each
PASS  ref s2: run finished and the flush step ran (timing.json t_end present; flush_episodes 0 == episodes with row_last 1250: 0)
PASS  ajax: config.json has the protocol values
PASS  10 records per seed per side at rows 2000 k
PASS  updates = 1 + floor((N - 1040) / 2) at every N on both sides (9481 at 20000)
PASS  9481 updates at 20000 rows on both sides
PASS  ajax s0: logged S2 108.519 == final buffer mean 108.519 (tol 1e-4)
PASS  ajax s1: logged S2 107.013 == final buffer mean 107.012 (tol 1e-4)
PASS  ajax s2: logged S2 107.378 == final buffer mean 107.378 (tol 1e-4)
PASS  ajax s0: final state rows 20000, n_updates 9481
PASS  ajax s1: final state rows 20000, n_updates 9481
PASS  ajax s2: final state rows 20000, n_updates 9481

Pre-registered rule

  • A = mean over Ajax seeds of S1 = 204.1
  • reference S1 range [min, max] = [265.2, 307.0]
  • LEARNS = False (A >= min reference S1)
  • WITHIN_RANGE = False (min <= A <= max)

Per-seed S1 and S2

seed ref S1 ajax S1 ref S2 ajax S2 (logged) ajax S2 (audit)
0 307.0 180.6 96.7 108.5 108.5
1 265.2 143.5 109.9 107.0 107.0
2 297.6 288.3 105.8 107.4 107.4
mean 289.9 204.1 104.1 107.6

Known differences between the two sides (read before interpreting)

  • D1 / U1, asynchrony (reference only). The reference acts with policy parameters that lag the trained ones by about 2 vector steps (about 16 of 9481 updates; measured on the real reference in refrun/audit_loop), its first training batch is prefetched before the gate opens (11 distinct items of 16 in the audit), and its write-back and metrics are one call late. Ajax acts with the latest parameters. Evaluation uses the latest parameters on both sides.
  • D2 / U2, shared replay-sampler stream (reference only). Stock make_replay passes no seed, so Uniform(seed=0) samples the replay for every reference seed: reference seeds [0, 1, 2] share one index stream (the replay contents differ). The reference S1 [min, max] range therefore comes from runs that are not fully independent. Ajax seeds its sampler from the run seed.
  • Report prefetch stream (stock, D1 / D2). With run.eval_every = -1 the stock report stream (train.py:76-77) is created but never read: when the first replay items appear (about row 1040) its prefetch thread draws 2 batches of 16 (32 samples, popping up to 32 of the earliest online-queue items, plus Uniform-RNG and agent.rng draws) and then blocks for good. A stock run (eval_every 180 s) would draw 16 more every 180 s of wall time. Negligible: training draws 128 samples per vector step against 16 new items.
  • NaN in training-metric windows. The reference Agg skips NaN values; Ajax's accumulator propagates them, so one NaN makes the Ajax window mean NaN. A nan Ajax window against a finite reference window means 'a NaN occurred in that Ajax window', not a divergence of the finite values. Non-finite Ajax window means here: none.
  • Reference metric windows hold updates n0-1 .. n1-1 (one-call lag); Ajax per-episode evaluation returns are not logged (only the mean).

Evaluation curve (mean return of the evaluation episodes)

rows ref s0 ref s1 ref s2 ref mean ajax s0 ajax s1 ajax s2 ajax mean
2000 19.8 23.3 26.5 23.2 22.4 19.4 15.9 19.2
4000 122.8 137.9 214.5 158.4 102.3 189.3 60.9 117.5
6000 151.5 334.5 163.7 216.6 480.5 230.5 476.4 395.8
8000 62.0 114.0 122.3 99.4 142.6 322.4 336.5 267.2
10000 134.1 209.8 150.6 164.8 163.9 500.0 149.1 271.0
12000 29.0 118.7 116.8 88.2 147.4 118.4 389.5 218.4
14000 260.0 152.4 197.4 203.3 224.7 174.1 246.5 215.1
16000 300.1 33.6 125.1 152.9 290.0 84.7 212.9 195.9
18000 150.4 266.6 298.8 238.6 108.6 167.5 151.9 142.7
20000 470.4 495.3 469.0 478.2 143.3 178.3 500.0 273.9

S2 curve (training-episode rolling mean, Ajax definition)

rows ref s0 ref s1 ref s2 ref mean ajax s0 ajax s1 ajax s2 ajax mean
2000 23.3 23.4 21.1 22.6 19.5 21.4 20.5 20.5
4000 25.7 27.8 25.6 26.3 24.1 23.9 26.4 24.8
6000 31.8 34.8 26.5 31.0 29.7 29.9 29.3 29.7
8000 48.0 38.9 33.2 40.0 36.0 31.5 31.5 33.0
10000 56.4 64.3 61.8 60.8 52.7 36.2 46.6 45.2
12000 62.5 68.7 69.5 66.9 62.2 50.2 69.1 60.5
14000 74.1 83.4 78.7 78.7 69.4 71.7 79.5 73.5
16000 78.4 96.3 84.2 86.3 78.1 95.0 89.7 87.6
18000 84.8 102.6 96.6 94.7 89.1 96.2 92.4 92.6
20000 96.7 109.9 105.8 104.1 108.5 107.0 107.4 107.6

Updates per checkpoint

rows expected ref s0 ref s1 ref s2 ajax s0 ajax s1 ajax s2
2000 481 481 481 481 481 481 481
4000 1481 1481 1481 1481 1481 1481 1481
6000 2481 2481 2481 2481 2481 2481 2481
8000 3481 3481 3481 3481 3481 3481 3481
10000 4481 4481 4481 4481 4481 4481 4481
12000 5481 5481 5481 5481 5481 5481 5481
14000 6481 6481 6481 6481 6481 6481 6481
16000 7481 7481 7481 7481 7481 7481 7481
18000 8481 8481 8481 8481 8481 8481 8481
20000 9481 9481 9481 9481 9481 9481 9481

Training metrics, window means (mean over seeds [min, max])

Reference windows hold updates n0-1 .. n1-1 (JAXAgent returns the previous call's metrics); Ajax windows n0 .. n1. Loss names mapped: vector_loss->rec_loss, reward_loss->rew_loss, cont_loss->con_loss, replay_critic_loss->repval_loss.

window ending at 2000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.4055 [0.3864, 0.4277] 0.4264 [0.3866, 0.4518] 1.052
rew_loss 5.2850 [5.2791, 5.2942] 5.2867 [5.2816, 5.2906] 1.000
con_loss 0.3908 [0.3656, 0.4086] 0.3454 [0.3227, 0.3630] 0.884
dyn_loss 2.5951 [2.5333, 2.7118] 2.3860 [2.1729, 2.6847] 0.919
rep_loss 2.5951 [2.5333, 2.7118] 2.3860 [2.1729, 2.6847] 0.919
actor_loss -0.0078 [-0.0093, -0.0071] -0.0096 [-0.0106, -0.0087] 1.221
critic_loss 3.9076 [3.7415, 4.0629] 3.9265 [3.7747, 4.0231] 1.005
repval_loss 10.1960 [10.1596, 10.2355] 10.1612 [10.1355, 10.1844] 0.997
ent/action/mean 0.6917 [0.6910, 0.6928] 0.6921 [0.6909, 0.6928] 1.001
rand/action/mean 0.9980 [0.9968, 0.9995] 0.9985 [0.9967, 0.9994] 1.001
ret/mean 0.0731 [0.0719, 0.0752] 0.0758 [0.0730, 0.0793] 1.037
ret/std 0.0085 [0.0078, 0.0096] 0.0099 [0.0091, 0.0109] 1.165
ret_normed/std 0.0085 [0.0078, 0.0096] 0.0099 [0.0091, 0.0109] 1.165
val/mean 0.0937 [0.0897, 0.1003] 0.1004 [0.0965, 0.1075] 1.072
adv/std 0.0086 [0.0079, 0.0098] 0.0101 [0.0095, 0.0110] 1.171
replay_ret/mean 7.9552 [7.5994, 8.2257] 7.9780 [7.8962, 8.1061] 1.003
weight/mean 0.3626 [0.3476, 0.3764] 0.3642 [0.3512, 0.3721] 1.004
rew/mean 0.0625 [0.0623, 0.0627] 0.0624 [0.0623, 0.0625] 0.999
prior_ent/mean 43.3118 [43.1853, 43.5029] 43.3131 [43.2701, 43.3656] 1.000
post_ent/mean 41.9823 [41.6774, 42.3990] 42.3208 [42.1189, 42.4362] 1.008
td_error 0.0206 [0.0178, 0.0251] 0.0246 [0.0223, 0.0281] 1.197
data_rew/mean 0.9572 [0.9536, 0.9599] 0.9562 [0.9556, 0.9574] 0.999
pred_rew/mean 0.0625 [0.0623, 0.0627] 0.0624 [0.0623, 0.0625] 0.999
activation/embed 0.4286 [0.4196, 0.4385] 0.3867 [0.3754, 0.3931] 0.902
opt_loss 15.8944 [15.6821, 16.1616] 15.6484 [15.2804, 16.0810] 0.985
opt_grad_norm 151.4155 [141.7651, 163.1922] 152.1094 [131.6112, 191.8375] 1.005
opt_update_norm 0.0073 [0.0072, 0.0074] 0.0072 [0.0072, 0.0073] 0.991
opt_param_norm 59.8884 [59.8758, 59.9083] 59.8509 [59.7766, 59.9060] 0.999
opt_grad_steps 240.5000 [240.5000, 240.5000] 241.0000 [241.0000, 241.0000] 1.002
retnorm_scale_derived 1.0000 [1.0000, 1.0000] 1.0000 [1.0000, 1.0000] 1.000
retnorm_scale_exact 1.0000 [1.0000, 1.0000] nan [nan, nan] nan

window ending at 4000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0620 [0.0576, 0.0654] 0.0653 [0.0605, 0.0736] 1.053
rew_loss 2.4972 [2.4815, 2.5186] 2.5103 [2.4903, 2.5275] 1.005
con_loss 0.1483 [0.1414, 0.1518] 0.1529 [0.1510, 0.1554] 1.031
dyn_loss 1.0117 [1.0113, 1.0120] 1.0094 [1.0086, 1.0104] 0.998
rep_loss 1.0117 [1.0113, 1.0120] 1.0094 [1.0086, 1.0104] 0.998
actor_loss 0.1545 [0.1449, 0.1674] 0.1488 [0.1455, 0.1520] 0.963
critic_loss 5.6574 [5.5416, 5.7442] 5.5446 [5.4760, 5.6279] 0.980
repval_loss 7.5490 [7.5282, 7.5769] 7.4848 [7.4255, 7.5486] 0.991
ent/action/mean 0.4591 [0.4481, 0.4765] 0.4565 [0.4485, 0.4611] 0.994
rand/action/mean 0.6624 [0.6465, 0.6875] 0.6586 [0.6470, 0.6652] 0.994
ret/mean 3.3147 [3.2842, 3.3560] 3.2713 [3.2168, 3.3266] 0.987
ret/std 0.7908 [0.7443, 0.8304] 0.7844 [0.7489, 0.8179] 0.992
ret_normed/std 0.2766 [0.2687, 0.2812] 0.2796 [0.2674, 0.2863] 1.011
val/mean 1.7883 [1.7630, 1.8172] 1.7864 [1.7566, 1.8313] 0.999
adv/std 0.2766 [0.2686, 0.2812] 0.2796 [0.2674, 0.2862] 1.011
replay_ret/mean 10.6399 [10.3391, 10.9321] 10.3835 [10.2506, 10.5989] 0.976
weight/mean 0.7391 [0.7262, 0.7491] 0.7314 [0.7292, 0.7344] 0.990
rew/mean 0.3529 [0.3477, 0.3570] 0.3483 [0.3438, 0.3538] 0.987
prior_ent/mean 43.5162 [43.4054, 43.5913] 43.5896 [43.3493, 43.7277] 1.002
post_ent/mean 42.5759 [42.3722, 42.7006] 42.6449 [42.3132, 42.8424] 1.002
td_error 1.5497 [1.5226, 1.5695] 1.5174 [1.4662, 1.5504] 0.979
data_rew/mean 0.9600 [0.9578, 0.9622] 0.9581 [0.9569, 0.9601] 0.998
pred_rew/mean 0.3529 [0.3477, 0.3570] 0.3483 [0.3438, 0.3538] 0.987
activation/embed 0.4314 [0.4206, 0.4396] 0.3911 [0.3842, 0.4010] 0.907
opt_loss 11.8970 [11.7591, 11.9668] 11.7777 [11.6617, 11.8918] 0.990
opt_grad_norm 9.0143 [8.5708, 9.4733] 8.6889 [8.3670, 9.0024] 0.964
opt_update_norm 0.0140 [0.0135, 0.0143] 0.0140 [0.0138, 0.0143] 1.003
opt_param_norm 60.5532 [60.5383, 60.5764] 60.5191 [60.4579, 60.5654] 0.999
opt_grad_steps 980.5000 [980.5000, 980.5000] 981.5000 [981.5000, 981.5000] 1.001
retnorm_scale_derived 2.8576 [2.7701, 2.9534] 2.8050 [2.7469, 2.8676] 0.982
retnorm_scale_exact 6.1740 [5.9703, 6.3421] nan [nan, nan] nan

window ending at 6000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0194 [0.0179, 0.0217] 0.0209 [0.0192, 0.0237] 1.079
rew_loss 0.8089 [0.8035, 0.8120] 0.8058 [0.8017, 0.8111] 0.996
con_loss 0.0776 [0.0747, 0.0804] 0.0826 [0.0821, 0.0836] 1.065
dyn_loss 1.0365 [1.0354, 1.0377] 1.0360 [1.0341, 1.0391] 0.999
rep_loss 1.0365 [1.0354, 1.0377] 1.0360 [1.0341, 1.0391] 0.999
actor_loss 0.1525 [0.1457, 0.1620] 0.1502 [0.1460, 0.1554] 0.985
critic_loss 4.1176 [4.0645, 4.1543] 4.0536 [3.9570, 4.1150] 0.984
repval_loss 6.0580 [6.0258, 6.0794] 5.9926 [5.9322, 6.0432] 0.989
ent/action/mean 0.2634 [0.2508, 0.2814] 0.2658 [0.2575, 0.2701] 1.009
rand/action/mean 0.3801 [0.3618, 0.4060] 0.3835 [0.3715, 0.3896] 1.009
ret/mean 13.6022 [13.5042, 13.7266] 13.5813 [13.5365, 13.6476] 0.998
ret/std 2.8719 [2.8233, 2.9061] 2.9491 [2.8584, 3.1098] 1.027
ret_normed/std 0.3727 [0.3658, 0.3795] 0.3781 [0.3715, 0.3878] 1.014
val/mean 8.7545 [8.6341, 8.8658] 8.8127 [8.7020, 9.0187] 1.007
adv/std 0.3725 [0.3657, 0.3790] 0.3778 [0.3714, 0.3874] 1.014
replay_ret/mean 20.9343 [20.6709, 21.4294] 20.2949 [20.0715, 20.6341] 0.969
weight/mean 0.8674 [0.8588, 0.8757] 0.8592 [0.8488, 0.8670] 0.991
rew/mean 0.9066 [0.9041, 0.9079] 0.9053 [0.9033, 0.9076] 0.999
prior_ent/mean 42.0368 [41.6401, 42.2854] 42.1852 [41.5852, 42.4880] 1.004
post_ent/mean 41.0067 [40.6029, 41.2556] 41.2104 [40.4710, 41.5911] 1.005
td_error 5.0567 [4.9808, 5.1212] 5.0347 [5.0007, 5.0780] 0.996
data_rew/mean 0.9725 [0.9713, 0.9744] 0.9705 [0.9696, 0.9720] 0.998
pred_rew/mean 0.9066 [0.9041, 0.9079] 0.9053 [0.9033, 0.9076] 0.999
activation/embed 0.4287 [0.4184, 0.4464] 0.3935 [0.3854, 0.4088] 0.918
opt_loss 8.1336 [8.0898, 8.1687] 8.0505 [7.9333, 8.1283] 0.990
opt_grad_norm 7.7154 [7.5540, 7.9404] 7.5068 [7.0433, 8.0157] 0.973
opt_update_norm 0.0079 [0.0076, 0.0081] 0.0079 [0.0078, 0.0081] 1.011
opt_param_norm 61.8625 [61.8457, 61.8818] 61.8299 [61.7560, 61.8739] 0.999
opt_grad_steps 1980.5000 [1980.5000, 1980.5000] 1981.5000 [1981.5000, 1981.5000] 1.001
retnorm_scale_derived 7.7062 [7.6058, 7.7949] 7.7976 [7.6791, 8.0191] 1.012
retnorm_scale_exact 8.4401 [8.3847, 8.4786] nan [nan, nan] nan

window ending at 8000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0151 [0.0135, 0.0172] 0.0160 [0.0150, 0.0173] 1.063
rew_loss 0.6926 [0.6914, 0.6934] 0.6915 [0.6900, 0.6932] 0.998
con_loss 0.0542 [0.0518, 0.0567] 0.0570 [0.0558, 0.0579] 1.053
dyn_loss 1.0539 [1.0469, 1.0589] 1.0566 [1.0445, 1.0652] 1.003
rep_loss 1.0539 [1.0469, 1.0589] 1.0566 [1.0445, 1.0652] 1.003
actor_loss 0.1310 [0.1238, 0.1379] 0.1323 [0.1263, 0.1356] 1.010
critic_loss 3.1810 [3.1362, 3.2281] 3.1401 [3.0905, 3.1805] 0.987
repval_loss 5.3572 [5.3058, 5.3936] 5.2863 [5.2558, 5.3350] 0.987
ent/action/mean 0.2268 [0.2105, 0.2357] 0.2287 [0.2160, 0.2368] 1.008
rand/action/mean 0.3271 [0.3037, 0.3400] 0.3299 [0.3117, 0.3416] 1.008
ret/mean 22.1062 [22.0232, 22.2165] 22.2497 [22.1054, 22.4649] 1.006
ret/std 3.5031 [3.3910, 3.6568] 3.6700 [3.5701, 3.8527] 1.048
ret_normed/std 0.4058 [0.3937, 0.4181] 0.4270 [0.4138, 0.4478] 1.052
val/mean 16.8608 [16.6233, 16.9963] 17.0731 [16.8829, 17.3766] 1.013
adv/std 0.3909 [0.3856, 0.3968] 0.4119 [0.4082, 0.4174] 1.054
replay_ret/mean 30.0908 [29.7472, 30.4508] 29.4273 [28.8639, 30.0003] 0.978
weight/mean 0.9096 [0.9031, 0.9143] 0.9034 [0.8957, 0.9111] 0.993
rew/mean 0.9724 [0.9718, 0.9728] 0.9718 [0.9712, 0.9725] 0.999
prior_ent/mean 40.4343 [39.8539, 40.7436] 40.0529 [38.7624, 40.8279] 0.991
post_ent/mean 39.1935 [38.6281, 39.5011] 38.8630 [37.4371, 39.6257] 0.992
td_error 5.6004 [5.5370, 5.7088] 5.6239 [5.5903, 5.6449] 1.004
data_rew/mean 0.9809 [0.9802, 0.9820] 0.9789 [0.9778, 0.9804] 0.998
pred_rew/mean 0.9724 [0.9718, 0.9728] 0.9718 [0.9712, 0.9725] 0.999
activation/embed 0.4171 [0.4050, 0.4326] 0.3901 [0.3807, 0.4054] 0.935
opt_loss 6.8403 [6.7750, 6.8920] 6.7850 [6.7356, 6.8454] 0.992
opt_grad_norm 7.4620 [6.9387, 7.7813] 7.1955 [6.3448, 7.6734] 0.964
opt_update_norm 0.0093 [0.0092, 0.0096] 0.0087 [0.0078, 0.0093] 0.935
opt_param_norm 62.5702 [62.5534, 62.5940] 62.5264 [62.4456, 62.5727] 0.999
opt_grad_steps 2980.5000 [2980.5000, 2980.5000] 2981.5000 [2981.5000, 2981.5000] 1.000
retnorm_scale_derived 8.6317 [8.5353, 8.7459] 8.5938 [8.5509, 8.6268] 0.996
retnorm_scale_exact 8.5519 [8.0623, 9.0591] nan [nan, nan] nan

window ending at 10000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0221 [0.0206, 0.0241] 0.0171 [0.0166, 0.0177] 0.776
rew_loss 0.6726 [0.6721, 0.6729] 0.6717 [0.6710, 0.6731] 0.999
con_loss 0.0533 [0.0494, 0.0606] 0.0519 [0.0497, 0.0541] 0.973
dyn_loss 1.0713 [1.0657, 1.0779] 1.0810 [1.0521, 1.0963] 1.009
rep_loss 1.0713 [1.0657, 1.0779] 1.0810 [1.0521, 1.0963] 1.009
actor_loss 0.1205 [0.1041, 0.1412] 0.1308 [0.1115, 0.1567] 1.085
critic_loss 3.7306 [3.3833, 4.2013] 2.8739 [2.8098, 2.9514] 0.770
repval_loss 5.5841 [5.3865, 5.8249] 5.1610 [4.7928, 5.3489] 0.924
ent/action/mean 0.2675 [0.2239, 0.3308] 0.2360 [0.2174, 0.2701] 0.882
rand/action/mean 0.3859 [0.3230, 0.4772] 0.3405 [0.3136, 0.3897] 0.882
ret/mean 23.0995 [20.7810, 24.8877] 29.5013 [29.0529, 29.9280] 1.277
ret/std 3.5098 [3.4467, 3.5909] 3.8739 [3.7988, 3.9985] 1.104
ret_normed/std 0.3726 [0.3557, 0.3825] 0.4364 [0.4338, 0.4380] 1.171
val/mean 18.4013 [16.0371, 20.1592] 24.1270 [23.6146, 24.5681] 1.311
adv/std 0.3464 [0.3398, 0.3527] 0.3971 [0.3931, 0.4013] 1.146
replay_ret/mean 32.0974 [29.4892, 33.6266] 36.9148 [36.5909, 37.1556] 1.150
weight/mean 0.9001 [0.8910, 0.9053] 0.9154 [0.9110, 0.9193] 1.017
rew/mean 0.9856 [0.9850, 0.9862] 0.9862 [0.9858, 0.9866] 1.001
prior_ent/mean 36.6881 [36.1362, 37.2451] 37.8955 [37.3968, 38.5125] 1.033
post_ent/mean 34.8842 [34.2875, 35.5943] 36.5290 [35.8578, 37.3137] 1.047
td_error 5.0343 [4.9191, 5.1653] 5.6590 [5.6107, 5.6847] 1.124
data_rew/mean 0.9845 [0.9830, 0.9859] 0.9839 [0.9827, 0.9850] 0.999
pred_rew/mean 0.9856 [0.9850, 0.9862] 0.9862 [0.9858, 0.9866] 1.001
activation/embed 0.4026 [0.3827, 0.4180] 0.3796 [0.3616, 0.3945] 0.943
opt_loss 7.4528 [7.0195, 8.0247] 6.4828 [6.3038, 6.6267] 0.870
opt_grad_norm 13.6103 [11.7731, 15.0260] 9.3156 [8.1343, 10.5228] 0.684
opt_update_norm 0.0135 [0.0125, 0.0154] 0.0119 [0.0103, 0.0128] 0.880
opt_param_norm 63.4467 [63.3995, 63.4814] 63.2954 [63.2447, 63.3218] 0.998
opt_grad_steps 3980.5000 [3980.5000, 3980.5000] 3981.5000 [3981.5000, 3981.5000] 1.000
retnorm_scale_derived 9.4339 [9.0761, 10.0959] 8.8773 [8.6723, 9.1441] 0.941
retnorm_scale_exact 9.6607 [9.3692, 9.8252] nan [nan, nan] nan

window ending at 12000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0218 [0.0146, 0.0310] 0.0205 [0.0174, 0.0236] 0.939
rew_loss 0.6640 [0.6635, 0.6647] 0.6643 [0.6636, 0.6650] 1.000
con_loss 0.0520 [0.0460, 0.0632] 0.0504 [0.0468, 0.0537] 0.969
dyn_loss 1.0782 [1.0695, 1.0922] 1.0869 [1.0753, 1.0955] 1.008
rep_loss 1.0782 [1.0695, 1.0922] 1.0869 [1.0753, 1.0955] 1.008
actor_loss 0.1003 [0.0812, 0.1200] 0.1095 [0.0929, 0.1333] 1.092
critic_loss 3.0320 [2.5848, 3.6424] 2.5396 [2.5048, 2.5626] 0.838
repval_loss 4.9160 [4.4863, 5.3032] 4.6666 [4.4581, 4.8484] 0.949
ent/action/mean 0.2211 [0.1878, 0.2738] 0.2133 [0.2047, 0.2283] 0.965
rand/action/mean 0.3189 [0.2709, 0.3949] 0.3077 [0.2954, 0.3294] 0.965
ret/mean 30.6650 [23.4510, 35.7573] 38.4030 [37.6805, 39.5081] 1.252
ret/std 4.0992 [3.4197, 4.6644] 4.5209 [4.2877, 4.8910] 1.103
ret_normed/std 0.3890 [0.3477, 0.4181] 0.4537 [0.4116, 0.4962] 1.166
val/mean 25.5937 [18.7523, 30.5098] 32.8359 [32.1808, 33.7802] 1.283
adv/std 0.3528 [0.3264, 0.3909] 0.3840 [0.3068, 0.4295] 1.088
replay_ret/mean 38.9354 [32.3867, 43.0934] 45.0535 [43.5066, 46.5038] 1.157
weight/mean 0.9210 [0.9030, 0.9310] 0.9257 [0.9203, 0.9286] 1.005
rew/mean 0.9934 [0.9929, 0.9937] 0.9930 [0.9927, 0.9934] 1.000
prior_ent/mean 35.0370 [32.0803, 37.0921] 36.5117 [35.7166, 37.3028] 1.042
post_ent/mean 33.1694 [29.4615, 35.5571] 35.0096 [34.1714, 35.8994] 1.055
td_error 5.4972 [4.9728, 5.7623] 5.8297 [5.7252, 5.8845] 1.060
data_rew/mean 0.9865 [0.9852, 0.9879] 0.9864 [0.9851, 0.9870] 1.000
pred_rew/mean 0.9934 [0.9929, 0.9937] 0.9930 [0.9927, 0.9934] 1.000
activation/embed 0.3981 [0.3731, 0.4165] 0.3798 [0.3629, 0.3928] 0.954
opt_loss 6.5310 [5.9136, 7.2914] 5.9800 [5.9199, 6.0433] 0.916
opt_grad_norm 15.3753 [9.7750, 19.2086] 10.2577 [9.8210, 10.5847] 0.667
opt_update_norm 0.0092 [0.0085, 0.0102] 0.0107 [0.0101, 0.0116] 1.174
opt_param_norm 64.2878 [64.2381, 64.3333] 64.1683 [64.0784, 64.2904] 0.998
opt_grad_steps 4980.5000 [4980.5000, 4980.5000] 4981.5000 [4981.5000, 4981.5000] 1.000
retnorm_scale_derived 10.5125 [9.8349, 11.6236] 10.0659 [8.6411, 11.8821] 0.958
retnorm_scale_exact 12.0872 [10.8630, 14.4546] nan [nan, nan] nan

window ending at 14000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0184 [0.0163, 0.0213] 0.0213 [0.0174, 0.0252] 1.157
rew_loss 0.6612 [0.6607, 0.6621] 0.6616 [0.6608, 0.6622] 1.000
con_loss 0.0503 [0.0444, 0.0574] 0.0486 [0.0442, 0.0529] 0.967
dyn_loss 1.0824 [1.0767, 1.0887] 1.0866 [1.0816, 1.0947] 1.004
rep_loss 1.0824 [1.0767, 1.0887] 1.0866 [1.0816, 1.0947] 1.004
actor_loss 0.0595 [0.0304, 0.0779] 0.0665 [0.0307, 0.0865] 1.117
critic_loss 2.4885 [2.2229, 2.8456] 2.3058 [2.1927, 2.4777] 0.927
repval_loss 4.4561 [4.0837, 4.7583] 4.5631 [4.3203, 4.8157] 1.024
ent/action/mean 0.2136 [0.1781, 0.2689] 0.1988 [0.1681, 0.2465] 0.930
rand/action/mean 0.3082 [0.2569, 0.3880] 0.2867 [0.2425, 0.3557] 0.930
ret/mean 38.4189 [31.0900, 43.2656] 46.7378 [45.5936, 48.1135] 1.217
ret/std 6.6936 [5.3542, 9.0016] 6.3234 [5.1351, 7.9023] 0.945
ret_normed/std 0.3648 [0.3363, 0.4091] 0.4222 [0.3209, 0.5266] 1.158
val/mean 33.5387 [26.3796, 38.5006] 41.3328 [39.9229, 42.7495] 1.232
adv/std 0.2475 [0.1765, 0.3399] 0.3155 [0.1984, 0.4527] 1.275
replay_ret/mean 45.7154 [39.2884, 49.1743] 52.7677 [51.5549, 54.4375] 1.154
weight/mean 0.9238 [0.9135, 0.9347] 0.9299 [0.9266, 0.9319] 1.007
rew/mean 0.9961 [0.9959, 0.9963] 0.9960 [0.9959, 0.9962] 1.000
prior_ent/mean 34.9873 [31.2971, 37.5473] 35.3672 [34.4461, 36.2464] 1.011
post_ent/mean 33.4484 [29.4271, 36.3723] 33.8216 [32.7594, 34.7699] 1.011
td_error 5.4663 [5.1900, 5.8117] 5.7563 [5.5178, 6.0344] 1.053
data_rew/mean 0.9876 [0.9867, 0.9891] 0.9880 [0.9868, 0.9887] 1.000
pred_rew/mean 0.9961 [0.9959, 0.9963] 0.9960 [0.9959, 0.9962] 1.000
activation/embed 0.4029 [0.3676, 0.4211] 0.3784 [0.3632, 0.3878] 0.939
opt_loss 5.8054 [5.3890, 6.2878] 5.6680 [5.5184, 5.8712] 0.976
opt_grad_norm 14.1391 [11.0843, 18.2200] 12.2113 [10.4470, 14.5420] 0.864
opt_update_norm 0.0095 [0.0090, 0.0098] 0.0104 [0.0098, 0.0111] 1.098
opt_param_norm 65.2318 [65.1372, 65.3964] 65.1407 [65.0182, 65.2818] 0.999
opt_grad_steps 5980.5000 [5980.5000, 5980.5000] 5981.5000 [5981.5000, 5981.5000] 1.000
retnorm_scale_derived 18.6382 [13.0871, 25.8031] 16.1778 [9.7505, 24.6291] 0.868
retnorm_scale_exact 26.7607 [19.1344, 36.9630] nan [nan, nan] nan

window ending at 16000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0192 [0.0182, 0.0199] 0.0193 [0.0147, 0.0234] 1.001
rew_loss 0.6602 [0.6596, 0.6610] 0.6604 [0.6598, 0.6607] 1.000
con_loss 0.0489 [0.0438, 0.0521] 0.0480 [0.0436, 0.0512] 0.982
dyn_loss 1.0875 [1.0809, 1.0983] 1.1045 [1.0989, 1.1104] 1.016
rep_loss 1.0875 [1.0809, 1.0983] 1.1045 [1.0989, 1.1104] 1.016
actor_loss 0.0265 [0.0182, 0.0361] 0.0329 [0.0197, 0.0519] 1.241
critic_loss 2.3658 [2.2272, 2.5371] 1.9897 [1.9344, 2.0729] 0.841
repval_loss 4.3228 [4.2180, 4.4333] 4.1513 [3.9327, 4.4051] 0.960
ent/action/mean 0.1825 [0.1604, 0.2207] 0.1500 [0.1350, 0.1608] 0.822
rand/action/mean 0.2632 [0.2313, 0.3184] 0.2164 [0.1948, 0.2320] 0.822
ret/mean 47.0313 [41.1534, 49.9942] 57.5719 [56.7033, 58.0750] 1.224
ret/std 9.9390 [8.7451, 11.3307] 8.6063 [5.1580, 11.6254] 0.866
ret_normed/std 0.3204 [0.3144, 0.3319] 0.3863 [0.3310, 0.4908] 1.206
val/mean 41.8066 [35.8814, 44.9817] 52.2728 [51.3622, 52.8784] 1.250
adv/std 0.1614 [0.1329, 0.1962] 0.2629 [0.1277, 0.4912] 1.629
replay_ret/mean 53.1812 [47.9805, 56.5700] 62.4652 [60.7286, 63.8527] 1.175
weight/mean 0.9313 [0.9272, 0.9357] 0.9370 [0.9314, 0.9445] 1.006
rew/mean 0.9975 [0.9973, 0.9977] 0.9975 [0.9975, 0.9976] 1.000
prior_ent/mean 35.6103 [33.5334, 37.7436] 35.2595 [33.9141, 36.9597] 0.990
post_ent/mean 34.1449 [31.8672, 36.5272] 33.7996 [32.2987, 35.6512] 0.990
td_error 5.7671 [5.3998, 6.2072] 5.7009 [5.6041, 5.8640] 0.989
data_rew/mean 0.9887 [0.9880, 0.9900] 0.9890 [0.9882, 0.9895] 1.000
pred_rew/mean 0.9975 [0.9973, 0.9977] 0.9975 [0.9975, 0.9976] 1.000
activation/embed 0.4060 [0.3677, 0.4257] 0.3807 [0.3690, 0.3924] 0.938
opt_loss 5.6137 [5.4494, 5.7912] 5.2105 [5.1074, 5.3846] 0.928
opt_grad_norm 12.0443 [10.5490, 14.3380] 11.5522 [10.4987, 12.6389] 0.959
opt_update_norm 0.0103 [0.0093, 0.0116] 0.0101 [0.0097, 0.0104] 0.984
opt_param_norm 66.1915 [66.0331, 66.4361] 66.0416 [65.8934, 66.3352] 0.998
opt_grad_steps 6980.5000 [6980.5000, 6980.5000] 6981.5000 [6981.5000, 6981.5000] 1.000
retnorm_scale_derived 31.0526 [27.7686, 36.0422] 24.1433 [10.5097, 35.1233] 0.777
retnorm_scale_exact 30.5203 [26.0859, 33.2625] nan [nan, nan] nan

window ending at 18000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0214 [0.0198, 0.0229] 0.0184 [0.0129, 0.0248] 0.860
rew_loss 0.6595 [0.6591, 0.6601] 0.6598 [0.6596, 0.6602] 1.000
con_loss 0.0495 [0.0464, 0.0515] 0.0470 [0.0444, 0.0489] 0.949
dyn_loss 1.0991 [1.0925, 1.1073] 1.1099 [1.0992, 1.1192] 1.010
rep_loss 1.0991 [1.0925, 1.1073] 1.1099 [1.0992, 1.1192] 1.010
actor_loss 0.0178 [0.0134, 0.0228] 0.0216 [0.0104, 0.0379] 1.214
critic_loss 2.0699 [1.9038, 2.3326] 1.8503 [1.8148, 1.9143] 0.894
repval_loss 3.9176 [3.6928, 4.2501] 3.8770 [3.5701, 4.0893] 0.990
ent/action/mean 0.1596 [0.1356, 0.1823] 0.1352 [0.1307, 0.1406] 0.847
rand/action/mean 0.2303 [0.1956, 0.2630] 0.1950 [0.1886, 0.2028] 0.847
ret/mean 57.8162 [52.6999, 62.0324] 67.7322 [65.7089, 70.0568] 1.172
ret/std 12.9139 [11.5943, 13.8887] 10.9500 [6.2882, 16.1412] 0.848
ret_normed/std 0.3364 [0.3233, 0.3569] 0.3525 [0.3257, 0.4029] 1.048
val/mean 52.7591 [47.1716, 56.8058] 62.6696 [60.6629, 65.1633] 1.188
adv/std 0.1294 [0.1235, 0.1405] 0.1969 [0.1050, 0.3257] 1.521
replay_ret/mean 62.1822 [57.9170, 66.4070] 71.1455 [69.0065, 73.5685] 1.144
weight/mean 0.9349 [0.9310, 0.9389] 0.9385 [0.9361, 0.9398] 1.004
rew/mean 0.9983 [0.9982, 0.9984] 0.9983 [0.9982, 0.9984] 1.000
prior_ent/mean 35.1675 [33.0074, 36.9673] 34.8135 [34.4458, 35.4919] 0.990
post_ent/mean 33.7520 [31.4949, 35.6049] 33.4034 [33.0307, 34.1167] 0.990
td_error 5.5812 [5.0256, 5.9382] 5.6056 [5.5147, 5.6763] 1.004
data_rew/mean 0.9893 [0.9887, 0.9901] 0.9897 [0.9893, 0.9902] 1.000
pred_rew/mean 0.9983 [0.9982, 0.9984] 0.9983 [0.9982, 0.9984] 1.000
activation/embed 0.4083 [0.3684, 0.4344] 0.3842 [0.3720, 0.3979] 0.941
opt_loss 5.2025 [4.9742, 5.5660] 4.9811 [4.8552, 5.0781] 0.957
opt_grad_norm 10.1187 [8.6297, 10.9022] 9.8298 [9.2521, 10.2911] 0.971
opt_update_norm 0.0108 [0.0100, 0.0116] 0.0104 [0.0094, 0.0110] 0.957
opt_param_norm 66.9227 [66.7928, 67.1615] 66.7709 [66.4631, 67.2166] 0.998
opt_grad_steps 7980.5000 [7980.5000, 7980.5000] 7981.5000 [7981.5000, 7981.5000] 1.000
retnorm_scale_derived 38.4084 [35.8606, 42.2197] 32.2216 [15.6079, 49.0654] 0.839
retnorm_scale_exact 42.5097 [39.1885, 47.9422] nan [nan, nan] nan

window ending at 20000 rows

metric (Ajax name) ref mean [min, max] ajax mean [min, max] ajax/ref
rec_loss 0.0208 [0.0197, 0.0215] 0.0186 [0.0144, 0.0230] 0.895
rew_loss 0.6597 [0.6593, 0.6601] 0.6598 [0.6595, 0.6604] 1.000
con_loss 0.0480 [0.0464, 0.0492] 0.0463 [0.0449, 0.0487] 0.964
dyn_loss 1.1234 [1.1036, 1.1432] 1.1120 [1.0854, 1.1319] 0.990
rep_loss 1.1234 [1.1036, 1.1432] 1.1120 [1.0854, 1.1319] 0.990
actor_loss 0.0100 [0.0081, 0.0118] 0.0157 [0.0088, 0.0214] 1.570
critic_loss 1.7873 [1.6568, 1.9421] 1.8680 [1.7326, 2.0645] 1.045
repval_loss 3.5200 [3.3229, 3.6609] 3.6394 [3.2858, 3.8443] 1.034
ent/action/mean 0.1296 [0.1205, 0.1428] 0.1442 [0.1354, 0.1585] 1.112
rand/action/mean 0.1870 [0.1738, 0.2060] 0.2080 [0.1954, 0.2287] 1.112
ret/mean 70.9764 [67.7061, 74.2263] 74.0936 [68.1780, 78.8971] 1.044
ret/std 16.5698 [15.1234, 17.4961] 14.6161 [10.2626, 20.1408] 0.882
ret_normed/std 0.3459 [0.3327, 0.3631] 0.3495 [0.3330, 0.3785] 1.011
val/mean 66.3418 [63.1841, 68.9627] 69.0034 [63.0574, 74.0279] 1.040
adv/std 0.1098 [0.1073, 0.1127] 0.1565 [0.0918, 0.2213] 1.426
replay_ret/mean 73.6483 [71.0874, 76.8707] 76.9562 [71.4557, 81.9675] 1.045
weight/mean 0.9385 [0.9361, 0.9429] 0.9403 [0.9376, 0.9437] 1.002
rew/mean 0.9988 [0.9987, 0.9988] 0.9988 [0.9987, 0.9988] 1.000
prior_ent/mean 35.5052 [33.7287, 36.9722] 34.5989 [32.9537, 35.6120] 0.974
post_ent/mean 34.1391 [32.3469, 35.7094] 33.0877 [31.0357, 34.3483] 0.969
td_error 5.4134 [4.9820, 6.0644] 5.8345 [5.6067, 6.0531] 1.078
data_rew/mean 0.9904 [0.9897, 0.9910] 0.9904 [0.9899, 0.9914] 1.000
pred_rew/mean 0.9988 [0.9987, 0.9988] 0.9988 [0.9987, 0.9988] 1.000
activation/embed 0.4126 [0.3749, 0.4370] 0.3886 [0.3753, 0.4055] 0.942
opt_loss 4.8175 [4.6485, 5.0123] 4.9235 [4.7665, 5.1797] 1.022
opt_grad_norm 9.6321 [8.8011, 10.0734] 11.2106 [9.7753, 13.1750] 1.164
opt_update_norm 0.0117 [0.0107, 0.0128] 0.0104 [0.0101, 0.0107] 0.887
opt_param_norm 67.4997 [67.2872, 67.8669] 67.3097 [66.8619, 67.8456] 0.997
opt_grad_steps 8980.5000 [8980.5000, 8980.5000] 8981.5000 [8981.5000, 8981.5000] 1.000
retnorm_scale_derived 47.9331 [44.2428, 51.3660] 42.4144 [27.1152, 59.7552] 0.885
retnorm_scale_exact 57.1577 [54.4640, 60.0010] 52.6593 [42.5683, 69.0615] 0.921

Training metrics per seed (ref / ajax)

seed 0

metric 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
rec_loss 0.403 / 0.452 0.065 / 0.074 0.022 / 0.019 0.015 / 0.015 0.024 / 0.017 0.031 / 0.024 0.021 / 0.025 0.020 / 0.023 0.023 / 0.025 0.021 / 0.023
rew_loss 5.294 / 5.291 2.519 / 2.513 0.812 / 0.805 0.693 / 0.691 0.673 / 0.671 0.663 / 0.664 0.661 / 0.661 0.660 / 0.660 0.659 / 0.660 0.660 / 0.659
con_loss 0.409 / 0.323 0.152 / 0.155 0.080 / 0.084 0.057 / 0.058 0.061 / 0.054 0.063 / 0.054 0.057 / 0.053 0.052 / 0.051 0.052 / 0.049 0.049 / 0.049
dyn_loss 2.533 / 2.685 1.012 / 1.010 1.038 / 1.034 1.047 / 1.044 1.070 / 1.052 1.073 / 1.075 1.089 / 1.095 1.083 / 1.099 1.093 / 1.111 1.104 / 1.119
rep_loss 2.533 / 2.685 1.012 / 1.010 1.038 / 1.034 1.047 / 1.044 1.070 / 1.052 1.073 / 1.075 1.089 / 1.095 1.083 / 1.099 1.093 / 1.111 1.104 / 1.119
actor_loss -0.009 / -0.009 0.167 / 0.152 0.162 / 0.155 0.131 / 0.135 0.141 / 0.157 0.120 / 0.102 0.078 / 0.087 0.036 / 0.027 0.023 / 0.017 0.012 / 0.017
critic_loss 3.741 / 4.023 5.687 / 5.530 4.065 / 4.089 3.136 / 3.149 4.201 / 2.810 3.642 / 2.563 2.846 / 2.193 2.537 / 1.934 2.333 / 1.914 1.763 / 2.065
repval_loss 10.193 / 10.164 7.542 / 7.480 6.026 / 6.002 5.306 / 5.256 5.825 / 4.793 5.303 / 4.848 4.758 / 4.320 4.317 / 4.116 4.250 / 3.971 3.576 / 3.788
ent/action/mean 0.691 / 0.693 0.477 / 0.460 0.281 / 0.270 0.234 / 0.233 0.331 / 0.270 0.274 / 0.205 0.269 / 0.247 0.221 / 0.161 0.182 / 0.134 0.143 / 0.159
rand/action/mean 0.997 / 0.999 0.687 / 0.663 0.406 / 0.389 0.338 / 0.337 0.477 / 0.390 0.395 / 0.295 0.388 / 0.356 0.318 / 0.232 0.263 / 0.194 0.206 / 0.229
ret/mean 0.075 / 0.073 3.284 / 3.271 13.504 / 13.560 22.079 / 22.105 20.781 / 29.928 23.451 / 37.680 31.090 / 46.506 41.153 / 56.703 52.700 / 65.709 67.706 / 68.178
ret/std 0.010 / 0.010 0.744 / 0.786 2.886 / 2.879 3.657 / 3.587 3.591 / 3.799 3.420 / 4.384 5.725 / 5.933 8.745 / 9.035 11.594 / 10.421 15.123 / 13.445
ret_normed/std 0.010 / 0.010 0.269 / 0.286 0.380 / 0.375 0.418 / 0.420 0.356 / 0.438 0.348 / 0.453 0.336 / 0.419 0.315 / 0.337 0.323 / 0.326 0.342 / 0.333
val/mean 0.100 / 0.097 1.785 / 1.757 8.763 / 8.702 16.963 / 16.883 16.037 / 24.568 18.752 / 32.181 26.380 / 41.326 35.881 / 51.362 47.172 / 60.663 63.184 / 63.057
adv/std 0.010 / 0.010 0.269 / 0.286 0.379 / 0.375 0.397 / 0.408 0.340 / 0.397 0.326 / 0.429 0.226 / 0.295 0.155 / 0.170 0.141 / 0.160 0.107 / 0.157
replay_ret/mean 8.040 / 7.896 10.648 / 10.251 20.671 / 20.072 29.747 / 28.864 29.489 / 36.591 32.387 / 43.507 39.288 / 51.555 47.980 / 60.729 57.917 / 69.007 71.087 / 71.456
weight/mean 0.348 / 0.372 0.742 / 0.731 0.859 / 0.862 0.903 / 0.903 0.891 / 0.916 0.903 / 0.920 0.914 / 0.927 0.931 / 0.931 0.935 / 0.936 0.936 / 0.938
rew/mean 0.062 / 0.062 0.348 / 0.347 0.904 / 0.905 0.972 / 0.971 0.985 / 0.986 0.993 / 0.993 0.996 / 0.996 0.997 / 0.997 0.998 / 0.998 0.999 / 0.999
prior_ent/mean 43.247 / 43.304 43.552 / 43.692 41.640 / 42.488 39.854 / 40.569 36.683 / 38.512 32.080 / 36.516 31.297 / 35.409 33.533 / 34.905 33.007 / 34.503 33.729 / 32.954
post_ent/mean 42.399 / 42.119 42.655 / 42.779 40.603 / 41.591 38.628 / 39.526 34.771 / 37.314 29.461 / 34.958 29.427 / 33.936 31.867 / 33.449 31.495 / 33.031 32.347 / 31.036
td_error 0.025 / 0.024 1.523 / 1.550 4.981 / 5.078 5.537 / 5.636 5.165 / 5.682 4.973 / 5.879 5.190 / 5.518 5.694 / 5.635 5.938 / 5.626 5.194 / 6.053
data_rew/mean 0.958 / 0.956 0.960 / 0.957 0.972 / 0.970 0.980 / 0.978 0.983 / 0.983 0.985 / 0.985 0.987 / 0.987 0.988 / 0.988 0.989 / 0.989 0.990 / 0.990
pred_rew/mean 0.062 / 0.062 0.348 / 0.347 0.904 / 0.905 0.972 / 0.971 0.985 / 0.986 0.993 / 0.993 0.996 / 0.996 0.997 / 0.997 0.998 / 0.998 0.999 / 0.999
activation/embed 0.420 / 0.393 0.421 / 0.388 0.418 / 0.386 0.414 / 0.384 0.407 / 0.383 0.405 / 0.384 0.421 / 0.384 0.426 / 0.381 0.422 / 0.382 0.426 / 0.385
opt_loss 15.682 / 16.081 11.965 / 11.779 8.090 / 8.090 6.775 / 6.774 8.025 / 6.304 7.291 / 6.043 6.288 / 5.518 5.791 / 5.139 5.566 / 5.078 4.792 / 5.180
opt_grad_norm 163.192 / 131.611 8.999 / 8.367 7.652 / 7.043 6.939 / 6.345 14.032 / 8.134 19.209 / 10.585 18.220 / 10.447 14.338 / 11.519 10.824 / 10.291 10.022 / 13.175
opt_update_norm 0.007 / 0.007 0.014 / 0.014 0.008 / 0.008 0.009 / 0.009 0.015 / 0.010 0.010 / 0.012 0.009 / 0.010 0.009 / 0.010 0.010 / 0.011 0.011 / 0.010
opt_param_norm 59.881 / 59.906 60.545 / 60.565 61.860 / 61.874 62.563 / 62.573 63.481 / 63.322 64.333 / 64.290 65.137 / 65.282 66.033 / 66.335 66.793 / 67.217 67.287 / 67.846
opt_grad_steps 240.500 / 241.000 980.500 / 981.500 1980.500 / 1981.500 2980.500 / 2981.500 3980.500 / 3981.500 4980.500 / 4981.500 5980.500 / 5981.500 6980.500 / 6981.500 7980.500 / 7981.500 8980.500 / 8981.500
retnorm_scale_derived 1.000 / 1.000 2.770 / 2.747 7.606 / 7.679 8.746 / 8.551 10.096 / 8.672 9.835 / 9.675 17.024 / 14.154 27.769 / 26.797 35.861 / 31.992 44.243 / 40.373
retnorm_scale_exact 1.000 / nan 5.970 / nan 8.385 / nan 9.059 / nan 9.788 / nan 10.863 / nan 24.185 / nan 33.262 / nan 39.188 / nan 54.464 / 46.348

seed 1

metric 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
rec_loss 0.386 / 0.441 0.063 / 0.061 0.018 / 0.024 0.013 / 0.016 0.021 / 0.017 0.020 / 0.021 0.018 / 0.021 0.018 / 0.020 0.020 / 0.017 0.020 / 0.018
rew_loss 5.279 / 5.282 2.481 / 2.490 0.811 / 0.802 0.693 / 0.690 0.673 / 0.671 0.665 / 0.664 0.662 / 0.662 0.661 / 0.661 0.660 / 0.660 0.660 / 0.659
con_loss 0.398 / 0.363 0.141 / 0.152 0.075 / 0.082 0.054 / 0.057 0.049 / 0.052 0.046 / 0.047 0.044 / 0.044 0.044 / 0.044 0.046 / 0.044 0.046 / 0.045
dyn_loss 2.540 / 2.173 1.011 / 1.009 1.036 / 1.039 1.059 / 1.060 1.078 / 1.095 1.092 / 1.090 1.082 / 1.084 1.081 / 1.104 1.098 / 1.099 1.123 / 1.085
rep_loss 2.540 / 2.173 1.011 / 1.009 1.036 / 1.039 1.059 / 1.060 1.078 / 1.095 1.092 / 1.090 1.082 / 1.084 1.081 / 1.104 1.098 / 1.099 1.123 / 1.085
actor_loss -0.007 / -0.011 0.151 / 0.146 0.150 / 0.146 0.138 / 0.136 0.116 / 0.112 0.100 / 0.133 0.070 / 0.082 0.025 / 0.052 0.017 / 0.038 0.010 / 0.021
critic_loss 3.918 / 3.775 5.744 / 5.476 4.154 / 3.957 3.179 / 3.090 3.607 / 2.860 2.869 / 2.505 2.397 / 2.247 2.333 / 2.073 1.973 / 1.815 1.942 / 1.733
repval_loss 10.236 / 10.136 7.577 / 7.425 6.069 / 5.932 5.372 / 5.268 5.541 / 5.341 4.958 / 4.693 4.526 / 4.553 4.433 / 4.405 3.810 / 4.089 3.661 / 3.844
ent/action/mean 0.691 / 0.691 0.448 / 0.448 0.258 / 0.270 0.236 / 0.237 0.248 / 0.217 0.202 / 0.228 0.194 / 0.182 0.160 / 0.135 0.161 / 0.141 0.126 / 0.135
rand/action/mean 0.998 / 0.997 0.646 / 0.647 0.372 / 0.390 0.340 / 0.342 0.358 / 0.314 0.291 / 0.329 0.280 / 0.262 0.231 / 0.195 0.232 / 0.203 0.181 / 0.195
ret/mean 0.072 / 0.079 3.356 / 3.327 13.727 / 13.648 22.216 / 22.465 23.630 / 29.523 32.787 / 39.508 40.901 / 48.114 49.994 / 57.937 62.032 / 67.431 74.226 / 75.206
ret/std 0.008 / 0.011 0.830 / 0.818 2.906 / 3.110 3.462 / 3.853 3.492 / 3.998 4.214 / 4.288 5.354 / 5.135 9.741 / 5.158 13.889 / 6.288 17.090 / 10.263
ret_normed/std 0.008 / 0.011 0.281 / 0.285 0.373 / 0.388 0.406 / 0.448 0.382 / 0.437 0.418 / 0.496 0.409 / 0.527 0.332 / 0.491 0.329 / 0.403 0.333 / 0.378
val/mean 0.091 / 0.107 1.817 / 1.831 8.866 / 9.019 16.996 / 17.377 19.008 / 24.198 27.519 / 33.780 35.736 / 42.749 44.557 / 52.578 56.806 / 62.183 68.963 / 69.925
adv/std 0.008 / 0.011 0.281 / 0.285 0.373 / 0.387 0.390 / 0.417 0.353 / 0.401 0.391 / 0.416 0.340 / 0.453 0.196 / 0.491 0.124 / 0.326 0.113 / 0.221
replay_ret/mean 8.226 / 7.932 10.932 / 10.301 21.429 / 20.179 30.451 / 29.418 33.176 / 36.998 41.326 / 46.504 48.684 / 54.438 56.570 / 63.853 66.407 / 70.862 76.871 / 77.445
weight/mean 0.364 / 0.351 0.749 / 0.729 0.876 / 0.849 0.911 / 0.896 0.904 / 0.911 0.929 / 0.929 0.935 / 0.932 0.936 / 0.945 0.939 / 0.940 0.943 / 0.940
rew/mean 0.063 / 0.062 0.357 / 0.354 0.908 / 0.908 0.973 / 0.973 0.986 / 0.987 0.994 / 0.993 0.996 / 0.996 0.998 / 0.998 0.998 / 0.998 0.999 / 0.999
prior_ent/mean 43.503 / 43.270 43.591 / 43.349 42.185 / 41.585 40.705 / 38.762 37.245 / 37.777 37.092 / 37.303 36.118 / 36.246 35.554 / 36.960 35.528 / 35.492 36.972 / 35.231
post_ent/mean 41.871 / 42.436 42.701 / 42.313 41.162 / 40.471 39.501 / 37.437 35.594 / 36.415 35.557 / 35.899 34.546 / 34.770 34.040 / 35.651 34.156 / 34.117 35.709 / 33.879
td_error 0.019 / 0.028 1.557 / 1.535 5.068 / 5.001 5.555 / 5.590 4.919 / 5.611 5.756 / 5.884 5.812 / 5.717 6.207 / 5.864 5.780 / 5.676 6.064 / 5.844
data_rew/mean 0.960 / 0.956 0.962 / 0.957 0.974 / 0.970 0.982 / 0.979 0.986 / 0.984 0.988 / 0.987 0.989 / 0.988 0.990 / 0.990 0.990 / 0.989 0.991 / 0.990
pred_rew/mean 0.063 / 0.062 0.357 / 0.354 0.908 / 0.908 0.973 / 0.973 0.986 / 0.987 0.994 / 0.993 0.996 / 0.996 0.998 / 0.998 0.998 / 0.998 0.999 / 0.999
activation/embed 0.438 / 0.375 0.434 / 0.384 0.421 / 0.385 0.405 / 0.381 0.383 / 0.362 0.373 / 0.363 0.368 / 0.363 0.368 / 0.369 0.368 / 0.372 0.375 / 0.375
opt_loss 15.840 / 15.280 11.967 / 11.662 8.169 / 7.933 6.854 / 6.736 7.314 / 6.518 6.388 / 5.977 5.739 / 5.614 5.600 / 5.385 5.067 / 5.010 5.012 / 4.824
opt_grad_norm 149.289 / 191.838 8.571 / 9.002 7.554 / 8.016 7.666 / 7.568 15.026 / 9.290 17.142 / 10.368 13.113 / 14.542 11.246 / 12.639 10.902 / 9.946 10.073 / 9.775
opt_update_norm 0.007 / 0.007 0.014 / 0.014 0.008 / 0.008 0.010 / 0.009 0.013 / 0.013 0.009 / 0.010 0.010 / 0.010 0.010 / 0.010 0.011 / 0.009 0.012 / 0.010
opt_param_norm 59.908 / 59.777 60.576 / 60.458 61.882 / 61.756 62.594 / 62.446 63.459 / 63.245 64.238 / 64.078 65.162 / 65.018 66.105 / 65.893 66.814 / 66.633 67.345 / 67.222
opt_grad_steps 240.500 / 241.000 980.500 / 981.500 1980.500 / 1981.500 2980.500 / 2981.500 3980.500 / 3981.500 4980.500 / 4981.500 5980.500 / 5981.500 6980.500 / 6981.500 7980.500 / 7981.500 8980.500 / 8981.500
retnorm_scale_derived 1.000 / 1.000 2.953 / 2.868 7.795 / 8.019 8.535 / 8.604 9.130 / 9.144 10.079 / 8.641 13.087 / 9.751 29.347 / 10.510 42.220 / 15.608 51.366 / 27.115
retnorm_scale_exact 1.000 / nan 6.342 / nan 8.479 / nan 8.062 / nan 9.825 / nan 10.944 / nan 19.134 / nan 32.213 / nan 47.942 / nan 60.001 / 42.568

seed 2

metric 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
rec_loss 0.428 / 0.387 0.058 / 0.062 0.019 / 0.020 0.017 / 0.017 0.022 / 0.018 0.015 / 0.017 0.016 / 0.017 0.020 / 0.015 0.021 / 0.013 0.021 / 0.014
rew_loss 5.282 / 5.288 2.491 / 2.528 0.804 / 0.811 0.691 / 0.693 0.672 / 0.673 0.664 / 0.665 0.661 / 0.662 0.660 / 0.661 0.659 / 0.660 0.659 / 0.660
con_loss 0.366 / 0.351 0.152 / 0.151 0.078 / 0.082 0.052 / 0.056 0.050 / 0.050 0.047 / 0.051 0.049 / 0.049 0.051 / 0.049 0.051 / 0.048 0.048 / 0.045
dyn_loss 2.712 / 2.300 1.012 / 1.009 1.035 / 1.035 1.056 / 1.065 1.066 / 1.096 1.069 / 1.095 1.077 / 1.082 1.098 / 1.110 1.107 / 1.119 1.143 / 1.132
rep_loss 2.712 / 2.300 1.012 / 1.009 1.035 / 1.035 1.056 / 1.065 1.066 / 1.096 1.069 / 1.095 1.077 / 1.082 1.098 / 1.110 1.107 / 1.119 1.143 / 1.132
actor_loss -0.007 / -0.009 0.145 / 0.149 0.146 / 0.149 0.124 / 0.126 0.104 / 0.124 0.081 / 0.093 0.030 / 0.031 0.018 / 0.020 0.013 / 0.010 0.008 / 0.009
critic_loss 4.063 / 3.982 5.542 / 5.628 4.134 / 4.115 3.228 / 3.180 3.383 / 2.951 2.585 / 2.551 2.223 / 2.478 2.227 / 1.962 1.904 / 1.822 1.657 / 1.807
repval_loss 10.160 / 10.184 7.528 / 7.549 6.079 / 6.043 5.394 / 5.335 5.387 / 5.349 4.486 / 4.458 4.084 / 4.816 4.218 / 3.933 3.693 / 3.570 3.323 / 3.286
ent/action/mean 0.693 / 0.693 0.453 / 0.461 0.251 / 0.258 0.211 / 0.216 0.224 / 0.221 0.188 / 0.207 0.178 / 0.168 0.166 / 0.154 0.136 / 0.131 0.120 / 0.139
rand/action/mean 1.000 / 0.999 0.653 / 0.665 0.362 / 0.372 0.304 / 0.312 0.323 / 0.318 0.271 / 0.298 0.257 / 0.243 0.240 / 0.222 0.196 / 0.189 0.174 / 0.200
ret/mean 0.072 / 0.075 3.304 / 3.217 13.576 / 13.536 22.023 / 22.179 24.888 / 29.053 35.757 / 38.020 43.266 / 45.594 49.946 / 58.075 58.716 / 70.057 70.997 / 78.897
ret/std 0.008 / 0.009 0.798 / 0.749 2.823 / 2.858 3.391 / 3.570 3.447 / 3.824 4.664 / 4.891 9.002 / 7.902 11.331 / 11.625 13.259 / 16.141 17.496 / 20.141
ret_normed/std 0.008 / 0.009 0.280 / 0.267 0.366 / 0.371 0.394 / 0.414 0.380 / 0.434 0.401 / 0.412 0.349 / 0.321 0.314 / 0.331 0.357 / 0.329 0.363 / 0.337
val/mean 0.090 / 0.097 1.763 / 1.771 8.634 / 8.717 16.623 / 16.960 20.159 / 23.615 30.510 / 32.547 38.501 / 39.923 44.982 / 52.878 54.300 / 65.163 66.879 / 74.028
adv/std 0.008 / 0.009 0.280 / 0.267 0.366 / 0.371 0.386 / 0.410 0.347 / 0.393 0.341 / 0.307 0.176 / 0.198 0.133 / 0.128 0.124 / 0.105 0.109 / 0.092
replay_ret/mean 7.599 / 8.106 10.339 / 10.599 20.702 / 20.634 30.074 / 30.000 33.627 / 37.156 43.093 / 45.150 49.174 / 52.311 54.993 / 62.814 62.223 / 73.569 72.987 / 81.967
weight/mean 0.376 / 0.369 0.726 / 0.734 0.868 / 0.867 0.914 / 0.911 0.905 / 0.919 0.931 / 0.928 0.923 / 0.931 0.927 / 0.935 0.931 / 0.940 0.937 / 0.944
rew/mean 0.062 / 0.062 0.354 / 0.344 0.908 / 0.903 0.973 / 0.972 0.986 / 0.986 0.994 / 0.993 0.996 / 0.996 0.997 / 0.998 0.998 / 0.998 0.999 / 0.999
prior_ent/mean 43.185 / 43.366 43.405 / 43.728 42.285 / 42.482 40.744 / 40.828 36.136 / 37.397 35.939 / 35.717 37.547 / 34.446 37.744 / 33.914 36.967 / 34.446 35.815 / 35.612
post_ent/mean 41.677 / 42.407 42.372 / 42.842 41.256 / 41.569 39.451 / 39.626 34.287 / 35.858 34.490 / 34.171 36.372 / 32.759 36.527 / 32.299 35.605 / 33.063 34.361 / 34.348
td_error 0.018 / 0.022 1.569 / 1.466 5.121 / 5.025 5.709 / 5.645 5.018 / 5.685 5.762 / 5.725 5.397 / 6.034 5.400 / 5.604 5.026 / 5.515 4.982 / 5.607
data_rew/mean 0.954 / 0.957 0.958 / 0.960 0.971 / 0.972 0.981 / 0.980 0.985 / 0.985 0.986 / 0.987 0.987 / 0.989 0.988 / 0.989 0.989 / 0.990 0.990 / 0.991
pred_rew/mean 0.062 / 0.062 0.354 / 0.344 0.908 / 0.903 0.973 / 0.972 0.986 / 0.986 0.994 / 0.993 0.996 / 0.996 0.997 / 0.998 0.998 / 0.998 0.999 / 0.999
activation/embed 0.428 / 0.392 0.440 / 0.401 0.446 / 0.409 0.433 / 0.405 0.418 / 0.395 0.416 / 0.393 0.420 / 0.388 0.425 / 0.392 0.434 / 0.398 0.437 / 0.406
opt_loss 16.162 / 15.584 11.759 / 11.892 8.142 / 8.128 6.892 / 6.845 7.020 / 6.627 5.914 / 5.920 5.389 / 5.871 5.449 / 5.107 4.974 / 4.855 4.648 / 4.767
opt_grad_norm 141.765 / 132.879 9.473 / 8.697 7.940 / 7.461 7.781 / 7.673 11.773 / 10.523 9.775 / 9.821 11.084 / 11.645 10.549 / 10.499 8.630 / 9.252 8.801 / 10.682
opt_update_norm 0.007 / 0.007 0.013 / 0.014 0.008 / 0.008 0.009 / 0.008 0.013 / 0.013 0.009 / 0.011 0.010 / 0.011 0.012 / 0.010 0.012 / 0.011 0.013 / 0.011
opt_param_norm 59.876 / 59.870 60.538 / 60.534 61.846 / 61.860 62.553 / 62.561 63.399 / 63.320 64.292 / 64.136 65.396 / 65.122 66.436 / 65.896 67.162 / 66.463 67.867 / 66.862
opt_grad_steps 240.500 / 241.000 980.500 / 981.500 1980.500 / 1981.500 2980.500 / 2981.500 3980.500 / 3981.500 4980.500 / 4981.500 5980.500 / 5981.500 6980.500 / 6981.500 7980.500 / 7981.500 8980.500 / 8981.500
retnorm_scale_derived 1.000 / 1.000 2.849 / 2.800 7.718 / 7.694 8.614 / 8.627 9.076 / 8.816 11.624 / 11.882 25.803 / 24.629 36.042 / 35.123 37.145 / 49.065 48.191 / 59.755
retnorm_scale_exact 1.000 / nan 6.210 / nan 8.457 / nan 8.534 / nan 9.369 / nan 14.455 / nan 36.963 / nan 26.086 / nan 40.399 / nan 57.008 / 69.062

Reference evaluation returns per episode

  • s0 rows 2000: [36.0, 18.0, 14.0, 10.0, 19.0, 28.0, 9.0, 23.0, 19.0, 22.0]
  • s0 rows 4000: [131.0, 176.0, 90.0, 157.0, 194.0, 119.0, 89.0, 103.0, 93.0, 76.0]
  • s0 rows 6000: [142.0, 127.0, 135.0, 158.0, 149.0, 143.0, 156.0, 160.0, 146.0, 199.0]
  • s0 rows 8000: [55.0, 55.0, 66.0, 65.0, 55.0, 63.0, 60.0, 63.0, 64.0, 74.0]
  • s0 rows 10000: [149.0, 150.0, 127.0, 129.0, 155.0, 135.0, 131.0, 117.0, 134.0, 114.0]
  • s0 rows 12000: [28.0, 35.0, 29.0, 31.0, 26.0, 25.0, 24.0, 30.0, 27.0, 35.0]
  • s0 rows 14000: [273.0, 267.0, 270.0, 248.0, 292.0, 243.0, 262.0, 257.0, 263.0, 225.0]
  • s0 rows 16000: [319.0, 414.0, 227.0, 238.0, 202.0, 270.0, 257.0, 312.0, 294.0, 468.0]
  • s0 rows 18000: [154.0, 141.0, 141.0, 148.0, 140.0, 152.0, 154.0, 153.0, 149.0, 172.0]
  • s0 rows 20000: [424.0, 324.0, 500.0, 500.0, 500.0, 500.0, 456.0, 500.0, 500.0, 500.0]
  • s1 rows 2000: [28.0, 21.0, 26.0, 40.0, 11.0, 11.0, 30.0, 12.0, 17.0, 37.0]
  • s1 rows 4000: [135.0, 42.0, 103.0, 457.0, 125.0, 81.0, 101.0, 93.0, 111.0, 131.0]
  • s1 rows 6000: [500.0, 166.0, 303.0, 241.0, 477.0, 302.0, 258.0, 250.0, 500.0, 348.0]
  • s1 rows 8000: [117.0, 102.0, 113.0, 119.0, 117.0, 114.0, 108.0, 119.0, 112.0, 119.0]
  • s1 rows 10000: [225.0, 500.0, 199.0, 140.0, 168.0, 217.0, 172.0, 144.0, 190.0, 143.0]
  • s1 rows 12000: [116.0, 110.0, 118.0, 124.0, 114.0, 116.0, 124.0, 128.0, 111.0, 126.0]
  • s1 rows 14000: [156.0, 174.0, 151.0, 144.0, 157.0, 154.0, 140.0, 139.0, 159.0, 150.0]
  • s1 rows 16000: [35.0, 33.0, 35.0, 34.0, 21.0, 38.0, 41.0, 31.0, 32.0, 36.0]
  • s1 rows 18000: [246.0, 204.0, 271.0, 315.0, 271.0, 261.0, 252.0, 340.0, 238.0, 268.0]
  • s1 rows 20000: [500.0, 500.0, 500.0, 500.0, 500.0, 453.0, 500.0, 500.0, 500.0, 500.0]
  • s2 rows 2000: [12.0, 21.0, 37.0, 17.0, 47.0, 46.0, 29.0, 20.0, 15.0, 21.0]
  • s2 rows 4000: [228.0, 222.0, 191.0, 316.0, 104.0, 208.0, 249.0, 304.0, 174.0, 149.0]
  • s2 rows 6000: [182.0, 191.0, 180.0, 121.0, 117.0, 147.0, 173.0, 174.0, 180.0, 172.0]
  • s2 rows 8000: [124.0, 128.0, 126.0, 110.0, 121.0, 123.0, 120.0, 125.0, 121.0, 125.0]
  • s2 rows 10000: [157.0, 155.0, 148.0, 139.0, 147.0, 152.0, 154.0, 151.0, 145.0, 158.0]
  • s2 rows 12000: [118.0, 120.0, 118.0, 111.0, 113.0, 119.0, 115.0, 119.0, 116.0, 119.0]
  • s2 rows 14000: [206.0, 218.0, 197.0, 161.0, 177.0, 217.0, 186.0, 191.0, 202.0, 219.0]
  • s2 rows 16000: [127.0, 126.0, 127.0, 120.0, 126.0, 127.0, 120.0, 128.0, 121.0, 129.0]
  • s2 rows 18000: [281.0, 248.0, 268.0, 423.0, 318.0, 282.0, 289.0, 276.0, 315.0, 288.0]
  • s2 rows 20000: [500.0, 500.0, 469.0, 417.0, 486.0, 500.0, 430.0, 500.0, 433.0, 455.0]