{ "approach": "Oracle diagnostic, no candidate. B (sha baef955e..., file unmodified) is loaded twice via load_strategy_factory; for the oracle copy the harness overwrites B's module-level P['lam_r'] and P['m'] before each simulation with that simulation's true retail arrival_rate and mean_notional, obtained from runner.sample_config(ChallengeConfig(), ParameterVariance(), seed=s) (a pure function using its own random.Random(seed)). B's size_sigma assumption (sig_n=1.2) already equals the true, unvaried value 1.2, and buy_probability is fixed at 0.5, so lam_r and m are the only retail parameters B lacks. Each seed: run_batch(factory, base_config=ChallengeConfig(), variance=ParameterVariance(), n_simulations=1, seed_start=s, workers=1), serial in-process, simulator at b244530 unmodified; 6 OS processes on disjoint seed ranges. Instrumentation check: on seeds 236000..236019, B via plain run_batch and B via the harness path (P set by the harness to B's own 0.25/5.0) gave identical total/retail/arb edge on all 20 seeds (exact float equality).", "tried": [ "Full block 236000..236999 (1000 seeds), B and oracle paired on every seed", "Oracle overall: +1.06 (SE 0.56) per simulation; retail -0.64, arbitrage +1.70. Not significant at 2 SE; matches wave 3's +1.05 on an older base", "Buckets by true lam_r, m and lam_r*m (see findings): the gain is mostly in low-flow sims (lam_r*m < 0.9: +2.47, SE 1.11, from arbitrage +6.69 against retail -4.22, i.e. smaller quotes) and the highest-flow sims (lam_r*m >= 1.7: +2.94, SE 1.88, from retail +10.42 against arbitrage -7.48, i.e. larger quotes); middle buckets are ~0" ], "next": [ "Little to gain here: perfect knowledge of lam_r and m is worth about +1.06 (SE 0.56) per simulation, versus +24 measured in wave 3 for next-step probability. Put rounds into probability/jump estimation and arbitrage loss instead.", "If anything: an online estimate of retail notional flow (count and size of retail fills on B's own resting quotes) that scales the quote-size rule's rho*E[notional] down when flow is low. The low-flow third of sims (lam_r*m < 0.9) is where this gains, at most about +2.5 there, so about +0.8 overall; an estimate made from noisy fills will get less than the oracle." ], "code": "# ===== harness.py =====\n\"\"\"Oracle harness: B given each simulation's true retail arrival_rate and mean_notional.\nUsage: python harness.py SIM_DIR B_PATH SEED_LO SEED_HI OUT [mode]\nmode: 'both' (B and oracle), 'check' (plain run_batch vs harness-with-defaults).\"\"\"\nimport sys, json, time, hashlib\nsys.path.insert(0, sys.argv[1])\nfrom orderbook_pm_challenge.config import ChallengeConfig, ParameterVariance\nfrom orderbook_pm_challenge.runner import run_batch, sample_config\nfrom orderbook_pm_challenge.loader import load_strategy_factory\n\nbpath, lo, hi, out = sys.argv[2], int(sys.argv[3]), int(sys.argv[4]), sys.argv[5]\nmode = sys.argv[6] if len(sys.argv) > 6 else 'both'\nassert hashlib.sha256(open(bpath, 'rb').read()).hexdigest() == 'baef955ec10728839f7c3357016ec9eb0550e7ec689bbd5171b1cdd5c9e37273'\nassert not (900000 <= lo <= 903999 or 900000 <= hi - 1 <= 903999)\n\ndef P_of(factory):\n # B's parameter dict lives in its module globals; the strategy file itself is unmodified\n return factory().__class__.on_step.__globals__['P']\n\ndef one(factory, s):\n r = run_batch(factory, base_config=ChallengeConfig(), variance=ParameterVariance(),\n n_simulations=1, seed_start=s, workers=1).simulation_results[0]\n return dict(total=r.total_edge, retail=r.retail_edge, arb=r.arb_edge, failed=bool(r.failed))\n\nfB = load_strategy_factory(bpath)\nfO = load_strategy_factory(bpath) # separate module instance for the oracle\nPO = P_of(fO)\nassert PO is not P_of(fB)\nwith open(out, 'w') as f:\n for s in range(lo, hi):\n cfg = sample_config(ChallengeConfig(), ParameterVariance(), seed=s) # pure: own random.Random(seed)\n lam, m = cfg.retail.arrival_rate, cfg.retail.mean_notional\n t0 = time.time(); b = one(fB, s); tb = time.time() - t0\n if mode == 'check':\n PO['lam_r'], PO['m'] = 0.25, 5.0 # harness path with B's own values\n else:\n PO['lam_r'], PO['m'] = lam, m # oracle: true retail parameters\n t0 = time.time(); o = one(fO, s); to = time.time() - t0\n f.write(json.dumps(dict(seed=s, lam=lam, m=m, B=b, O=o, tB=tb, tO=to)) + '\\n'); f.flush()\n\n# ===== an.py (analysis) =====\nimport json, math, glob\nL=sorted([json.loads(l) for f in glob.glob('part*.jsonl') for l in open(f)], key=lambda x:x['seed'])\nn=len(L); assert [x['seed'] for x in L]==list(range(236000,237000))\ndef ms(v):\n k=len(v); mu=sum(v)/k; sd=math.sqrt(sum((x-mu)**2 for x in v)/(k-1)) if k>1 else 0; return mu, sd/math.sqrt(k)\nr=lambda x:round(x,3)\nB=[x['B']['total'] for x in L]\ninc=dict(mean=r(ms(B)[0]),retail_mean=r(ms([x['B']['retail'] for x in L])[0]),arb_mean=r(ms([x['B']['arb'] for x in L])[0]),se=r(ms(B)[1]))\ndef pair(S):\n d=[x['O']['total']-x['B']['total'] for x in S]\n m,se=ms(d)\n return dict(n=len(S),mean_diff=r(m),se=r(se),retail_diff=r(ms([x['O']['retail']-x['B']['retail'] for x in S])[0]),arb_diff=r(ms([x['O']['arb']-x['B']['arb'] for x in S])[0]))\nfails=sum(x['B']['failed'] for x in L)+sum(x['O']['failed'] for x in L)\nrows=[]\ndef row(name,S):\n b=[x['B']['total'] for x in S]; p=pair(S)\n rows.append(dict(name=name,count=len(S),B_mean=r(ms(b)[0]),B_retail=r(ms([x['B']['retail'] for x in S])[0]),B_arb=r(ms([x['B']['arb'] for x in S])[0]),B_se=r(ms(b)[1]),\n oracle_mean=r(ms([x['O']['total'] for x in S])[0]),oracle_retail=r(ms([x['O']['retail'] for x in S])[0]),oracle_arb=r(ms([x['O']['arb'] for x in S])[0]),\n diff=p['mean_diff'],diff_se=p['se'],retail_diff=p['retail_diff'],arb_diff=p['arb_diff']))\nrow('all',L)\nfor nm,key,cuts in [('lam_r',lambda x:x['lam'],[0.154,0.22,0.286,0.352]),('m',lambda x:x['m'],[2.64,3.872,5.104,6.336]),('lam_r*m',lambda x:x['lam']*x['m'],[0,0.9,1.25,1.7,9])]:\n for a,b in zip(cuts,cuts[1:]):\n S=[x for x in L if a<=key(x)=0.25)==la and (x['m']>=5.0)==mm]\n row(f\"lam_r {'>=' if la else '<'}0.25, m {'>=' if mm else '<'}5.0\",S)\ntsec=sum(x['tB']+x['tO'] for x in L)/(2*n)\nprint(json.dumps(dict(inc=inc,paired=pair(L),fails=fails,tsec=tsec),indent=1))\nfor x in rows: print(x)\njson.dump(dict(inc=inc,paired=pair(L),rows=rows,fails=fails,tsec=round(tsec,3)),open('summary.json','w'))\n", "code_sha256": "cccb3a808243c504f4700f6044d5ad73ee27c4196c5d4c519f2f5305783a87cf", "incumbent_code_sha256": "baef955ec10728839f7c3357016ec9eb0550e7ec689bbd5171b1cdd5c9e37273", "tuning": null, "seeds": "236000..236999", "n": 1000, "complete": true, "incumbent": { "mean": 95.152, "retail_mean": 121.143, "arb_mean": -25.99, "se": 5.873 }, "findings": [ { "name": "all", "count": 1000, "B_mean": 95.152, "B_retail": 121.143, "B_arb": -25.99, "B_se": 5.873, "oracle_mean": 96.212, "oracle_retail": 120.5, "oracle_arb": -24.288, "diff": 1.06, "diff_se": 0.557, "retail_diff": -0.643, "arb_diff": 1.703 }, { "name": "lam_r in [0.154,0.22)", "count": 340, "B_mean": 65.229, "B_retail": 85.587, "B_arb": -20.358, "B_se": 5.525, "oracle_mean": 65.556, "oracle_retail": 82.423, "oracle_arb": -16.866, "diff": 0.327, "diff_se": 0.369, "retail_diff": -3.164, "arb_diff": 3.492 }, { "name": "lam_r in [0.22,0.286)", "count": 336, "B_mean": 77.371, "B_retail": 109.955, "B_arb": -32.584, "B_se": 11.105, "oracle_mean": 79.264, "oracle_retail": 108.736, "oracle_arb": -29.473, "diff": 1.893, "diff_se": 1.422, "retail_diff": -1.219, "arb_diff": 3.112 }, { "name": "lam_r in [0.286,0.352)", "count": 324, "B_mean": 144.993, "B_retail": 170.056, "B_arb": -25.063, "B_se": 12.305, "oracle_mean": 145.957, "oracle_retail": 172.656, "oracle_arb": -26.699, "diff": 0.964, "diff_se": 0.794, "retail_diff": 2.6, "arb_diff": -1.636 }, { "name": "m in [2.64,3.872)", "count": 336, "B_mean": 67.226, "B_retail": 90.276, "B_arb": -23.05, "B_se": 6.93, "oracle_mean": 69.382, "oracle_retail": 86.54, "oracle_arb": -17.158, "diff": 2.156, "diff_se": 1.025, "retail_diff": -3.736, "arb_diff": 5.892 }, { "name": "m in [3.872,5.104)", "count": 356, "B_mean": 98.796, "B_retail": 124.717, "B_arb": -25.921, "B_se": 9.398, "oracle_mean": 98.703, "oracle_retail": 123.112, "oracle_arb": -24.41, "diff": -0.093, "diff_se": 0.62, "retail_diff": -1.605, "arb_diff": 1.512 }, { "name": "m in [5.104,6.336)", "count": 308, "B_mean": 121.405, "B_retail": 150.683, "B_arb": -29.278, "B_se": 13.571, "oracle_mean": 122.601, "oracle_retail": 154.526, "oracle_arb": -31.925, "diff": 1.196, "diff_se": 1.226, "retail_diff": 3.843, "arb_diff": -2.647 }, { "name": "lam_r*m in [0,0.9)", "count": 313, "B_mean": 53.049, "B_retail": 78.734, "B_arb": -25.685, "B_se": 6.294, "oracle_mean": 55.523, "oracle_retail": 74.514, "oracle_arb": -18.991, "diff": 2.474, "diff_se": 1.111, "retail_diff": -4.22, "arb_diff": 6.694 }, { "name": "lam_r*m in [0.9,1.25)", "count": 358, "B_mean": 91.607, "B_retail": 113.682, "B_arb": -22.075, "B_se": 7.842, "oracle_mean": 91.959, "oracle_retail": 111.713, "oracle_arb": -19.754, "diff": 0.352, "diff_se": 0.549, "retail_diff": -1.969, "arb_diff": 2.321 }, { "name": "lam_r*m in [1.25,1.7)", "count": 245, "B_mean": 119.172, "B_retail": 150.904, "B_arb": -31.732, "B_se": 15.956, "oracle_mean": 118.814, "oracle_retail": 152.975, "oracle_arb": -34.16, "diff": -0.358, "diff_se": 1.442, "retail_diff": 2.071, "arb_diff": -2.428 }, { "name": "lam_r*m in [1.7,9)", "count": 84, "B_mean": 197.088, "B_retail": 224.157, "B_arb": -27.068, "B_se": 29.189, "oracle_mean": 200.028, "oracle_retail": 234.579, "oracle_arb": -34.551, "diff": 2.94, "diff_se": 1.883, "retail_diff": 10.423, "arb_diff": -7.483 }, { "name": "lam_r <0.25, m <5.0", "count": 338, "B_mean": 54.249, "B_retail": 77.266, "B_arb": -23.017, "B_se": 5.723, "oracle_mean": 55.275, "oracle_retail": 73.393, "oracle_arb": -18.118, "diff": 1.026, "diff_se": 0.627, "retail_diff": -3.872, "arb_diff": 4.899 }, { "name": "lam_r <0.25, m >=5.0", "count": 165, "B_mean": 80.685, "B_retail": 113.136, "B_arb": -32.452, "B_se": 18.244, "oracle_mean": 79.061, "oracle_retail": 112.778, "oracle_arb": -33.717, "diff": -1.624, "diff_se": 1.891, "retail_diff": -0.358, "arb_diff": -1.266 }, { "name": "lam_r >=0.25, m <5.0", "count": 329, "B_mean": 113.987, "B_retail": 140.608, "B_arb": -26.621, "B_se": 10.603, "oracle_mean": 115.387, "oracle_retail": 139.077, "oracle_arb": -23.69, "diff": 1.4, "diff_se": 1.033, "retail_diff": -1.531, "arb_diff": 2.931 }, { "name": "lam_r >=0.25, m >=5.0", "count": 168, "B_mean": 154.77, "B_retail": 179.163, "B_arb": -24.393, "B_se": 17.177, "oracle_mean": 157.865, "oracle_retail": 186.476, "oracle_arb": -28.611, "diff": 3.095, "diff_se": 1.352, "retail_diff": 7.313, "arb_diff": -4.218 } ], "paired": { "n": 1000, "mean_diff": 1.06, "se": 0.557, "retail_diff": -0.643, "arb_diff": 1.703 }, "instrumentation_check": { "seeds": "236000..236019", "identical": true, "B_total_edges": [ 74.163835, 100.552125, 61.214442, 35.701817, 90.258764, 196.542113, 155.013031, 937.465285, 43.560351, 551.537875, 484.494141, 0.065099, 11.810733, 367.899919, 97.058728, -0.122445, 40.078348, 291.528389, 44.934201, 367.801638 ] }, "failures": { "B": 0, "oracle": 0, "note": "failed flag false in all 2000 simulations; no errors" }, "seconds_per_simulation": 0.796, "notes": "Oracle result is a diagnostic of what information is worth, not a strategy to submit. findings rows: B_* and oracle_* are per-simulation means (total, retail, arb); diff = oracle minus B paired, diff_se its standard error; B_se is SE of B's per-sim mean in the bucket." }