# latt-c5: Taylor's lattice filter at 61 cells of 5 bps, screened against qt1's reference The policy, the lattice filter, the replica and the qm tables are all Taylor's (gtmcknight): [amm-sim research/solved @4812a638](https://github.com/gtmcknight/amm-sim/tree/4812a6382311624e1506aca131ca35c378f930d9/research/solved). This item only coarsens his `latt.py` and measures it. Everything ran on his pure-Python replica with `python3 -I`, standard library only. No engine seed was used and no Solidity was written. **Result.** On replica seeds 60000..60511 (n = 512, each policy run once per seed), latt-c5 scores **+0.5788 (se 0.0245)** over qt1's reference, ahead on 440 of 512 seeds. It gives up nothing measurable against the exact lattice: **−0.0151 (se 0.0106)** on the same 512 seeds. **It does not fit the gas limit as specified.** By the rule below, the worst case of one callback is about 1.62M gas (analytic bound) and 1.06M gas (largest seen), against a peak limit of 245,000. The mean is about 174k, but 0.53% of callbacks are over 245,000, and every one of the 512 runs has some. The cost is in the silent-step loop (up to 30 diffuse + cut + renormalise passes), not in the resolution. A side measurement below shows that capping that loop at 2 to 4 steps keeps the gain, but only a cap of 0 or 1 comes near the budget under this rule. All figures are local measurements on this machine. Replica gaps do not convert to board scores. ## Design What changed against `latt.py` (sha256 `925c0fe9…0cc8` at the pinned commit): | | latt.py | latt_c5.py | |---|---|---| | Cell width `HC` | 2.5e-4 (2.5 bps) | **5e-4 (5 bps)** | | Cells `NC` | 121 (±150 bps) | **61 (±150 bps)** | | Kernel rule | sampled Gaussian, sigma 9.45 bps, cut at 4 sd, normalised | unchanged rule; it now yields 17 taps (`KM` = 8) instead of 33 (`KM` = 16) | | Cut, renormalise, arb/retail mixture, pin, deposit, moments | | unchanged, line for line | | Silent-step cap (30), `hard` 0.999, thresholds 1e-15 / 1e-12 / 1e-300 | | unchanged | | Window-edge recentre trigger | `abs(m - c) > 40 * HC` = 100 bps | `abs(m - c) > EDGE`, `EDGE` = 100 bps (20 cells) | | Tables | `pol_mid.json`, `qm_k1e-5.json` | the same files, untouched | | Harness | `HERE` = own folder, `Pool(14)` | `HERE` = `$T_SOLVED` (Taylor's folder), `Pool(NPROC or cpu count)` | One judgment call: the recentre trigger. `latt.py` writes it as `40 * HC`. Copied literally at 5 bps that is 200 bps, which is outside the ±150 bps window, so the rule could never fire. I kept the physical distance (100 bps) instead. It barely matters: this file minus the literal `40 * HC` version is +0.0041 (se 0.0023) on the 512 seeds, largest single-seed gap 0.85. Consequences of the coarser grid: - A full-window diffusion is 965 multiply-adds instead of 3,721 (3.86 times fewer). - The belief is 61 numbers instead of 121. - Filter quality is unchanged within noise: arb/retail misclassification 16.26% (exact lattice 16.21%), price rms error 13.67 bps (exact 13.64), both over the 512 seeds. Check values for a later port, latt_c5 with the full qm tables on replica seeds 10000 / 10001 / 10002: **443.890699 / 571.689889 / 526.388880**. Unmodified `latt.py` gives 443.663520 / 571.977293 / 526.411631 there, as Taylor's README states. ## Screen Preconditions, all met: - **Gate P:** unmodified `python3 -I gates.py` printed every digit of the README's table (four columns, seeds 10000..10002). - **qmref:** 443.101741 / 571.454483 / 525.908441 on seeds 10000..10002, as the brief and qt1's result state. - **Hashes:** qt1's `Strategy.sol` block is 42,990 bytes, sha256 `c68887cff61180e12ba2b1ceb333edf76a830f91f63b5d775a8806fb9ec08aca`. `blob.bin` is 10,167 bytes, sha256 `3ac7cb8811ecfb6a17f499aa4e9b70dd55f7fb6432b3aec3b904e11a3dece38e`. Both match. - **Timing:** the first 16 seeds took 2.0 s of wall time on 6 processes, and the other 496 took 59.0 s. The whole window ran; nothing was cut short. ```json { "variant": "latt-c5", "base": "qt1-ref", "seeds": "60000..60511", "n": 512, "mean_diff": 0.578835, "se": 0.024537, "worst5": [ [ 60446, -1.019152 ], [ 60205, -0.972253 ], [ 60053, -0.760694 ], [ 60202, -0.718922 ], [ 60183, -0.712195 ] ], "mean_edge_variant": 529.854741, "mean_edge_base": 529.275906, "vs_exact_lattice": { "n": 128, "mean_diff": -0.015777, "se": 0.021515 }, "cost": { "cells": 6104, "mul": 34454, "div": 1941, "explog": 12, "lookups": 2, "state_words": 11, "est_gas": 1619800, "rule": "est_gas = (2100 SLOAD + 2900 SSTORE) * state_words + 40 * (mul + div) + 2000 * explog + 42500 * lookups. Figures are the analytic worst case of one callback: 30 silent steps plus the current one (31 diffusions), all 61 cells occupied at every pass, mixture branch with pi > 0.5, window-edge recentre. cells = cell visits summed over all passes (61 distinct cells). state_words = 8 words of 61 cells at 32 bits + 3 scalar words as in qt1 (assumed). Largest callback seen on the 512 seeds: cells 4049, mul 21032, div 1435, explog 12, est_gas 1062680; mean est_gas 174402." } } ``` Notes on the block: - `vs_exact_lattice` is the first 128 seeds, as the brief asks. Time allowed all 512: **−0.0151, se 0.0106, n = 512** (latt-c5 minus unmodified `latt.py`). On those 512 seeds the exact lattice is +0.5939 (se 0.0251) over qt1-ref, with mean edge 529.8698. - `cost` is the analytic worst case of one callback; the rule and the observed figures are in the next subsection. - Split halves of the 512 paired differences: +0.5618 (se 0.0357) and +0.5958 (se 0.0337). ### What the +0.58 is made of The base (qt1-ref) reads the decoded qm-spline tables. The variant, as the brief specifies, reads the full `qm_k1e-5.json`. So part of the gap is tables, not filter. Three extra runs on the same 512 seeds separate them (each once per seed, paired): | Policy | minus qt1-ref | se | Reading | |---|---:|---:|---| | latt-c5, full qm tables (the variant) | +0.5788 | 0.0245 | the headline | | latt-c5 filter, decoded qm-spline tables | +0.5218 | 0.0254 | what a contract holding qt1's tables would get | | Taylor's Gaussian filter (`real.py`, soft=1), full qm tables | +0.0744 | 0.0150 | the table effect alone | | unmodified `latt.py`, full qm tables | +0.5939 | 0.0251 | the exact lattice | - Lattice c5 over Gaussian with the same full tables: +0.5045 (se 0.0233). That matches the goal's +0.49 for the exact lattice. - Full tables over spline tables under the c5 filter: +0.0571 (se 0.0129). qm-spline reported a loss of 0.0632 (se 0.0072, n = 2048) for its decoded tables on the holdout. ### Cost of one callback **How I counted.** A wrapper around the unmodified methods (in `screen.py`, below) counts operations as the Python executes them, per callback. It reads state only, and the counted class reproduces the uncounted edges on seeds 10000..10002 to the last digit. - `cells` is cell visits summed over every pass (a diffusion visits the support once, `moments` twice), not distinct cells. There are 61 distinct cells; the widest support seen after a diffusion was 55. - `mul` counts one per kernel tap, plus the per-cell multiplies in cut, renormalise, moments, recentre and the mixture scaling. - `div` counts one per non-zero cell in cut (the Python divides by `HC` per cell) plus the scalar divisions. - `explog` is 12 on the first trade of a step: 7 ln (lo, hi, the arb reading, s/sp, d, s, the rival spot), 4 exp (the lognormal density, exp(m), two fee conversions) and 1 square root. It is 5 on later trades of a step. - Scalar arithmetic outside the lattice methods is a fixed allowance: 14 mul and 10 div on a first trade, 4 and 2 otherwise. - `lookups` is 2: the ask and the bid surface, as in qt1. - `state_words` is 11: 8 words for 61 cells at 32 bits each, plus 3 scalar words as qt1 has. This is an assumption, not a measurement; see Problems. **Rule.** est_gas = (2,100 + 2,900) × state_words + 40 × (mul + div) + 2,000 × explog + 42,500 × lookups. The 42,500 is the top of the 35–43k per spline surface that qt1's result reports. The fixed part is 55,000 + 24,000 + 85,000 = 164,000 on a first trade, which leaves 81,000 gas, about 2,025 multiplies or divisions, for the lattice. | latt-c5, 512 seeds, 5,530,283 callbacks | cells | mul | div | explog | est_gas | |---|---:|---:|---:|---:|---:| | Mean per callback | 113.2 | 343.8 | 40.5 | 9.51 | 174,402 | | Mean, first trade of a step (64.5% of callbacks) | | | | 12 | 187,706 | | Largest seen (one callback, 31 diffusions) | 4,049 | 21,032 | 1,435 | 12 | 1,062,680 | | Analytic worst case (the JSON) | 6,104 | 34,454 | 1,941 | 12 | 1,619,800 | - Distribution of est_gas: 5,143,603 callbacks under 200,000; 311,613 in 200–225k; 51,502 in 225–250k; 23,565 at 250,000 or more. **29,373 (0.531%) are over 245,000, in 512 of 512 runs.** - The analytic worst case assumes 30 silent steps plus the current one, all 61 cells occupied at every pass, the mixture branch with pi > 0.5, and a window-edge recentre. It is loose, because a cut normally shrinks the support to the no-arb interval. - The same rule on unmodified `latt.py` (19 words, 512 seeds): mean 243,130, first-trade mean 272,265, largest seen 3,329,280, analytic worst case 5,326,840, and 62.4% of callbacks over 245,000. Mean multiplies per callback are 1,034 against 344 here. ### Side measurement: capping the silent-step loop This is not the c5 variant. It is the same file with `min(dt - 1, 30)` replaced by `min(dt - 1, K)`, run to see where the gas goes and what the loop is worth. Same 512 seeds, once each, counted the same way. A cap of 30 reproduces c5 exactly (checked on 32 seeds). | Cap K | minus qt1-ref (se) | minus c5 (se) | Largest est_gas seen | Analytic worst case | Callbacks over 245,000 | |---:|---:|---:|---:|---:|---:| | 0 | −1.0291 (0.0421) | −1.6079 (0.0401) | 203,040 | 238,600 | 0 | | 1 | +0.4283 (0.0278) | −0.1505 (0.0196) | 229,200 | 284,640 | 0 | | 2 | +0.5252 (0.0262) | −0.0536 (0.0139) | 254,680 | 330,680 | 1,143 | | 3 | +0.5643 (0.0255) | −0.0145 (0.0100) | 283,840 | 376,720 | 8,690 | | 4 | +0.5760 (0.0251) | −0.0029 (0.0078) | 313,560 | 422,760 | 15,236 | | 6 | +0.5813 (0.0245) | +0.0025 (0.0035) | 373,720 | 514,840 | 28,740 | | 8 | +0.5800 (0.0244) | +0.0011 (0.0023) | 429,320 | 606,920 | 29,403 | | 12 | +0.5823 (0.0245) | +0.0035 (0.0012) | 542,120 | 791,080 | 29,378 | | 30 (c5) | +0.5788 (0.0245) | 0 | 1,062,680 | 1,619,800 | 29,373 | Reading: - The first silent step carries most of the value (cap 0 loses 1.61, cap 1 loses 0.15). From cap 4 up the loss is inside the noise. - Under this rule only cap 0 is inside 245,000 in the worst case, and it is worse than qt1. Cap 1 stays under 245,000 on every callback seen (peak 229,200) while its analytic worst case is 284,640. Cap 2 already has callbacks over the limit. - So the resolution is not what stands between the lattice and the budget. With 85,000 gas in two spline lookups, there is room for about one silent step. Cheaper lookups or a cheaper per-step update would be needed for cap 2 to 4. ## Code The complete variant file, `latt_c5.py`. It imports Taylor's `replica2`, `real` and `evalpol` from the folder named by `T_SOLVED`. ```python """latt_c5: Taylor's (gtmcknight) lattice filter, coarse variant c5: 61 cells of 5 bps instead of 121 of 2.5 bps. All of the policy is Taylor's work: github.com/gtmcknight/amm-sim @4812a6382311624e1506aca131ca35c378f930d9, research/solved/latt.py (LatticePolicy over real.RealPolicy with the qm tables). This file is latt.py with HC 2.5e-4 -> 5e-4, NC 121 -> 61 (still +-150 bps), the recentre trigger written in bps (EDGE = 100 bps = latt.py's 40 cells of 2.5 bps = 20 cells here), and nothing else: kernel rule (sigma 9.45 bps, 4 sd, sampled Gaussian), cut and mixture rules are unchanged. No table constant is touched. Exact Bayes on a price lattice, real observables only. Belief = probability mass on cells of width HC around a moving centre. Each elapsed step convolves with the price-move kernel; a step with no own trade zeroes the cells where the arb would have traded; the first trade of a step is a mixture of "arb" (p exactly at the post-trade band edge) and "retail" (p inside the no-arb interval), weighted by likelihood. Nothing is thrown away, so a later pin resolves an earlier ambiguous trade. Quotes: moment-matched (mean, sd) into the uncertainty-aware tables. usage: T_SOLVED= python3 -I latt_c5.py pol_mid.json name:key=val,... [more] (as latt.py: python3 -I latt_c5.py pol_mid.json lattice:qm=qm_k1e-5.json) """ import sys, os, math, json, time from multiprocessing import Pool HERE = os.environ.get('T_SOLVED') or os.path.dirname(os.path.abspath(__file__)) # Taylor's research/solved sys.path.insert(0, HERE) import replica2 as R import real from real import RealPolicy, phi, G2, RHO, LD_M, LD_S, H0, LD_FULL from evalpol import load, stats, CALIB HC = 5e-4 # cell width (log) [latt.py: 2.5e-4] NC = 61 # cells: +-150 bps [latt.py: 121] N0 = NC // 2 SIG = 0.000945 EDGE = 100e-4 # recentre when |mean - centre| exceeds this [latt.py: 40 * HC = 100 bps] def kernel(sig, hc): s = sig / hc; M = int(math.ceil(4.0 * s)) w = [math.exp(-0.5 * (k / s) ** 2) for k in range(-M, M + 1)] t = sum(w) return [v / t for v in w], M KER, KM = kernel(SIG, HC) # 17 taps (KM = 8) at 5 bps; latt.py has 33 (KM = 16) class LatticePolicy(RealPolicy): def __init__(self, T, opt=None): RealPolicy.__init__(self, T, opt) self.c = math.log(100.0) # log price of cell N0 self.P = [0.0] * NC; self.P[N0] = 1.0 self.lo_i = N0; self.hi_i = N0 # support bounds (inclusive) # ---- lattice ops def diffuse(self): P = self.P; lo = max(0, self.lo_i - KM); hi = min(NC - 1, self.hi_i + KM) out = [0.0] * NC for i in range(self.lo_i, self.hi_i + 1): pi_ = P[i] if pi_ <= 1e-15: continue j0 = i - KM for k in range(2 * KM + 1): j = j0 + k if 0 <= j < NC: out[j] += pi_ * KER[k] self.P = out; self.lo_i = lo; self.hi_i = hi def cut(self, lo, hi): """keep only mass with lo < ln p < hi (the no-arb interval); returns the kept mass before renormalising""" P = self.P; c = self.c; tot = 0.0 for i in range(self.lo_i, self.hi_i + 1): if P[i] == 0.0: continue x0 = c + (i - N0 - 0.5) * HC; x1 = x0 + HC a = lo if lo > x0 else x0; b = hi if hi < x1 else x1 f = (b - a) / HC if f <= 0.0: P[i] = 0.0 else: if f < 1.0: P[i] *= f tot += P[i] return tot def norm(self, tot): if tot <= 1e-300: return False P = self.P; inv = 1.0 / tot lo = None; hi = None for i in range(self.lo_i, self.hi_i + 1): if P[i] > 0.0: P[i] *= inv if P[i] < 1e-12: P[i] = 0.0 else: if lo is None: lo = i hi = i if lo is None: return False self.lo_i, self.hi_i = lo, hi return True def dens(self, x): pos = (x - self.c) / HC + N0 i = int(math.floor(pos)); f = pos - i if i < 0 or i >= NC - 1: return 0.0 return (self.P[i] * (1 - f) + self.P[i + 1] * f) / HC def moments(self): P = self.P; m = 0.0; tot = 0.0 for i in range(self.lo_i, self.hi_i + 1): m += P[i] * (i - N0); tot += P[i] if tot <= 0: return self.c, 0.0 m /= tot; v = 0.0 for i in range(self.lo_i, self.hi_i + 1): v += P[i] * (i - N0 - m) ** 2 v /= tot return self.c + m * HC, v * HC * HC def recentre(self, cnew): """move the lattice so that cell N0 sits at cnew (linear re-binning)""" sh = (cnew - self.c) / HC P = self.P; out = [0.0] * NC for i in range(self.lo_i, self.hi_i + 1): if P[i] == 0.0: continue pos = i - sh; j = int(math.floor(pos)); f = pos - j if 0 <= j < NC: out[j] += P[i] * (1 - f) if 0 <= j + 1 < NC: out[j + 1] += P[i] * f self.P = out; self.c = cnew self.lo_i = max(0, int(math.floor(self.lo_i - sh)) - 1); self.hi_i = min(NC - 1, int(math.ceil(self.hi_i - sh)) + 1) def pin(self, x): self.c = x; self.P = [0.0] * NC; self.P[N0] = 1.0; self.lo_i = self.hi_i = N0 def deposit(self, x, w): pos = (x - self.c) / HC + N0 i = int(math.floor(pos)); f = pos - i if 0 <= i < NC: self.P[i] += w * (1 - f); self.lo_i = min(self.lo_i, i); self.hi_i = max(self.hi_i, i) if 0 <= i + 1 < NC: self.P[i + 1] += w * f; self.lo_i = min(self.lo_i, i + 1); self.hi_i = max(self.hi_i, i + 1) # ---- callback def on_trade(self, is_buy, ax, ay, t, x, y, arb, p, s2, j): o = self.o s = y / x dt = t - self.tlast is_arb = False if dt >= 1: self.cnt = 0 lo = math.log(self.sp * (1.0 - self.fb)); hi = math.log(self.sp / (1.0 - self.fa)) ok = True for _ in range(min(dt - 1, 30)): self.diffuse() ok = self.norm(self.cut(lo, hi)) if not ok: break if ok: self.diffuse() lp_arb = math.log(s * (1.0 - self.fb)) if is_buy else math.log(s / (1.0 - self.fa)) if not ok: # belief died (model mismatch): restart wide around the arb reading self.pin(lp_arb); pi = 1.0 else: l_arb = self.dens(lp_arb) eps = o.get('tail', 0.0) if eps > 0.0: mm, vv = self.moments() l_arb = (1.0 - eps) * l_arb + eps * phi((lp_arb - mm) / 40e-4) / 40e-4 d = abs(math.log(s / self.sp)) # retail branch: p inside the no-arb interval Pkeep = self.P[:]; lo_i, hi_i = self.lo_i, self.hi_i Z = self.cut(lo, hi) l_ret = Z * o.get('rho', RHO) * phi((math.log(max(d, 1e-9)) - LD_M) / LD_S) / (max(d, 1e-9) * LD_S) pi = l_arb / (l_arb + l_ret) if (l_arb + l_ret) > 0 else 1.0 if o.get('oraclearb', 0): pi = 1.0 if arb else 0.0 hard = o.get('hard', 0.999) if pi >= hard or Z <= 1e-300: self.pin(lp_arb) elif pi <= 1.0 - hard: self.norm(Z) else: # mixture: (1 - pi) * retail posterior + pi * point mass at the arb reading self.norm(Z) P = self.P for i in range(self.lo_i, self.hi_i + 1): P[i] *= (1.0 - pi) if pi > 0.5: self.recentre(lp_arb) self.deposit(lp_arb, pi) is_arb = pi > 0.5 self.ncls += 1 if is_arb != arb: self.nmis += 1 self.m, self.v = self.moments() # keep the mass away from the window edge if abs(self.m - self.c) > EDGE: self.recentre(self.m); self.m, self.v = self.moments() pe = math.exp(self.m) if self.s2h < pe * G2: self.s2h = pe * G2 elif self.s2h > pe / G2: self.s2h = pe / G2 self.synced = False if not is_arb: if is_buy: cand = s * (1.0 - self.fb) / G2 if cand < self.s2h: self.s2h = cand; self.synced = True else: cand = s * G2 / (1.0 - self.fa) if cand > self.s2h: self.s2h = cand; self.synced = True self.cnt += 1 phs = 1 if self.cnt <= 1 else 2 else: phs = 0 err = (self.m - math.log(p)) * 1e4 self.se += err * err; self.nse += 1 self.tlast = t; self.sp = s w = (math.log(s) - self.m) * 1e4; w2 = (math.log(self.s2h) - self.m) * 1e4 fb, fa = self.lookup(w, w2, phs, math.sqrt(self.v) * 1e4) self.fb, self.fa = R.clampf(fb), R.clampf(fa) return fb, fa def ev(a): path, opt, seeds = a T = load(path); out = [] for s in seeds: pol = LatticePolicy(T, opt) r = R.run(s, pol) out.append((s, r['edge'], r['retail'], r['arb'], r['rv'], r['av'], pol.nmis, pol.ncls, pol.se, pol.nse)) return out if __name__ == '__main__': path = sys.argv[1] if not os.path.exists(path): path = os.path.join(HERE, path) n = int(os.environ.get('NSEED', '2048')) seeds = list(range(10000, 10000 + n)) rows = json.load(open(CALIB)); ref = {} for r in rows: ref.setdefault(r['kind'], {})[r['seed']] = r real.ev = ev with Pool(int(os.environ.get('NPROC', os.cpu_count() or 1))) as pool: for spec in sys.argv[2:]: name, _, kv = spec.partition(':') opt = {} for item in kv.split(','): if item: k, _, val = item.partition('=') opt[k] = os.path.join(HERE, val) if k == 'qm' else float(val) real.report(pool, path, opt, seeds, ref, spec) ``` The screen driver with the operation counter, `screen.py` (it expects `T/amm-sim/research/solved`, `ref/qmref.py` and `ref/blob.bin` beside its own folder): ```python """Screen driver for latt_c5 (Taylor gtmcknight's lattice filter, 61 cells of 5 bps) against qt1's reference qmref. Resumable, one seed = one task, ordered, so the state file always holds a contiguous prefix. usage: python3 -I screen.py gate python3 -I screen.py main qmref + latt_c5 (counted) per seed -> main.json python3 -I screen.py exact unmodified latt.py (counted) per seed -> exact.json python3 -I screen.py edge40 latt_c5 with the literal 40*HC recentre trigger -> edge40.json python3 -I screen.py report """ import sys, os, time, json, math from multiprocessing import Pool HERE = os.path.dirname(os.path.abspath(__file__)) ROOT = os.path.dirname(HERE) TDIR = os.path.join(ROOT, 'T', 'amm-sim', 'research', 'solved') REF = os.path.join(ROOT, 'ref') os.environ['T_SOLVED'] = TDIR sys.path.insert(0, REF); sys.path.insert(0, HERE); sys.path.insert(0, TDIR) import replica2 as R from evalpol import load import latt import latt_c5 from qmref import QMRef POL = os.path.join(TDIR, 'pol_mid.json'); QM = os.path.join(TDIR, 'qm_k1e-5.json'); BLOB = os.path.join(REF, 'blob.bin') NPROC = int(os.environ.get('NPROC', os.cpu_count() or 1)) # ---- gas rule (stated in result.md) G_SLOAD, G_SSTORE, G_MULDIV, G_EXPLOG, G_LOOKUP = 2100, 2900, 40, 2000, 42500 CELLS_PER_WORD, SCALAR_WORDS, LOOKUPS = 8, 3, 2 BUDGET = 245000 KEYS = ('cells', 'mul', 'div', 'explog', 'ndiff') def words(nc): return -(-nc // CELLS_PER_WORD) + SCALAR_WORDS def gas(nc, k): return (G_SLOAD + G_SSTORE) * words(nc) + G_MULDIV * (k['mul'] + k['div']) + G_EXPLOG * k['explog'] + G_LOOKUP * LOOKUPS def counted(mod): """LatticePolicy of `mod` with per-callback operation counts. The wrappers only read state before calling the unmodified method, so fees are those of the unmodified class.""" Base = mod.LatticePolicy; NC, KM = mod.NC, mod.KM class C(Base): def __init__(self, T, opt=None): Base.__init__(self, T, opt) self.k = dict.fromkeys(KEYS, 0) self.ncb = 0; self.sum = dict.fromkeys(KEYS, 0); self.max = dict.fromkeys(KEYS, 0) self.gsum = 0; self.gmax = 0; self.gmaxk = None; self.nover = 0; self.maxspan = 0 self.hist = [0] * 64 # est_gas in bins of 25k self.ndt = 0; self.gsum_dt = 0 def _span(self): return self.hi_i - self.lo_i + 1 def diffuse(self): P = self.P; taps = 0 for i in range(self.lo_i, self.hi_i + 1): if P[i] > 1e-15: a = i - KM; b = i + KM taps += (b if b < NC - 1 else NC - 1) - (a if a > 0 else 0) + 1 k = self.k; k['mul'] += taps; k['cells'] += self._span(); k['ndiff'] += 1 Base.diffuse(self) sp = self._span() if sp > self.maxspan: self.maxspan = sp def cut(self, lo, hi): P = self.P; nz = sum(1 for i in range(self.lo_i, self.hi_i + 1) if P[i] != 0.0) k = self.k; k['mul'] += nz + 2; k['div'] += nz; k['cells'] += self._span() return Base.cut(self, lo, hi) def norm(self, tot): if tot > 1e-300: P = self.P; nz = sum(1 for i in range(self.lo_i, self.hi_i + 1) if P[i] > 0.0) k = self.k; k['mul'] += nz; k['div'] += 1; k['cells'] += self._span() return Base.norm(self, tot) def dens(self, x): k = self.k; k['mul'] += 2; k['div'] += 2; k['cells'] += 2 return Base.dens(self, x) def moments(self): sp = self._span(); k = self.k; k['mul'] += 3 * sp + 3; k['div'] += 2; k['cells'] += 2 * sp return Base.moments(self) def recentre(self, cnew): P = self.P; nz = sum(1 for i in range(self.lo_i, self.hi_i + 1) if P[i] != 0.0) k = self.k; k['mul'] += 2 * nz; k['div'] += 1; k['cells'] += self._span() Base.recentre(self, cnew) def deposit(self, x, w): # only called in the mixture branch, right after the (1 - pi) scaling loop over the support k = self.k; k['mul'] += 2 + self._span(); k['div'] += 1; k['cells'] += 2 + self._span() Base.deposit(self, x, w) def on_trade(self, is_buy, ax, ay, t, x, y, arb, p, s2, j): k = self.k for key in KEYS: k[key] = 0 newstep = (t - self.tlast) >= 1 # scalar work outside the lattice methods if newstep: k['explog'] += 7; k['mul'] += 10; k['div'] += 8 # ln lo, hi, lp_arb, s/sp, d; exp in phi, exp(m) k['explog'] += 5; k['mul'] += 4; k['div'] += 2 # ln s, ln s2h, sqrt v, 2 exp in the fee conversion out = Base.on_trade(self, is_buy, ax, ay, t, x, y, arb, p, s2, j) self.ncb += 1 for key in KEYS: v = k[key]; self.sum[key] += v if v > self.max[key]: self.max[key] = v g = gas(NC, k); self.gsum += g if newstep: self.ndt += 1; self.gsum_dt += g if g > self.gmax: self.gmax = g; self.gmaxk = dict(k) if g > BUDGET: self.nover += 1 self.hist[min(63, g // 25000)] += 1 return out def summary(self): return dict(ncb=self.ncb, sum=self.sum, max=self.max, gsum=self.gsum, gmax=self.gmax, gmaxk=self.gmaxk, nover=self.nover, hist=self.hist, maxspan=self.maxspan, ndt=self.ndt, gsum_dt=self.gsum_dt, nmis=self.nmis, ncls=self.ncls, se=self.se, nse=self.nse) return C C5 = counted(latt_c5) EX = counted(latt) def t_main(seed): t0 = time.time(); eq = R.run(seed, QMRef(BLOB))['edge']; t1 = time.time() pol = C5(load(POL), {'qm': QM}); ec = R.run(seed, pol)['edge']; t2 = time.time() return seed, dict(q=eq, c5=ec, tq=t1 - t0, tc=t2 - t1, k=pol.summary()) def t_exact(seed): t0 = time.time(); pol = EX(load(POL), {'qm': QM}); e = R.run(seed, pol)['edge'] return seed, dict(ex=e, t=time.time() - t0, k=pol.summary()) def t_edge40(seed): t0 = time.time(); pol = latt_c5.LatticePolicy(load(POL), {'qm': QM}); e = R.run(seed, pol)['edge'] return seed, dict(e=e, t=time.time() - t0) def t_gate(a): kind, seed = a; T = load(POL) if kind == 'qmref': pol = QMRef(BLOB) elif kind == 'c5': pol = latt_c5.LatticePolicy(T, {'qm': QM}) elif kind == 'c5-counted': pol = C5(T, {'qm': QM}) elif kind == 'latt': pol = latt.LatticePolicy(T, {'qm': QM}) elif kind == 'latt-counted': pol = EX(T, {'qm': QM}) t0 = time.time(); e = R.run(seed, pol)['edge'] return kind, seed, e, time.time() - t0 def resumable(name, fn, first, n, budget): path = os.path.join(ROOT, name + '.json') st = json.load(open(path)) if os.path.exists(path) else {} todo = [s for s in range(first, first + n) if str(s) not in st] t0 = time.time(); done = 0 if todo: pool = Pool(NPROC) it = pool.imap(fn, todo, chunksize=1) try: for _ in todo: left = budget - (time.time() - t0) if left <= 0: break try: seed, r = it.next(timeout=left) except Exception as e: if type(e).__name__ != 'TimeoutError': raise break st[str(seed)] = r; done += 1 finally: pool.terminate(); pool.join() tmp = path + '.tmp'; json.dump(st, open(tmp, 'w')); os.replace(tmp, path) have = sorted(int(s) for s in st); pre = 0 while pre < len(have) and have[pre] == have[0] + pre: pre += 1 print(f'{name}: +{done} seeds in {time.time() - t0:.1f}s wall, {len(have)} stored, contiguous prefix {have[0] if have else None}..{have[0] + pre - 1 if have else None} ({pre}), {NPROC} procs', flush=True) def stats(d): n = len(d); m = sum(d) / n sd = (sum((v - m) ** 2 for v in d) / (n - 1)) ** 0.5 return m, sd / n ** 0.5 def agg(rows, nc): ncb = sum(r['k']['ncb'] for r in rows) out = dict(callbacks=ncb, words=words(nc)) out['mean'] = {k: sum(r['k']['sum'][k] for r in rows) / ncb for k in KEYS} out['max'] = {k: max(r['k']['max'][k] for r in rows) for k in KEYS} out['gas_mean'] = sum(r['k']['gsum'] for r in rows) / ncb out['gas_mean_first_of_step'] = sum(r['k']['gsum_dt'] for r in rows) / sum(r['k']['ndt'] for r in rows) out['first_of_step_share'] = sum(r['k']['ndt'] for r in rows) / ncb w = max(rows, key=lambda r: r['k']['gmax']); out['gas_max'] = w['k']['gmax']; out['gas_max_ops'] = w['k']['gmaxk'] out['over_budget'] = sum(r['k']['nover'] for r in rows); out['over_budget_share'] = out['over_budget'] / ncb out['runs_with_over_budget'] = sum(1 for r in rows if r['k']['nover'] > 0) out['maxspan'] = max(r['k']['maxspan'] for r in rows) h = [sum(r['k']['hist'][i] for r in rows) for i in range(64)]; c = 0; q = {} for i, v in enumerate(h): c += v for p in (0.5, 0.9, 0.99, 0.999): if p not in q and c >= p * ncb: q[p] = (i + 1) * 25000 out['gas_quantile_upper_bin_edge'] = q out['misclass'] = sum(r['k']['nmis'] for r in rows) / max(1, sum(r['k']['ncls'] for r in rows)) out['p_rms_bps'] = math.sqrt(sum(r['k']['se'] for r in rows) / max(1, sum(r['k']['nse'] for r in rows))) return out def bound(mod): """analytic worst case of one callback: dt - 1 >= 30 silent steps, full window occupied at every pass, mixture branch with pi > 0.5, plus the window-edge recentre""" NC, KM = mod.NC, mod.KM taps = sum(min(NC - 1, i + KM) - max(0, i - KM) + 1 for i in range(NC)) k = dict.fromkeys(KEYS, 0) k['ndiff'] = 31; k['mul'] += 31 * taps; k['cells'] += 31 * NC k['mul'] += 30 * (NC + 2); k['div'] += 30 * NC; k['cells'] += 30 * NC # 30 cuts k['mul'] += 30 * NC; k['div'] += 30; k['cells'] += 30 * NC # 30 norms k['mul'] += 2; k['div'] += 2; k['cells'] += 2 # dens k['mul'] += NC + 2; k['div'] += NC; k['cells'] += NC # cut (retail branch) k['mul'] += NC; k['div'] += 1; k['cells'] += NC # norm k['mul'] += 2 * NC; k['div'] += 1; k['cells'] += NC # recentre (pi > .5) k['mul'] += 2 + NC; k['div'] += 1; k['cells'] += 2 + NC # scaling + deposit k['mul'] += 2 * (3 * NC + 3); k['div'] += 4; k['cells'] += 4 * NC # moments twice k['mul'] += 2 * NC; k['div'] += 1; k['cells'] += NC # window-edge recentre k['explog'] += 12; k['mul'] += 14; k['div'] += 10 return dict(taps_per_full_diffusion=taps, ops=k, est_gas=gas(NC, k), words=words(NC)) if __name__ == '__main__': mode = sys.argv[1] load(POL); load(QM); QMRef(BLOB) # parse once, shared with the forked workers if mode == 'gate': seeds = (10000, 10001, 10002) jobs = [(k, s) for k in ('qmref', 'c5', 'c5-counted', 'latt', 'latt-counted') for s in seeds] with Pool(NPROC) as pool: res = pool.map(t_gate, jobs, chunksize=1) for k, s, e, t in res: print(f'{k:13s} {s} {e:.6f} ({t:.1f}s)') print('kernel c5: KM', latt_c5.KM, 'taps', len(latt_c5.KER), ' latt: KM', latt.KM, 'taps', len(latt.KER)) elif mode in ('main', 'exact', 'edge40'): if mode == 'edge40': latt_c5.EDGE = 40 * latt_c5.HC resumable(mode, {'main': t_main, 'exact': t_exact, 'edge40': t_edge40}[mode], int(sys.argv[2]), int(sys.argv[3]), float(sys.argv[4])) elif mode == 'report': st = json.load(open(os.path.join(ROOT, 'main.json'))) seeds = sorted(int(s) for s in st); n = 0 while n < len(seeds) and seeds[n] == seeds[0] + n: n += 1 seeds = seeds[:n]; rows = [st[str(s)] for s in seeds] d = [(s, r['c5'] - r['q']) for s, r in zip(seeds, rows)] m, se = stats([v for _, v in d]) out = dict(variant='latt-c5', base='qt1-ref', seeds=f'{seeds[0]}..{seeds[-1]}', n=n, mean_diff=m, se=se, worst5=[[s, v] for s, v in sorted(d, key=lambda a: a[1])[:5]], mean_edge_variant=sum(r['c5'] for r in rows) / n, mean_edge_base=sum(r['q'] for r in rows) / n) extra = dict(ahead=sum(1 for _, v in d if v > 0), cpu_s_qmref=sum(r['tq'] for r in rows), cpu_s_c5=sum(r['tc'] for r in rows)) for half in (0, 1): dd = [v for _, v in d[half * (n // 2):(half + 1) * (n // 2)]] if len(dd) > 1: extra[f'half{half}'] = stats(dd) extra['c5_counts'] = agg(rows, latt_c5.NC); extra['c5_bound'] = bound(latt_c5); extra['exact_bound'] = bound(latt) pe = os.path.join(ROOT, 'exact.json'); out['vs_exact_lattice'] = None if os.path.exists(pe): ex = json.load(open(pe)); es = [s for s in seeds if str(s) in ex]; k = 0 while k < len(es) and es[k] == seeds[0] + k: k += 1 es = es[:k] if k > 1: m2, se2 = stats([st[str(s)]['c5'] - ex[str(s)]['ex'] for s in es]) out['vs_exact_lattice'] = dict(n=k, mean_diff=m2, se=se2) m3, se3 = stats([ex[str(s)]['ex'] - st[str(s)]['q'] for s in es]) m4, se4 = stats([st[str(s)]['c5'] - st[str(s)]['q'] for s in es]) extra['exact_minus_qmref'] = dict(n=k, mean_diff=m3, se=se3, mean_edge_exact=sum(ex[str(s)]['ex'] for s in es) / k) extra['c5_minus_qmref_same_seeds'] = dict(n=k, mean_diff=m4, se=se4) extra['exact_counts'] = agg([ex[str(s)] for s in es], latt.NC) extra['c5_counts_same_seeds'] = agg([st[str(s)] for s in es], latt_c5.NC) extra['cpu_s_exact'] = sum(ex[str(s)]['t'] for s in es) p4 = os.path.join(ROOT, 'edge40.json') if os.path.exists(p4): e4 = json.load(open(p4)); es = [s for s in seeds if str(s) in e4] if len(es) > 1: dd = [st[str(s)]['c5'] - e4[str(s)]['e'] for s in es]; m5, se5 = stats(dd) extra['c5_minus_literal40'] = dict(n=len(es), seeds=f'{es[0]}..{es[-1]}', mean_diff=m5, se=se5, max_abs=max(abs(v) for v in dd)) w = extra['c5_bound'] out['cost'] = dict(cells=w['ops']['cells'], mul=w['ops']['mul'], div=w['ops']['div'], explog=w['ops']['explog'], lookups=LOOKUPS, state_words=w['words'], est_gas=w['est_gas'], rule='see result.md') print(json.dumps(out, indent=1)); print('--- extra'); print(json.dumps(extra, indent=1)) ``` The side measurements, `extra.py`: ```python """Side measurements for latt_c5 (Taylor gtmcknight's lattice filter at 61 cells of 5 bps); same seeds as the screen. usage: python3 -I extra.py c5s|gq|cap python3 -I extra.py report c5s latt_c5 filter over the DECODED qm-spline tables (the tables qt1's contract holds) gq Taylor's Gaussian filter (real.RealPolicy soft=1) over the full qm tables cap latt_c5 with the silent-step loop capped at K instead of 30 (counted) """ import sys, os, time, json HERE = os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0, HERE) import screen as S from screen import R, load, POL, QM, BLOB, latt_c5, QMRef, C5 import real, qmref _dec = {} def decoded(): if 'q' not in _dec: T = QMRef(BLOB).T; q = {'S': T.P['S']} for tag in T.A: q['A' + tag] = T.A[tag]; q['B' + tag] = T.B[tag] _dec['q'] = q return _dec['q'] def t_c5s(seed): t0 = time.time(); pol = latt_c5.LatticePolicy(load(POL), {}); pol.Q = decoded() return seed, dict(e=R.run(seed, pol)['edge'], t=time.time() - t0) def t_gq(seed): t0 = time.time(); pol = real.RealPolicy(load(POL), {'soft': 1.0, 'qm': QM}) return seed, dict(e=R.run(seed, pol)['edge'], t=time.time() - t0) CAP = [30] class Capped(C5): def on_trade(self, is_buy, ax, ay, t, *a): if t - self.tlast - 1 > CAP[0]: self.tlast = t - 1 - CAP[0] # tlast is only used for dt: same as min(dt - 1, CAP) return C5.on_trade(self, is_buy, ax, ay, t, *a) def t_cap(seed): t0 = time.time(); pol = Capped(load(POL), {'qm': QM}) return seed, dict(e=R.run(seed, pol)['edge'], t=time.time() - t0, k=pol.summary()) def bound_cap(K): NC = latt_c5.NC; b = S.bound(latt_c5); k = dict(b['ops']); taps = b['taps_per_full_diffusion']; d = 30 - K k['ndiff'] -= d; k['mul'] -= d * (taps + NC + 2 + NC); k['div'] -= d * (NC + 1); k['cells'] -= d * 3 * NC return dict(ops=k, est_gas=S.gas(NC, k)) if __name__ == '__main__': mode = sys.argv[1]; load(POL); load(QM); QMRef(BLOB); decoded() if mode == 'report': st = json.load(open(os.path.join(S.ROOT, 'main.json'))); out = {} for f in sorted(os.listdir(S.ROOT)): name = f[:-5] if not f.endswith('.json') or not (name in ('c5s', 'gq', 'exact') or name.startswith('cap')): continue x = json.load(open(os.path.join(S.ROOT, f))); seeds = sorted(int(s) for s in x) key = 'ex' if name == 'exact' else 'e' r = dict(n=len(seeds), seeds=f'{seeds[0]}..{seeds[-1]}', mean_edge=sum(x[str(s)][key] for s in seeds) / len(seeds)) for lab, base in (('minus_qmref', 'q'), ('minus_c5', 'c5')): d = [x[str(s)][key] - st[str(s)][base] for s in seeds]; m, se = S.stats(d) r[lab] = dict(mean_diff=m, se=se, min=min(d), max=max(d)) if name.startswith('cap'): a = S.agg([x[str(s)] for s in seeds], latt_c5.NC) r['counts'] = {k: a[k] for k in ('mean', 'max', 'gas_mean', 'gas_max', 'gas_max_ops', 'over_budget', 'over_budget_share', 'runs_with_over_budget', 'misclass', 'p_rms_bps')} r['bound'] = bound_cap(int(name[3:])) out[name] = r print(json.dumps(out, indent=1)) else: if mode.startswith('cap'): CAP[0] = int(mode[3:]); fn = t_cap else: fn = {'c5s': t_c5s, 'gq': t_gq}[mode] S.resumable(mode, fn, int(sys.argv[2]), int(sys.argv[3]), float(sys.argv[4])) ``` ## Commands Every command run, in order. `$S` is `test/scratch` in the workspace; all state lives there, nothing under `/tmp`. ``` git clone -q https://github.com/gtmcknight/amm-sim $S/T/amm-sim git -C $S/T/amm-sim checkout -q 4812a6382311624e1506aca131ca35c378f930d9 curl -sS -o $S/q/Q.md https://api.imdnotdead.fun/artifacts/8a6e303b662e1a5ea9bb47f31acedd3988ef65b3ddb66d074517aa59ad41ca40 curl -sS -o $S/q/B.md https://api.imdnotdead.fun/artifacts/17d249ab0c33fc44931d9b105cf3f9e3846a560bd518b67310c2a56defacdecc python3 -I - (inline: cut the '## Strategy.sol' and '## Python reference' fenced blocks of Q.md to $S/ref/Strategy.sol and $S/ref/qmref.py, base64-decode the '## Data' block of B.md to $S/ref/blob.bin, print lengths and sha256s) python3 -I $S/T/amm-sim/research/solved/gates.py # Gate P, 4.4 s NPROC=5 python3 -I $S/work/screen.py gate # qmref, c5, c5 counted, latt, latt counted on 10000..10002 python3 -I $S/work/screen.py main 60000 16 400 # timing: 2.0 s python3 -I $S/work/screen.py main 60000 512 400 # the other 496 seeds: 59.0 s python3 -I $S/work/screen.py exact 60000 128 400 # unmodified latt.py: 21.8 s python3 -I $S/work/screen.py edge40 60000 512 400 # literal 40*HC trigger: 38.5 s python3 -I $S/work/screen.py report python3 -I $S/work/extra.py c5s 60000 512 200 # 36.3 s python3 -I $S/work/extra.py gq 60000 512 200 # 9.9 s python3 -I $S/work/screen.py exact 60000 512 200 # adds seeds 60128..60511: 67.7 s python3 -I $S/work/extra.py cap30 60000 32 100 # self-check, equal to c5 on 32 seeds; then rm $S/cap30.json for K in 0 1 2 3 4; do python3 -I $S/work/extra.py cap$K 60000 512 90; done # 41 to 47 s each for K in 6 8 12; do python3 -I $S/work/extra.py cap$K 60000 512 100; done # 49 to 53 s each python3 -I $S/work/extra.py report; python3 -I $S/work/screen.py report python3 -I - (inline: print the two reports as one-line summaries, the first-128 pairing and the est_gas histogram) cd $S/work && NSEED=3 T_SOLVED=../T/amm-sim/research/solved python3 -I latt_c5.py pol_mid.json lattice:qm=qm_k1e-5.json # the variant's own CLI on seeds 10000..10002: edge 513.990 python3 -I $S/work/src.py # fetch each source URL, check each quote, write sources.json ``` The longest command took 68 s. Seeds used: replica 60000..60511, and 10000..10002 for check values only. The holdout 10003..12047 and seeds 60512..61023 were not touched. ## Problems - **Over the gas limit.** As specified (30 silent steps), c5 is far over a 245,000 peak by this rule: about 4 times over on the largest callback seen, and over on 0.53% of all callbacks. This is the main finding, alongside the fact that the coarser grid itself costs nothing measurable in edge. - **The gas figures are estimates from a counting rule, not EVM measurements.** The rule charges 40 gas per multiply or divide and nothing for loop control, memory access, bounds checks or packing and unpacking cells. I expect a Solidity inner loop to cost several times that per tap, so the estimates are more likely low than high. I did not measure it. - **I could not reconcile the rule with the 453k quoted for the exact lattice.** The rule gives the exact lattice a mean of 243,130 and a first-trade mean of 272,265. I do not know how the 453k was measured. If it is a mean, this rule is low by roughly a factor of 1.9. - **State packing is assumed.** 32 bits per cell and 3 scalar words are not verified. The reference is in floats and zeroes cells below 1e-12, which 32-bit cells cannot represent. A contract would need a fixed-point reference of its own, screened again, before Gate F means anything. The lattice also adds a centre and two support bounds to qt1's scalars; if they need a fourth word, add 5,000 gas. - **Tables differ between variant and base**, as the brief specifies. About 0.06 of the +0.58 is the full tables against the spline tables; the like-for-like figure is +0.5218 (se 0.0254). - **The operation counts are approximate** in two places: scalar arithmetic outside the lattice methods is a fixed allowance, and each cut is charged two partial-cell multiplies whether or not both edges fall inside the support. - **The cap table is a side measurement on the choosing seeds.** No cap was selected and none was run on the holdout. - **No engine work.** The item is replica-only, so the amm-challenge checkout and `setup.sh` were not run. - **Working folder.** State is under the workspace's `test/scratch`, the scratch area this run was given, not a top-level `scratch/`. It is not delivered. - **One transient network error:** the first fetch of the qm-spline artifact during the source check ended with "Remote end closed connection without response". A retry succeeded, and all 17 quotes were found at their URLs.