lujingze/SystemSimulationApp
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

优化原生结果编码传输与浏览器缓存，记录八路性能基线

Browse Source 
原生结果series通过字节索引直传，C端使用Ryu精确回读编码和64 KiB批量写出；网页采用Float64缓存和CSV工作线程，减少结果处理与保存等待。

补充八路AME曲线核查、全流程分阶段计时、独立编码基准和复现工具，固定后续优化采用修正八路及rtol=1e-8。C写出1.1808→0.1638 s，点击到可查看8.0100→6.9756 s。

验证：最终10项编码专项、29项相关后端回归通过；8份原生结果逐位一致，16次网页结果/CSV/刷新恢复通过。前端构建及缓存/CSV专项在本轮结果处理工作中通过。环境、原始大结果与临时构建不纳入Git。
This commit is contained in:
lujingze committed 2026-09-11 15:09:15 +00:00
1 parent 808c484f5b
commit 3bc4be3c06
55 files changed
+20522 -141
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/data/README.md
+2
View File
@@ -9,6 +9,8 @@
网页当前只支持导入工程 JSON；XML 可从网页下载，用作后端输入。网页使用原生 BDF 和当前默认 `rtol=1e-8`，与 AME 内部积分器及历史数值设置不能视为相同。这里的参数对齐指模型物理参数与初值，不声称不同仿真器所有数值输出完全一致。
后续优化测试优先使用八路 corrected 工程；仅当八路跑不通且短期无法解决时，再使用四路。具体记录要求见 [优化验证约定](../../docs/standard/optimization-benchmark-model.md)。
2026-09-11 整理时，两份工程内容保持不变。其他 JSON/XML 已按用途移出本目录，移动前后字节数及 SHA256 相同：
| 原文件 | 当前路径 | 用途 |
tests/fixtures/native-skill-test.xml
Vendored
+325
View File
@@ -0,0 +1,325 @@
<?xml version="1.0" encoding="UTF-8"?>
<System name="skill-test" schemaVersion="3" unitSystem="SI">
<Simulation tStart="0" tStop="10" sampleStep="0.02" maxStep="0.001" method="RK45"/>
<Components>
<Component id="amesim_pnch023_1" type="amesim_pnch023" modelVersion="0.1.0">
<Parameter name="gi" value="1"/>
<Parameter name="cvol" value="0.057"/>
<Parameter name="kth" value="0"/>
<Parameter name="sth" value="0.1"/>
<Parameter name="extemp" value="293.15"/>
<Parameter name="p0" value="15300000"/>
<Parameter name="T0" value="293.15"/>
</Component>
<Component id="amesim_pnvo001_1" type="amesim_pnvo001" modelVersion="0.2.0">
<Parameter name="gi" value="1"/>
<Parameter name="cq" value="0.45"/>
<Parameter name="area0" value="0.0000785"/>
<Parameter name="Cv" value="0.5"/>
<Parameter name="Kv" value="0.4"/>
<Parameter name="flowset" value="1"/>
<Parameter name="opening0" value="0"/>
</Component>
<Component id="amesim_step0_1" type="amesim_step0" modelVersion="0.1.0">
<Parameter name="initial" value="0"/>
<Parameter name="final" value="1"/>
<Parameter name="time" value="0.04"/>
</Component>
<Component id="amesim_pnch012_1" type="amesim_pnch012" modelVersion="0.1.0">
<Parameter name="gi" value="1"/>
<Parameter name="cvol0" value="0.015"/>
<Parameter name="kth" value="1500"/>
<Parameter name="sth" value="0.7"/>
<Parameter name="extemp" value="293.15"/>
<Parameter name="p0" value="100000"/>
<Parameter name="T0" value="293.15"/>
<Parameter name="vol1" value="0"/>
<Parameter name="vol2" value="0"/>
<Parameter name="vol3" value="0"/>
<Parameter name="vol4" value="0"/>
<Parameter name="dvol1" value="0"/>
<Parameter name="dvol2" value="0"/>
<Parameter name="dvol3" value="0"/>
<Parameter name="dvol4" value="0"/>
</Component>
<Component id="amesim_pnpl01_2" type="amesim_pnpl01" modelVersion="0.1.0"/>
<Component id="amesim_pnpl01_3" type="amesim_pnpl01" modelVersion="0.1.0"/>
<Component id="amesim_pnpl01_4" type="amesim_pnpl01" modelVersion="0.1.0"/>
<Component id="amesim_ud00_1" type="amesim_ud00" modelVersion="0.2.0">
<Parameter name="tstart" value="0"/>
<Parameter name="start1" value="100000000000000000"/>
<Parameter name="end1" value="100000000000000000"/>
<Parameter name="t1" value="0.8"/>
<Parameter name="start2" value="49000"/>
<Parameter name="end2" value="49000"/>
<Parameter name="t2" value="10"/>
<Parameter name="start3" value="1"/>
<Parameter name="end3" value="1"/>
<Parameter name="t3" value="0"/>
<Parameter name="start4" value="1"/>
<Parameter name="end4" value="1"/>
<Parameter name="t4" value="0"/>
<Parameter name="start5" value="1"/>
<Parameter name="end5" value="1"/>
<Parameter name="t5" value="0"/>
<Parameter name="start6" value="1"/>
<Parameter name="end6" value="1"/>
<Parameter name="t6" value="0"/>
<Parameter name="start7" value="1"/>
<Parameter name="end7" value="1"/>
<Parameter name="t7" value="0"/>
<Parameter name="start8" value="1"/>
<Parameter name="end8" value="1"/>
<Parameter name="t8" value="0"/>
<Parameter name="nstages" value="2"/>
<Parameter name="iscyclic" value="0"/>
</Component>
<Component id="amesim_forc_1" type="amesim_forc" modelVersion="0.2.0">
<Parameter name="direction" value="1"/>
</Component>
<Component id="amesim_pnrp17_1" type="amesim_pnrp17" modelVersion="0.1.0">
<Parameter name="gi" value="1"/>
<Parameter name="dp" value="0.2"/>
<Parameter name="dr" value="0.001"/>
<Parameter name="x0" value="0"/>
</Component>
<Component id="amesim_helium_medium_1" type="amesim_helium_medium" modelVersion="0.1.0">
<Parameter name="gi" value="1"/>
<Parameter name="property_model" value="0"/>
</Component>
<Component id="amesim_pnch012_2" type="amesim_pnch012" modelVersion="0.1.0">
<Parameter name="gi" value="1"/>
<Parameter name="cvol0" value="0.015"/>
<Parameter name="kth" value="1500"/>
<Parameter name="sth" value="0.7"/>
<Parameter name="extemp" value="293.15"/>
<Parameter name="p0" value="100000"/>
<Parameter name="T0" value="293.15"/>
<Parameter name="vol1" value="0"/>
<Parameter name="vol2" value="0"/>
<Parameter name="vol3" value="0"/>
<Parameter name="vol4" value="0"/>
<Parameter name="dvol1" value="0"/>
<Parameter name="dvol2" value="0"/>
<Parameter name="dvol3" value="0"/>
<Parameter name="dvol4" value="0"/>
</Component>
<Component id="amesim_pnpl01_6" type="amesim_pnpl01" modelVersion="0.1.0"/>
<Component id="amesim_pnpl01_7" type="amesim_pnpl01" modelVersion="0.1.0"/>
<Component id="amesim_pnpl01_8" type="amesim_pnpl01" modelVersion="0.1.0"/>
<Component id="amesim_pnpl01_9" type="amesim_pnpl01" modelVersion="0.1.0"/>
<Component id="amesim_mecmas21_2" type="amesim_mecmas21" modelVersion="0.2.0">
<Parameter name="mass" value="50"/>
<Parameter name="fstick" value="0"/>
<Parameter name="fcoul" value="0"/>
<Parameter name="rvisc" value="0"/>
<Parameter name="wind" value="0"/>
<Parameter name="dvel" value="0.000001"/>
<Parameter name="restdvel" value="0.000001"/>
<Parameter name="restcoeff" value="0.65"/>
<Parameter name="astrib" value="0.001"/>
<Parameter name="xmin" value="-1"/>
<Parameter name="Kbmin" value="1000000000"/>
<Parameter name="Dbmin" value="10000"/>
<Parameter name="Pdmin" value="0.0001"/>
<Parameter name="xmax" value="0.8"/>
<Parameter name="Kbmax" value="1000000000"/>
<Parameter name="Dbmax" value="10000"/>
<Parameter name="Pdmax" value="0.0001"/>
<Parameter name="theta" value="0"/>
<Parameter name="useFriction" value="1"/>
<Parameter name="stoptype" value="4"/>
<Parameter name="discContactOption" value="1"/>
<Parameter name="strib" value="1"/>
<Parameter name="frictionType" value="1"/>
<Parameter name="v0" value="0"/>
<Parameter name="x0" value="0"/>
</Component>
<Component id="amesim_f000_1" type="amesim_f000" modelVersion="0.1.0"/>
<Component id="amesim_f000_2" type="amesim_f000" modelVersion="0.1.0"/>
<Component id="amesim_lstp00a_1" type="amesim_lstp00a" modelVersion="0.2.0">
<Parameter name="na" value="10"/>
<Parameter name="gap0" value="0"/>
<Parameter name="kcont" value="100000000000"/>
<Parameter name="G" value="85700000000"/>
<Parameter name="sdiam" value="0.02"/>
<Parameter name="wdiam" value="0.002"/>
<Parameter name="rcont" value="100000000000"/>
<Parameter name="Pdis" value="1e-7"/>
<Parameter name="stiffmode" value="1"/>
<Parameter name="discContactOption" value="1"/>
</Component>
<Component id="amesim_forc_2" type="amesim_forc" modelVersion="0.2.0">
<Parameter name="direction" value="1"/>
</Component>
<Component id="amesim_ud00_2" type="amesim_ud00" modelVersion="0.2.0">
<Parameter name="tstart" value="0"/>
<Parameter name="start1" value="1000000000000"/>
<Parameter name="end1" value="1000000000000"/>
<Parameter name="t1" value="0.8"/>
<Parameter name="start2" value="0"/>
<Parameter name="end2" value="0"/>
<Parameter name="t2" value="10"/>
<Parameter name="start3" value="1"/>
<Parameter name="end3" value="1"/>
<Parameter name="t3" value="0"/>
<Parameter name="start4" value="1"/>
<Parameter name="end4" value="1"/>
<Parameter name="t4" value="0"/>
<Parameter name="start5" value="1"/>
<Parameter name="end5" value="1"/>
<Parameter name="t5" value="0"/>
<Parameter name="start6" value="1"/>
<Parameter name="end6" value="1"/>
<Parameter name="t6" value="0"/>
<Parameter name="start7" value="1"/>
<Parameter name="end7" value="1"/>
<Parameter name="t7" value="0"/>
<Parameter name="start8" value="1"/>
<Parameter name="end8" value="1"/>
<Parameter name="t8" value="0"/>
<Parameter name="nstages" value="2"/>
<Parameter name="iscyclic" value="0"/>
</Component>
<Component id="amesim_mecmas21_5" type="amesim_mecmas21" modelVersion="0.2.0">
<Parameter name="mass" value="170000"/>
<Parameter name="fstick" value="0"/>
<Parameter name="fcoul" value="0"/>
<Parameter name="rvisc" value="0"/>
<Parameter name="wind" value="0"/>
<Parameter name="dvel" value="0.000001"/>
<Parameter name="restdvel" value="0.000001"/>
<Parameter name="restcoeff" value="0.65"/>
<Parameter name="astrib" value="0.001"/>
<Parameter name="xmin" value="0"/>
<Parameter name="Kbmin" value="1000000000"/>
<Parameter name="Dbmin" value="10000"/>
<Parameter name="Pdmin" value="0.0001"/>
<Parameter name="xmax" value="0.37"/>
<Parameter name="Kbmax" value="1000000000"/>
<Parameter name="Dbmax" value="10000"/>
<Parameter name="Pdmax" value="0.0001"/>
<Parameter name="theta" value="0"/>
<Parameter name="useFriction" value="1"/>
<Parameter name="stoptype" value="1"/>
<Parameter name="discContactOption" value="1"/>
<Parameter name="strib" value="1"/>
<Parameter name="frictionType" value="1"/>
<Parameter name="v0" value="0"/>
<Parameter name="x0" value="0"/>
</Component>
<Component id="amesim_mecmas21_7" type="amesim_mecmas21" modelVersion="0.2.0">
<Parameter name="mass" value="90000"/>
<Parameter name="fstick" value="0"/>
<Parameter name="fcoul" value="0"/>
<Parameter name="rvisc" value="0"/>
<Parameter name="wind" value="0"/>
<Parameter name="dvel" value="0.000001"/>
<Parameter name="restdvel" value="0.000001"/>
<Parameter name="restcoeff" value="0.65"/>
<Parameter name="astrib" value="0.001"/>
<Parameter name="xmin" value="-0.72"/>
<Parameter name="Kbmin" value="1000000000"/>
<Parameter name="Dbmin" value="10000"/>
<Parameter name="Pdmin" value="0.0001"/>
<Parameter name="xmax" value="0"/>
<Parameter name="Kbmax" value="1000000000"/>
<Parameter name="Dbmax" value="10000"/>
<Parameter name="Pdmax" value="0.0001"/>
<Parameter name="theta" value="0"/>
<Parameter name="useFriction" value="1"/>
<Parameter name="stoptype" value="1"/>
<Parameter name="discContactOption" value="1"/>
<Parameter name="strib" value="1"/>
<Parameter name="frictionType" value="1"/>
<Parameter name="v0" value="0"/>
<Parameter name="x0" value="0"/>
</Component>
</Components>
<Connections>
<Connection id="edge-amesim_pnch023_1-port_2-amesim_pnvo001_1-port_2-1786524999267">
<Endpoint component="amesim_pnch023_1" port="port_2"/>
<Endpoint component="amesim_pnvo001_1" port="port_2"/>
</Connection>
<Connection id="edge-amesim_pnvo001_1-port_3-amesim_pnch012_1-port_1-1786525009973">
<Endpoint component="amesim_pnvo001_1" port="port_3"/>
<Endpoint component="amesim_pnch012_1" port="port_1"/>
</Connection>
<Connection id="edge-contact-amesim_pnpl01_2-port_1-amesim_pnch012_1-port_2-1786525035706-0">
<Endpoint component="amesim_pnpl01_2" port="port_1"/>
<Endpoint component="amesim_pnch012_1" port="port_2"/>
</Connection>
<Connection id="edge-contact-amesim_pnpl01_3-port_1-amesim_pnch012_1-port_3-1786525054054-0">
<Endpoint component="amesim_pnpl01_3" port="port_1"/>
<Endpoint component="amesim_pnch012_1" port="port_3"/>
</Connection>
<Connection id="edge-contact-amesim_pnpl01_4-port_1-amesim_pnch012_1-port_4-1786525061349-0">
<Endpoint component="amesim_pnpl01_4" port="port_1"/>
<Endpoint component="amesim_pnch012_1" port="port_4"/>
</Connection>
<Connection id="edge-amesim_step0_1-out-amesim_pnvo001_1-res-1786525074494">
<Endpoint component="amesim_step0_1" port="out"/>
<Endpoint component="amesim_pnvo001_1" port="res"/>
</Connection>
<Connection id="edge-amesim_ud00_1-out-amesim_forc_1-res-1786525088412">
<Endpoint component="amesim_ud00_1" port="out"/>
<Endpoint component="amesim_forc_1" port="res"/>
</Connection>
<Connection id="edge-contact-amesim_pnpl01_6-port_1-amesim_pnch023_1-port_1-1786525164428-0">
<Endpoint component="amesim_pnpl01_6" port="port_1"/>
<Endpoint component="amesim_pnch023_1" port="port_1"/>
</Connection>
<Connection id="edge-contact-amesim_pnpl01_7-port_1-amesim_pnch012_2-port_1-1786525181969-0">
<Endpoint component="amesim_pnpl01_7" port="port_1"/>
<Endpoint component="amesim_pnch012_2" port="port_1"/>
</Connection>
<Connection id="edge-contact-amesim_pnpl01_8-port_1-amesim_pnch012_2-port_4-1786525185493-0">
<Endpoint component="amesim_pnpl01_8" port="port_1"/>
<Endpoint component="amesim_pnch012_2" port="port_4"/>
</Connection>
<Connection id="edge-contact-amesim_pnpl01_9-port_1-amesim_pnch012_2-port_2-1786525188482-0">
<Endpoint component="amesim_pnpl01_9" port="port_1"/>
<Endpoint component="amesim_pnch012_2" port="port_2"/>
</Connection>
<Connection id="edge-amesim_mecmas21_2-port_1-amesim_pnrp17_1-port_2-1786525203778">
<Endpoint component="amesim_mecmas21_2" port="port_1"/>
<Endpoint component="amesim_pnrp17_1" port="port_2"/>
</Connection>
<Connection id="edge-amesim_pnrp17_1-port_5-amesim_lstp00a_1-port_1-1786525213548">
<Endpoint component="amesim_pnrp17_1" port="port_5"/>
<Endpoint component="amesim_lstp00a_1" port="port_1"/>
</Connection>
<Connection id="edge-amesim_ud00_2-out-amesim_forc_2-res-1786525226761">
<Endpoint component="amesim_ud00_2" port="out"/>
<Endpoint component="amesim_forc_2" port="res"/>
</Connection>
<Connection id="edge-contact-amesim_pnrp17_1-port_1-amesim_pnch012_2-port_3-1786525729832-1">
<Endpoint component="amesim_pnrp17_1" port="port_1"/>
<Endpoint component="amesim_pnch012_2" port="port_3"/>
</Connection>
<Connection id="edge-amesim_forc_1-port_2-amesim_mecmas21_7-port_2-1786525915163">
<Endpoint component="amesim_forc_1" port="port_2"/>
<Endpoint component="amesim_mecmas21_7" port="port_2"/>
</Connection>
<Connection id="edge-amesim_mecmas21_7-port_1-amesim_pnrp17_1-port_3-1786525916652">
<Endpoint component="amesim_mecmas21_7" port="port_1"/>
<Endpoint component="amesim_pnrp17_1" port="port_3"/>
</Connection>
<Connection id="edge-amesim_mecmas21_5-port_1-amesim_forc_2-port_2-1786525922010">
<Endpoint component="amesim_mecmas21_5" port="port_1"/>
<Endpoint component="amesim_forc_2" port="port_2"/>
</Connection>
<Connection id="edge-amesim_lstp00a_1-port_2-amesim_mecmas21_5-port_2-1786525924224">
<Endpoint component="amesim_lstp00a_1" port="port_2"/>
<Endpoint component="amesim_mecmas21_5" port="port_2"/>
</Connection>
<Connection id="edge-contact-amesim_f000_2-port_1-amesim_pnrp17_1-port_4-1788429018340-0">
<Endpoint component="amesim_f000_2" port="port_1"/>
<Endpoint component="amesim_pnrp17_1" port="port_4"/>
</Connection>
<Connection id="edge-contact-amesim_f000_1-port_1-amesim_mecmas21_2-port_2-1788429022851-0">
<Endpoint component="amesim_f000_1" port="port_1"/>
<Endpoint component="amesim_mecmas21_2" port="port_2"/>
</Connection>
</Connections>
</System>
tests/manual/backend_stage_profile.py
+300
View File
@@ -0,0 +1,300 @@
"""Serve the real app with opt-in, request-scoped stage measurements.
All C changes are sparse clocks in an isolated copy of runtime/main.c. Component
kernels and the solver are copied unchanged. No production modules are edited.
Use --plain for the uninstrumented HTTP/browser control. Output must be fresh.
"""
from __future__ import annotations
import argparse
from contextvars import ContextVar
from functools import wraps
from hashlib import sha256
import json
import os
import resource
from pathlib import Path
import shutil
import subprocess
import sys
import threading
import time
from uuid import uuid4
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT))
LOCAL = threading.local()
REQUEST = ContextVar('benchmark_request', default=None)
def isolate_runtime(output):
target = output/'native'
shutil.copytree(ROOT/'native', target)
p = target/'runtime/main.c'; source = p.read_text()
source = '#include <time.h>\n' + source
signature = next(line for line in source.splitlines() if line.startswith('static int write_result(NativeRun *r,'))
source = source.replace(signature,
'static double profile_projection_seconds, profile_json_write_seconds;\n'
'static double profile_projection_cpu_seconds, profile_json_write_cpu_seconds;\n'
+ signature + '\n'
' double profile_output_start=native_wall_time();\n'
' double profile_output_cpu_start=(double)clock()/CLOCKS_PER_SEC;')
source = source.replace(' FILE *f=fopen(path,"wb");',
' profile_projection_seconds=native_wall_time()-profile_output_start;\n'
' profile_projection_cpu_seconds=(double)clock()/CLOCKS_PER_SEC-profile_output_cpu_start;\n'
' double profile_write_start=native_wall_time();\n'
' double profile_write_cpu_start=(double)clock()/CLOCKS_PER_SEC;\n FILE *f=fopen(path,"wb");')
source = source.replace(' free(values); return ok;',
' profile_json_write_seconds=native_wall_time()-profile_write_start;\n'
' profile_json_write_cpu_seconds=(double)clock()/CLOCKS_PER_SEC-profile_write_cpu_start;\n'
' free(values); return ok;')
source = source.replace('int main(int argc, char **argv) {',
'int main(int argc, char **argv) {\n double profile_main_start=native_wall_time();')
source = source.replace(' native_solve(&r);',
' double profile_solve_call_start=native_wall_time();\n native_solve(&r);\n'
' double profile_solve_call_end=native_wall_time();')
source = source.replace(' native_run_free(&r); return code;', r''' native_run_free(&r);
fprintf(stderr,"{\"event\":\"native-stage-profile\",\"mainStartMonotonic\":%.17g,"
"\"solveCallStartMonotonic\":%.17g,\"integrationStartMonotonic\":%.17g,"
"\"integrationSeconds\":%.17g,\"solveCallEndMonotonic\":%.17g,"
"\"argumentPreparationSeconds\":%.17g,\"initializationSeconds\":%.17g,"
"\"finalSampleAndStatusSeconds\":%.17g,\"projectionSeconds\":%.17g,"
"\"jsonWriteSeconds\":%.17g,\"projectionCpuSeconds\":%.17g,\"jsonWriteCpuSeconds\":%.17g,\"mainTotalSeconds\":%.17g}\n",
profile_main_start,profile_solve_call_start,r.wall_start,r.solve_seconds,profile_solve_call_end,
profile_solve_call_start-profile_main_start,r.wall_start-profile_solve_call_start,
profile_solve_call_end-r.wall_start-r.solve_seconds,profile_projection_seconds,
profile_json_write_seconds,profile_projection_cpu_seconds,profile_json_write_cpu_seconds,
native_wall_time()-profile_main_start);
return code;''')
if source.count('native-stage-profile') != 1:
raise ValueError('Unrecognized runtime layout')
p.write_text(source)
# The measurement copy must preserve all numerical files verbatim.
for original in (ROOT/'native').rglob('*'):
if original.is_file() and original.relative_to(ROOT/'native').as_posix() != 'runtime/main.c':
assert original.read_bytes() == (target/original.relative_to(ROOT/'native')).read_bytes()
return target
class Profile:
def __init__(self, output):
self.output=output; self.records={}; self.trackers={}; self.results={}
def current(self):
return getattr(LOCAL,'record',None) or REQUEST.get()
def record(self, record, name, start, end):
record['spans'].append({'name':name,'startMs':(start-record['startNs'])/1e6,
'endMs':(end-record['startNs'])/1e6,'seconds':(end-start)/1e9})
def wrap(self, function, name):
@wraps(function)
def measured(*args, **kwargs):
record=self.current()
if record is None: return function(*args, **kwargs)
start=time.perf_counter_ns()
try:return function(*args, **kwargs)
finally:self.record(record,name,start,time.perf_counter_ns())
return measured
def install(self, api, builder, runner):
profile=self
api.validate_system_xml_document=self.wrap(api.validate_system_xml_document,'xml_validation')
api.compile_system_xml_network=self.wrap(api.compile_system_xml_network,'network_compilation')
runner.compile_native_program=self.wrap(runner.compile_native_program,'c_generation')
if hasattr(runner,'read_indexed_result'):
runner.read_indexed_result=self.wrap(runner.read_indexed_result,'native_indexed_result_read')
from app.simulation.native_codegen import transport
class TransportJson:
def __getattr__(self,key):return getattr(json,key)
def loads(self,value,*args,**kwargs):
return profile.wrap(json.loads,'native_result_metadata_json_parse')(value,*args,**kwargs)
transport.json=TransportJson()
api.serialize_result_parts=self.wrap(api.serialize_result_parts,'response_result_json_serialization')
original_read_bytes=Path.read_bytes
def read_bytes(path,*args,**kwargs):
name='native_result_read_bytes' if path.name=='result.json' else 'native_result_index_read_bytes'
if path.name in ('result.json','result-index.json') and self.current() is not None:
return self.wrap(original_read_bytes,name)(path,*args,**kwargs)
return original_read_bytes(path,*args,**kwargs)
Path.read_bytes=read_bytes
original_build=runner.build_native
def isolated_build(program, **kwargs):
kwargs['cache_dir']=self.output/'cache'
result=original_build(program,**kwargs)
rec=self.current()
if rec is not None:
rec['build']={'cacheHit':result.cache_hit,'buildKey':result.manifest['buildKey'],
'reportedSeconds':result.seconds,'executable':str(result.executable)}
return result
runner.build_native=self.wrap(isolated_build,'native_build_or_cache_validation')
builder.build_native=runner.build_native
# Observe process creation and reaping without polling more often or
# changing the production stderr-reader/cancellation loop.
class RunnerSubprocess:
def __getattr__(self, key): return getattr(subprocess, key)
def Popen(self, *args, **kwargs):
rec=profile.current()
if rec is None: return subprocess.Popen(*args, **kwargs)
start=time.perf_counter_ns()
usage=resource.getrusage(resource.RUSAGE_CHILDREN)
process=subprocess.Popen(*args, **kwargs)
profile.record(rec,'native_process_spawn',start,time.perf_counter_ns())
rec['process']={'pid':process.pid,'command':list(args[0]),
'startMs':(start-rec['startNs'])/1e6}
original_poll,original_wait=process.poll,process.wait
def observe(code):
if code is not None and 'exitObservedMs' not in rec['process']:
now=time.perf_counter_ns()
after=resource.getrusage(resource.RUSAGE_CHILDREN)
rec['process'].update(exitCode=code,exitObservedMs=(now-rec['startNs'])/1e6,
childrenUserCpuSeconds=after.ru_utime-usage.ru_utime,
childrenSystemCpuSeconds=after.ru_stime-usage.ru_stime)
profile.record(rec,'native_process_lifetime_observed',start,now)
return code
def poll(*a,**k):return observe(original_poll(*a,**k))
def wait(*a,**k):return observe(original_wait(*a,**k))
process.poll,process.wait=poll,wait
return process
runner.subprocess=RunnerSubprocess()
original_execute=runner.execute_native
def execute(*args, **kwargs):
data=original_execute(*args,**kwargs)
rec=self.current()
if rec is not None:
rec['native']={k:v for k,v in data.items() if k not in ('series','final','finalState')}
series=data['series']
rec['sampleCount']=series.sample_count if hasattr(series,'sample_count') else len(series.get('time',[]))
if hasattr(series,'data'):rec['rawSeriesBytes']=len(series.data)
started=time.perf_counter_ns()
directory=self.output/'requests'/rec['id'];directory.mkdir(parents=True,exist_ok=True)
source=kwargs['run_dir']/'result.json';target=directory/'native-result.json'
try:os.link(source,target)
except OSError:shutil.copyfile(source,target)
shutil.copyfile(kwargs['run_dir']/'worker.log',directory/'worker.log')
index=kwargs['run_dir']/'result-index.json'
if index.exists():shutil.copyfile(index,directory/'result-index.json')
self.record(rec,'profile_artifact_preservation',started,time.perf_counter_ns())
return data
runner.execute_native=self.wrap(execute,'native_execution_with_result_read')
runner.simulate_native=self.wrap(runner.simulate_native,'native_orchestration_total')
original_read=Path.read_text
@wraps(original_read)
def read_text(path,*args,**kwargs):
if path.name=='result.json' and self.current() is not None:
return self.wrap(original_read,'native_result_read_utf8')(path,*args,**kwargs)
return original_read(path,*args,**kwargs)
Path.read_text=read_text
class RunnerJson:
def __getattr__(self,key):return getattr(json,key)
def loads(self,text,*args,**kwargs):
rec=profile.current()
if rec is not None and len(text)>100000:
return profile.wrap(json.loads,'native_result_json_parse')(text,*args,**kwargs)
value=json.loads(text,*args,**kwargs)
if rec is not None and isinstance(value,dict) and value.get('event')=='native-stage-profile':
rec['nativeStages']=value
return value
runner.json=RunnerJson()
original_run=api.run_system_xml_simulation
def run(xml,*args,**kwargs):
tracker=kwargs.get('activity_tracker') or (args[2] if len(args)>2 else None)
rec=self.trackers.get(id(tracker)) or REQUEST.get()
if rec is None:return original_run(xml,*args,**kwargs)
LOCAL.record=rec
started=time.perf_counter_ns()
try:
result=original_run(xml,*args,**kwargs)
self.results[id(result)]=rec
rec['existingPerformance']=result['diagnostics'].get('performance')
return result
finally:
self.record(rec,'simulation_worker_total',started,time.perf_counter_ns())
LOCAL.record=None
api.run_system_xml_simulation=run
original_stream=api.simulation_event_stream
def stream(xml,*,task=None,**kwargs):
rec=self.records.get(task.simulation_id) if task else None
if rec is not None:
self.trackers[id(task.activity_tracker)]=rec
rec['xmlSha256']=sha256(xml).hexdigest();rec['xmlBytes']=len(xml)
directory=self.output/'requests'/rec['id'];directory.mkdir(parents=True,exist_ok=True)
(directory/'input.xml').write_bytes(xml)
yield from original_stream(xml,task=task,**kwargs)
api.simulation_event_stream=stream
class ApiJson:
def __getattr__(self,key):return getattr(json,key)
def dumps(self,value,*args,**kwargs):
rec=profile.results.get(id(value.get('result'))) if isinstance(value,dict) and value.get('event')=='result' else None
if rec is None:return json.dumps(value,*args,**kwargs)
start=time.perf_counter_ns();result=json.dumps(value,*args,**kwargs)
profile.record(rec,'response_result_json_serialization',start,time.perf_counter_ns())
rec['resultJsonCharacters']=len(result)
return result
api.json=ApiJson()
api.build_simulation_results_csv=self.wrap(api.build_simulation_results_csv,'csv_assembly')
def app(self, underlying):
async def wrapped(scope,receive,send):
if scope['type']!='http' or scope.get('path') not in ('/api/system-xml/simulate-stream','/api/simulation-results/csv'):
return await underlying(scope,receive,send)
headers=dict(scope.get('headers',[]));ident=headers.get(b'x-simulation-id',uuid4().hex.encode()).decode()
if not all(c.isalnum() or c in '-_' for c in ident): raise ValueError('Invalid profiling request ID')
rec={'id':ident,'path':scope['path'],'startNs':time.perf_counter_ns(),'spans':[],
'responseBodyBytes':0,'responseSendAwaitSeconds':0.0}
self.records[ident]=rec;token=REQUEST.set(rec)
async def observed_receive():
message=await receive()
if message['type']=='http.request' and not message.get('more_body',False):
rec['requestBodyCompleteMs']=(time.perf_counter_ns()-rec['startNs'])/1e6
return message
async def observed_send(message):
start=time.perf_counter_ns()
if message['type']=='http.response.start':
rec['responseHeadersMs']=(start-rec['startNs'])/1e6;rec['httpStatus']=message['status']
if message['type']=='http.response.body':
rec['responseBodyBytes']+=len(message.get('body',b''))
if len(message.get('body',b''))>100000:
rec['largeResultBodySendStartMs']=(start-rec['startNs'])/1e6
await send(message)
rec['responseSendAwaitSeconds']+=(time.perf_counter_ns()-start)/1e9
if message['type']=='http.response.body' and not message.get('more_body',False):
rec['responseBodyCompleteMs']=(time.perf_counter_ns()-rec['startNs'])/1e6
try:await underlying(scope,observed_receive,observed_send)
finally:
rec['httpTotalSeconds']=(time.perf_counter_ns()-rec['startNs'])/1e9
REQUEST.reset(token)
target=self.output/'requests'/ident;target.mkdir(parents=True,exist_ok=True)
(target/'stages.json').write_text(json.dumps(rec,ensure_ascii=False,indent=2)+'\n')
return wrapped
def main():
parser=argparse.ArgumentParser(description=__doc__)
parser.add_argument('--output-dir',type=Path,required=True)
parser.add_argument('--port',type=int,default=8012)
parser.add_argument('--plain',action='store_true')
parser.add_argument('--frontend-dist',type=Path,default=ROOT/'frontend/dist')
args=parser.parse_args();out=args.output_dir.resolve()
if out.exists():parser.error('Choose a fresh output directory')
out.mkdir(parents=True)
# Snapshot static assets so another workspace build cannot change a run.
shutil.copytree(args.frontend_dist,out/'frontend')
if not args.plain:os.environ['SIMULATIONAPP_PROFILE']='standard'
import app.main as api
from app.simulation.native_codegen import build as builder,runner
api.FRONTEND_DIST_DIR=out/'frontend'
profile=None
if not args.plain:
builder.NATIVE=isolate_runtime(out)
profile=Profile(out);profile.install(api,builder,runner)
metadata={'gitHead':subprocess.check_output(['git','rev-parse','HEAD'],cwd=ROOT,text=True).strip(),
'mode':'plain' if args.plain else 'profile','port':args.port,
'python':sys.version,'platform':sys.platform,
'frontendFiles':{str(p.relative_to(out/'frontend')):sha256(p.read_bytes()).hexdigest() for p in (out/'frontend').rglob('*') if p.is_file()},
'productionKernelSha256':sha256((ROOT/'native/components/kernels.c').read_bytes()).hexdigest()}
(out/'environment.json').write_text(json.dumps(metadata,indent=2)+'\n')
import uvicorn
uvicorn.run(profile.app(api.app) if profile else api.app,host='127.0.0.1',port=args.port)
if __name__=='__main__':main()
tests/manual/benchmark_native_result_encoding.py
+318
View File
@@ -0,0 +1,318 @@
"""Replay native result encoding in isolated C programs, without model evaluation.
Prepare only by default. Explicit --run builds a small C replay and serially
runs one warmup and three measured real-file outputs per variant. --dev-null
adds sink-only runs after real-file verification; these never stand in for I/O.
.venv/bin/python tests/manual/benchmark_native_result_encoding.py \
--result-json test/web-cost-20260911/native-compute-profile/control/run-1/result.json \
--output-dir test/c-result-encoding-20260911 --ryu-root /path/to/ryu --run
ryu-root must contain ryu/d2s.c and ryu/ryu.h (the ryu/ subdirectory itself is
also accepted). No dependency is downloaded and no production source is edited.
Every series cell, final scalar and finalState cell is encoded in C. Static
metadata, JSON structure and escaped keys are prepared outside timing. Values
are loaded contiguously before timing; this isolates decimal encoding/write
cost, excluding projection and the production writer's strided matrix reads.
Timers include fopen, buffer setup, formatting, write, flush and fclose, but
not fsync durability. All real-file outputs are parsed and compared as complete
binary64 values, including signed zero, outside the measured interval.
"""
from __future__ import annotations
import argparse
from array import array
from hashlib import sha256
import json
import math
import os
from pathlib import Path
import shutil
import statistics
import struct
import subprocess
import sys
import time
ROOT = Path(__file__).resolve().parents[2]
VARIANTS = ["fprintf-default", "fprintf-1m", "snprintf-64k", "ryu-64k"]
C_SOURCE = r'''
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>
#include <math.h>
#include <time.h>
#if HAVE_RYU
#include "ryu/ryu.h"
#endif
#include "replay-layout.h"
#define CHUNK (64u*1024u)
typedef struct { FILE *f; char block[CHUNK]; size_t used; unsigned long long bytes; int failed; } Writer;
static double wall_now(void) { struct timespec t; if(clock_gettime(CLOCK_MONOTONIC,&t))exit(72); return t.tv_sec+t.tv_nsec*1e-9; }
static double cpu_now(void) { struct timespec t; if(clock_gettime(CLOCK_PROCESS_CPUTIME_ID,&t))exit(72); return t.tv_sec+t.tv_nsec*1e-9; }
static void flush_block(Writer *w) {
if(w->used && fwrite(w->block,1,w->used,w->f)!=w->used)w->failed=1;
w->used=0;
}
static void block_bytes(Writer *w,const char *text,size_t n) {
w->bytes+=n;
while(n) {
size_t left=CHUNK-w->used, part=n<left?n:left;
memcpy(w->block+w->used,text,part); w->used+=part; text+=part; n-=part;
if(w->used==CHUNK)flush_block(w);
}
}
static void direct_bytes(Writer *w,const char *text,size_t n) {
w->bytes+=n;
if(fwrite(text,1,n,w->f)!=n)w->failed=1;
}
static void encode(Writer *w,const double *values,int mode) {
for(size_t g=0;g<GROUP_COUNT;g++) {
const Group *group=&groups[g];
if(mode<2)direct_bytes(w,group->prefix,group->prefix_length);
else block_bytes(w,group->prefix,group->prefix_length);
for(size_t i=0;i<group->count;i++) {
double value=values[group->offset+i];
if(mode<2) {
/* Same number/separator formatting call as native main.c. */
int n=fprintf(w->f,"%s%.17g",i?",":"",value);
if(n<0)w->failed=1; else w->bytes+=(unsigned)n;
} else if(mode==2) {
/* Format directly into the remaining batch buffer. */
if(CHUNK-w->used<64)flush_block(w);
int n=snprintf(w->block+w->used,CHUNK-w->used,"%s%.17g",i?",":"",value);
if(n<0 || (size_t)n>=CHUNK-w->used){w->failed=1;return;}
w->used+=(unsigned)n;w->bytes+=(unsigned)n;
} else {
#if HAVE_RYU
if(CHUNK-w->used<64)flush_block(w);
if(i){w->block[w->used++]=',';w->bytes++;}
int n=d2s_buffered_n(value,w->block+w->used);
if(n<1 || n>32){w->failed=1;return;}
w->used+=(unsigned)n;w->bytes+=(unsigned)n;
#else
w->failed=1;return;
#endif
}
}
}
if(mode<2)direct_bytes(w,tail,TAIL_LENGTH);
else {block_bytes(w,tail,TAIL_LENGTH);flush_block(w);}
}
int main(int argc,char **argv) {
if(argc!=4)return 64;
int mode=-1;
const char *names[]={"fprintf-default","fprintf-1m","snprintf-64k","ryu-64k"};
for(int i=0;i<4;i++)if(!strcmp(argv[1],names[i]))mode=i;
if(mode<0 || (mode==3 && !HAVE_RYU))return 64;
if(sizeof(double)!=8 || sizeof(uint64_t)!=8)return 65;
FILE *input=fopen(argv[2],"rb"); if(!input)return 66;
double *values=malloc(VALUE_COUNT*sizeof(double));
if(!values){fclose(input);return 67;}
int loaded=fread(values,sizeof(double),VALUE_COUNT,input)==VALUE_COUNT && fgetc(input)==EOF && !ferror(input);
if(fclose(input))loaded=0;
if(!loaded){free(values);return 68;}
for(size_t i=0;i<VALUE_COUNT;i++)if(!isfinite(values[i])){free(values);return 69;}
Writer *writer=calloc(1,sizeof(Writer)); char *stdio_buffer=malloc(1024u*1024u);
if(!writer || !stdio_buffer){free(values);free(writer);free(stdio_buffer);return 67;}
/* Loading, allocation and input validation are intentionally outside timing. */
double wall_start=wall_now(), cpu_start=cpu_now();
writer->f=fopen(argv[3],"wb");
if(!writer->f){free(values);free(writer);free(stdio_buffer);return 70;}
if(mode==1 && setvbuf(writer->f,stdio_buffer,_IOFBF,1024u*1024u))writer->failed=1;
/* Manual batch variants use identical unbuffered FILE sinks. */
if(mode>=2 && setvbuf(writer->f,NULL,_IONBF,0))writer->failed=1;
if(!writer->failed)encode(writer,values,mode);
if(ferror(writer->f))writer->failed=1;
if(fclose(writer->f))writer->failed=1;
double cpu_seconds=cpu_now()-cpu_start, wall_seconds=wall_now()-wall_start;
printf("{\"variant\":\"%s\",\"wallSeconds\":%.17g,\"cpuSeconds\":%.17g,\"encodedBytes\":%llu,\"valueCount\":%zu,\"success\":%s}\n",
names[mode],wall_seconds,cpu_seconds,writer->bytes,(size_t)VALUE_COUNT,writer->failed?"false":"true");
int code=writer->failed?71:0;
free(values);free(stdio_buffer);free(writer);return code;
}
'''
def digest(path: Path) -> str:
return sha256(path.read_bytes()).hexdigest()
def write_json(path: Path, value: object) -> None:
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def read_json_numbers(path: Path) -> dict:
# JSON's integer spelling -0 must retain the sign before conversion to double.
return json.loads(path.read_bytes(), parse_int=lambda token: -0.0 if token == "-0" else int(token))
def c_string(value: bytes) -> str:
return '"' + ''.join(f"\\x{byte:02x}" for byte in value) + '"'
def numeric_blocks(result: dict) -> list[tuple[str, list]]:
blocks = [(f"series/{key}", value) for key, value in result["series"].items()]
blocks.extend((f"final/{key}", [value]) for key, value in result["final"].items())
blocks.append(("finalState", result["finalState"]))
return blocks
def prepare(args: argparse.Namespace) -> dict:
output = args.output_dir.resolve()
if not output.is_relative_to(ROOT / "test"):
raise RuntimeError("Output must be beneath the repository's ignored test/ directory")
if sys.byteorder != "little" or array('d').itemsize != 8 or not sys.platform.startswith("linux"):
raise RuntimeError("This isolated replay currently requires Linux and little-endian binary64")
result = read_json_numbers(args.result_json)
if not isinstance(result.get("series"), dict) or not isinstance(result.get("final"), dict) or not isinstance(result.get("finalState"), list):
raise RuntimeError("Input must be a complete native result.json")
blocks = numeric_blocks(result)
all_values = array('d')
descriptors = []
for name, values in blocks:
if not isinstance(values, list):
raise RuntimeError(f"Expected numeric array: {name}")
for value in values:
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value):
raise RuntimeError(f"Nonfinite or nonnumeric input: {name}")
if isinstance(value, int) and int(float(value)) != value:
raise RuntimeError(f"Integer does not fit exactly in binary64: {name}")
descriptors.append({"name": name, "offset": len(all_values), "count": len(values)})
all_values.extend(values)
if not all_values:
raise RuntimeError("No numeric output values")
metadata = {key: value for key, value in result.items() if key not in {"series", "final", "finalState"}}
pending = json.dumps(metadata, ensure_ascii=True, separators=(",", ":"), allow_nan=False)[:-1]
pending += ("," if metadata else "") + '"series":{'
prefixes = []
first = True
for key in result["series"]:
pending += ("" if first else ",") + json.dumps(key, ensure_ascii=True) + ":["
prefixes.append(pending.encode()); pending = "]"; first = False
pending += '},"final":{'
first = True
for key in result["final"]:
pending += ("" if first else ",") + json.dumps(key, ensure_ascii=True) + ":"
prefixes.append(pending.encode()); pending = ""; first = False
pending += '},"finalState":['
prefixes.append(pending.encode())
tail = b"]}\n"
output.mkdir(parents=True, exist_ok=True)
raw = output / "values.f64le"
raw.write_bytes(all_values.tobytes())
header = ["/* Generated test data: all numeric payload cells, no projection. */", "typedef struct { const char *prefix; size_t prefix_length, offset, count; } Group;", f"#define GROUP_COUNT {len(descriptors)}u", f"#define VALUE_COUNT {len(all_values)}u", "static const Group groups[]={"]
for desc, prefix in zip(descriptors, prefixes, strict=True):
header.append(f" {{{c_string(prefix)},{len(prefix)}u,{desc['offset']}u,{desc['count']}u}},")
header += ["};", f"static const char tail[]={c_string(tail)};", f"#define TAIL_LENGTH {len(tail)}u"]
(output / "replay-layout.h").write_text("\n".join(header) + "\n")
(output / "replay.c").write_text(C_SOURCE)
compiler = os.environ.get("SIMULATION_NATIVE_CC") or shutil.which("gcc")
if not compiler:
raise RuntimeError("GCC is required")
ryu = args.ryu_root.resolve() if args.ryu_root else None
if ryu and not (ryu / "ryu/d2s.c").is_file() and (ryu / "d2s.c").is_file():
ryu = ryu.parent
if ryu and not all((ryu / name).is_file() for name in ("ryu/d2s.c", "ryu/ryu.h")):
raise RuntimeError("--ryu-root must contain ryu/d2s.c and ryu/ryu.h")
command = [compiler, "-std=c11", "-O3", "-Wall", "-Wextra", "-Werror", "-ffp-contract=off", "-fno-fast-math", "-D_POSIX_C_SOURCE=200809L", f"-DHAVE_RYU={int(ryu is not None)}", "-I", str(output), str(output / "replay.c")]
ryu_hashes = {}
if ryu:
command += ["-I", str(ryu), str(ryu / "ryu/d2s.c")]
ryu_hashes = {str(p.relative_to(ryu)): digest(p) for p in sorted((ryu / "ryu").glob("*")) if p.is_file() and p.suffix in {".c", ".h"}}
command += ["-lm", "-o", str(output / "replay")]
prepared = {"sourceResult": str(args.result_json.resolve()), "sourceSha256": digest(args.result_json), "sourceBytes": args.result_json.stat().st_size, "rawValuesSha256": digest(raw), "rawValueBytes": raw.stat().st_size, "valueCount": len(all_values), "seriesColumns": len(result["series"]), "seriesValues": sum(len(v) for v in result["series"].values()), "finalValues": len(result["final"]), "finalStateValues": len(result["finalState"]), "blocks": descriptors, "variants": VARIANTS if ryu else VARIANTS[:3], "ryuRoot": str(ryu) if ryu else None, "ryuSourceHashes": ryu_hashes, "buildCommand": command, "compiler": subprocess.run([compiler, "--version"], capture_output=True, text=True, check=True).stdout.splitlines()[0], "warmups": args.warmups, "repeats": args.repeats, "devNullRequested": args.dev_null, "precisionContract": "All finite payload values must decode to identical little-endian binary64 bytes, including signed zero. Shortest output may have different length/exponent spelling.", "timingContract": "C wall/process-CPU from before fopen through fclose, including buffer setup, all numeric payload formatting and writing. Excludes extraction, preload, allocation, static JSON framing preparation and verification. Ordinary files/page cache; no fsync durability. Contiguous replay does not reproduce production matrix strides or output projection. Block variants both use a 64KiB application buffer with unbuffered FILE sink."}
write_json(output / "prepared.json", prepared)
return prepared
def verify_file(path: Path, expected: dict, raw: bytes, prepared: dict) -> dict:
actual = read_json_numbers(path)
# This catches missing columns, changed metadata, duplicates in array values,
# order differences, and scalar value drift before the exact signed-zero pass.
if actual != expected:
raise RuntimeError(f"Full result value/structure parity failed: {path}")
actual_blocks = numeric_blocks(actual)
if [name for name, _ in actual_blocks] != [d["name"] for d in prepared["blocks"]]:
raise RuntimeError(f"Numeric block order differs: {path}")
negative_zeroes = 0
for (_, values), desc in zip(actual_blocks, prepared["blocks"], strict=True):
binary = array('d', values).tobytes()
start, end = desc["offset"] * 8, (desc["offset"] + desc["count"]) * 8
if binary != raw[start:end]:
raise RuntimeError(f"Binary64 parity failed at {desc['name']}: {path}")
negative_zeroes += sum(value == 0 and math.copysign(1.0, value) < 0 for value in values)
return {"fullResultParity": True, "allPayloadBinary64Parity": True, "checkedValues": prepared["valueCount"], "negativeZeroCount": negative_zeroes, "sha256": digest(path)}
def execute(args: argparse.Namespace, prepared: dict) -> None:
output = args.output_dir.resolve()
built = subprocess.run(prepared["buildCommand"], capture_output=True, text=True, timeout=120)
(output / "build.log").write_text(built.stdout + built.stderr)
if built.returncode:
raise RuntimeError(f"Compilation failed: {output / 'build.log'}")
expected = read_json_numbers(args.result_json)
raw = (output / "values.f64le").read_bytes()
rows = []
for sink in (["file", "dev-null"] if args.dev_null else ["file"]):
for index in range(-args.warmups, args.repeats):
label = f"warmup-{index + args.warmups + 1}" if index < 0 else f"run-{index + 1}"
for variant in prepared["variants"]:
run = output / sink / variant / label
run.mkdir(parents=True, exist_ok=True)
target = run / "result.json" if sink == "file" else Path("/dev/null")
started = time.perf_counter()
process = subprocess.run([str(output / "replay"), variant, str(output / "values.f64le"), str(target)], capture_output=True, text=True, timeout=120)
process_wall = time.perf_counter() - started
(run / "stdout.log").write_text(process.stdout)
(run / "stderr.log").write_text(process.stderr)
if process.returncode:
raise RuntimeError(f"Replay failed ({process.returncode}): {run}")
record = json.loads(process.stdout)
if record["success"] is not True:
raise RuntimeError(f"Encoding reported failure: {run}")
record.update(sink=sink, run=label, warmup=index < 0, processWallSeconds=process_wall)
if sink == "file":
if target.stat().st_size != record["encodedBytes"]:
raise RuntimeError(f"Written byte count differs: {target}")
record["verification"] = verify_file(target, expected, raw, prepared)
else:
record["verification"] = {"actualSinkFileReadback": False, "sameEncoderPassedRealFileReadback": True}
write_json(run / "run.json", record)
rows.append(record)
print(f"{sink}/{variant}/{label}: wall={record['wallSeconds']:.6f}s cpu={record['cpuSeconds']:.6f}s bytes={record['encodedBytes']}", flush=True)
medians = {}
for sink in {row["sink"] for row in rows}:
medians[sink] = {}
for variant in prepared["variants"]:
selected = [row for row in rows if row["sink"] == sink and row["variant"] == variant and not row["warmup"]]
medians[sink][variant] = {key: statistics.median(row[key] for row in selected) for key in ("wallSeconds", "cpuSeconds", "encodedBytes")}
baseline = medians[sink]["fprintf-default"]
for data in medians[sink].values():
data["wallReductionFractionVsDefault"] = 1 - data["wallSeconds"] / baseline["wallSeconds"]
data["byteReductionFractionVsDefault"] = 1 - data["encodedBytes"] / baseline["encodedBytes"]
write_json(output / "summary.json", {"prepared": prepared, "runs": rows, "medians": medians, "allRealFileBinary64Parity": True, "limitation": "This replay isolates formatting and ordinary file writes on preloaded contiguous doubles. It is not an end-to-end native/application speedup and excludes projection, strided output reads and durable storage flush. /dev/null metrics, if present, are separate sink-only observations."})
print(f"Summary: {output / 'summary.json'}", flush=True)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--result-json", required=True, type=Path)
parser.add_argument("--output-dir", type=Path, default=ROOT / "test/c-result-encoding-20260911")
parser.add_argument("--ryu-root", type=Path)
parser.add_argument("--run", action="store_true")
parser.add_argument("--dev-null", action="store_true")
parser.add_argument("--warmups", type=int, default=1)
parser.add_argument("--repeats", type=int, default=3)
args = parser.parse_args()
if args.warmups < 0 or args.repeats < 1:
parser.error("warmups must be nonnegative and repeats positive")
prepared = prepare(args)
print(f"Prepared {prepared['valueCount']} binary64 values and {len(prepared['variants'])} variants: {args.output_dir.resolve()}", flush=True)
if args.run:
execute(args, prepared)
if __name__ == "__main__":
main()
tests/manual/browser_stage_profile.mjs
+967
View File
@@ -0,0 +1,967 @@
// Real production-page profiling. No route mocks, response cloning, or duplicate body parsing.
// All stage timestamps use the active document's performance.now(). A reload starts a new axis.
import { chromium } from '../../frontend/node_modules/playwright/index.mjs';
import fs from 'node:fs/promises';
import path from 'node:path';
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
const usage = `node tests/manual/browser_stage_profile.mjs --output DIR [--input tests/data/test-mql-8-corrected.json] [--url http://127.0.0.1:8011] [--runs 3 (0 for one smoke run)] [--mode both|profiled|control] [--deep] [--cpu-interval-us 1000] [--source-map-dir DIR] [--check]
Offline only: node tests/manual/browser_stage_profile.mjs --summarize-cpu-only EXISTING_DIRECTORY [--source-map-dir DIR] [--check]
Each mode runs one warmup followed by RUNS measured runs, sequentially. --check validates inputs without launching a browser.
Optional --deep (alias --cpu-profile) records renderer-main-thread .cpuprofile diagnostics separately from ordinary endpoint timing.
--source-map-dir accepts an offline hidden-source-map build only when its generated JS bytes exactly match served assets.
Run with the repository Node 24 and Chromium runtime library environment. No application code is modified.`;
const args = process.argv.slice(2);
if (args.includes('--help')) { console.log(usage); process.exit(0); }
const options = { input: 'tests/data/test-mql-8-corrected.json', url: 'http://127.0.0.1:8011', runs: '3', mode: 'both', cpuIntervalUs: '1000' };
for (let i = 0; i < args.length; i++) {
if (['--deep', '--cpu-profile'].includes(args[i])) { options.deep = true; continue; }
if (args[i] === '--check') { options.check = true; continue; }
const cliKey = args[i].replace(/^--/, '');
const key = ({ 'cpu-interval-us': 'cpuIntervalUs', 'source-map-dir': 'sourceMapDir', 'summarize-cpu-only': 'summarizeCpuOnly' })[cliKey] ?? cliKey;
if (!['input', 'output', 'url', 'runs', 'mode', 'cpuIntervalUs', 'sourceMapDir', 'summarizeCpuOnly'].includes(key) || !args[i + 1]) throw new Error(usage);
options[key] = args[++i];
}
if (!['both', 'profiled', 'control'].includes(options.mode) || !/^(0|[1-9]\d*)$/.test(options.runs)) throw new Error(usage);
if (!/^\d+$/.test(options.cpuIntervalUs) || Number(options.cpuIntervalUs) < 100 || Number(options.cpuIntervalUs) > 100000) throw new Error('--cpu-interval-us must be between 100 and 100000.');
if (options.sourceMapDir && !options.deep && !options.summarizeCpuOnly) throw new Error('--source-map-dir requires --deep or --summarize-cpu-only.');
const sha = data => createHash('sha256').update(data).digest('hex');
let inputText; let project; let curveNodeId;
if (!options.summarizeCpuOnly) {
inputText = await fs.readFile(options.input, 'utf8');
project = JSON.parse(inputText);
assert.equal(project.projectSchemaVersion, 1);
assert.ok(project.nodes.length && project.edges.length);
curveNodeId = project.nodes.find(n => n.id === 'amesim_pnl0002_10')?.id
?? project.nodes.find(n => n.data.modelType === 'amesim_pnl0002')?.id;
assert.ok(curveNodeId, 'A PNL0002 temperature component is required.');
if (!options.check) {
if (!options.output) throw new Error(usage);
await fs.mkdir(options.output, { recursive: true });
await fs.writeFile(path.join(options.output, 'input.json'), inputText);
}
}
// Serialized by Playwright. Only the profiled mode wraps production APIs; control observes
// clicks and DOM readiness, with storage-pointer polling at 16 ms resolution.
function installStageObserver({ profiled }) {
const storageKey = 'system-simulation-flow:latest-result';
const nativeParse = JSON.parse;
const data = { profiled, timeOrigin: performance.timeOrigin, marks: {}, requests: [], reads: [], parses: [],
decodes: [], transactions: [], workers: [], downloads: [], longTasks: [], streamActive: false, activeRun: false };
window.__stageProfile = data;
const mark = (name, value = performance.now()) => {
if (data.marks[name] === undefined) {
data.marks[name] = value;
performance.mark(`stage:${name}`, { startTime: value });
}
return value;
};
data.mark = mark;
data.reset = () => {
Object.assign(data, { marks: {}, requests: [], reads: [], parses: [], decodes: [], transactions: [], workers: [], runFailure: undefined,
downloads: [], longTasks: [], streamActive: false, activeRun: true });
performance.clearMarks();
};
data.armClick = (name, selector) => {
const listener = event => {
if (event.target instanceof Element && event.target.closest(selector)) {
mark(name);
document.removeEventListener('click', listener, true);
}
};
document.addEventListener('click', listener, true);
};
data.watchRunReady = () => {
const previous = sessionStorage.getItem(storageKey);
let observedBusy = false;
const observer = new MutationObserver(check);
observer.observe(document.documentElement, { subtree: true, childList: true, attributes: true, characterData: true });
function check() {
if (data.marks.runClick === undefined) return;
const failure = document.querySelector('.simulation-console-dock-progress.error, .simulation-console-progress.error');
if (failure) { data.runFailure = failure.textContent; mark('runFailure'); observer.disconnect(); return; }
const button = document.querySelector('button[aria-label="运行仿真"]');
observedBusy ||= Boolean(button?.disabled || document.querySelector('.simulation-console-dock-progress.running, .simulation-console-progress.running'));
const success = document.querySelector('.simulation-console-dock-progress.success, .simulation-console-progress.success');
if (observedBusy && button && !button.disabled && success?.textContent.includes('仿真完成')) {
mark('resultReadyDom');
data.streamActive = false;
requestAnimationFrame(() => requestAnimationFrame(() => mark('resultReadyPaintOpportunity')));
observer.disconnect();
}
}
const poll = () => {
const raw = sessionStorage.getItem(storageKey);
if (raw && raw !== previous && nativeParse(raw).storage === 'indexeddb') {
mark('indexedDbPointerObserved');
} else setTimeout(poll, 16);
};
poll();
};
// DOM quiet is an operational threshold, not measured GPU work. No SVG serialization.
data.watchDom = (name, selector, quietMs = 120) => {
let target;
let quietTimer;
let quietObserver;
const foundObserver = new MutationObserver(find);
function find() {
const next = document.querySelector(selector);
if (!next || !next.getClientRects().length) return;
target = next;
mark(`${name}Dom`);
foundObserver.disconnect();
requestAnimationFrame(() => requestAnimationFrame(() => mark(`${name}PaintOpportunity`)));
const settle = () => {
clearTimeout(quietTimer);
data.marks[`${name}LastMutation`] = performance.now();
quietTimer = setTimeout(() => requestAnimationFrame(() => requestAnimationFrame(() => {
// A later mutation cancels the pending quiet interval as well as its timer.
if (performance.now() - data.marks[`${name}LastMutation`] < quietMs) return;
mark(`${name}Stable`);
quietObserver.disconnect();
})), quietMs);
};
quietObserver = new MutationObserver(settle);
quietObserver.observe(target, { subtree: true, childList: true, attributes: true, characterData: true });
settle();
}
foundObserver.observe(document, { subtree: true, childList: true, attributes: true });
find();
};
data.watchImport = expectedName => {
const previous = [...document.querySelectorAll('[data-entry-id]')].at(-1)?.getAttribute('data-entry-id');
const fileInput = document.querySelector('input[type="file"][accept*=".json"]');
fileInput.addEventListener('change', () => mark('importChange'), { once: true, capture: true });
const observer = new MutationObserver(() => {
if (data.marks.importChange === undefined) return;
const entry = [...document.querySelectorAll('[data-entry-id]')].at(-1);
if (entry?.getAttribute('data-entry-id') !== previous && entry?.textContent.includes('已导入工程') &&
[...document.querySelectorAll('input')].some(input => input.value === expectedName)) {
mark('importReadyDom');
requestAnimationFrame(() => requestAnimationFrame(() => mark('importReadyPaintOpportunity')));
observer.disconnect();
}
});
observer.observe(document, { subtree: true, childList: true, characterData: true });
};
data.snapshot = () => ({ ...data, mark: undefined, reset: undefined, armClick: undefined, watchRunReady: undefined,
watchDom: undefined, watchImport: undefined, snapshot: undefined, resources: performance.getEntriesByType('resource')
.filter(e => /simulate-stream|simulation-results\/csv/.test(e.name))
.map(e => ({ name: e.name, startTime: e.startTime, requestStart: e.requestStart, responseStart: e.responseStart,
responseEnd: e.responseEnd, duration: e.duration, transferSize: e.transferSize,
encodedBodySize: e.encodedBodySize, decodedBodySize: e.decodedBodySize })) });
if (location.hash === '#/results' && sessionStorage.getItem(storageKey)) {
data.watchDom('restoredResults', '.results-shell .results-system-panel');
}
if (!profiled) return;
const bodies = new WeakMap();
const originalFetch = window.fetch;
window.fetch = function (...args) {
const url = typeof args[0] === 'string' ? args[0] : args[0] instanceof Request ? args[0].url : String(args[0]);
const kind = url.includes('/api/system-xml/simulate-stream') ? 'simulation'
: url.includes('/api/simulation-results/csv') ? 'csv' : null;
if (!kind) return Reflect.apply(originalFetch, this, args);
const row = { kind, fetchStart: performance.now(), simulationId: new Headers(args[1]?.headers ?? (args[0] instanceof Request ? args[0].headers : undefined)).get('X-Simulation-Id'), requestStringLength: typeof args[1]?.body === 'string' ? args[1].body.length : null };
data.requests.push(row);
if (kind === 'simulation') { mark('fetchStart', row.fetchStart); data.streamActive = true; }
return Reflect.apply(originalFetch, this, args).then(response => {
row.headers = performance.now(); row.status = response.status;
if (kind === 'simulation') mark('headers', row.headers);
if (response.body) bodies.set(response.body, row);
return response;
});
};
const originalGetReader = ReadableStream.prototype.getReader;
ReadableStream.prototype.getReader = function (...args) {
const reader = Reflect.apply(originalGetReader, this, args);
const request = bodies.get(this);
if (!request) return reader;
const originalRead = reader.read;
reader.read = function (...readArgs) {
const row = { kind: request.kind, start: performance.now() };
return Reflect.apply(originalRead, this, readArgs).then(value => {
row.end = performance.now(); row.bytes = value.value?.byteLength ?? 0; row.done = value.done;
data.reads.push(row);
if (request.kind === 'simulation') {
if (row.bytes) { mark('firstChunk', row.end); data.marks.lastChunk = row.end; }
if (row.done) mark('streamEof', row.end);
}
return value;
});
};
return reader;
};
JSON.parse = function (...args) {
if (!data.streamActive) return Reflect.apply(nativeParse, this, args);
const start = performance.now();
const parsed = Reflect.apply(nativeParse, this, args);
const end = performance.now();
if (parsed && ['progress', 'result', 'error'].includes(parsed.event)) {
data.parses.push({ event: parsed.event, phase: parsed.phase, start, end, characters: typeof args[0] === 'string' ? args[0].length : null });
if (parsed.event === 'result') {
mark('resultParseStart', start); mark('resultParseEnd', end);
// This microtask is only a checkpoint after the current consumer continuation,
// not a claim that all EOF/finally/publish/React work has completed.
queueMicrotask(() => mark('resultConsumerMicrotaskCheckpoint'));
}
}
return parsed;
};
const originalDecode = TextDecoder.prototype.decode;
TextDecoder.prototype.decode = function (...args) {
if (!data.streamActive) return Reflect.apply(originalDecode, this, args);
const start = performance.now();
const value = Reflect.apply(originalDecode, this, args);
data.decodes.push({ start, end: performance.now(), bytes: args[0]?.byteLength ?? 0 });
return value;
};
const originalTransaction = IDBDatabase.prototype.transaction;
IDBDatabase.prototype.transaction = function (...args) {
const transaction = Reflect.apply(originalTransaction, this, args);
if (this.name === 'system-simulation-results') {
const row = { start: performance.now(), mode: transaction.mode, stores: Array.from(transaction.objectStoreNames) };
data.transactions.push(row);
transaction.addEventListener('complete', () => { row.end = performance.now(); row.outcome = 'complete'; });
transaction.addEventListener('abort', () => { row.end = performance.now(); row.outcome = 'abort'; });
}
return transaction;
};
const originalSetItem = Storage.prototype.setItem;
Storage.prototype.setItem = function (...args) {
const value = Reflect.apply(originalSetItem, this, args);
if (this === sessionStorage && args[0] === storageKey && data.activeRun) mark('indexedDbCommittedPointer');
return value;
};
const NativeWorker = window.Worker;
window.Worker = new Proxy(NativeWorker, {
construct(target, args, newTarget) {
const url = String(args[0]);
const isCsv = /resultCsv/i.test(url);
const start = performance.now();
const worker = Reflect.construct(target, args, newTarget);
if (!isCsv) return worker;
const row = { url, constructStart: start, constructEnd: performance.now(), posts: [] };
data.workers.push(row);
worker.addEventListener('message', event => {
const message = event.data;
if (message?.type === 'complete') { row.completeReceived = performance.now(); row.blobBytes = message.blob?.size ?? null; }
if (message?.type === 'error') { row.errorReceived = performance.now(); row.error = message.message; }
});
const originalPost = worker.postMessage;
worker.postMessage = function (...postArgs) {
const message = postArgs[0];
// Read scalar metadata before transfer detaches the original buffer. Never
// inspect/copy values, Blob contents, or inject code into the worker.
const post = { type: message?.type, start: performance.now(), offset: message?.offset,
bytes: message?.values?.byteLength ?? 0 };
const result = Reflect.apply(originalPost, this, postArgs);
post.end = performance.now();
row.posts.push(post);
return result;
};
return worker;
},
});
const originalAnchorClick = HTMLAnchorElement.prototype.click;
HTMLAnchorElement.prototype.click = function (...args) {
if (this.download) data.downloads.push({ name: this.download, anchorClick: performance.now(), blobUrl: this.href.startsWith('blob:') });
return Reflect.apply(originalAnchorClick, this, args);
};
if (PerformanceObserver.supportedEntryTypes.includes('longtask')) {
new PerformanceObserver(list => {
for (const e of list.getEntries()) if (data.activeRun) data.longTasks.push({ start: e.startTime, duration: e.duration, name: e.name });
}).observe({ type: 'longtask', buffered: false });
}
}
// Optional sampling diagnostics. These deliberately remain independent of production
// code and existing stage marks. Analysis/source-map loading runs after all timings.
async function calibrateCpuClock(session) {
const samples = [];
for (let i = 0; i < 3; i++) {
const read = async () => (await session.send('Runtime.evaluate', {
expression: '({now: performance.now(), timeOrigin: performance.timeOrigin})', returnByValue: true,
})).result.value;
const before = await read();
const metrics = await session.send('Performance.getMetrics');
const after = await read();
const timestamp = metrics.metrics.find(metric => metric.name === 'Timestamp')?.value;
if (timestamp === undefined || before.timeOrigin !== after.timeOrigin) continue;
samples.push({ timeOrigin: before.timeOrigin, pageBeforeMs: before.now, pageAfterMs: after.now,
cdpTimestampMs: timestamp * 1000, offsetMs: timestamp * 1000 - (before.now + after.now) / 2,
uncertaintyMs: (after.now - before.now) / 2 });
}
if (!samples.length) throw new Error('Unable to calibrate CDP sampling against the page clock.');
samples.sort((a, b) => a.uncertaintyMs - b.uncertaintyMs);
return { chosen: samples[0], probes: samples };
}
async function createCpuRecorder(context, page, intervalUs) {
const session = await context.newCDPSession(page);
await session.send('Performance.enable', { timeDomain: 'timeTicks' });
await session.send('Profiler.enable');
await session.send('Profiler.setSamplingInterval', { interval: intervalUs });
let startCalibration;
let running = false;
return {
async start() {
if (running) throw new Error('CPU profiler already running.');
startCalibration = await calibrateCpuClock(session);
await session.send('Profiler.start');
running = true;
},
async stop() {
const { profile } = await session.send('Profiler.stop');
running = false;
const endCalibration = await calibrateCpuClock(session);
return { profile, calibration: { start: startCalibration, end: endCalibration }, intervalUs };
},
async close() {
if (running) await session.send('Profiler.stop').catch(() => {});
await session.detach().catch(() => {});
},
};
}
const cpuFunctionCategories = {
buildSystemXml: 'xml_generation', validateModel: 'model_validation', checkModel: 'model_validation',
componentParameterValidationMessage: 'model_validation', projectExecutionContractIssues: 'model_validation',
resolveSimulationConfig: 'model_validation', modelValidationSignature: 'model_validation',
projectConnectionMetadata: 'model_contract_and_endpoints', buildCurrentProject: 'project_snapshot_copy',
buildProjectPayload: 'project_snapshot_copy', cloneValue: 'project_snapshot_copy',
publishSimulationResult: 'result_publication', normalizeSimulationProgressEvent: 'progress_normalization',
storeResultSnapshot: 'persistence_pack_and_save', writeSnapshot: 'indexeddb_request_submission',
restorePacked: 'persistence_unpack', loadStoredResultSnapshot: 'persistence_restore',
openDatabase: 'indexeddb_open', deleteCache: 'persistence_cleanup',
};
const cpuSourceRanges = new Map();
function originalFunctionAt(content, line) {
if (!content) return null;
let ranges = cpuSourceRanges.get(content);
if (!ranges) {
const lines = content.split('\n');
const declarations = [];
for (let i = 0; i < lines.length; i++) {
const match = lines[i].match(/^(\s*)(?:(?:export\s+)?(?:async\s+)?function\s+|const\s+)([A-Za-z_$][\w$]*)/);
if (match) declarations.push({ name: match[2], indent: match[1].length, line: i + 1 });
}
ranges = [];
for (const [index, declaration] of declarations.entries()) {
if (!(declaration.name in cpuFunctionCategories)) continue;
const next = declarations.slice(index + 1).find(peer => peer.indent <= declaration.indent);
ranges.push({ name: declaration.name, startLine: declaration.line, endLineExclusive: next?.line ?? lines.length + 1 });
}
cpuSourceRanges.set(content, ranges);
}
// sourcesContent comes from the byte-verified bundle. Declaration ranges are
// cached once per source, rather than rescanning App.tsx for every sampled frame.
return ranges.findLast(range => line >= range.startLine && line < range.endLineExclusive) ?? null;
}
async function loadVerifiedCpuMaps(sourceMapDirectory, servedAssets) {
const consumers = new Map();
const evidence = [];
if (!sourceMapDirectory) return { consumers, evidence };
const module = await import('../../frontend/node_modules/source-map-js/source-map.js');
const { SourceMapConsumer } = module.default ?? module;
const root = path.resolve(sourceMapDirectory);
for (const asset of servedAssets.filter(asset => new URL(asset.url).pathname.endsWith('.js'))) {
const pathname = decodeURIComponent(new URL(asset.url).pathname);
const candidates = [path.resolve(root, `.${pathname}`), path.join(root, path.basename(pathname))];
let generated = candidates[0];
for (const candidate of candidates) {
if (!candidate.startsWith(`${root}${path.sep}`)) continue;
if (await fs.stat(candidate).then(stat => stat.isFile()).catch(() => false)) { generated = candidate; break; }
}
if (!generated.startsWith(`${root}${path.sep}`)) continue;
const mapPath = `${generated}.map`;
try {
const generatedBytes = await fs.readFile(generated);
const generatedSha256 = sha(generatedBytes);
if (generatedSha256 !== asset.sha256) {
evidence.push({ url: asset.url, generated, status: 'rejected-generated-bytes-differ', generatedSha256, servedSha256: asset.sha256 });
continue;
}
const mapBytes = await fs.readFile(mapPath);
const consumer = new SourceMapConsumer(JSON.parse(mapBytes));
consumers.set(asset.url, consumer);
evidence.push({ url: asset.url, generated, mapPath, status: 'verified', generatedSha256, mapSha256: sha(mapBytes) });
} catch (error) {
evidence.push({ url: asset.url, generated, mapPath, status: 'unavailable', reason: String(error) });
}
}
return { consumers, evidence };
}
function cpuFrameInfo(frame, maps, cache) {
const key = `${frame.scriptId}:${frame.url}:${frame.lineNumber}:${frame.columnNumber}:${frame.functionName}`;
if (cache.has(key)) return cache.get(key);
let original = null;
let sourceFunction = null;
const consumer = maps.get(frame.url);
if (consumer && frame.lineNumber >= 0 && frame.columnNumber >= 0) {
const position = consumer.originalPositionFor({ line: frame.lineNumber + 1, column: frame.columnNumber });
if (position.source && position.line) {
original = position;
sourceFunction = originalFunctionAt(consumer.sourceContentFor(position.source, true), position.line);
}
}
const name = sourceFunction?.name ?? original?.name ?? frame.functionName;
const source = original?.source ?? frame.url ?? '';
let category = 'unclassified';
let basis = 'unclassified';
if (frame.functionName === '(idle)') { category = 'idle'; basis = 'v8-system-frame'; }
else if (frame.functionName === '(garbage collector)') { category = 'garbage_collection'; basis = 'v8-system-frame'; }
else if (['(program)', '(root)'].includes(frame.functionName)) { category = 'unclassified_program'; basis = 'v8-system-frame'; }
else if (name in cpuFunctionCategories) { category = cpuFunctionCategories[name]; basis = sourceFunction ? 'verified-map-declaration-range' : original ? 'verified-map-name' : 'runtime-function-name'; }
else if (/JSON\.parse/.test(name)) { category = 'json_parse'; basis = 'runtime-function-name'; }
else if (/TextDecoder.*decode/.test(name)) { category = 'utf8_decode'; basis = 'runtime-function-name'; }
else if (/XMLSerializer|serializeToString/.test(name)) { category = 'xml_native_serialization'; basis = 'runtime-function-name'; }
else if (/structuredClone/.test(name)) { category = 'structured_clone'; basis = 'runtime-function-name'; }
else if (/react-dom|react\/cjs|scheduler\/cjs/.test(source)) { category = 'react_runtime'; basis = original ? 'verified-map-source' : 'runtime-source-url'; }
else if (/@xyflow/.test(source)) { category = 'diagram_reactflow'; basis = original ? 'verified-map-source' : 'runtime-source-url'; }
else if (/ndjsonStream\./.test(source)) { category = 'ndjson_scan_join_dispatch'; basis = 'verified-map-source'; }
else if (/resultPersistence\./.test(source)) { category = 'persistence_other'; basis = 'verified-map-source'; }
else if (/resultCsvExport\./.test(source)) { category = 'csv_prepare_and_transfer'; basis = 'verified-map-source'; }
else if (/chartData\.|resultEventSeries\./.test(source)) { category = 'chart_data_preparation'; basis = 'verified-map-source'; }
else if (/SimulationResultsView\./.test(source)) { category = 'results_view_preparation'; basis = 'verified-map-source'; }
else if (/componentSymbols|edgeRouting|ContactAwareEdge/.test(source)) { category = 'diagram_geometry'; basis = 'verified-map-source'; }
else if (/App\.tsx/.test(source)) { category = 'application_other'; basis = 'verified-map-source'; }
else if (/__playwright|utilityScript|evaluate@/.test(source + name)) { category = 'measurement_or_automation'; basis = 'runtime-frame'; }
const result = { key, functionName: frame.functionName, hasScript: Boolean(frame.scriptId && frame.scriptId !== '0'), generated: { url: frame.url, line: frame.lineNumber + 1, column: frame.columnNumber },
original, sourceFunction, category, classificationBasis: basis };
cache.set(key, result);
return result;
}
function cpuWindows(trace, kind) {
const m = trace.marks;
const endCommit = m.indexedDbCommittedPointer ?? m.indexedDbPointerObserved;
const windows = kind === 'restore' ? [
['restore_navigation_to_dom', 0, m.restoredResultsDom],
['restore_navigation_to_paint_opportunity', 0, m.restoredResultsPaintOpportunity],
] : [
['import', m.importChange, m.importReadyPaintOpportunity],
['pre_submit', m.runClick, m.fetchStart],
['response_before_result_parse', m.headers, m.resultParseStart],
['result_json_parse', m.resultParseStart, m.resultParseEnd],
['result_publish_to_dom', m.resultParseEnd, m.resultReadyDom],
['result_ready_to_saved_pointer', m.resultReadyDom, endCommit],
['result_parse_through_saved_pointer', m.resultParseStart, endCommit],
['result_parsed_to_saved_pointer', m.resultParseEnd, endCommit],
['result_tab_preparation', m.resultTabClick, m.resultsPaintOpportunity],
['temperature_chart_preparation', m.temperatureClick, m.curvePaintOpportunity],
['csv_export_through_download_saved', m.csvClick, m.csvDownloadSaved],
['result_export_through_download_saved', m.resultFileClick, m.resultFileDownloadSaved],
['whole_run_click_through_download_saved', m.runClick, m.resultFileDownloadSaved],
];
return windows.filter(([, start, end]) => Number.isFinite(start) && Number.isFinite(end) && end >= start)
.map(([name, startMs, endMs]) => ({ name, startMs, endMs, wallMs: endMs - startMs }));
}
function cpuExecutionState(stack) {
const leaf = stack[0];
if (!leaf) return 'unknown_runtime';
if (['idle', 'garbage_collection', 'unclassified_program'].includes(leaf.category)) return leaf.category;
if (leaf.hasScript) return 'active_js';
// Native call frames (for example IDBObjectStore.put or Blob) may be sampled
// under a real JS caller. A bare runtime/native frame has no such evidence.
return stack.some(frame => frame.hasScript) ? 'active_native_call' : 'unknown_runtime';
}
function summarizeCpu(record, maps) {
const { profile, trace, intervalUs, calibration, kind } = record;
const anchors = [calibration.start.chosen, calibration.end.chosen].filter(anchor => anchor.timeOrigin === trace.timeOrigin);
if (!anchors.length) throw new Error('CPU profile calibration does not match its document timeOrigin.');
const anchor = anchors.reduce((best, candidate) => candidate.uncertaintyMs < best.uncertaintyMs ? candidate : best);
const offsetSpreadMs = Math.max(...anchors.map(a => a.offsetMs)) - Math.min(...anchors.map(a => a.offsetMs));
const frameCache = new Map();
const nodes = new Map(profile.nodes.map(node => [node.id, { ...node, info: cpuFrameInfo(node.callFrame, maps, frameCache) }]));
const parent = new Map();
for (const node of profile.nodes) for (const child of node.children ?? []) parent.set(child, node.id);
const stacks = new Map();
for (const node of profile.nodes) {
const stack = []; let id = node.id;
while (nodes.has(id) && stack.length < 256) { stack.push(nodes.get(id).info); id = parent.get(id); }
stacks.set(node.id, stack);
}
let clockMs = profile.startTime / 1000 - anchor.offsetMs;
const samples = (profile.samples ?? []).map((id, index) => {
const deltaMs = (profile.timeDeltas?.[index] ?? intervalUs) / 1000;
const startMs = clockMs; clockMs += deltaMs;
// Sampling gaps can include OS descheduling; never credit a long unsampled
// gap wholesale to the currently sampled JavaScript function.
return { id, startMs, endMs: clockMs, representedMs: Math.min(deltaMs, 2 * intervalUs / 1000) };
});
const windows = cpuWindows(trace, kind).map(window => {
const categories = new Map(); const functions = new Map();
const executionSampleMs = { active_js: 0, active_native_call: 0, idle: 0,
garbage_collection: 0, unclassified_program: 0, unknown_runtime: 0 };
let sampleCount = 0; let representedMs = 0; let samplingGapMs = 0;
for (const sample of samples) {
const overlap = Math.max(0, Math.min(window.endMs, sample.endMs) - Math.max(window.startMs, sample.startMs));
if (!overlap) continue;
const interval = sample.endMs - sample.startMs;
const weight = interval > 0 ? overlap * sample.representedMs / interval : 0;
sampleCount++; representedMs += weight; samplingGapMs += Math.max(0, overlap - weight);
const stack = stacks.get(sample.id) ?? [];
const state = cpuExecutionState(stack);
executionSampleMs[state] += weight;
const active = state === 'active_js' || state === 'active_native_call';
// Never walk up to V8 (root)/(program) to classify unknown JavaScript.
// GC/idle/program samples also never accrue to an application caller.
const activeStack = stack.filter(frame => frame.classificationBasis !== 'v8-system-frame');
const owner = activeStack.find(frame => !['unclassified', 'application_other'].includes(frame.category)) ?? activeStack[0];
const category = active ? owner?.category ?? 'unclassified' : state;
categories.set(category, (categories.get(category) ?? 0) + weight);
if (!active) continue;
const seen = new Set();
for (const [index, frame] of activeStack.entries()) {
if (seen.has(frame.key)) continue;
seen.add(frame.key);
const entry = functions.get(frame.key) ?? { ...frame, selfActiveEstimatedMs: 0, inclusiveActiveEstimatedMs: 0 };
if (index === 0) entry.selfActiveEstimatedMs += weight;
entry.inclusiveActiveEstimatedMs += weight;
functions.set(frame.key, entry);
}
}
return { ...window, sampleCount, representedSampleMs: representedMs, executionSampleMs,
activeJsAndNativeCallSampleMs: executionSampleMs.active_js + executionSampleMs.active_native_call,
samplingGapMs, unattributedWallMs: Math.max(0, window.wallMs - representedMs - samplingGapMs),
exclusiveOwnerCategories: Object.fromEntries([...categories].sort((a, b) => b[1] - a[1])),
functions: [...functions.values()].sort((a, b) => b.inclusiveActiveEstimatedMs - a.inclusiveActiveEstimatedMs) };
});
return { schemaVersion: 2, kind, intervalUs, timeOrigin: trace.timeOrigin, profileStartUs: profile.startTime, profileEndUs: profile.endTime,
profilePageStartMs: profile.startTime / 1000 - anchor.offsetMs, profilePageEndMs: profile.endTime / 1000 - anchor.offsetMs,
clockCalibration: { anchor, offsetSpreadMs, probes: calibration }, windows,
limitations: [
'Sampling estimates renderer-main-thread execution attribution; it is not a function stopwatch or an OS thread CPU clock.',
'representedSampleMs includes idle, GC and unknown runtime samples. It is NOT an active CPU total.',
'activeJsAndNativeCallSampleMs includes sampled JavaScript and native calls with a JavaScript ancestor; bare native/runtime frames remain unknown.',
'V8 (program)/(root) samples are unclassified_program, not evidence of JavaScript or native CPU activity. Do not attribute this interval to receiving, parsing, painting or another residual category.',
'Each sample represents at most two configured sampling intervals; larger gaps are unassigned scheduling/sampling gaps.',
'Function selfActiveEstimatedMs/inclusiveActiveEstimatedMs exclude idle, GC, program and unknown-runtime samples even if they have application ancestors.',
'Active inclusive estimates include active children and cannot be added together; stage windows also overlap and cannot be added.',
'Synchronous JSON.parse/decoder durations in trace.json remain direct wrapper measurements; do not add sampling estimates to them.',
'Renderer main-thread profiling excludes CSV Worker CPU, IndexedDB background/disk work, compositor/GPU work and backend CPU.',
'React/source categories describe JavaScript preparation, not actual paint completion.',
'Without byte-verified source maps, minified function ownership stays unknown; original declaration ranges are a source attribution aid, not exact instruction boundaries.',
'Clock alignment uses CDP timeTicks and bracketing page-clock reads; uncertainty and before/after offset spread are reported.',
'Deep profiling perturbs execution; compare endpoint timings using separate runs without --deep.',
] };
}
function cpuSummaryMarkdown(summary, filename) {
const lines = [`CPU sampling diagnostic: ${filename}`, '', `Schema ${summary.schemaVersion}; interval ${summary.intervalUs} us. Main-thread samples only.`,
`Clock uncertainty: ${summary.clockCalibration.anchor.uncertaintyMs.toFixed(3)} ms; offset spread: ${summary.clockCalibration.offsetSpreadMs.toFixed(3)} ms.`, '',
'Active = JavaScript plus native calls sampled under JavaScript. Program/unknown and idle are not counted as active CPU.', '',
'| Window (overlaps allowed) | Wall ms | Active sample ms | Idle ms | GC ms | Program/unknown ms | Sampling gap ms | Leading active owner categories |',
'|---|---:|---:|---:|---:|---:|---:|---|'];
for (const window of summary.windows) {
const states = window.executionSampleMs;
const top = Object.entries(window.exclusiveOwnerCategories).filter(([name]) => !['idle', 'garbage_collection', 'unclassified_program', 'unknown_runtime'].includes(name))
.slice(0, 5).map(([name, ms]) => `${name}: ${ms.toFixed(2)}`).join('; ');
lines.push(`| ${window.name} | ${window.wallMs.toFixed(2)} | ${window.activeJsAndNativeCallSampleMs.toFixed(2)} | ${states.idle.toFixed(2)} | ${states.garbage_collection.toFixed(2)} | ${(states.unclassified_program + states.unknown_runtime).toFixed(2)} | ${window.samplingGapMs.toFixed(2)} | ${top} |`);
}
for (const window of summary.windows) {
lines.push('', `Top active sampled functions: ${window.name}`, '');
lines.push(...window.functions.filter(frame => frame.selfActiveEstimatedMs > 0).sort((a, b) => b.selfActiveEstimatedMs - a.selfActiveEstimatedMs).slice(0, 8).map(frame => {
const location = frame.original ? `${frame.original.source}:${frame.original.line}:${frame.original.column}`
: `${frame.generated.url || '(native/injected)'}:${frame.generated.line}:${frame.generated.column}`;
return `- ${frame.sourceFunction?.name ?? frame.original?.name ?? frame.functionName ?? '(anonymous)'} — active self ${frame.selfActiveEstimatedMs.toFixed(2)} ms, active inclusive ${frame.inclusiveActiveEstimatedMs.toFixed(2)} ms; ${location}; ${frame.classificationBasis}`;
}));
}
lines.push('', 'Limitations:', '', ...summary.limitations.map(text => `- ${text}`), '');
return lines.join('\n');
}
// Small synthetic sampling contract: nested GC is separate, a bare program frame
// remains unknown, minified JS cannot inherit (root), and long gaps stay unassigned.
function checkCpuSummaryContract() {
const frame = (functionName, scriptId = '0') => ({ functionName, scriptId, url: scriptId === '0' ? '' : 'fixture.js', lineNumber: scriptId === '0' ? -1 : 0, columnNumber: 0 });
const profile = { startTime: 0, endTime: 14000, nodes: [
{ id: 1, callFrame: frame('(root)'), children: [2, 3, 4, 8] },
{ id: 2, callFrame: frame('(idle)') }, { id: 3, callFrame: frame('(program)') },
{ id: 4, callFrame: frame('buildSystemXml', '1'), children: [5, 6] },
{ id: 5, callFrame: frame('(garbage collector)') },
{ id: 6, callFrame: frame('anonymousMinified', '1'), children: [7] },
{ id: 7, callFrame: frame('put') }, { id: 8, callFrame: frame('unresolvedNative') },
], samples: [2, 3, 4, 5, 6, 7, 8], timeDeltas: [1000, 1000, 1000, 1000, 1000, 1000, 8000] };
const chosen = { timeOrigin: 0, offsetMs: 0, uncertaintyMs: 0 };
const summary = summarizeCpu({ profile, kind: 'interaction', trace: { timeOrigin: 0, marks: { runClick: 0, fetchStart: 14 } },
calibration: { start: { chosen }, end: { chosen } }, intervalUs: 1000 }, new Map());
const window = summary.windows[0];
assert.deepEqual(window.executionSampleMs, { active_js: 2, active_native_call: 1, idle: 1, garbage_collection: 1, unclassified_program: 1, unknown_runtime: 2 });
assert.equal(window.activeJsAndNativeCallSampleMs, 3);
assert.equal(window.samplingGapMs, 6);
assert.equal(window.exclusiveOwnerCategories.xml_generation, 3);
assert.equal(window.functions.find(item => item.functionName === 'buildSystemXml').inclusiveActiveEstimatedMs, 3);
assert.equal(window.functions.some(item => item.classificationBasis === 'v8-system-frame'), false);
// A JS leaf directly under root must be active with unknown ownership.
const isolated = structuredClone(profile);
isolated.nodes[0].children.push(9);
isolated.nodes.push({ id: 9, callFrame: frame('minifiedUnknown', '1') });
isolated.samples = [9]; isolated.timeDeltas = [1000]; isolated.endTime = 1000;
const unknown = summarizeCpu({ profile: isolated, kind: 'interaction', trace: { timeOrigin: 0, marks: { runClick: 0, fetchStart: 1 } },
calibration: { start: { chosen }, end: { chosen } }, intervalUs: 1000 }, new Map()).windows[0];
assert.equal(unknown.exclusiveOwnerCategories.unclassified, 1);
assert.equal(unknown.executionSampleMs.unclassified_program, 0);
}
function cpuMeasuredMedians(records) {
const median = values => { const sorted = values.sort((a, b) => a - b); const mid = Math.floor(sorted.length / 2);
return sorted.length % 2 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2; };
const groups = new Map();
for (const record of records.filter(record => /(?:^|[/\\])(?:control|profiled)-run-\d+(?:[/\\]|$)/.test(record.profile))) {
const mode = record.profile.match(/(?:control|profiled)-run-\d+/)[0].split('-')[0];
for (const window of record.summary.windows) {
const key = `${mode}:${record.kind}:${window.name}`;
if (!groups.has(key)) groups.set(key, []);
groups.get(key).push(window);
}
}
return Object.fromEntries([...groups].map(([key, windows]) => {
const categories = new Set(windows.flatMap(window => Object.keys(window.exclusiveOwnerCategories)));
return [key, { samples: windows.length,
...Object.fromEntries(['wallMs', 'activeJsAndNativeCallSampleMs', 'representedSampleMs', 'samplingGapMs', 'unattributedWallMs']
.map(field => [field, median(windows.map(window => window[field]))])),
executionSampleMs: Object.fromEntries(Object.keys(windows[0].executionSampleMs).map(state => [state, median(windows.map(window => window.executionSampleMs[state]))])),
exclusiveOwnerCategories: Object.fromEntries([...categories].map(category => [category, median(windows.map(window => window.exclusiveOwnerCategories[category] ?? 0))])
.sort((a, b) => b[1] - a[1])),
}];
}));
}
async function summarizeExistingCpu(directory, sourceMapDirectory) {
const summaryFile = path.join(directory, 'summary.json');
const timingBytes = await fs.readFile(summaryFile);
const timing = JSON.parse(timingBytes);
assert.ok(timing.cpuDiagnostics?.profiles?.length, 'No captured CPU profiles in summary.json.');
const sourceDirectory = sourceMapDirectory ?? timing.cpuDiagnostics.sourceMaps.find(item => item.status === 'verified')?.generated;
const { consumers, evidence: sourceMaps } = await loadVerifiedCpuMaps(sourceMapDirectory ?? (sourceDirectory && path.dirname(sourceDirectory)), timing.servedAssets);
const records = [];
const index = { schemaVersion: 2, browserLaunched: false, sourceTimingSummary: summaryFile, sourceTimingSha256: sha(timingBytes),
scriptSha256: sha(await fs.readFile(new URL(import.meta.url))), sourceMaps, profiles: [] };
for (const entry of timing.cpuDiagnostics.profiles) {
// Prefer paths relative to the supplied directory so captured runs can be moved.
const profileFile = path.join(directory, path.basename(path.dirname(entry.profile)), path.basename(entry.profile));
const oldSummaryFile = path.join(path.dirname(profileFile), path.basename(entry.summary));
const traceFile = path.join(path.dirname(profileFile), entry.kind === 'restore' ? 'restore-trace.json' : 'trace.json');
const [profileBytes, previousBytes, traceBytes] = await Promise.all([fs.readFile(profileFile), fs.readFile(oldSummaryFile), fs.readFile(traceFile)]);
const previous = JSON.parse(previousBytes);
const record = { profile: JSON.parse(profileBytes), trace: JSON.parse(traceBytes), kind: entry.kind,
intervalUs: previous.intervalUs, calibration: previous.clockCalibration.probes };
const summary = summarizeCpu(record, consumers);
const output = profileFile.replace(/\.cpuprofile$/, '-cpu-summary-v2.json');
await fs.writeFile(output, JSON.stringify(summary, null, 2));
await fs.writeFile(output.replace(/\.json$/, '.md'), cpuSummaryMarkdown(summary, profileFile));
index.profiles.push({ profile: profileFile, profileSha256: sha(profileBytes), trace: traceFile, traceSha256: sha(traceBytes),
sourceSummary: oldSummaryFile, sourceSummarySha256: sha(previousBytes), summary: output, kind: entry.kind });
records.push({ profile: profileFile, kind: entry.kind, summary });
// Offline analysis must never change recorded timing, traces or raw profiles.
assert.equal(sha(await fs.readFile(profileFile)), sha(profileBytes));
assert.equal(sha(await fs.readFile(traceFile)), sha(traceBytes));
}
index.measuredMedians = cpuMeasuredMedians(records);
index.medianDefinition = 'Per-field medians over measured runs only; warmups excluded. Median columns and overlapping windows are not additive. Active includes JS plus native calls under JS, excluding idle/GC/program/unknown runtime.';
assert.equal(sha(await fs.readFile(summaryFile)), sha(timingBytes));
const indexFile = path.join(directory, 'cpu-resummary-v2.json');
await fs.writeFile(indexFile, JSON.stringify(index, null, 2));
console.log(JSON.stringify({ indexFile, profiles: records.length, originalTimingAndProfilesUnchanged: true, browserLaunched: false }));
}
if (options.check) {
checkCpuSummaryContract();
if (options.summarizeCpuOnly) {
const manifest = JSON.parse(await fs.readFile(path.join(options.summarizeCpuOnly, 'summary.json')));
assert.ok(manifest.cpuDiagnostics?.profiles?.length);
console.log(JSON.stringify({ directory: options.summarizeCpuOnly, profiles: manifest.cpuDiagnostics.profiles.length,
cpuClassificationContractPassed: true, browserLaunched: false }));
} else {
console.log(JSON.stringify({ input: options.input, inputSha256: sha(inputText), nodes: project.nodes.length,
edges: project.edges.length, curveNodeId, mode: options.mode, deep: Boolean(options.deep), cpuIntervalUs: Number(options.cpuIntervalUs),
sourceMapDir: options.sourceMapDir ?? null, warmupsPerMode: 1, measuredRunsPerMode: Number(options.runs),
cpuClassificationContractPassed: true, browserLaunched: false }));
}
process.exit(0);
}
if (options.summarizeCpuOnly) {
checkCpuSummaryContract();
await summarizeExistingCpu(options.summarizeCpuOnly, options.sourceMapDir);
process.exit(0);
}
const definitions = {
clock: 'All timestamps are performance.now() in the stated document timeOrigin; reload uses a separate time axis.',
clickToFetch: 'Includes model checking, snapshot construction, XML generation and submission setup; not isolated XML CPU time.',
headersAndReads: 'fetch resolution and consumer read delivery. Outstanding-read intervals include backend production, transport and browser scheduling; not pure network time.',
unobservedCpu: 'NDJSON fragment scanning/join/trim and React handler CPU are not isolated. Their residual intervals can also include scheduling and cannot be attributed wholesale to parsing, transport or drawing.',
jsonParse: 'Only the original synchronous JSON.parse call, called once per application parse. No duplicate body read, decode, scan or parse.',
resultReady: 'First DOM observation of successful completion plus an enabled run button after busy state. React state/handler boundaries are not directly instrumented.',
indexedDb: 'Profiled: session pointer publication immediately after all save transactions commit. Control: pointer polling, up to 16 ms plus scheduling delay. Transaction windows also include asynchronous waiting and may include old-cache cleanup.',
render: 'First visible DOM, then two requestAnimationFrame callbacks (paint opportunity, not GPU completion); stable means scoped DOM quiet for 120 ms followed by two frames.',
export: 'User click to blob anchor invocation (profiled) and Playwright download completion observed back on the page clock. Completion includes automation notification and filesystem saveAs overhead.',
worker: 'Profiled only: native Worker construction, original postMessage calls with unchanged transfer lists, and complete-message receipt on the document clock. No worker injection, payload copy or Blob read. Finish-post to receipt includes worker scheduling/encoding/Blob creation/message delivery, not isolated worker CPU. Main-thread preparation between posts overlaps worker activity and includes deliberate yields.',
control: 'Same user workflow and minimal click/DOM observations, without fetch/reader/JSON/decoder/IDB/Storage/Worker/anchor wrappers or long-task observer.',
overlap: 'Intervals overlap. Do not add read waits, parse, main-thread tasks, rendering, persistence, backend native time or residual differences as exclusive costs.',
import: 'Every run reimports the fixed input before runClick. Timing is native file-input change through import-success DOM and two frames. Public project export then verifies node/edge counts, all parameters and endpoints outside simulation timing.',
warmup: 'One complete warmup per mode excluded from measured summaries. Modes run sequentially; ordering/cache/thermal effects remain possible.',
};
const cpuRecords = [];
const rows = [];
const errors = [];
const browser = await chromium.launch({ headless: true });
const evidence = { input: path.resolve(options.input), inputSha256: sha(inputText), baseURL: options.url,
browser: browser.version(), node: process.version, deep: Boolean(options.deep), cpuIntervalUs: options.deep ? Number(options.cpuIntervalUs) : null, initialNavigation: {}, scriptSha256: sha(await fs.readFile(new URL(import.meta.url))), definitions, rows, errors, servedAssets: [] };
const writeSummary = () => fs.writeFile(path.join(options.output, 'summary.json'), JSON.stringify(evidence, null, 2));
const waitMark = async (page, name) => {
await page.waitForFunction(key => window.__stageProfile?.marks[key] !== undefined || window.__stageProfile?.marks.runFailure !== undefined, name, { timeout: 180000 });
const failure = await page.evaluate(() => window.__stageProfile.runFailure);
if (failure) throw new Error(`Page simulation failed: ${failure}`);
};
const pageMark = (page, name) => page.evaluate(key => window.__stageProfile.mark(key), name);
const arm = (page, name, selector) => page.evaluate(({ name, selector }) => window.__stageProfile.armClick(name, selector), { name, selector });
const watch = (page, name, selector) => page.evaluate(({ name, selector }) => window.__stageProfile.watchDom(name, selector), { name, selector });
const clickTab = async (page, name) => { await page.getByRole('tab', { name }).click(); };
const resultDigest = result => sha(JSON.stringify(result));
const delta = (m, a, b) => m[a] === undefined || m[b] === undefined ? null : m[b] - m[a];
function metrics(trace) {
const m = trace.marks;
const csvAnchor = trace.downloads.find(d => d.name.endsWith('.csv'))?.anchorClick;
const resultAnchor = trace.downloads.find(d => d.name.endsWith('.simresult'))?.anchorClick;
const csvRequest = trace.requests.find(r => r.kind === 'csv');
const worker = trace.workers?.[0];
const workerStart = worker?.posts.find(p => p.type === 'start');
const workerFinish = worker?.posts.find(p => p.type === 'finish');
return {
importToDomMs: delta(m, 'importChange', 'importReadyDom'),
importToPaintOpportunityMs: delta(m, 'importChange', 'importReadyPaintOpportunity'),
clickToFetchMs: delta(m, 'runClick', 'fetchStart'), fetchToHeadersMs: delta(m, 'fetchStart', 'headers'),
headersToEofMs: delta(m, 'headers', 'streamEof'), clickToReadyDomMs: delta(m, 'runClick', 'resultReadyDom'),
clickToReadyPaintOpportunityMs: delta(m, 'runClick', 'resultReadyPaintOpportunity'),
resultParseMs: delta(m, 'resultParseStart', 'resultParseEnd'),
resultParseEndToReadyDomMs: delta(m, 'resultParseEnd', 'resultReadyDom'),
clickToIndexedDbCommitMs: delta(m, 'runClick', 'indexedDbCommittedPointer'),
clickToIndexedDbObservedMs: delta(m, 'runClick', 'indexedDbPointerObserved'),
resultTabToDomMs: delta(m, 'resultTabClick', 'resultsDom'),
resultTabToPaintOpportunityMs: delta(m, 'resultTabClick', 'resultsPaintOpportunity'),
resultTabToDomStableMs: delta(m, 'resultTabClick', 'resultsStable'),
curveSelectToDomMs: delta(m, 'temperatureClick', 'curveDom'),
curveSelectToPaintOpportunityMs: delta(m, 'temperatureClick', 'curvePaintOpportunity'),
curveSelectToDomStableMs: delta(m, 'temperatureClick', 'curveStable'),
csvClickToWorkerConstructMs: worker && m.csvClick !== undefined ? worker.constructStart - m.csvClick : null,
csvWorkerConstructMs: worker ? worker.constructEnd - worker.constructStart : null,
csvWorkerStartToFinishPostMs: workerStart && workerFinish ? workerFinish.end - workerStart.start : null,
csvWorkerFinishPostToCompleteReceivedMs: workerFinish && worker.completeReceived !== undefined ? worker.completeReceived - workerFinish.end : null,
csvWorkerStartToCompleteReceivedMs: workerStart && worker.completeReceived !== undefined ? worker.completeReceived - workerStart.start : null,
csvWorkerPostSyncTotalMs: worker ? worker.posts.reduce((total, post) => total + post.end - post.start, 0) : null,
csvWorkerTransferredBytes: worker ? worker.posts.reduce((total, post) => total + post.bytes, 0) : null,
csvClickToFetchMs: csvRequest && m.csvClick !== undefined ? csvRequest.fetchStart - m.csvClick : null,
csvFetchToHeadersMs: csvRequest?.headers === undefined ? null : csvRequest.headers - csvRequest.fetchStart,
csvClickToBlobAnchorMs: csvAnchor === undefined ? null : csvAnchor - m.csvClick,
resultClickToBlobAnchorMs: resultAnchor === undefined ? null : resultAnchor - m.resultFileClick,
csvClickToDownloadSavedMs: delta(m, 'csvClick', 'csvDownloadSaved'),
resultClickToDownloadSavedMs: delta(m, 'resultFileClick', 'resultFileDownloadSaved'),
streamReadCount: trace.profiled ? trace.reads.filter(r => r.kind === 'simulation').length : null,
streamBytes: trace.profiled ? trace.reads.reduce((n, r) => n + (r.kind === 'simulation' ? r.bytes : 0), 0) : null,
streamOutstandingReadMs: trace.profiled ? trace.reads.reduce((n, r) => n + (r.kind === 'simulation' ? r.end - r.start : 0), 0) : null,
synchronousStreamJsonParseMs: trace.profiled ? trace.parses.reduce((n, r) => n + r.end - r.start, 0) : null,
synchronousStreamDecodeMs: trace.profiled ? trace.decodes.reduce((n, r) => n + r.end - r.start, 0) : null,
};
}
try {
for (const mode of options.mode === 'both' ? ['control', 'profiled'] : [options.mode]) {
const context = await browser.newContext({ viewport: { width: 1600, height: 1000 }, acceptDownloads: true });
await context.addInitScript(installStageObserver, { profiled: mode === 'profiled' });
const page = await context.newPage();
const cpuRecorder = options.deep ? await createCpuRecorder(context, page, Number(options.cpuIntervalUs)) : null;
page.setDefaultTimeout(30000);
const simulationRequests = [];
const workerUrls = new Set();
page.on('worker', worker => workerUrls.add(worker.url()));
page.on('request', request => {
if (request.url().includes('/api/system-xml/simulate-stream')) {
simulationRequests.push({ url: request.url(), simulationId: request.headers()['x-simulation-id'] ?? null });
}
});
page.on('pageerror', error => errors.push({ mode, error: String(error) }));
page.on('dialog', dialog => { errors.push({ mode, dialog: dialog.message() }); void dialog.dismiss(); });
try {
await page.goto(options.url);
await page.locator('input[type="file"][accept*=".json"]').waitFor({ state: 'attached' });
evidence.initialNavigation[mode] = await page.evaluate(() => ({ timeOrigin: performance.timeOrigin,
appControlObservedAt: performance.now(), navigation: performance.getEntriesByType('navigation').map(e => e.toJSON()) }));
const assetUrls = await page.evaluate(() => [...document.querySelectorAll('script[src],link[rel="stylesheet"][href]')]
.map(e => e.src || e.href));
if (assetUrls.some(url => /@vite\/client|\/src\//.test(url))) throw new Error('Expected a production build, found Vite development assets.');
for (const url of assetUrls) {
const response = await context.request.get(url);
assert.ok(response.ok(), `Asset HTTP ${response.status()}: ${url}`);
const bytes = await response.body();
const previous = evidence.servedAssets.find(asset => asset.url === url);
if (previous) assert.equal(previous.sha256, sha(bytes), 'Production asset changed between modes.');
else evidence.servedAssets.push({ url, sha256: sha(bytes), bytes: bytes.length });
}
assert.ok(evidence.servedAssets.length, 'No production assets found.');
evidence.buildAssetSetSha256 = sha(JSON.stringify(evidence.servedAssets
.map(({ url, ...asset }) => ({ path: new URL(url).pathname, ...asset }))
.sort((a, b) => a.path.localeCompare(b.path))));
for (let run = 0; run <= Number(options.runs); run++) {
const prefix = `${mode}-${run === 0 ? 'warmup' : `run-${run}`}`;
const runDir = path.join(options.output, prefix);
await fs.mkdir(runDir, { recursive: true });
await clickTab(page, '建模');
const expand = page.getByRole('button', { name: '展开仿真控制台', exact: true });
if (await expand.isVisible()) await expand.click();
const requestOffset = simulationRequests.length;
if (cpuRecorder) await cpuRecorder.start();
await page.evaluate(name => { window.__stageProfile.reset(); window.__stageProfile.watchImport(name); },
path.basename(options.input, path.extname(options.input)));
await page.locator('input[type="file"][accept*=".json"]').setInputFiles(path.resolve(options.input));
await waitMark(page, 'importReadyPaintOpportunity');
// Verify the public project export outside simulation timing; importing after each
// reload avoids assuming that result recovery also restores the modeling workspace.
const projectDownload = page.waitForEvent('download');
await page.getByRole('button', { name: '导出工程 JSON', exact: true }).click();
await (await projectDownload).saveAs(path.join(runDir, 'imported-project.json'));
const imported = JSON.parse(await fs.readFile(path.join(runDir, 'imported-project.json'), 'utf8'));
assert.equal(imported.nodes.length, project.nodes.length);
assert.equal(imported.edges.length, project.edges.length);
assert.equal(imported.name, path.basename(options.input, path.extname(options.input)));
assert.deepEqual(imported.simulation, project.simulation);
for (const node of project.nodes) assert.deepEqual(imported.nodes.find(n => n.id === node.id)?.data.parameters, node.data.parameters);
for (const edge of project.edges) {
const actual = imported.edges.find(e => e.id === edge.id);
for (const key of ['source', 'target', 'sourceHandle', 'targetHandle']) assert.equal(actual?.[key], edge[key]);
}
await page.evaluate(() => window.__stageProfile.watchRunReady());
await arm(page, 'runClick', 'button[aria-label="运行仿真"]');
await page.getByRole('button', { name: '运行仿真', exact: true }).click();
await waitMark(page, 'resultReadyPaintOpportunity');
// Open results immediately; persistence is allowed to overlap exactly as in real use.
await arm(page, 'resultTabClick', '[role="tab"]');
await watch(page, 'results', '.results-shell .results-system-panel');
await clickTab(page, /^结果/);
await waitMark(page, 'resultsStable');
await page.getByRole('button', { name: '适应系统图窗口', exact: true }).click();
await page.locator(`.results-system-panel .react-flow__node[data-id=${JSON.stringify(curveNodeId)}]`).click();
// Remove persisted chart windows outside the curve-selection timing interval.
const close = page.locator('.result-chart-window .result-chart-window-actions button.close');
while (await close.count()) await close.first().click();
assert.equal(await page.locator('.results-chart-panel svg[data-result-chart="true"]').count(), 0,
'Curve timing requires no existing chart; update close-window selector if the UI changed.');
await arm(page, 'temperatureClick', '.results-variable-list button');
await watch(page, 'curve', '.results-chart-panel svg[data-result-chart="true"]');
await page.locator('.results-variable-list button').filter({ has: page.locator('small', { hasText: /^K$/ }) }).first().click();
await waitMark(page, 'curveStable');
await waitMark(page, 'indexedDbPointerObserved');
const saveDownload = async (buttonName, clickName, completedName, filename) => {
await arm(page, clickName, 'button');
const pending = page.waitForEvent('download', { timeout: 180000 });
await page.getByRole('button', { name: buttonName, exact: true }).click();
const download = await pending;
await download.saveAs(path.join(runDir, filename));
await pageMark(page, completedName);
assert.equal(await download.failure(), null);
};
await saveDownload('下载结果 CSV', 'csvClick', 'csvDownloadSaved', 'result.csv');
await saveDownload('下载结果文件', 'resultFileClick', 'resultFileDownloadSaved', 'result.simresult');
const trace = await page.evaluate(() => window.__stageProfile.snapshot());
if (cpuRecorder) {
const capture = await cpuRecorder.stop();
const file = path.join(runDir, 'interaction.cpuprofile');
await fs.writeFile(file, JSON.stringify(capture.profile));
cpuRecords.push({ ...capture, trace, kind: 'interaction', file });
}
const exportedBytes = await fs.readFile(path.join(runDir, 'result.simresult'));
const exported = JSON.parse(exportedBytes);
const result = exported.snapshot.result;
assert.equal(result.success, true, result.message);
assert.equal(result.simulatedUntil, Number(project.simulation.t_stop));
const expectedDigest = resultDigest(result);
const csv = await fs.readFile(path.join(runDir, 'result.csv'));
await fs.writeFile(path.join(runDir, 'trace.json'), JSON.stringify(trace, null, 2));
await page.screenshot({ path: path.join(runDir, 'result.png'), fullPage: true });
// Reload the real persisted snapshot without changing its session pointer or result.
if (cpuRecorder) await cpuRecorder.start();
await page.reload();
await page.evaluate(() => { window.__stageProfile.activeRun = true; });
await waitMark(page, 'restoredResultsStable');
await page.getByRole('button', { name: '下载结果文件', exact: true }).waitFor();
if (cpuRecorder) {
const restoreCpuTrace = await page.evaluate(() => window.__stageProfile.snapshot());
const capture = await cpuRecorder.stop();
const file = path.join(runDir, 'restore.cpuprofile');
await fs.writeFile(file, JSON.stringify(capture.profile));
cpuRecords.push({ ...capture, trace: restoreCpuTrace, kind: 'restore', file });
}
const restoredDownload = page.waitForEvent('download');
await page.getByRole('button', { name: '下载结果文件', exact: true }).click();
await (await restoredDownload).saveAs(path.join(runDir, 'restored.simresult'));
const restoredBytes = await fs.readFile(path.join(runDir, 'restored.simresult'));
const restored = JSON.parse(restoredBytes);
assert.equal(resultDigest(restored.snapshot.result), expectedDigest, 'Restored result changed.');
const restoreTrace = await page.evaluate(() => ({ ...window.__stageProfile.snapshot(),
navigation: performance.getEntriesByType('navigation').map(e => e.toJSON()) }));
await fs.writeFile(path.join(runDir, 'restore-trace.json'), JSON.stringify(restoreTrace, null, 2));
const requests = simulationRequests.slice(requestOffset);
assert.equal(requests.length, 1, 'Expected exactly one real simulation request.');
assert.ok(requests[0].simulationId, 'Missing X-Simulation-Id correlation key.');
const row = { mode, deep: Boolean(options.deep), run, warmup: run === 0, simulationId: requests[0].simulationId, timeOrigin: trace.timeOrigin, ...metrics(trace),
restoreNavigationToDomMs: restoreTrace.marks.restoredResultsDom,
restoreNavigationToPaintOpportunityMs: restoreTrace.marks.restoredResultsPaintOpportunity,
restoreNavigationToDomStableMs: restoreTrace.marks.restoredResultsStable,
sampleCount: result.series.time.length, variableCount: result.variables.length,
native: result.diagnostics.native, integration: result.diagnostics.integration,
resultSha256: sha(exportedBytes), numericalResultSha256: expectedDigest, resultBytes: exportedBytes.length,
csvSha256: sha(csv), csvBytes: csv.length, restoredIdentical: true, artifacts: prefix };
rows.push(row);
await writeSummary();
console.log(JSON.stringify(row));
if (errors.length) throw new Error(`Browser errors: ${JSON.stringify(errors)}`);
}
// Collect actual worker build evidence after all timing intervals. Worker code
// is loaded dynamically and does not appear among the initial document tags.
for (const url of workerUrls) {
if (!/^https?:/.test(url)) continue;
const response = await context.request.get(url);
assert.ok(response.ok(), `Worker asset HTTP ${response.status()}: ${url}`);
const bytes = await response.body();
const previous = evidence.servedAssets.find(asset => asset.url === url);
if (previous) assert.equal(previous.sha256, sha(bytes), 'Worker build changed between modes.');
else evidence.servedAssets.push({ url, sha256: sha(bytes), bytes: bytes.length });
}
evidence.buildAssetSetSha256 = sha(JSON.stringify(evidence.servedAssets
.map(({ url, ...asset }) => ({ path: new URL(url).pathname, ...asset }))
.sort((a, b) => a.path.localeCompare(b.path))));
await writeSummary();
} catch (error) {
await page.screenshot({ path: path.join(options.output, `${mode}-failure.png`), fullPage: true }).catch(() => {});
const trace = await page.evaluate(() => window.__stageProfile?.snapshot()).catch(() => null);
const body = await page.locator('body').innerText().catch(() => null);
await fs.writeFile(path.join(options.output, `${mode}-failure.json`), JSON.stringify({ error: error.stack, trace, errors, body }, null, 2));
throw error;
} finally { if (cpuRecorder) await cpuRecorder.close(); await context.close(); }
}
if (cpuRecords.length) {
const { consumers, evidence: sourceMaps } = await loadVerifiedCpuMaps(options.sourceMapDir, evidence.servedAssets);
evidence.cpuDiagnostics = { sourceMaps, profiles: [] };
for (const record of cpuRecords) {
const summary = summarizeCpu(record, consumers);
const output = record.file.replace(/\.cpuprofile$/, '-cpu-summary.json');
await fs.writeFile(output, JSON.stringify(summary, null, 2));
await fs.writeFile(output.replace(/\.json$/, '.md'), cpuSummaryMarkdown(summary, record.file));
evidence.cpuDiagnostics.profiles.push({ profile: record.file, summary: output, kind: record.kind });
}
}
const median = values => { const v = values.filter(n => typeof n === 'number').sort((a, b) => a - b);
return v.length ? v.length % 2 ? v[Math.floor(v.length / 2)] : (v[v.length / 2 - 1] + v[v.length / 2]) / 2 : null; };
evidence.measuredMedians = Object.fromEntries(['control', 'profiled'].map(mode => [mode,
Object.fromEntries(Object.keys(rows.find(r => r.mode === mode) ?? {}).filter(k => k.endsWith('Ms'))
.map(k => [k, median(rows.filter(r => r.mode === mode && !r.warmup).map(r => r[k]))]))]));
await writeSummary();
} finally { await writeSummary(); await browser.close(); }
tests/manual/compare_browser_stage_outputs.py
+135
View File
@@ -0,0 +1,135 @@
"""Compare all real browser CSV/result/restore values with a native execution.
Run only after performance measurements finish; JSON/CSV verification is CPU intensive.
No tolerance or resampling is used. Diagnostic timing fields intentionally differ.
"""
from __future__ import annotations
import argparse
import csv
import hashlib
import json
from pathlib import Path
def sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open('rb') as source:
for chunk in iter(lambda: source.read(1024 * 1024), b''):
digest.update(chunk)
return digest.hexdigest()
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('directory', type=Path)
parser.add_argument('--native', type=Path, help='Full native result baseline; defaults to DIRECTORY/native-production/run-1/result.json')
parser.add_argument('--group', action='append', default=[], metavar='LABEL=PATH', help='Explicit browser output group; repeat for old/new/profiled groups')
parser.add_argument('--output', type=Path, help='Defaults to DIRECTORY/equality.json')
args = parser.parse_args()
base = args.directory
native_path = args.native or base / 'native-production/run-1/result.json'
groups = []
for group in args.group:
label, separator, directory = group.partition('=')
if not separator or not label or not directory:
parser.error('--group requires LABEL=PATH')
groups.append((label, Path(directory)))
if not groups:
groups = [(mode, base / f'browser-{mode}') for mode in ('control', 'profiled')]
assert len({label for label, _ in groups}) == len(groups), 'Group labels must be unique'
native = json.loads(native_path.read_text())
assert native['success'] and native['simulatedUntil'] == 10
expected_series, expected_final = native['series'], native['final']
point_count = len(expected_series['time'])
assert len(expected_series) == 1785 and point_count == 1002
report = {
'native': str(native_path), 'nativeSha256': sha256(native_path),
'comparison': 'Numeric equality using == on parsed numbers, with identical keys and lengths; no tolerance, conversion, interpolation or rounding. Diagnostic timings are excluded.',
'sampleCount': point_count, 'seriesCountIncludingTime': len(expected_series),
'finalValueCount': len(expected_final), 'runs': [], 'groups': [],
}
input_hashes, build_hashes, csv_hashes = set(), set(), set()
for label, group in groups:
group_csv_hashes = set()
summary = json.loads((group / 'summary.json').read_text())
assert not summary['errors'], summary['errors']
assert len(summary['rows']) == 4
assert sum(bool(row['warmup']) for row in summary['rows']) == 1
assert len({row['run'] for row in summary['rows']}) == 4
input_hashes.add(summary['inputSha256'])
build_hashes.add(summary['buildAssetSetSha256'])
for row in summary['rows']:
directory = group / row['artifacts']
result_path = directory / 'result.simresult'
result = json.loads(result_path.read_text())['snapshot']['result']
assert result['success'] and result['simulatedUntil'] == 10
assert set(result['series']) == set(expected_series), directory
assert result['series'] == expected_series, f'{directory}: series differs from native'
assert result['final'] == expected_final, f'{directory}: final differs from native'
variables = [variable['key'] for variable in result['variables']]
assert len(variables) == 1784 and len(set(variables)) == 1784
restored_path = directory / 'restored.simresult'
restored = json.loads(restored_path.read_text())['snapshot']['result']
assert restored == result, f'{directory}: restored result changed'
del restored
csv_path = directory / 'result.csv'
checked = 0
with csv_path.open(newline='', encoding='utf-8-sig') as source:
reader = csv.reader(source)
headers = next(reader)
assert headers == ['time', *variables], f'{directory}: CSV variable order differs'
assert len(headers) == 1785 and set(headers) == set(expected_series)
columns = [expected_series[key] for key in headers]
row_count = 0
for index, cells in enumerate(reader):
assert index < point_count and len(cells) == len(headers), (directory, index)
for column_index, (cell, column) in enumerate(zip(cells, columns, strict=True)):
assert float(cell) == column[index], (directory, index, headers[column_index], cell, column[index])
checked += 1
row_count += 1
assert row_count == point_count
csv_hash = sha256(csv_path)
csv_hashes.add(csv_hash)
group_csv_hashes.add(csv_hash)
item = {
'group': label, 'mode': row['mode'], 'run': row['run'], 'warmup': row['warmup'],
'simulationId': row['simulationId'], 'directory': str(directory),
'seriesComparedValues': sum(map(len, expected_series.values())),
'finalComparedValues': len(expected_final), 'csvComparedCells': checked,
'csvColumns': len(headers), 'csvRows': row_count,
'seriesExactlyEqualToNative': True, 'finalExactlyEqualToNative': True,
'csvAllCellsExactlyEqualToNative': True, 'restoredResultExactlyEqual': True,
'resultSha256': sha256(result_path), 'restoredSha256': sha256(restored_path),
'csvSha256': csv_hash,
}
report['runs'].append(item)
print(json.dumps({'group': label, 'mode': row['mode'], 'run': row['run'], 'checkedCsvCells': checked, 'exact': True}), flush=True)
assert len(group_csv_hashes) == 1, f'{label}: CSV bytes differ between repeated runs'
report['groups'].append({
'label': label, 'directory': str(group), 'runCount': len(summary['rows']),
'buildAssetSetSha256': summary['buildAssetSetSha256'],
'inputSha256': summary['inputSha256'], 'allCsvFilesByteIdentical': True,
'csvSha256': next(iter(group_csv_hashes)),
})
assert len(input_hashes) == 1
# Python and JavaScript emit different round-tripping spellings (e.g. 0.0/0).
# Across versions CSV numeric equality is required; text hashes are per group.
report.update({
'allPassed': True, 'inputSha256': next(iter(input_hashes)),
'buildAssetSetSha256': next(iter(build_hashes)) if len(build_hashes) == 1 else None,
'buildAssetSetSha256Values': sorted(build_hashes),
'allGroupsCsvFilesByteIdentical': True,
'allCsvFilesByteIdentical': len(csv_hashes) == 1,
'csvSha256': next(iter(csv_hashes)) if len(csv_hashes) == 1 else None,
'totalCsvCellsCompared': sum(row['csvComparedCells'] for row in report['runs']),
'totalSeriesValuesCompared': sum(row['seriesComparedValues'] for row in report['runs']),
'totalFinalValuesCompared': sum(row['finalComparedValues'] for row in report['runs']),
})
output = args.output or base / 'equality.json'
output.write_text(json.dumps(report, indent=2) + '\n')
print(json.dumps({'output': str(output), 'allPassed': True, 'totalCsvCellsCompared': report['totalCsvCellsCompared']}))
if __name__ == '__main__':
main()
tests/manual/compare_native_result_bits.py
+236
View File
@@ -0,0 +1,236 @@
"""Compare complete native numerical results as binary64, outside benchmark timing.
.venv/bin/python tests/manual/compare_native_result_bits.py \
--baseline old/result.json --candidate new/run-1/result.json \
--candidate new/run-2/result.json --output test/result-bit-parity.json
JSON number spelling may change. In particular, the integer token -0 must be
parsed as negative floating zero before packing. All series columns (including
time), final scalars and finalState entries are compared without sampling.
Solver status/configuration/counters must also agree. Only solve wall/CPU timing
metadata is intentionally ignored. Nonfinite payload or metadata numbers fail.
"""
from __future__ import annotations
import argparse
from hashlib import sha256
import json
import math
from pathlib import Path
import struct
import sys
PAYLOAD_KEYS = ("series", "final", "finalState")
FLOAT_METADATA = ("simulatedUntil", "maxAcceptedStep")
COUNT_METADATA = ("nfev", "acceptedSteps", "rejectedSteps", "njev", "nlu", "stateTransitions", "solverStarts")
VALUE_METADATA = ("success", "status", "message", "method", "backend", "solver", "sundialsVersion")
TIMING_METADATA = ("solveSeconds", "solveCpuSeconds")
NEGATIVE_ZERO = struct.pack("<Q", 1 << 63)
POSITIVE_ZERO = b"\0" * 8
class ComparisonError(ValueError):
def __init__(self, path: str, reason: str, **details: object) -> None:
super().__init__(f"{path}: {reason}")
self.detail = {"path": path, "reason": reason, **details}
def pointer(*parts: object) -> str:
return "/" + "/".join(str(part).replace("~", "~0").replace("/", "~1") for part in parts)
def read_result(path: Path) -> tuple[dict, str]:
raw = path.read_bytes()
def unique_object(items: list[tuple[str, object]]) -> dict:
result = {}
for key, value in items:
if key in result:
raise ComparisonError("/", "Duplicate JSON object key", key=key)
result[key] = value
return result
def reject_constant(token: str) -> object:
raise ComparisonError("/", "Nonfinite JSON token", token=token)
value = json.loads(raw, parse_int=lambda token: -0.0 if token == "-0" else int(token),
parse_constant=reject_constant, object_pairs_hook=unique_object)
if not isinstance(value, dict):
raise ComparisonError("/", "Expected a native result object")
return value, sha256(raw).hexdigest()
def bits(value: object, path: str) -> bytes:
if isinstance(value, bool) or not isinstance(value, (int, float)):
raise ComparisonError(path, "Expected a finite numeric value", actualType=type(value).__name__)
try:
number = float(value)
except (OverflowError, ValueError):
raise ComparisonError(path, "Number cannot be represented as finite binary64") from None
if not math.isfinite(number):
raise ComparisonError(path, "Nonfinite binary64 value")
return struct.pack("<d", number)
def ensure_finite_tree(value: object, path: str = "") -> None:
"""Reject overflow-to-infinity tokens even in metadata excluded from parity."""
if isinstance(value, dict):
for key, child in value.items():
ensure_finite_tree(child, path + pointer(key))
elif isinstance(value, list):
for index, child in enumerate(value):
ensure_finite_tree(child, path + pointer(index))
elif isinstance(value, (int, float)) and not isinstance(value, bool):
bits(value, path or "/")
def validate_result(result: dict) -> dict:
required = set(PAYLOAD_KEYS + FLOAT_METADATA + COUNT_METADATA + VALUE_METADATA)
if missing := required - result.keys():
raise ComparisonError("/", "Missing native result fields", missing=sorted(missing))
if not isinstance(result["series"], dict) or not isinstance(result["final"], dict):
raise ComparisonError("/", "series and final must be objects")
if not isinstance(result["finalState"], list):
raise ComparisonError("/finalState", "Expected an array")
for key, values in result["series"].items():
if not isinstance(values, list):
raise ComparisonError(pointer("series", key), "Expected a numeric array")
if result["series"] and "time" not in result["series"]:
raise ComparisonError("/series", "Nonempty series has no time column")
if result["series"]:
samples = len(result["series"]["time"])
for key, values in result["series"].items():
if len(values) != samples:
raise ComparisonError(pointer("series", key), "Column length differs from time", expectedLength=samples, actualLength=len(values))
for key in COUNT_METADATA:
value = result[key]
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
raise ComparisonError(pointer(key), "Expected a nonnegative integer solver counter")
if not isinstance(result["success"], bool):
raise ComparisonError("/success", "Expected a boolean")
for key in VALUE_METADATA[1:]:
if not isinstance(result[key], str):
raise ComparisonError(pointer(key), "Expected string metadata")
ensure_finite_tree(result)
for key in FLOAT_METADATA:
bits(result[key], pointer(key))
# Scalars in final and all payload cells must be numeric, never bool/null.
count = negative_zeroes = positive_zeroes = 0
for path, value in payload_values(result):
packed = bits(value, path)
count += 1
negative_zeroes += packed == NEGATIVE_ZERO
positive_zeroes += packed == POSITIVE_ZERO
return {"payloadValues": count, "seriesColumns": len(result["series"]),
"samples": len(result["series"].get("time", [])), "finalScalars": len(result["final"]),
"finalStateValues": len(result["finalState"]), "negativeZeroValues": negative_zeroes,
"positiveZeroValues": positive_zeroes}
def payload_values(result: dict):
for key, values in result["series"].items():
for index, value in enumerate(values):
yield pointer("series", key, index), value
for key, value in result["final"].items():
yield pointer("final", key), value
for index, value in enumerate(result["finalState"]):
yield pointer("finalState", index), value
def match_keys(baseline: dict, candidate: dict, path: str) -> None:
if baseline.keys() != candidate.keys():
raise ComparisonError(path, "Object key sets differ", missing=sorted(baseline.keys() - candidate.keys()),
extra=sorted(candidate.keys() - baseline.keys()))
def compare(baseline: dict, candidate: dict) -> dict:
# Validate the complete structure before comparing any payload bit patterns.
match_keys(baseline, candidate, "/")
for key in ("series", "final"):
match_keys(baseline[key], candidate[key], pointer(key))
for key, values in baseline["series"].items():
if len(values) != len(candidate["series"][key]):
raise ComparisonError(pointer("series", key), "Array lengths differ", baselineLength=len(values), candidateLength=len(candidate["series"][key]))
if len(baseline["finalState"]) != len(candidate["finalState"]):
raise ComparisonError("/finalState", "Array lengths differ", baselineLength=len(baseline["finalState"]), candidateLength=len(candidate["finalState"]))
metadata_comparisons = 0
for key in VALUE_METADATA + COUNT_METADATA:
if baseline[key] != candidate[key]:
raise ComparisonError(pointer(key), "Solver metadata or counter differs", baseline=baseline[key], candidate=candidate[key])
metadata_comparisons += 1
for key in FLOAT_METADATA:
left, right = bits(baseline[key], pointer(key)), bits(candidate[key], pointer(key))
if left != right:
raise ComparisonError(pointer(key), "Numeric metadata binary64 bits differ", baselineBitsLE=left.hex(), candidateBitsLE=right.hex())
metadata_comparisons += 1
# Any future top-level metadata field must also agree unless explicitly timed.
known = set(PAYLOAD_KEYS + FLOAT_METADATA + COUNT_METADATA + VALUE_METADATA + TIMING_METADATA)
for key in baseline.keys() - known:
if baseline[key] != candidate[key]:
raise ComparisonError(pointer(key), "Additional metadata differs")
metadata_comparisons += 1
comparisons = negative_zeroes = positive_zeroes = 0
def compare_number(left_value: object, right_value: object, path: str) -> None:
nonlocal comparisons, negative_zeroes, positive_zeroes
left, right = bits(left_value, path), bits(right_value, path)
if left != right:
raise ComparisonError(path, "Payload binary64 bits differ", baselineBitsLE=left.hex(), candidateBitsLE=right.hex(),
baselineValue=repr(left_value), candidateValue=repr(right_value),
signedZeroMismatch=left in (POSITIVE_ZERO, NEGATIVE_ZERO) and right in (POSITIVE_ZERO, NEGATIVE_ZERO))
comparisons += 1
negative_zeroes += left == NEGATIVE_ZERO
positive_zeroes += left == POSITIVE_ZERO
# Look up columns by their validated key; JSON object order is immaterial.
for key, values in baseline["series"].items():
for index, (left, right) in enumerate(zip(values, candidate["series"][key], strict=True)):
compare_number(left, right, pointer("series", key, index))
for key, value in baseline["final"].items():
compare_number(value, candidate["final"][key], pointer("final", key))
for index, (left, right) in enumerate(zip(baseline["finalState"], candidate["finalState"], strict=True)):
compare_number(left, right, pointer("finalState", index))
return {"passed": True, "comparisons": comparisons, "metadataComparisons": metadata_comparisons,
"negativeZeroComparisons": negative_zeroes, "positiveZeroComparisons": positive_zeroes,
"allPayloadBinary64BitsEqual": True}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--baseline", required=True, type=Path)
parser.add_argument("--candidate", required=True, action="append", type=Path)
parser.add_argument("--output", required=True, type=Path)
args = parser.parse_args()
if args.output.resolve() in {args.baseline.resolve(), *(path.resolve() for path in args.candidate)}:
parser.error("--output must differ from every input file")
report = {"version": 1, "baseline": str(args.baseline.resolve()), "candidates": [], "allPassed": False,
"comparisonContract": "Exact finite binary64 payload bits, including signed zero; complete structure plus solver metadata/counters. JSON object ordering is ignored. solveSeconds and solveCpuSeconds are excluded. Parsing and comparison are diagnostic work outside benchmark timing."}
try:
baseline, baseline_hash = read_result(args.baseline)
report["baselineSha256"] = baseline_hash
report["baselineStatistics"] = validate_result(baseline)
for path in args.candidate:
item = {"path": str(path.resolve()), "passed": False}
try:
candidate, candidate_hash = read_result(path)
item["sha256"] = candidate_hash
item["statistics"] = validate_result(candidate)
item.update(compare(baseline, candidate))
except ComparisonError as error:
item["error"] = error.detail
except (OSError, ValueError, TypeError) as error:
item["error"] = {"reason": str(error), "type": type(error).__name__}
report["candidates"].append(item)
report["allPassed"] = all(item["passed"] for item in report["candidates"])
except ComparisonError as error:
report["baselineError"] = error.detail
except (OSError, ValueError, TypeError) as error:
report["baselineError"] = {"reason": str(error), "type": type(error).__name__}
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(report, ensure_ascii=False, indent=2, allow_nan=False) + "\n", encoding="utf-8")
print(json.dumps({"allPassed": report["allPassed"], "candidates": len(report["candidates"]), "output": str(args.output.resolve())}))
return 0 if report["allPassed"] else 1
if __name__ == "__main__":
sys.exit(main())
tests/manual/native_compute_profile.py
+314
View File
@@ -0,0 +1,314 @@
"""Isolated, Linux/GCC-only CVODE cost diagnostic; never a production benchmark.
Example (prepare only by default; --run builds and executes serially)::
.venv/bin/python tests/manual/native_compute_profile.py \
--cache-dir test/.../cache/BUILD_KEY --request-stages test/.../stages.json \
--output-dir test/native-compute-profile --run --warmups 1 --repeats 3
The cached executable is the unmodified control. Only a private native source
copy receives wall-clock scopes and sparse CVODE counter reads. The generated
model and numerical expressions, compiler FP flags, solver and libraries stay
unchanged. Full output/state/counter equality is checked outside run timing.
Inclusive durations are nested: only exclusiveSeconds may be added. Clock and
bookkeeping overhead remain in measured totals; compare against the control.
No property/pipe/libc allocation is inferred from this outer-only diagnostic.
"""
from __future__ import annotations
import argparse
from hashlib import sha256
import json
import os
from pathlib import Path
import re
import shutil
import statistics
import subprocess
import sys
import time
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT))
from app.simulation.native_codegen.build import LIBRARIES, toolchain
CATEGORIES = ["integration", "cvode_step", "rhs", "poll", "accept", "append", "dense_output", "dense_setup", "dense_solve"]
COUNTERS = ["rhs", "linear_rhs", "nonlinear_iterations", "nonlinear_failures"]
PROFILE_HEADER = r'''
#ifndef NATIVE_COMPUTE_PROFILE_H
#define NATIVE_COMPUTE_PROFILE_H
#include <stddef.h>
enum { @CATEGORIES@, PROFILE_CATEGORY_COUNT };
typedef struct ProfileScope { double start, children; int id, domain, active; struct ProfileScope *parent; } ProfileScope;
ProfileScope profile_begin(int id);
void profile_link(ProfileScope *scope);
void profile_end(ProfileScope *scope);
void profile_counter(int slot, int status, long int value);
void profile_counter_segment(void);
void profile_dump(void);
#define PROFILE_SCOPE(id) ProfileScope profile_scope __attribute__((cleanup(profile_end)))=profile_begin(id); profile_link(&profile_scope)
#endif
'''
PROFILE_SOURCE = r'''
#include "compute_profile.h"
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <stdint.h>
typedef struct { unsigned long long count; double inclusive, exclusive; } ProfileTotal;
static ProfileTotal totals[2][PROFILE_CATEGORY_COUNT];
static ProfileScope *parent;
static unsigned long long counters[4], segments, counter_errors;
static double now(void) { struct timespec t; clock_gettime(CLOCK_MONOTONIC,&t); return t.tv_sec+t.tv_nsec*1e-9; }
ProfileScope profile_begin(int id) {
ProfileScope s={0}; s.id=id; s.domain=(id==PROFILE_INTEGRATION || (parent && parent->domain));
s.parent=parent; s.active=1; s.start=now(); return s;
}
void profile_link(ProfileScope *s) { parent=s; }
void profile_end(ProfileScope *s) {
if(!s->active) return;
double elapsed=now()-s->start;
if(parent!=s) { fputs("Invalid profile scope nesting\n",stderr); exit(74); }
ProfileTotal *t=&totals[s->domain][s->id];
t->count++; t->inclusive+=elapsed; t->exclusive+=elapsed-s->children;
parent=s->parent;
if(parent) parent->children+=elapsed;
s->active=0;
}
void profile_counter(int slot,int status,long int value) {
if(status || value<0) counter_errors++; else counters[slot]+=(unsigned long long)value;
}
void profile_counter_segment(void) { segments++; }
void profile_dump(void) {
const char *path=getenv("NATIVE_COMPUTE_PROFILE"); if(!path)return;
FILE *f=fopen(path,"wb"); if(!f){perror(path);exit(73);}
const char *names[]={@NAMES@};
fprintf(f,"{\"version\":1,\"counterSegments\":%llu,\"counterErrors\":%llu,\"cvodeCounters\":{",segments,counter_errors);
const char *counter_names[]={"rhs","linear_rhs","nonlinear_iterations","nonlinear_failures"};
for(int i=0;i<4;i++)fprintf(f,"%s\"%s\":%llu",i?",":"",counter_names[i],counters[i]);
fprintf(f,"},\"scopes\":{");
for(int d=0;d<2;d++) {
fprintf(f,"%s\"%s\":{",d?",":"",d?"integration":"outsideIntegration");
for(int i=0;i<PROFILE_CATEGORY_COUNT;i++) {
ProfileTotal *t=&totals[d][i];
fprintf(f,"%s\"%s\":{\"calls\":%llu,\"inclusiveSeconds\":%.17g,\"exclusiveSeconds\":%.17g}",
i?",":"",names[i],t->count,t->inclusive,t->exclusive);
}
fputc('}',f);
}
fprintf(f,"}}\n"); int ok=!ferror(f); if(fclose(f))ok=0; if(!ok)exit(73);
}
'''
LINEAR_WRAPPERS = r'''
/* Preserve the exact original Dense ops; only the call boundary is timed. */
static int (*profile_original_setup)(SUNLinearSolver,SUNMatrix);
static int (*profile_original_solve)(SUNLinearSolver,SUNMatrix,N_Vector,N_Vector,sunrealtype);
static int profile_dense_setup(SUNLinearSolver linear,SUNMatrix matrix) {
PROFILE_SCOPE(PROFILE_DENSE_SETUP);
return profile_original_setup(linear,matrix);
}
static int profile_dense_solve(SUNLinearSolver linear,SUNMatrix matrix,N_Vector x,N_Vector b,sunrealtype tolerance) {
PROFILE_SCOPE(PROFILE_DENSE_SOLVE);
return profile_original_solve(linear,matrix,x,b,tolerance);
}
static int profile_cvode(void *solver,sunrealtype end,N_Vector y,sunrealtype *next,int task) {
PROFILE_SCOPE(PROFILE_CVODE_STEP);
return CVode(solver,end,y,next,task);
}
'''
def digest(path: Path) -> str:
return sha256(path.read_bytes()).hexdigest()
def write_json(path: Path, value: object) -> None:
path.write_text(json.dumps(value, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
def replace_once(text: str, old: str, new: str) -> str:
if text.count(old) != 1:
raise RuntimeError(f"Source anchor count changed: {old!r}")
return text.replace(old, new)
def scope_function(text: str, function: str, category: str) -> str:
pattern = rf"(?m)^[A-Za-z_][A-Za-z0-9_ \t*]*\b{re.escape(function)}\s*\([^;{{}}]*\)\s*\{{"
matches = list(re.finditer(pattern, text))
if len(matches) != 1:
raise RuntimeError(f"Cannot identify unique function {function}")
pos = matches[0].end()
return text[:pos] + f"\n PROFILE_SCOPE(PROFILE_{category.upper()});" + text[pos:]
def instrument(native: Path) -> None:
(native / "include/compute_profile.h").write_text(PROFILE_HEADER.replace("@CATEGORIES@", ", ".join("PROFILE_" + c.upper() for c in CATEGORIES)))
(native / "runtime/compute_profile.c").write_text(PROFILE_SOURCE.replace("@NAMES@", ",".join(json.dumps(c) for c in CATEGORIES)))
for filename, functions in {
"common.c": {"native_rhs": "rhs", "native_poll": "poll", "native_accept": "accept", "native_append": "append"},
"cvode_solver.c": {"native_bdf": "integration", "cv_dense": "dense_output"},
"rk45.c": {"native_rk45": "integration"},
}.items():
path = native / "runtime" / filename
text = '#include "compute_profile.h"\n' + path.read_text()
for function, category in functions.items():
text = scope_function(text, function, category)
if filename == "cvode_solver.c":
text = replace_once(text, "typedef struct { void *solver;", LINEAR_WRAPPERS + "\ntypedef struct { void *solver;")
text = replace_once(text, " if (!linear) goto cleanup;", " if (!linear) goto cleanup;\n profile_original_setup=linear->ops->setup; profile_original_solve=linear->ops->solve;\n linear->ops->setup=profile_dense_setup; linear->ops->solve=profile_dense_solve;")
text = replace_once(text, "int flag=CVode(solver,end,y,&next,CV_ONE_STEP);", "int flag=profile_cvode(solver,end,y,&next,CV_ONE_STEP);")
extra = "\n profile_counter_segment();\n"
for slot, api in enumerate(("CVodeGetNumRhsEvals", "CVodeGetNumLinRhsEvals", "CVodeGetNumNonlinSolvIters", "CVodeGetNumNonlinSolvConvFails")):
extra += f" value=0; int profile_status_{slot}={api}(solver,&value); profile_counter({slot},profile_status_{slot},value);\n"
text = replace_once(text, "CVodeGetNumLinSolvSetups(solver,&value); r->nlu+=(unsigned long)value;", "CVodeGetNumLinSolvSetups(solver,&value); r->nlu+=(unsigned long)value;" + extra)
path.write_text(text)
path = native / "runtime/main.c"
text = '#include "compute_profile.h"\n' + path.read_text()
text = replace_once(text, " native_run_free(&r); return code;", " native_run_free(&r); profile_dump(); return code;")
path.write_text(text)
def runtime_arguments(stages: Path | None) -> list[str]:
if stages:
original = json.loads(stages.read_text())["process"]["command"]
args = original[1:]
else:
args = ["--method", "BDF", "--start", "0", "--stop", "10", "--sample-step", ".01", "--max-step", "1e30", "--rtol", "1e-8", "--timeout", "300"]
safe, index = [], 0
value_options = {"--method", "--start", "--stop", "--sample-step", "--max-step", "--rtol", "--timeout"}
while index < len(args):
key = args[index]
if key == "--solve-only":
safe.append(key); index += 1; continue
if key not in value_options | {"--output", "--result-index", "--cancel-file"} or index + 1 >= len(args):
raise RuntimeError(f"Unsupported replay argument: {key}")
if key in value_options:
safe.extend(args[index:index + 2])
index += 2
return safe
def prepare(args: argparse.Namespace) -> dict:
cache, output = args.cache_dir.resolve(), args.output_dir.resolve()
if not output.is_relative_to(ROOT / "test"):
raise RuntimeError("Diagnostic output must be in the repository's ignored test/ directory")
manifest = json.loads((cache / "manifest.json").read_text())
for name in ("model", "model.c", "model.h"):
if digest(cache / name) != manifest["artifacts"][name]:
raise RuntimeError(f"Cache artifact integrity failure: {name}")
# Reject numerical/runtime drift; a sparse timing-only cached main is allowed
# because control and our current writer share the numeric model contract.
differences = []
for name, expected in manifest["sourceHashes"].items():
relative = name.split("native/", 1)[-1]
current = ROOT / "native" / relative
if digest(current) != expected:
differences.append(relative)
if any(name != "runtime/main.c" for name in differences):
raise RuntimeError(f"Cached numerical sources differ from current sources: {differences}")
compiler, sundials, compiler_version = toolchain()
if not sys.platform.startswith("linux"):
raise RuntimeError("This test-only cleanup-scope profiler requires Linux/GCC")
libraries = [sundials / "lib" / f"libsundials_{name}.a" for name in LIBRARIES]
for name, expected in manifest["dependencyHashes"].items():
path = sundials / ("include" if "/" in name else "lib") / name
if digest(path) != expected:
raise RuntimeError(f"SUNDIALS dependency changed: {name}")
if compiler_version != manifest["compiler"]:
raise RuntimeError("Use the cached model's compiler version for this comparison")
output.mkdir(parents=True, exist_ok=True)
native = output / "native"
shutil.copytree(ROOT / "native", native, dirs_exist_ok=True)
for name in ("model.c", "model.h", "manifest.json"):
shutil.copy2(cache / name, output / name)
instrument(native)
command = [compiler, *manifest["compilerFlags"], "-I", str(output), "-I", str(native / "include"), "-I", str(sundials / "include"), str(output / "model.c"), *map(str, sorted(native.rglob("*.c"))), "-Wl,--start-group", *map(str, libraries), "-Wl,--end-group", "-lm", "-o", str(output / "profiled-model")]
prepared = {"controlExecutable": str(cache / "model"), "profiledExecutable": str(output / "profiled-model"), "buildCommand": command, "runtimeArguments": runtime_arguments(args.request_stages), "cacheDir": str(cache), "cachedMainDifference": differences, "stateCount": len(manifest["stateKeys"]), "modelSha256": digest(output / "model.c"), "compiler": compiler_version, "warmups": args.warmups, "repeats": args.repeats, "scope": "Inclusive times overlap. Exclusive scope times partition integration including instrumentation overhead. No component or libc sub-cost inference.", "sourceHashes": {str(p.relative_to(output)): digest(p) for p in sorted(native.rglob("*")) if p.is_file()}}
write_json(output / "prepared.json", prepared)
return prepared
def parity_payload(result: dict) -> dict:
return {key: value for key, value in result.items() if key not in {"solveSeconds", "solveCpuSeconds"}}
def execute(args: argparse.Namespace, prepared: dict) -> None:
output = args.output_dir.resolve()
build = subprocess.run(prepared["buildCommand"], capture_output=True, text=True, timeout=180)
(output / "build.log").write_text(build.stdout + build.stderr)
if build.returncode:
raise RuntimeError(f"Compilation failed: {output / 'build.log'}")
baseline = None
rows = []
# Serial paired control/profile runs; warmups excluded from overhead figures.
for index in range(-args.warmups, args.repeats):
label = f"warmup-{index + args.warmups + 1}" if index < 0 else f"run-{index + 1}"
for variant in ("control", "profiled"):
run = output / variant / label
run.mkdir(parents=True, exist_ok=True)
result_path, profile_path = run / "result.json", run / "profile.json"
for stale in (result_path, profile_path, run / "cancel.request"):
stale.unlink(missing_ok=True)
command = [prepared[f"{variant}Executable"], *prepared["runtimeArguments"], "--output", str(result_path), "--result-index", str(run / "result-index.json"), "--cancel-file", str(run / "cancel.request")]
environment = dict(os.environ)
environment.pop("NATIVE_COMPUTE_PROFILE", None)
# Disable independent sparse-stage profilers in a cached control.
environment.pop("NATIVE_STAGE_PROFILE", None)
if variant == "profiled":
environment["NATIVE_COMPUTE_PROFILE"] = str(profile_path)
started = time.perf_counter()
process = subprocess.run(command, env=environment, capture_output=True, timeout=args.process_timeout)
wall = time.perf_counter() - started
(run / "stdout.log").write_bytes(process.stdout)
(run / "stderr.log").write_bytes(process.stderr)
if process.returncode:
raise RuntimeError(f"{variant}/{label} exit {process.returncode}; see stderr.log")
result = json.loads(result_path.read_bytes())
if result.get("success") is not True:
raise RuntimeError(f"{variant}/{label} did not complete")
comparable = parity_payload(result)
if baseline is None:
baseline = comparable
if comparable != baseline:
mismatches = [k for k in baseline.keys() | comparable.keys() if baseline.get(k) != comparable.get(k)]
write_json(run / "parity-failure.json", mismatches)
raise RuntimeError(f"Numerical/counter parity failed: {mismatches}")
row = {"variant": variant, "run": label, "warmup": index < 0, "processWallSeconds": wall, "solveSeconds": result["solveSeconds"], "solveCpuSeconds": result["solveCpuSeconds"], "fullParity": True, "resultBytes": result_path.stat().st_size, "nfev": result["nfev"], "njev": result["njev"], "nlu": result["nlu"], "acceptedSteps": result["acceptedSteps"], "solverStarts": result["solverStarts"]}
if variant == "profiled":
profile = json.loads(profile_path.read_text())
counters = profile["cvodeCounters"]
checks = {"counterReadsSucceeded": profile["counterErrors"] == 0, "rhsCountMatches": counters["rhs"] + counters["linear_rhs"] == result["nfev"], "linearRhsEqualsJacobianCountTimesStates": counters["linear_rhs"] == result["njev"] * prepared["stateCount"], "rhsClockCountMatches": profile["scopes"]["integration"]["rhs"]["calls"] == result["nfev"]}
row["profile"] = profile
row["counterChecks"] = checks
if not checks["counterReadsSucceeded"] or not checks["rhsClockCountMatches"] or (result["method"] == "BDF" and not checks["rhsCountMatches"]):
write_json(run / "counter-failure.json", row)
raise RuntimeError(f"Unexpected profiling counters: {checks}")
rows.append(row)
write_json(run / "run.json", row)
print(f"{variant}/{label}: solve={row['solveSeconds']:.6f}s wall={wall:.6f}s parity=true", flush=True)
medians = {variant: {key: statistics.median(row[key] for row in rows if row["variant"] == variant and not row["warmup"]) for key in ("processWallSeconds", "solveSeconds", "solveCpuSeconds")} for variant in ("control", "profiled")}
overhead = {key: medians["profiled"][key] / medians["control"][key] - 1 for key in medians["control"]}
write_json(output / "summary.json", {"prepared": prepared, "runs": rows, "medians": medians, "instrumentationOverheadFraction": overhead, "allFullParity": True, "interpretation": "CVODE linear_rhs counts finite-difference RHS calls independently of model nfev. Its multiplication by stateCount is checked, not assumed. RHS time includes all model work; no Jacobian-specific RHS time is inferred. cvode_step.exclusiveSeconds excludes nested RHS, polling and Dense ops; integration.exclusiveSeconds is outer setup/loop/cleanup work. Dense setup covers the original factorization operation; dense_solve covers the original triangular solve operation. Clock/bookkeeping costs remain in all measured totals. Production control may retain its pre-existing sparse main clocks; cachedMainDifference records this."})
print(f"Summary: {output / 'summary.json'}", flush=True)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--cache-dir", required=True, type=Path)
parser.add_argument("--request-stages", type=Path)
parser.add_argument("--output-dir", required=True, type=Path)
parser.add_argument("--run", action="store_true", help="Build and run serial warmups/repeats; default only prepares")
parser.add_argument("--warmups", type=int, default=1)
parser.add_argument("--repeats", type=int, default=3)
parser.add_argument("--process-timeout", type=float, default=360)
args = parser.parse_args()
if args.warmups < 0 or args.repeats < 1:
parser.error("warmups must be nonnegative and repeats positive")
prepared = prepare(args)
print(f"Prepared: {args.output_dir.resolve() / 'prepared.json'}", flush=True)
if args.run:
execute(args, prepared)
if __name__ == "__main__":
main()
tests/manual/profile_pipe_iterations.py
+440
View File
@@ -0,0 +1,440 @@
"""Instrument isolated pipe solvers; these runs are diagnostics, never benchmarks.
Prepare: python tests/manual/profile_pipe_iterations.py --output-dir test/pipe-profile --prepare-only
Run the prepared diagnostic programs: use the same command without --prepare-only.
All mutations except this test helper stay below the ignored output directory.
The replay contains EVERY resistance-law input from the guarded solver's RHS
trajectory, including its scalar low-Re analytic branch. Cache hits, zero dp,
and the separate PNL00R analytic law are counted but do not enter this replay.
"""
from __future__ import annotations
import argparse
from hashlib import sha256
import json
import os
from pathlib import Path
import subprocess
import sys
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT))
from app.main import compile_system_xml_network
from app.simulation.backends import simulation_config
from app.simulation.native_codegen import build as builder
from app.simulation.native_codegen.compiler import compile_native_program
from app.simulation.native_codegen.input import load_input
VARIANTS = ('guarded-newton', 'previous-newton', 'fixed-point')
FIELDS = '''cache_requests cache_hits cache_misses flow_calls zero_pressure_calls
pnl00r_analytic_calls resistance_calls scalar_analytic_calls iterative_calls
iterations_total iterations_max reached_last_iteration exhausted_limit
algorithm_converged failed_before_iteration finite_returns nonfinite_returns
bisections residual_pass residual_fail residual_nonfinite
residual_fail_after_algorithm_converged residual_pass_after_limit
invalid_inputs upper_bracket_evaluations upper_bracket_exhausted
bisection_after_poor_progress bisection_invalid_or_outside float_stagnation
'''.split()
PROFILE_PREFIX = r'''
/* Test-only instrumentation, injected in an isolated source tree. */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#define PROFILE_VARIANT "@VARIANT@"
#define PROFILE_FIXED @FIXED@
#define PROFILE_LIMIT(kind) @LIMIT@
#define PROFILE_FIELDS(X) @FIELDS@
typedef struct {
#define PROFILE_DECLARE(name) unsigned long long name;
PROFILE_FIELDS(PROFILE_DECLARE)
#undef PROFILE_DECLARE
unsigned long long histogram[129];
double maximum_relative_residual;
} PipeProfile;
static PipeProfile profile_stats[2][4];
static unsigned long long profile_rhs_calls,profile_capture_records;
static int profile_in_rhs,profile_kind,profile_last_exhausted;
static FILE *profile_capture;
static PipeProfile *profile_bucket(int kind) {
if(kind<0 || kind>3){fprintf(stderr,"Unexpected pipe kind %d\n",kind);exit(71);}
return &profile_stats[profile_in_rhs?0:1][kind];
}
void pipe_profile_rhs_enter(void){profile_in_rhs=1;profile_rhs_calls++;}
void pipe_profile_rhs_leave(void){profile_in_rhs=0;}
static void profile_add(PipeProfile *total,const PipeProfile *value) {
#define PROFILE_ADD(name) total->name+=value->name;
PROFILE_FIELDS(PROFILE_ADD)
#undef PROFILE_ADD
if(value->iterations_max>total->iterations_max)total->iterations_max=value->iterations_max;
for(int i=0;i<129;i++)total->histogram[i]+=value->histogram[i];
if(value->maximum_relative_residual>total->maximum_relative_residual)
total->maximum_relative_residual=value->maximum_relative_residual;
}
static void profile_write_bucket(FILE *f,const PipeProfile *value) {
fprintf(f,"{");
#define PROFILE_WRITE(name) fprintf(f,"\"" #name "\":%llu,",value->name);
PROFILE_FIELDS(PROFILE_WRITE)
#undef PROFILE_WRITE
fprintf(f,"\"maximum_relative_residual\":%.17g,\"iteration_histogram\":{",value->maximum_relative_residual);
int comma=0;
for(int i=0;i<129;i++)if(value->histogram[i]) {
fprintf(f,"%s\"%d\":%llu",comma?",":"",i,value->histogram[i]);comma=1;
}
fprintf(f,"}}");
}
static void profile_dump(void) {
const char *path=getenv("PIPE_PROFILE_JSON");
if(profile_capture){if(fclose(profile_capture))exit(73);profile_capture=NULL;}
if(!path)return;
FILE *f=fopen(path,"wb");if(!f){perror(path);exit(73);}
fprintf(f,"{\"variant\":\"%s\",\"rhs_calls\":%llu,\"capture_records\":%llu,",PROFILE_VARIANT,profile_rhs_calls,profile_capture_records);
for(int scope=0;scope<2;scope++) {
PipeProfile total={0};
for(int kind=0;kind<4;kind++)profile_add(&total,&profile_stats[scope][kind]);
/* iterations_max is a maximum, unlike the additive counters. */
total.iterations_max=0;
for(int kind=0;kind<4;kind++)if(profile_stats[scope][kind].iterations_max>total.iterations_max)
total.iterations_max=profile_stats[scope][kind].iterations_max;
fprintf(f,"%s\"%s\":{\"total\":",scope?",":"",scope?"non_rhs":"rhs");
profile_write_bucket(f,&total);fprintf(f,",\"by_kind\":{");
for(int kind=0;kind<4;kind++) {
fprintf(f,"%s\"%d\":",kind?",":"",kind);profile_write_bucket(f,&profile_stats[scope][kind]);
}
fprintf(f,"}}");
}
fprintf(f,"}\n");if(fclose(f))exit(73);
}
void pipe_profile_install(void) {
const char *path=getenv("PIPE_PROFILE_CAPTURE");
if(path){profile_capture=fopen(path,"wb");if(!profile_capture){perror(path);exit(73);}}
if(atexit(profile_dump)){fprintf(stderr,"Cannot register profile writer\n");exit(73);}
}
static void profile_save_input(double base,double d,double length,double rr,double den,int kind) {
if(profile_in_rhs && profile_capture) {
/* Six IEEE doubles, native endian; no sampling or deduplication. */
double input[]={base,d,length,rr,den,(double)kind};
if(fwrite(input,sizeof(input),1,profile_capture)!=1){perror("capture");exit(73);}
profile_capture_records++;
}
}
'''
PROFILE_SOLVE = r'''
/* Independently factored Darcy law: no production slope or solver status is
consulted. Long double reduces rounding noise in the returned-q residual. */
static long double profile_reference_friction(long double re,long double rr) {
if(!(re>0))return NAN;
long double laminar=64/re;
if(re<=89.96829989L)return laminar;
long double smooth=powl(-1.8L*log10l(6.9L/re),-2),turbulent=smooth;
if(rr>0) {
long double fully_rough=powl(-2*log10l(rr/3.7L),-2);
long double weight=1/(1+powl(180/(re*rr),2));
turbulent=(1-weight)*smooth+weight*fully_rough;
}
long double blend=powl((re-89.96829989L)/2741.96700831L,8.37293695L);
return (laminar+blend*turbulent)/(1+blend);
}
static void profile_returned_residual(PipeProfile *s,double q,double den,double K,double rr,
int converged,int exhausted) {
if(!isfinite(q)){s->nonfinite_returns++;s->residual_nonfinite++;return;}
s->finite_returns++;
long double re=fabsl((long double)q/(den/4)),residual;
if(K==0 && q==0)residual=0;
else residual=fabsl(re*re*profile_reference_friction(re,rr)/K-1);
if(!isfinite(residual)){s->residual_nonfinite++;return;}
if(residual>s->maximum_relative_residual)s->maximum_relative_residual=(double)residual;
if(residual<=1e-9L){s->residual_pass++;if(exhausted)s->residual_pass_after_limit++;}
else {s->residual_fail++;if(converged)s->residual_fail_after_algorithm_converged++;}
}
static double profile_solve(double base,double d,double length,double rr,double den,int kind) {
PipeProfile *s=profile_bucket(kind);
double K=pow(4*base/den,2)*d/length;
profile_save_input(base,d,length,rr,den,kind);
s->resistance_calls++;
if(!(K>=0 && rr>=0 && den/4>0) || !isfinite(K) || !isfinite(rr) || !isfinite(den/4))s->invalid_inputs++;
int iterations=0,converged=0,bisections=0,exhausted=0;
double q;
profile_kind=kind;profile_last_exhausted=0;
#if PROFILE_FIXED
double rough_limit=pipe_rough_limit(rr);
q=sqrt(d/(length*.02))*base;
for(int i=0;i<PROFILE_LIMIT(kind);i++) {
iterations=i+1;
double next=sqrt(d/(length*pipe_friction_prepared(4*fabs(q)/den,rr,rough_limit)))*base;
if(fabs(next-q)<=fmax(1e-12,fabs(q)*1e-9)){q=next;converged=1;break;}
q=.5*(q+next);
}
exhausted=!converged;
#else
NativePipeSolve status;
q=native_pipe_resistance(K,rr,den/4,&status)*den/4;
iterations=status.iterations;converged=status.converged;bisections=status.bisections;
exhausted=profile_last_exhausted;
#endif
if(iterations<0 || iterations>128){fprintf(stderr,"Unexpected iteration count\n");exit(71);}
s->histogram[iterations]++;s->iterations_total+=(unsigned)iterations;s->bisections+=(unsigned)bisections;
if((unsigned)iterations>s->iterations_max)s->iterations_max=(unsigned)iterations;
if(iterations)s->iterative_calls++;
else if(converged)s->scalar_analytic_calls++;
else s->failed_before_iteration++;
if(iterations==PROFILE_LIMIT(kind))s->reached_last_iteration++;
if(exhausted)s->exhausted_limit++;
if(converged)s->algorithm_converged++;
profile_returned_residual(s,q,den,K,rr,converged,exhausted);
return q;
}
'''
REPLAY_MAIN = r'''
#include "native/components/kernels.c"
int main(int argc,char **argv) {
if(argc!=2)return 64;
pipe_profile_install();profile_in_rhs=1;
FILE *f=fopen(argv[1],"rb");if(!f){perror(argv[1]);return 73;}
double input[6];size_t count;
while((count=fread(input,1,sizeof(input),f))==sizeof(input)) {
profile_solve(input[0],input[1],input[2],input[3],input[4],(int)input[5]);
}
int failed=count || ferror(f);fclose(f);return failed?74:0;
}
'''
def replace_once(source: str, old: str, new: str) -> str:
if source.count(old) != 1:
raise ValueError(f'Expected exactly one audited source fragment: {old[:100]!r}')
return source.replace(old, new, 1)
def instrument(source: str, variant: str) -> str:
fixed = variant == 'fixed-point'
limit = '(kind==0?64:16)' if fixed else '80' if variant == 'previous-newton' else '128'
prefix = PROFILE_PREFIX.replace('@VARIANT@', variant).replace('@FIXED@', str(int(fixed)))
prefix = prefix.replace('@LIMIT@', limit).replace('@FIELDS@', ' '.join(f'X({name})' for name in FIELDS))
source = replace_once(source, '#include <stddef.h>', '#include <stddef.h>\n' + prefix)
# Exhaustion is marked at the actual loop fall-through, not inferred from
# visiting the last allowed iteration (which can still converge).
start = source.index('double native_pipe_resistance(')
end = source.index('double native_pipe_flow(', start)
resistance = source[start:end]
ending = ' return NAN;\n}\n'
if not resistance.endswith(ending):
raise ValueError('Unexpected resistance function ending')
resistance = resistance[:-len(ending)] + ' profile_last_exhausted=1;return NAN;\n}\n'
if variant == 'guarded-newton':
resistance = replace_once(resistance, ' double value=hi*hi*pipe_friction_prepared(hi,rr,rough);',
' profile_bucket(profile_kind)->upper_bracket_evaluations++;\n double value=hi*hi*pipe_friction_prepared(hi,rr,rough);')
resistance = replace_once(resistance, ' if(!bracketed)return NAN;',
' if(!bracketed){profile_bucket(profile_kind)->upper_bracket_exhausted++;return NAN;}')
resistance = replace_once(resistance, ' if(bisect) {',
''' if(bisect) {
if(previous_newton && fabs(F)>.5*previous_residual)profile_bucket(profile_kind)->bisection_after_poor_progress++;
if(!(slope>0) || !isfinite(slope) || !isfinite(next) || next<=lo || next>=hi)
profile_bucket(profile_kind)->bisection_invalid_or_outside++;''')
resistance = replace_once(resistance, ' if(next<=lo || next>=hi) {',
' if(next<=lo || next>=hi) {\n profile_bucket(profile_kind)->float_stagnation++;')
elif variant == 'previous-newton':
# Count each evaluation of the original loop condition without changing
# its short-circuit behaviour or the original upper endpoint arithmetic.
resistance = replace_once(resistance, 'i<128 && hi*hi*pipe_friction_prepared(hi,rr,rough)<K',
'i<128 && (profile_bucket(profile_kind)->upper_bracket_evaluations++,hi*hi*pipe_friction_prepared(hi,rr,rough)<K)')
resistance = replace_once(resistance, 'status->bisections++;',
'status->bisections++;profile_bucket(profile_kind)->bisection_invalid_or_outside++;')
source = source[:start] + resistance + PROFILE_SOLVE + source[end:]
source = replace_once(source, ' if(fabs(p1-p2)<=1e-8) return 0;',
''' PipeProfile *profile=profile_bucket(kind);profile->flow_calls++;
if(fabs(p1-p2)<=1e-8){profile->zero_pressure_calls++;return 0;}''')
source = replace_once(source, ' if(4*lam/den<=1000) return sign*lam;',
' if(4*lam/den<=1000){profile->pnl00r_analytic_calls++;return sign*lam;}')
source = replace_once(source,
' double base=area*p*cm/sqrt(T),K=pow(4*base/den,2)*d/length;\n return sign*native_pipe_resistance(K,rr,den/4,NULL)*den/4;',
' double base=area*p*cm/sqrt(T);\n return sign*profile_solve(base,d,length,rr,den,kind);')
source = replace_once(source, ' if(cache->valid && cache->p1==p1 && cache->p2==p2 && cache->T==T &&',
' PipeProfile *profile=profile_bucket(kind);profile->cache_requests++;\n if(cache->valid && cache->p1==p1 && cache->p2==p2 && cache->T==T &&')
source = replace_once(source, ' return cache->flow;\n double result=properties?',
' {profile->cache_hits++;return cache->flow;}\n profile->cache_misses++;\n double result=properties?')
return source
def git(*arguments: str) -> str:
return subprocess.check_output(['git', *arguments], cwd=ROOT, text=True).strip()
def prepare(args, out: Path) -> dict:
if (out / 'prepared.json').exists():
metadata = json.loads((out / 'prepared.json').read_text())
if metadata['input_sha256'] != sha256(args.input.read_bytes()).hexdigest():
raise ValueError('Prepared model no longer matches the input file')
if metadata['script_sha256'] != sha256(Path(__file__).read_bytes()).hexdigest():
raise ValueError('Diagnostic helper changed; choose a fresh output directory')
return metadata
previous = git('rev-parse', args.previous_ref)
current = git('rev-parse', args.current_ref)
xml, doc = load_input(args.input)
config = simulation_config(doc.simulation)
if config.rtol != 1e-8 or config.t_start != 0 or config.t_stop != 10:
raise ValueError('This experiment requires API default rtol=1e-8 and 0–10 s model settings')
program = compile_native_program(compile_system_xml_network(doc))
out.mkdir(parents=True, exist_ok=True)
(out / 'input.xml').write_bytes(xml)
(out / 'input.json').write_bytes(args.input.read_bytes())
paths = git('ls-tree', '-r', '--name-only', current, 'native').splitlines()
metadata = dict(input=str(args.input.resolve()), input_sha256=sha256(args.input.read_bytes()).hexdigest(),
xml_sha256=sha256(xml).hexdigest(), script_sha256=sha256(Path(__file__).read_bytes()).hexdigest(),
previous_revision=previous, current_revision=current, settings=vars(config),
sample_step=doc.simulation.sample_step, variants={},
measurement_note='Instrumented times are diagnostic overhead and MUST NOT be used as production benchmark results.',
counter_scope='rhs counts native_rhs/model_eval calls including rejected trials and Jacobian differences; non_rhs includes initialization/output/probe evaluations.',
replay_scope='All guarded RHS resistance calls, including scalar analytic low-Re cases; no sampling or deduplication. Cache hits, zero pressure difference and direct PNL00R analytic calls are excluded and counted separately.',
capture_format='Native-endian IEEE-754 binary64 records: base, diameter, length, relative roughness, den=pi*d*mu, kind (six doubles, 48 bytes). Replay on the same host.',
residual_test='At actual returned q, independently factored long-double f(Re) evaluates abs(Re^2*f(Re)/K-1)<=1e-9. Separate from each algorithm stopping rule.')
original_native = builder.NATIVE
try:
for variant in VARIANTS:
directory = out / variant
revision = current if variant == 'guarded-newton' else previous
for name in paths:
target = directory / name
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(subprocess.check_output(['git', 'show', f'{revision}:{name}'], cwd=ROOT))
kernel = directory / 'native/components/kernels.c'
original_hash = sha256(kernel.read_bytes()).hexdigest()
kernel.write_text(instrument(kernel.read_text(), variant))
common = directory / 'native/runtime/common.c'
common.write_text(replace_once(common.read_text(), ' return model_eval(t,y,dy,w);',
''' extern void pipe_profile_rhs_enter(void),pipe_profile_rhs_leave(void);
pipe_profile_rhs_enter();int ok=model_eval(t,y,dy,w);pipe_profile_rhs_leave();return ok;'''))
main = directory / 'native/runtime/main.c'
main.write_text(replace_once(main.read_text(), 'int main(int argc, char **argv) {',
'int main(int argc, char **argv) {\n extern void pipe_profile_install(void);pipe_profile_install();'))
builder.NATIVE = directory / 'native'
build = builder.build_native(program, cache_dir=out / 'cache')
replay_source = directory / 'replay.c'
replay_source.write_text(REPLAY_MAIN)
replay = directory / ('replay.exe' if os.name == 'nt' else 'replay')
compiler, _, _ = builder.toolchain()
command = [compiler, '-std=c11', '-O3', '-Wall', '-Wextra', '-Werror', '-ffp-contract=off',
'-fno-fast-math', '-I', str(directory / 'native/include'), str(replay_source), '-lm', '-o', str(replay)]
compiled = subprocess.run(command, capture_output=True, text=True, timeout=60)
(directory / 'replay-build.log').write_text(compiled.stdout + compiled.stderr)
if compiled.returncode:
raise RuntimeError(f'Replay compilation failed: {compiled.stderr}')
metadata['variants'][variant] = dict(executable=str(build.executable), replay=str(replay),
build_key=build.manifest['buildKey'], original_kernel_sha256=original_hash,
instrumented_kernel_sha256=sha256(kernel.read_bytes()).hexdigest())
print(f'Prepared {variant}', flush=True)
finally:
builder.NATIVE = original_native
(out / 'prepared.json').write_text(json.dumps(metadata, ensure_ascii=False, indent=2) + '\n')
return metadata
def load_stats(path: Path) -> dict:
stats = json.loads(path.read_text())
for scope in ('rhs', 'non_rhs'):
for row in [stats[scope]['total'], *stats[scope]['by_kind'].values()]:
assert row['cache_requests'] == row['cache_hits'] + row['cache_misses']
assert row['flow_calls'] == row['zero_pressure_calls'] + row['pnl00r_analytic_calls'] + row['resistance_calls'] or not row['flow_calls']
assert row['resistance_calls'] == row['iterative_calls'] + row['scalar_analytic_calls'] + row['failed_before_iteration']
assert row['resistance_calls'] == sum(row['iteration_histogram'].values())
assert row['iterations_total'] == sum(int(k) * v for k, v in row['iteration_histogram'].items())
assert row['resistance_calls'] == row['finite_returns'] + row['nonfinite_returns']
assert row['resistance_calls'] == row['residual_pass'] + row['residual_fail'] + row['residual_nonfinite']
assert row['exhausted_limit'] <= row['reached_last_iteration']
return stats
def run(metadata: dict, out: Path, timeout: float):
summary_path = out / 'summary.json'
if summary_path.exists():
raise ValueError('Diagnostic results already exist; choose a fresh output directory')
settings = metadata['settings']
capture = out / 'guarded-rhs-resistance-inputs.bin'
rows = {}
for variant in VARIANTS:
directory = out / variant / 'trajectory'
directory.mkdir(parents=True, exist_ok=False)
result_path = directory / 'result.json'
stats_path = directory / 'profile.json'
env = os.environ.copy()
env.pop('PIPE_PROFILE_CAPTURE', None)
env['PIPE_PROFILE_JSON'] = str(stats_path)
if variant == 'guarded-newton':
env['PIPE_PROFILE_CAPTURE'] = str(capture)
executable = metadata['variants'][variant]['executable']
command = [executable, '--method', settings['method'], '--start', str(settings['t_start']),
'--stop', str(settings['t_stop']), '--sample-step', str(metadata['sample_step']),
'--max-step', str(settings['max_step']), '--rtol', str(settings['rtol']),
'--timeout', str(timeout), '--output', str(result_path)]
print(f'Starting diagnostic trajectory: {variant}', flush=True)
with (directory / 'worker.log').open('w') as log:
try:
process = subprocess.run(command, cwd=Path(executable).parent, env=env,
stdout=subprocess.DEVNULL, stderr=log, timeout=timeout+15)
exit_code = process.returncode
except subprocess.TimeoutExpired:
exit_code = 'external-timeout'
data = json.loads(result_path.read_text()) if result_path.exists() else {}
# Deliberately omit measured times from the cross-variant summary.
trajectory = {key:data.get(key) for key in ('success', 'status', 'message', 'simulatedUntil',
'nfev', 'acceptedSteps', 'rejectedSteps', 'njev', 'nlu', 'stateTransitions')}
trajectory['exit_code'] = exit_code
rows[variant] = dict(trajectory=trajectory, profile=load_stats(stats_path) if stats_path.exists() else None)
print(json.dumps(dict(variant=variant, **trajectory), ensure_ascii=False), flush=True)
if capture.stat().st_size % 48:
raise ValueError('Truncated replay capture')
count = capture.stat().st_size // 48
guarded = rows['guarded-newton']['profile']
if guarded is None or count != guarded['rhs']['total']['resistance_calls'] or count != guarded['capture_records']:
raise ValueError('Capture does not contain every guarded RHS resistance call')
for variant in VARIANTS:
directory = out / variant / 'replay-results'
directory.mkdir(exist_ok=False)
stats_path = directory / 'profile.json'
env = os.environ.copy()
env.pop('PIPE_PROFILE_CAPTURE', None)
env['PIPE_PROFILE_JSON'] = str(stats_path)
print(f'Replaying all {count} common inputs: {variant}', flush=True)
with (directory / 'worker.log').open('w') as log:
subprocess.run([metadata['variants'][variant]['replay'], str(capture)], env=env,
stdout=subprocess.DEVNULL, stderr=log, timeout=300, check=True)
replay = load_stats(stats_path)
if replay['rhs']['total']['resistance_calls'] != count:
raise ValueError('Replay count mismatch')
rows[variant]['replay'] = replay
# Identical input replay must reproduce every resistance-solve counter for
# the guarded solver, excluding cache/flow bookkeeping performed upstream.
exempt = {'cache_requests', 'cache_hits', 'cache_misses', 'flow_calls', 'zero_pressure_calls', 'pnl00r_analytic_calls'}
for kind in ('total', '0', '1', '2', '3'):
actual = guarded['rhs']['total'] if kind == 'total' else guarded['rhs']['by_kind'][kind]
replay = rows['guarded-newton']['replay']['rhs']['total'] if kind == 'total' else rows['guarded-newton']['replay']['rhs']['by_kind'][kind]
assert {k:v for k,v in actual.items() if k not in exempt} == {k:v for k,v in replay.items() if k not in exempt}
summary = dict(metadata=metadata, capture=dict(path=str(capture), records=count, bytes=capture.stat().st_size,
sha256=sha256(capture.read_bytes()).hexdigest(), sampling='none'), variants=rows)
summary_path.write_text(json.dumps(summary, ensure_ascii=False, indent=2) + '\n')
print(f'Diagnostic profile complete: {summary_path}', flush=True)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--input', type=Path, default=ROOT / 'tests/data/test-mql-8-corrected.json')
parser.add_argument('--output-dir', type=Path, required=True)
parser.add_argument('--previous-ref', default='5d5a2e1')
parser.add_argument('--current-ref', default='808c484')
parser.add_argument('--timeout', type=float, default=120)
parser.add_argument('--prepare-only', action='store_true')
args = parser.parse_args()
out = args.output_dir.resolve()
# Keep diagnostic native copies out of production sources and tracked data.
if not out.is_relative_to(ROOT / 'test'):
parser.error('--output-dir must be below the ignored repository test/ directory')
metadata = prepare(args, out)
if not args.prepare_only:
run(metadata, out, args.timeout)
if __name__ == '__main__':
main()
tests/manual/summarize_native_encoding.py
+342
View File
@@ -0,0 +1,342 @@
"""Summarize the four complete C-result-encoding browser groups.
.venv/bin/python tests/manual/summarize_native_encoding.py \
--root test/c-result-encoding-20260911
Reads small timing metadata only, never result arrays or CSV contents. Every
browser group must contain one warmup and three measured successful runs.
Missing/inconsistent evidence exits 2 and writes complete:false plus an empty
CSV, so an earlier successful summary cannot masquerade as current evidence.
"""
from __future__ import annotations
import argparse
import csv
import hashlib
import json
import math
from pathlib import Path
from statistics import median
import sys
from typing import Any
REPO = Path(__file__).resolve().parents[2]
GROUPS = {
"baseline": ("control", None),
"baseline-profiled": ("profiled", "baseline-source/profiled-backend/requests"),
"optimized": ("control", None),
"optimized-profiled": ("profiled", "backend-optimized-profiled/requests"),
}
GOALS = {
"ready": "clickToReadyDomMs",
"saved_observed": "clickToIndexedDbObservedMs",
"csv_download_saved": "csvClickToDownloadSavedMs",
}
COUNTERS = ("nfev", "acceptedSteps", "rejectedSteps", "stateTransitions", "solverStarts", "njev", "nlu")
NATIVE_IDENTITY = ("backend", "method", "solver", "sundialsVersion", "simulatedUntil", "maxAcceptedStep", *COUNTERS)
C_WALL = ("argumentPreparationSeconds", "initializationSeconds", "integrationSeconds",
"finalSampleAndStatusSeconds", "projectionSeconds", "jsonWriteSeconds", "mainTotalSeconds")
CSV_FIELDS = ("group", "phase", "run", "simulationId", "domain", "metric", "statistic", "value", "unit",
"parent", "denominatorValue", "percentOfParent", "baselineValue", "optimizedValue",
"count", "missingCount", "inclusion", "source")
DEFINITIONS = {
"scope": "One C-result-encoding experiment. Baseline/optimized control groups alone provide end-to-end comparisons; profiled C-write comparisons are separate diagnostics.",
"statistics": "Run values precede median/min/max. Stage/parent percentages use each run's own denominator before aggregation. Before/after changes use the ratio of independently collected group medians; medians and overlapping stages must not be added.",
"comparison": "Groups were collected separately, not as alternating paired trials. Duration reduction = (baseline median - optimized median) / baseline median; speedup = baseline median / optimized median. Run ordinals are not matched pairs. Ordering, scheduling and thermal variability remain possible.",
"ready": "Click to DOM-observed successful completion and an enabled Run button, not GPU completion.",
"saved": "End-to-end comparisons use pointer polling observation in BOTH control groups; includes polling and scheduling latency. Exact instrumented pointer publication remains a separate profiled metric.",
"csv": "Click through Playwright download notification and saveAs completion; includes automation and filesystem work.",
"backend": "Backend spans are inclusive wall intervals; children are included in their parents. ASGI send awaits are not pure network time. Response serialization includes metadata encoding and raw numeric-fragment joining.",
"cWrite": "C output write includes numeric encoding, stdio writes, close and index writing. CPU and wall are distinct observations; their difference is not an isolated disk-I/O measurement.",
"cSolve": "Integration includes CVODE setup, RHS/Jacobian/linear work, events and sampling. Counts are not CPU-time shares. Projection is outside integration and inside C main.",
"process": "Native reported processWallSeconds includes Python result reading after child exit; observed process lifetime spans include spawn and exit-observation latency.",
"overlap": "Browser reads overlap backend work. Parse/decode lie inside reception; persistence and rendering overlap. Per-stage percentages are inclusive and must not be added.",
"warmup": "Warmup rows are retained separately. Formal browser runs require build-cache hits. CacheHit, not the warmup label, identifies cold compilation.",
"replay": "Microbenchmark replays preloaded contiguous binary64 values, excluding model projection and production strided access. Its medians are separate and never substituted for end-to-end results.",
"validation": "Timing metadata validates group completeness, input/assets, counters, sample/variable counts and recorded success/restore flags. It does not independently prove numerical bitwise parity; use the separate full-result comparison artifact.",
}
def numeric(value: Any) -> bool:
return type(value) in (int, float) and math.isfinite(value)
def require(condition: bool, message: str) -> None:
if not condition:
raise ValueError(message)
def stats(values: list[Any]) -> dict:
present = [v for v in values if numeric(v)]
return {"count": len(present), "missingCount": len(values) - len(present), "values": values,
"median": median(present) if present else None,
"min": min(present) if present else None, "max": max(present) if present else None}
def percent(value: Any, denominator: Any) -> float | None:
return value / denominator * 100 if numeric(value) and numeric(denominator) and denominator > 0 else None
def finite_field(data: dict, name: str, context: str, *, positive: bool = False) -> float:
value = data.get(name)
require(numeric(value) and (value > 0 if positive else value >= 0), f"{context}: missing/invalid {name}")
return value
class Summarizer:
def __init__(self, root: Path):
self.root = root.resolve()
self.sources: dict[str, dict] = {}
self.observations: list[dict] = []
self.simulation_ids: set[str] = set()
def read(self, relative: str) -> dict:
path = (self.root / relative).resolve()
require(path.is_relative_to(self.root), f"Metadata path leaves experiment root: {relative}")
require(path.is_file(), f"Incomplete experiment: missing {relative}")
require(path.stat().st_size <= 4 * 1024 * 1024, f"Refusing large non-metadata input: {relative}")
raw = path.read_bytes()
value = json.loads(raw, parse_constant=lambda token: (_ for _ in ()).throw(ValueError(f"Invalid JSON number: {token}")))
require(isinstance(value, dict), f"Expected metadata object: {relative}")
self.sources[relative] = {"bytes": len(raw), "sha256": hashlib.sha256(raw).hexdigest()}
return value
def observe(self, run: dict, domain: str, metric: str, value: Any, unit: str = "ms", *,
parent: str = "", denominator: Any = None, inclusion: str = "inclusive/overlapping; not additive",
source: str = "", baseline: Any = None, optimized: Any = None) -> None:
require(value is None or numeric(value), f"Invalid observation {domain}.{metric}: {value!r}")
self.observations.append({k: run[k] for k in ("group", "phase", "run", "simulationId")} | {
"domain": domain, "metric": metric, "value": value, "unit": unit, "parent": parent,
"denominatorValue": denominator, "percentOfParent": percent(value, denominator),
"baselineValue": baseline, "optimizedValue": optimized, "inclusion": inclusion, "source": source})
def backend(self, run: dict, relative: str) -> dict:
data = self.read(relative)
context = f"{run['group']}/{run['run']} backend"
require(data.get("id") == run["simulationId"], f"{context}: simulationId mismatch")
require(data.get("httpStatus") == 200, f"{context}: HTTP did not succeed")
native = run["native"]
for key in (*NATIVE_IDENTITY, "solveSeconds", "solveCpuSeconds", "buildKey", "cacheHit"):
require(data.get("native", {}).get(key) == native.get(key), f"{context}: browser/backend mismatch for {key}")
require(data.get("sampleCount") == run["sampleCount"], f"{context}: backend sampleCount mismatch")
http = finite_field(data, "httpTotalSeconds", context, positive=True) * 1000
self.observe(run, "backend", "httpTotalMs", http, source=relative, inclusion=DEFINITIONS["backend"])
spans = data.get("spans", [])
require(bool(spans), f"{context}: missing backend spans")
totals: dict[str, float] = {}
for span in spans:
start = finite_field(span, "startMs", context)
end = finite_field(span, "endMs", context)
require(start <= end <= http + 1e-5, f"{context}: span outside HTTP interval: {span['name']}")
totals[span["name"]] = totals.get(span["name"], 0) + end - start
for name in ("native_indexed_result_read", "native_process_lifetime_observed", "response_result_json_serialization"):
require(name in totals, f"{context}: missing {name}")
for name, value in totals.items():
self.observe(run, "backend_span", name, value, parent="httpTotalMs", denominator=http,
inclusion=DEFINITIONS["backend"], source=relative)
c = data.get("nativeStages", {})
main_ms = finite_field(c, "mainTotalSeconds", context, positive=True) * 1000
for name in C_WALL:
self.observe(run, "c_wall", name.removesuffix("Seconds") + "Ms", finite_field(c, name, context) * 1000,
parent="cMainMs" if name != "mainTotalSeconds" else "",
denominator=main_ms if name != "mainTotalSeconds" else None,
inclusion=DEFINITIONS["cWrite"] if name == "jsonWriteSeconds" else DEFINITIONS["cSolve"], source=relative)
for name in ("projectionCpuSeconds", "jsonWriteCpuSeconds"):
self.observe(run, "c_cpu", name.removesuffix("Seconds") + "Ms", finite_field(c, name, context) * 1000,
inclusion="CPU duration; separate from wall intervals", source=relative)
self.observe(run, "backend", "responseSendAwaitMs", finite_field(data, "responseSendAwaitSeconds", context) * 1000,
parent="httpTotalMs", denominator=http, inclusion=DEFINITIONS["backend"], source=relative)
for name in ("rawSeriesBytes", "responseBodyBytes"):
self.observe(run, "size", name, finite_field(data, name, context, positive=True), "bytes", source=relative)
phases = data.get("existingPerformance", {}).get("phases", {})
for name, phase in phases.items():
self.observe(run, "backend_existing", name, finite_field(phase, "inclusiveNs", context) / 1e6,
parent="httpTotalMs", denominator=http,
inclusion="Inclusive duration without aligned start/end; not an exclusive extra cost", source=relative)
return {"source": relative, "xmlSha256": data.get("xmlSha256"), "httpTotalMs": http,
"spans": spans, "nativeStages": c, "process": data.get("process"), "build": data.get("build")}
def group(self, name: str, mode: str, backend_root: str | None) -> dict:
relative = f"browser-{name}/summary.json"
data = self.read(relative)
require(data.get("errors") == [], f"{name}: missing errors list or reported browser errors")
rows = data.get("rows", [])
require(len(rows) == 4, f"Incomplete {name}: expected 1 warmup + 3 measured rows, got {len(rows)}")
require(sorted(r.get("run", -1) for r in rows) == [0, 1, 2, 3], f"{name}: unexpected/duplicate run numbers")
runs = []
for row in sorted(rows, key=lambda r: r["run"]):
context = f"{name}/{row['run']}"
require(row.get("mode") == mode and row.get("deep") is False, f"{context}: wrong instrumentation mode")
require(row.get("warmup") is (row["run"] == 0), f"{context}: warmup label mismatch")
sid = row.get("simulationId")
require(isinstance(sid, str) and bool(sid) and sid not in self.simulation_ids, f"{context}: missing/duplicate simulationId")
self.simulation_ids.add(sid)
native = row.get("native", {})
require(native.get("success") is True and native.get("status") == "completed", f"{context}: native simulation failed")
require(row.get("restoredIdentical") is True, f"{context}: restore parity was not confirmed")
require(type(native.get("cacheHit")) is bool, f"{context}: missing cacheHit")
if row["run"]:
require(native["cacheHit"], f"{context}: measured run includes a cold build")
for key in NATIVE_IDENTITY:
require(key in native and native[key] is not None, f"{context}: missing native {key}")
for key in COUNTERS:
value = finite_field(native, key, context)
require(int(value) == value, f"{context}: noninteger counter {key}")
run = {"group": name, "phase": "warmup" if row["warmup"] else "measured", "run": row["run"],
"simulationId": sid, "native": native, "original": row,
"sampleCount": finite_field(row, "sampleCount", context, positive=True),
"variableCount": finite_field(row, "variableCount", context, positive=True)}
for key in GOALS.values():
finite_field(row, key, context, positive=True)
if mode == "profiled":
for key in ("resultParseMs", "synchronousStreamDecodeMs", "clickToIndexedDbCommitMs", "streamBytes"):
finite_field(row, key, context, positive=True)
for key, value in row.items():
if key.endswith(("Ms", "Bytes")) and (value is None or numeric(value)):
self.observe(run, "frontend", key, value, "bytes" if key.endswith("Bytes") else "ms", source=relative)
for key in ("solveSeconds", "solveCpuSeconds", "processWallSeconds", "buildSeconds"):
self.observe(run, "native", key.removesuffix("Seconds") + "Ms", finite_field(native, key, context) * 1000,
source=relative, inclusion=DEFINITIONS["process"] if key == "processWallSeconds" else "Native-reported timing; CPU and wall are separate")
run["backend"] = self.backend(run, f"{backend_root}/{sid}/stages.json") if backend_root else None
runs.append(run)
for key in ("inputSha256", "buildAssetSetSha256"):
value = data.get(key)
require(isinstance(value, str) and len(value) == 64 and all(c in "0123456789abcdef" for c in value), f"{name}: missing/invalid {key}")
require(bool(data.get("servedAssets")), f"{name}: missing served frontend assets")
return {"source": relative, "mode": mode, "inputSha256": data["inputSha256"],
"buildAssetSetSha256": data["buildAssetSetSha256"], "servedAssets": data["servedAssets"],
"browser": data.get("browser"), "node": data.get("node"), "scriptSha256": data.get("scriptSha256"),
"sourceDefinitions": data.get("definitions"), "measuredRunCount": 3, "warmupRunCount": 1, "runs": runs}
def comparisons(self, groups: dict, before: str, after: str, metrics: dict, *, diagnostic: bool) -> list[dict]:
comparison = []
for target, (source_domain, metric) in metrics.items():
summaries = []
for name in (before, after):
rows = [o for o in self.observations if o["group"] == name and o["phase"] == "measured"
and o["domain"] == source_domain and o["metric"] == metric]
require(len(rows) == 3 and all(numeric(r["value"]) and r["value"] > 0 for r in rows),
f"Missing comparison metric {name}/{target}")
summaries.append(stats([r["value"] for r in sorted(rows, key=lambda r: r["run"])]))
baseline, optimized = summaries
old, new = baseline["median"], optimized["median"]
comparison.append({"target": target, "metric": f"{source_domain}.{metric}", "before": before, "after": after,
"diagnosticOnly": diagnostic, "statistic": "ratio_of_group_medians", "definition": DEFINITIONS["comparison"],
"source": f"{groups[before]['source']} | {groups[after]['source']}",
"baselineMs": baseline, "optimizedMs": optimized, "savedMs": old - new,
"durationReductionPercent": (old - new) / old * 100, "speedupRatio": old / new})
return comparison
def replay(self) -> dict:
relative = "replay/summary.json"
data = self.read(relative)
require(data.get("allRealFileBinary64Parity") is True, "Replay real-file binary64 verification not complete")
supplied = data.get("medians", {})
require(bool(supplied.get("file")), "Replay real-file medians missing")
runs = data.get("runs", [])
require(bool(runs), "Replay run metadata missing")
for row in runs:
require(row.get("success") is True, "Replay includes a failed run")
run = {"group": f"replay:{row['sink']}:{row['variant']}", "phase": "warmup" if row["warmup"] else "measured",
"run": row["run"], "simulationId": ""}
for metric, unit in (("wallSeconds", "ms"), ("cpuSeconds", "ms"), ("encodedBytes", "bytes")):
value = finite_field(row, metric, run["group"], positive=True)
self.observe(run, "replay", metric.removesuffix("Seconds") + "Ms" if unit == "ms" else metric,
value * 1000 if unit == "ms" else value, unit, source=relative, inclusion=DEFINITIONS["replay"])
medians = {sink: {variant: {k: value[k] for k in ("wallSeconds", "cpuSeconds", "encodedBytes")}
for variant, value in variants.items()} for sink, variants in supplied.items()}
return {"source": relative, "suppliedMedians": medians, "runs": runs,
"timingContract": data.get("prepared", {}).get("timingContract"), "limitation": data.get("limitation"),
"usedForEndToEndComparison": False}
def build(self) -> dict:
groups = {name: self.group(name, *config) for name, config in GROUPS.items()}
reference = groups["baseline"]
first = reference["runs"][0]
for name, group in groups.items():
for key in ("inputSha256", "buildAssetSetSha256", "browser", "node"):
require(group[key] is not None and group[key] == reference[key], f"Cross-group {key} mismatch: {name}")
for run in group["runs"]:
for key in ("sampleCount", "variableCount"):
require(run[key] == first[key], f"Cross-run {key} mismatch: {name}/{run['run']}")
for key in NATIVE_IDENTITY:
require(run["native"][key] == first["native"][key], f"Cross-run native {key} mismatch: {name}/{run['run']}")
xml_hashes = [run["backend"]["xmlSha256"] for group in groups.values() for run in group["runs"] if run["backend"]]
require(all(isinstance(value, str) and len(value) == 64 for value in xml_hashes) and len(set(xml_hashes)) == 1,
"Profiled input XML SHA missing or mismatched")
goals = self.comparisons(groups, "baseline", "optimized",
{key: ("frontend", metric) for key, metric in GOALS.items()}, diagnostic=False)
stages = self.comparisons(groups, "baseline-profiled", "optimized-profiled",
{"write_wall": ("c_wall", "jsonWriteMs"), "write_cpu": ("c_cpu", "jsonWriteCpuMs")}, diagnostic=True)
replay = self.replay()
buckets: dict[tuple, list[dict]] = {}
keys = ("group", "phase", "domain", "metric", "unit", "parent")
for observation in self.observations:
buckets.setdefault(tuple(observation[key] for key in keys), []).append(observation)
aggregates = []
for key, rows in sorted(buckets.items()):
require(len({r["run"] for r in rows}) == len(rows), f"Duplicate per-run observation: {key}")
aggregates.append(dict(zip(keys, key)) | stats([r["value"] for r in rows]) | {
"percentOfParent": stats([r["percentOfParent"] for r in rows]),
"runs": [{k: row[k] for k in ("run", "simulationId")} for row in rows]})
return {"schemaVersion": 1, "complete": True, "errors": [], "experimentRoot": str(self.root),
"scriptSha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), "definitions": DEFINITIONS,
"sourceFiles": self.sources, "groups": groups, "changes": {"endToEnd": goals, "cWriteDiagnostics": stages},
"replay": replay, "observations": self.observations, "aggregates": aggregates,
"validation": {"groupCount": 4, "browserRunCount": 16, "warmupCount": 4, "measuredCount": 12,
"inputSha256": reference["inputSha256"], "frontendAssetSetSha256": reference["buildAssetSetSha256"],
"profiledXmlSha256": xml_hashes[0], "sampleCount": first["sampleCount"], "variableCount": first["variableCount"],
"nativeIdentity": {key: first["native"][key] for key in NATIVE_IDENTITY},
"fullNumericalParityIndependentlyChecked": False}}
def write_outputs(root: Path, summary: dict) -> None:
root.mkdir(parents=True, exist_ok=True)
with (root / "timings.csv").open("w", encoding="utf-8", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=CSV_FIELDS)
writer.writeheader()
for row in summary.get("observations", []):
writer.writerow(row | {"statistic": "run", "count": int(numeric(row["value"])), "missingCount": int(row["value"] is None)})
for entry in summary.get("aggregates", []):
for statistic in ("median", "min", "max"):
writer.writerow({k: entry[k] for k in ("group", "phase", "domain", "metric", "unit", "parent", "count", "missingCount")} | {
"statistic": statistic, "value": entry[statistic], "percentOfParent": entry["percentOfParent"][statistic],
"inclusion": "Per-run values and percentages aggregated separately; medians are not additive"})
for category, changes in summary.get("changes", {}).items():
for change in changes:
for key, unit in (("durationReductionPercent", "percent"), ("speedupRatio", "ratio"), ("savedMs", "ms")):
writer.writerow({"group": f"{change['after']}-vs-{change['before']}", "phase": "measured",
"domain": f"comparison:{category}", "metric": f"{change['target']}.{key}",
"statistic": "ratio_of_group_medians" if key != "savedMs" else "difference_of_group_medians",
"value": change[key], "unit": unit, "baselineValue": change["baselineMs"]["median"],
"optimizedValue": change["optimizedMs"]["median"], "count": 3, "missingCount": 0,
"inclusion": ("Profiled C-write diagnostic only; " if change["diagnosticOnly"] else "") + change["definition"],
"source": change["source"]})
(root / "summary.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2, allow_nan=False) + "\n", encoding="utf-8")
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--root", type=Path, default=REPO / "test/c-result-encoding-20260911")
args = parser.parse_args()
root = args.root.resolve()
if not root.is_relative_to(REPO / "test"):
parser.error("--root must be beneath the repository's ignored test/ directory")
summarizer = Summarizer(root)
try:
summary = summarizer.build()
except (OSError, ValueError, KeyError, TypeError) as error:
summary = {"schemaVersion": 1, "complete": False, "errors": [str(error)],
"experimentRoot": str(root), "sourceFiles": summarizer.sources,
"note": "No partial timing statistics are published. Complete/fix all groups and rerun."}
write_outputs(root, summary)
print(json.dumps(summary, ensure_ascii=False, indent=2), file=sys.stderr)
return 2
write_outputs(root, summary)
print(json.dumps({"complete": True, "validation": summary["validation"],
"observations": len(summary["observations"]), "outputs": [str(root / name) for name in ("summary.json", "timings.csv")]}, ensure_ascii=False, indent=2))
return 0
if __name__ == "__main__":
raise SystemExit(main())
tests/manual/summarize_web_cost.py
+385
View File
@@ -0,0 +1,385 @@
"""Summarize one web-cost experiment without reading numerical result files.
Usage: .venv/bin/python tests/manual/summarize_web_cost.py --root test/web-cost-20260911
Only small summary/trace/stages/environment JSON files are read. Warmups and
control/profiled groups stay separate. This script does not compute speedups.
"""
from __future__ import annotations
import argparse
import csv
import hashlib
import json
import math
from pathlib import Path
from statistics import median
from typing import Any
REPO = Path(__file__).resolve().parents[2]
AXIS = (
("runClick", "click"), ("fetchStart", "fetch"), ("headers", "headers"),
("lastChunk", "last_chunk"), ("resultParseStart", "parse_start"),
("resultParseEnd", "parse_end"), ("streamEof", "eof"),
("resultReadyDom", "ready"), ("indexedDbCommittedPointer", "commit"),
)
# Prune the reporting tree at these declared boundaries. The indexed-read and
# process-lifetime spans are atomic ONLY in this coarse HTTP partition; their
# measured children remain visible in the separate inclusive span hierarchy.
HTTP_BOUNDARIES = {
"xml_validation", "network_compilation", "c_generation",
"native_build_or_cache_validation", "native_process_lifetime_observed",
"native_indexed_result_read", "profile_artifact_preservation",
"response_result_json_serialization",
}
C_WALL_FIELDS = (
"argumentPreparationSeconds", "initializationSeconds", "integrationSeconds",
"finalSampleAndStatusSeconds", "projectionSeconds", "jsonWriteSeconds",
)
DEFINITIONS = {
"scope": "One current experiment; control/profiled are observation modes, not before/after implementations. No speedup is computed.",
"statistics": "Median/min/max are calculated from individual runs. Percentages divide each run by its own stated denominator before aggregation. Medians need not add to a median total.",
"warmup": "All warmup rows are retained separately and excluded from measured statistics. Cold build is identified by cacheHit=false, not by the warmup label.",
"frontendClock": "Browser performance.now() within one trace timeOrigin. Reload/restore has another time axis; no timestamps are subtracted across documents or across browser/backend clocks.",
"waterfall": "Adjacent requested marks are subtracted without clipping negatives. Missing marks remain null; commit is the instrumented pointer publication, never replaced by the control polling mark.",
"waterfallPercent": "Each adjacent interval / same-run click-to-commit. Negative intervals remain negative and reveal overlapping completion order; they are not exclusive CPU costs.",
"backendClock": "Backend spans use request-relative perf_counter_ns. Parent links are inferred by interval containment and indicate inclusive wall intervals, not a traced call stack or exclusive CPU work.",
"httpPartition": "Pruned reporting leaves use HTTP_BOUNDARIES, checked for overlap and containment before partitioning. Their union is subtracted from HTTP total to produce unnamed other time. Spawn/read/parse children must not be added again. Response assembly has a duration but no aligned timestamp and remains in other.",
"backendOther": "Unclassified HTTP wall intervals include uninstrumented work, scheduling, response assembly, inter-stage gaps and sending. They are not all transport time or CPU work.",
"process": "Observed lifetime runs from Popen entry to the first existing poll/wait reporting exit; includes spawn and exit-observation delay. The result processWallSeconds also includes Python result reading. Child CPU is a RUSAGE_CHILDREN delta and assumes no unrelated child is reaped in the same parent interval.",
"cInitialization": "The separately measured initialization covers model_init and first sample. CVODE allocation/init/reinit and cleanup are inside integration, along with RHS, Jacobian/linear solve, events and sampling.",
"cOutput": "Projection includes final/sample model_eval and allocation. JSON write includes float formatting, stdio, file close and index writing. CPU and wall are separately reported; their difference is not a measured disk-I/O stage.",
"cParent": "C phase wall percentages use same-run C main wall; CPU percentages use the same-run observed child user+system CPU. Initial/final/remaining CPU is not individually measured.",
"stream": "Read-wait wall overlaps backend production/transport/browser scheduling. Decode and JSON.parse are within headers-to-EOF. NDJSON scanning/join/trim, callbacks, GC and scheduler time are not independently timed.",
"persistence": "Save preparation/IndexedDB run in the background with readiness and rendering. Transaction windows include asynchronous waiting; pointer commit is publication after successful writes, control observation uses polling.",
"renderExport": "Two requestAnimationFrame callbacks give a paint opportunity, not GPU completion. Download completion includes automation delivery and saveAs. CSV Worker transfer/preparation overlaps Worker activity; finish-post-to-receipt is not isolated Worker CPU.",
"unmeasured": "No exclusive breakdown of click preprocessing, NDJSON join/trim, React/GC, CVODE internals, C formatting versus file writes, browser network stack or GPU work is invented. Deeper/native-only experiments are intentionally not read here.",
}
def number(value: Any) -> bool:
return type(value) in (int, float) and math.isfinite(value)
def ratio(value: Any, denominator: Any) -> float | None:
return 100.0 * value / denominator if number(value) and number(denominator) and denominator > 0 else None
def difference(marks: dict, left: str, right: str) -> float | None:
a, b = marks.get(left), marks.get(right)
return b - a if number(a) and number(b) else None
def statistics(values: list[Any]) -> dict:
valid = [float(value) for value in values if number(value)]
return {"count": len(valid), "missingCount": len(values) - len(valid),
"median": median(valid) if valid else None,
"min": min(valid) if valid else None, "max": max(valid) if valid else None,
"values": values}
def union_length(intervals: list[tuple[float, float]]) -> float:
total, end = 0.0, -math.inf
for start, stop in sorted(intervals):
total += max(0.0, stop - max(start, end))
end = max(end, stop)
return total
def span_hierarchy(raw_spans: list[dict], http_ms: float) -> list[dict]:
spans = [{"id": "http", "name": "http_total", "startMs": 0.0,
"endMs": http_ms, "durationMs": http_ms, "source": "httpTotalSeconds"}]
for index, span in enumerate(raw_spans):
start, end = span.get("startMs"), span.get("endMs")
if not number(start) or not number(end) or end < start:
raise ValueError(f"Invalid backend span: {span}")
spans.append({"id": f"span-{index}", "name": span["name"], "startMs": start,
"endMs": end, "durationMs": end - start,
"recordedSeconds": span.get("seconds"), "source": "spans"})
for current in spans:
candidates = [other for other in spans if other["id"] != current["id"]
and other["startMs"] <= current["startMs"]
and other["endMs"] >= current["endMs"]
and (other["durationMs"] > current["durationMs"]
or other["id"] == "http")]
parent = min(candidates, key=lambda s: s["durationMs"]) if current["id"] != "http" and candidates else None
current["parentId"] = parent["id"] if parent else None
current["parentName"] = parent["name"] if parent else None
current["percentOfParent"] = ratio(current["durationMs"], parent["durationMs"]) if parent else None
current["percentOfHttp"] = ratio(current["durationMs"], http_ms)
current["inclusion"] = "inclusive interval; do not add its children"
for current in spans:
children = [s for s in spans if s["parentId"] == current["id"]]
current["children"] = [s["id"] for s in children]
current["uncoveredByDirectChildrenMs"] = current["durationMs"] - union_length(
[(s["startMs"], s["endMs"]) for s in children])
current["partiallyOverlaps"] = [s["id"] for s in spans if s["id"] != current["id"]
and max(s["startMs"], current["startMs"]) < min(s["endMs"], current["endMs"])
and not (s["startMs"] <= current["startMs"] and s["endMs"] >= current["endMs"])
and not (current["startMs"] <= s["startMs"] and current["endMs"] >= s["endMs"])]
return spans
class Summary:
def __init__(self, root: Path):
self.root = root.resolve()
self.sources: dict[str, dict] = {}
self.observations: list[dict] = []
self.warnings: list[str] = []
def read(self, path: Path) -> dict:
path = path.resolve()
path.relative_to(self.root)
size = path.stat().st_size
if size > 4 * 1024 * 1024:
raise ValueError(f"Expected small metadata JSON, refusing {path} ({size} bytes)")
raw = path.read_bytes()
self.sources[str(path.relative_to(self.root))] = {
"sha256": hashlib.sha256(raw).hexdigest(), "bytes": len(raw)}
return json.loads(raw)
def observe(self, run: dict, domain: str, metric: str, value: Any,
unit: str = "ms", *, parent: str = "", denominator: Any = None,
inclusion: str = "inclusive or overlapping; not additive", source: str = "") -> None:
if value is not None and not number(value):
raise ValueError(f"Non-numeric metric {domain}.{metric}: {value!r}")
self.observations.append({"group": run["group"], "phase": run["phase"],
"run": run["run"], "simulationId": run["simulationId"], "domain": domain,
"metric": metric, "value": value, "unit": unit, "parent": parent,
"denominatorValue": denominator if number(denominator) else None,
"percentOfParent": ratio(value, denominator), "inclusion": inclusion, "source": source})
def backend(self, run: dict, data: dict, source: str) -> dict:
if data.get("id") != run["simulationId"]:
raise ValueError(f"Simulation ID mismatch in {source}")
http_ms = data["httpTotalSeconds"] * 1000
hierarchy = span_hierarchy(data.get("spans", []), http_ms)
by_name: dict[str, list[dict]] = {}
for span in hierarchy:
by_name.setdefault(span["name"], []).append(span)
for name, occurrences in by_name.items():
parent_names = sorted({s["parentName"] or "" for s in occurrences})
self.observe(run, "backend_spans", name, sum(s["durationMs"] for s in occurrences),
parent="http_total", denominator=http_ms, source=source,
inclusion=f"inclusive sum of {len(occurrences)} call(s); interval parents: {', '.join(parent_names)}")
chosen = [s for s in hierarchy if s["name"] in HTTP_BOUNDARIES]
intervals = [(s["startMs"], s["endMs"]) for s in chosen]
covered = union_length(intervals)
overlaps = sum(stop - start for start, stop in intervals) - covered
outside = [s["id"] for s in chosen if s["startMs"] < 0 or s["endMs"] > http_ms]
partition_ok = overlaps <= 1e-6 and not outside
partition = {"valid": partition_ok, "scope": "http_total", "totalMs": http_ms,
"selectedSpanIds": [s["id"] for s in chosen], "measuredUnionMs": covered,
"overlapMs": overlaps, "outsideHttpSpanIds": outside,
"otherMs": http_ms - covered if not outside else None,
"note": DEFINITIONS["httpPartition"], "segments": []}
if partition_ok:
partition_durations: dict[str, float] = {}
for span in sorted(chosen, key=lambda s: s["startMs"]):
partition["segments"].append({"metric": span["name"], "startMs": span["startMs"],
"endMs": span["endMs"], "durationMs": span["durationMs"],
"percentOfHttp": ratio(span["durationMs"], http_ms)})
partition_durations[span["name"]] = partition_durations.get(span["name"], 0) + span["durationMs"]
for name, duration in partition_durations.items():
self.observe(run, "http_partition", name, duration,
parent="http_total", denominator=http_ms, inclusion="non-overlapping at declared reporting depth", source=source)
self.observe(run, "http_partition", "other_unclassified", partition["otherMs"],
parent="http_total", denominator=http_ms, inclusion=DEFINITIONS["backendOther"], source=source)
else:
self.warnings.append(f"{run['simulationId']}: HTTP partition disabled; selected spans overlap or leave request bounds")
for key, value in data.items():
if key.endswith("Ms") or key in ("responseBodyBytes", "rawSeriesBytes", "xmlBytes", "sampleCount"):
self.observe(run, "backend_metrics", key, value,
"ms" if key.endswith("Ms") else "bytes" if key.endswith("Bytes") else "count", source=source)
elif key == "responseSendAwaitSeconds":
self.observe(run, "backend_metrics", key, value * 1000,
inclusion="ASGI send waits overlap HTTP and are not pure network time", source=source)
phases = data.get("existingPerformance", {}).get("phases", {})
for key, phase in phases.items():
self.observe(run, "backend_existing_performance", key, phase["inclusiveNs"] / 1e6,
parent="http_total", denominator=http_ms, source=source,
inclusion="inclusive duration without aligned start/end; excluded from HTTP partition")
c = dict(data.get("nativeStages", {}))
native = data.get("native", {})
c_main = c.get("mainTotalSeconds")
for key in C_WALL_FIELDS:
value = c.get(key)
self.observe(run, "c_wall", key, value * 1000 if number(value) else None,
parent="c_main_wall", denominator=c_main * 1000 if number(c_main) else None,
inclusion="sequential C phase wall duration; C main lies within observed process lifetime", source=source)
self.observe(run, "c_wall", "mainTotalSeconds", c_main * 1000 if number(c_main) else None,
inclusion="inclusive C main; does not include loader/exit observation", source=source)
c_other = c_main - sum(c[key] for key in C_WALL_FIELDS) if number(c_main) and all(number(c.get(k)) for k in C_WALL_FIELDS) else None
self.observe(run, "c_wall", "other_unclassified", c_other * 1000 if number(c_other) else None,
parent="c_main_wall", denominator=c_main * 1000 if number(c_main) else None,
inclusion="C main minus sequential measured wall phases, without clipping negative differences", source=source)
process = dict(data.get("process", {}))
child_cpu = (process["childrenUserCpuSeconds"] + process["childrenSystemCpuSeconds"]
if all(number(process.get(k)) for k in ("childrenUserCpuSeconds", "childrenSystemCpuSeconds")) else None)
for key, value in {"integrationCpuSeconds": native.get("solveCpuSeconds"),
"projectionCpuSeconds": c.get("projectionCpuSeconds"),
"jsonWriteCpuSeconds": c.get("jsonWriteCpuSeconds")}.items():
self.observe(run, "c_cpu", key, value * 1000 if number(value) else None,
parent="observed_child_cpu", denominator=child_cpu * 1000 if number(child_cpu) else None,
inclusion="CPU duration, separate from wall partition", source=source)
for key in ("startMs", "exitObservedMs", "childrenUserCpuSeconds", "childrenSystemCpuSeconds"):
value = process.get(key)
self.observe(run, "process", key, value * 1000 if number(value) and key.endswith("Seconds") else value,
inclusion=DEFINITIONS["process"], source=source)
self.observe(run, "process", "observed_lifetime", difference(process, "startMs", "exitObservedMs"),
inclusion="includes spawn; excludes subsequent Python result read", source=source)
self.observe(run, "process", "observed_child_cpu", child_cpu * 1000 if number(child_cpu) else None,
inclusion=DEFINITIONS["process"], source=source)
return {"source": source, "httpTotalMs": http_ms, "spanHierarchy": hierarchy,
"httpPartition": partition, "cStages": c, "process": process,
"build": data.get("build"), "native": native,
"existingPerformance": data.get("existingPerformance")}
def group(self, mode: str, expected_runs: int) -> dict:
folder = self.root / f"browser-{mode}"
source = folder / "summary.json"
summary = self.read(source)
rows = summary.get("rows", [])
seen: set[str] = set()
runs = []
for original in rows:
if original.get("mode") != mode:
raise ValueError(f"Unexpected mode {original.get('mode')!r} in {source}")
sid = original["simulationId"]
if sid in seen:
raise ValueError(f"Duplicate simulationId {sid} in {source}")
seen.add(sid)
run = {"group": mode, "phase": "warmup" if original.get("warmup") else "measured",
"run": original["run"], "simulationId": sid, "original": original}
trace_path = folder / original["artifacts"] / "trace.json"
trace = self.read(trace_path)
if trace.get("timeOrigin") != original.get("timeOrigin"):
raise ValueError(f"Browser timeOrigin mismatch: {trace_path}")
marks = trace.get("marks", {})
if mode == "profiled":
request_ids = {r.get("simulationId") for r in trace.get("requests", []) if r.get("kind") == "simulation"}
if request_ids != {sid}:
raise ValueError(f"Browser trace request ID mismatch: {trace_path}")
total = difference(marks, "runClick", "indexedDbCommittedPointer")
waterfall = []
for (left, left_name), (right, right_name) in zip(AXIS, AXIS[1:]):
value = difference(marks, left, right)
metric = f"{left_name}_to_{right_name}"
entry = {"metric": metric, "fromMark": left, "toMark": right,
"startMs": marks.get(left), "endMs": marks.get(right), "durationMs": value,
"percentOfClickToCommit": ratio(value, total), "negative": value is not None and value < 0}
waterfall.append(entry)
self.observe(run, "frontend_waterfall", metric, value, parent="click_to_commit",
denominator=total, source=str(trace_path.relative_to(self.root)), inclusion=DEFINITIONS["waterfall"])
complete = all(number(part["durationMs"]) for part in waterfall)
if complete and not math.isclose(sum(p["durationMs"] for p in waterfall), total, abs_tol=1e-6):
raise ValueError(f"Waterfall does not telescope for {sid}")
run["frontend"] = {"timeOrigin": trace["timeOrigin"], "marks": marks,
"waterfall": waterfall, "clickToCommitMs": total, "waterfallComplete": complete,
"negativeIntervals": [part["metric"] for part in waterfall if part["negative"]]}
for key, value in original.items():
if key in ("run", "timeOrigin") or isinstance(value, bool):
continue
if number(value) or value is None:
unit = "ms" if key.endswith("Ms") else "bytes" if key.endswith("Bytes") else "count" if key.endswith("Count") else "number"
self.observe(run, "frontend_metrics", key, value, unit, source=str(source.relative_to(self.root)))
native = original.get("native", {})
for key, value in native.items():
if number(value):
unit = "ms" if key.endswith("Seconds") else "simulated_s" if key in ("simulatedUntil", "maxAcceptedStep") else "count"
self.observe(run, "native_reported", key, value * 1000 if key.endswith("Seconds") else value, unit,
source=str(source.relative_to(self.root)), inclusion="original browser native diagnostic; processWallSeconds includes result read")
run["buildCost"] = {"cacheHit": native.get("cacheHit"), "buildKey": native.get("buildKey"),
"reportedSeconds": native.get("buildSeconds"),
"classification": "cache_hit" if native.get("cacheHit") is True else "cold_build" if native.get("cacheHit") is False else "unknown"}
self.observe(run, "build", run["buildCost"]["classification"],
native["buildSeconds"] * 1000 if number(native.get("buildSeconds")) else None,
source=str(source.relative_to(self.root)), inclusion="warmups separate; cacheHit determines cold/cache classification")
backend_path = self.root / "backend-profiled/requests" / sid / "stages.json"
if mode == "profiled" and backend_path.exists():
backend_data = self.read(backend_path)
run["backend"] = self.backend(run, backend_data, str(backend_path.relative_to(self.root)))
for key in ("buildKey", "nfev", "acceptedSteps", "solveSeconds"):
if backend_data.get("native", {}).get(key) != native.get(key):
raise ValueError(f"Browser/backend native diagnostic mismatch for {sid}: {key}")
else:
run["backend"] = None
if mode == "profiled": self.warnings.append(f"{sid}: completed browser row has no backend stages.json yet")
runs.append(run)
measured = sum(r["phase"] == "measured" for r in runs)
if measured != expected_runs:
self.warnings.append(f"{mode}: {measured} measured run(s), expected {expected_runs}")
if summary.get("errors"):
self.warnings.append(f"{mode}: browser summary contains errors; inspect source before interpreting results")
environment = self.root / f"backend-{mode}" / "environment.json"
return {"source": str(source.relative_to(self.root)), "browser": summary.get("browser"),
"node": summary.get("node"), "input": summary.get("input"), "inputSha256": summary.get("inputSha256"),
"buildAssetSetSha256": summary.get("buildAssetSetSha256"), "servedAssets": summary.get("servedAssets"),
"sourceDefinitions": summary.get("definitions"), "initialNavigation": summary.get("initialNavigation"),
"backendEnvironment": self.read(environment) if environment.exists() else None,
"measuredRunCount": measured, "warmupRunCount": len(runs) - measured, "runs": runs}
def aggregate(self) -> list[dict]:
buckets: dict[tuple, list[dict]] = {}
for observation in self.observations:
key = tuple(observation[k] for k in ("group", "phase", "domain", "metric", "unit", "parent"))
buckets.setdefault(key, []).append(observation)
result = []
for key, rows in sorted(buckets.items()):
entry = dict(zip(("group", "phase", "domain", "metric", "unit", "parent"), key))
entry.update(statistics([r["value"] for r in rows]))
entry["percentOfParent"] = statistics([r["percentOfParent"] for r in rows])
entry["runs"] = [{"run": r["run"], "simulationId": r["simulationId"]} for r in rows]
result.append(entry)
return result
def write(self, expected_runs: int) -> dict:
groups = {mode: self.group(mode, expected_runs) for mode in ("control", "profiled")}
if groups["control"]["inputSha256"] != groups["profiled"]["inputSha256"]:
raise ValueError("Control/profiled input SHA mismatch")
if groups["control"]["buildAssetSetSha256"] != groups["profiled"]["buildAssetSetSha256"]:
self.warnings.append("Control/profiled frontend asset sets differ")
aggregates = self.aggregate()
build_costs = [{"group": mode, "run": r["run"], "phase": r["phase"],
"simulationId": r["simulationId"], **r["buildCost"]}
for mode, group in groups.items() for r in group["runs"]]
output = {"schemaVersion": 1, "experimentRoot": str(self.root),
"script": str(Path(__file__).resolve().relative_to(REPO)),
"scriptSha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
"definitions": DEFINITIONS, "warnings": self.warnings, "sourceFiles": self.sources,
"groups": groups, "observations": self.observations, "aggregates": aggregates,
"complete": not self.warnings, "buildCosts": build_costs,
"warmupBuildCosts": [cost for cost in build_costs if cost["phase"] == "warmup"],
"coldBuildCosts": [cost for cost in build_costs if cost["classification"] == "cold_build"]}
(self.root / "summary.json").write_text(json.dumps(output, ensure_ascii=False, indent=2, allow_nan=False) + "\n")
fields = ["group", "phase", "run", "simulationId", "domain", "metric", "statistic", "value", "unit",
"parent", "denominatorValue", "percentOfParent", "count", "missingCount", "inclusion", "source"]
with (self.root / "timings.csv").open("w", encoding="utf-8", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=fields)
writer.writeheader()
for row in self.observations:
writer.writerow({**row, "statistic": "run", "count": int(number(row["value"])), "missingCount": int(row["value"] is None)})
for entry in aggregates:
for stat in ("median", "min", "max"):
writer.writerow({**{key: entry[key] for key in ("group", "phase", "domain", "metric", "unit", "parent")},
"statistic": stat, "value": entry[stat], "percentOfParent": entry["percentOfParent"][stat],
"count": entry["count"], "missingCount": entry["missingCount"],
"inclusion": "values and same-run percentages aggregated separately; do not add medians"})
return output
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--root", type=Path, default=REPO / "test/web-cost-20260911")
parser.add_argument("--expected-runs", type=int, default=3)
args = parser.parse_args()
if args.expected_runs < 1:
parser.error("--expected-runs must be positive")
summary = Summary(args.root).write(args.expected_runs)
print(json.dumps({"root": summary["experimentRoot"], "complete": summary["complete"],
"runs": {mode: {"measured": value["measuredRunCount"], "warmup": value["warmupRunCount"]}
for mode, value in summary["groups"].items()},
"observations": len(summary["observations"]), "warnings": summary["warnings"]}, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
tests/test_native_json_writer.py
+367
View File
@@ -0,0 +1,367 @@
"""Standalone JSON-number contracts; no model, solver or SUNDIALS is required.
The decimal text may change between correct shortest encoders. These tests use
Python's independent JSON decoder and binary64 bits, including the sign of zero,
instead of comparing against the old %.17g spelling.
"""
import ctypes
import json
import math
import os
from pathlib import Path
import random
import re
import shlex
import shutil
import struct
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
JSON_NUMBER = re.compile(rb'-?(?:0|[1-9][0-9]*)(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?\Z')
BUFFER_BYTES = 64 * 1024
SIGN = 1 << 63
FRACTION_MASK = (1 << 52) - 1
def double_from_bits(bits):
return struct.unpack('=d', struct.pack('=Q', bits))[0]
def double_bits(value):
return struct.unpack('=Q', struct.pack('=d', value))[0]
def finite_patterns(random_count=10000):
# Every finite exponent binade, with exact powers and significand edges.
values = {0, SIGN}
for exponent in range(0x7ff):
for fraction in (0, 1, (1 << 51) - 1, 1 << 51, FRACTION_MASK):
bits = (exponent << 52) | fraction
values.update((bits, bits | SIGN))
# Decimal carry/notation boundaries and adjacent representable values.
for exponent in range(-323, 309):
center = float(f'1e{exponent}')
for value in (math.nextafter(center, 0), center, math.nextafter(center, math.inf)):
if math.isfinite(value):
values.update((double_bits(value), double_bits(-value)))
random_source = random.Random(0x5259555F4A534F4E)
added = 0
while added < random_count:
bits = random_source.getrandbits(64)
if (bits >> 52) & 0x7ff != 0x7ff:
values.add(bits)
added += 1
return sorted(values)
class WriteStatus(ctypes.Structure):
_fields_ = [
('opened', ctypes.c_int), ('written', ctypes.c_int), ('closed', ctypes.c_int),
('start', ctypes.c_longlong), ('end', ctypes.c_longlong), ('final_position', ctypes.c_longlong),
]
HARNESS = r'''
#include <stdio.h>
#include <stdint.h>
#include <stddef.h>
#include <string.h>
#include "json_numbers.h"
/* Faults intercept only the production writer's fwrite calls in this temporary
translation unit. Real stdio still writes the accepted prefix. */
int test_write_fault_mode = 0;
int test_write_call_count = 0;
#ifdef TEST_FWRITE_FAULTS
static size_t test_fwrite(const void *data, size_t size, size_t count, FILE *stream) {
test_write_call_count++;
if (test_write_fault_mode == 1 ||
(test_write_fault_mode == 3 && test_write_call_count >= 2)) return 0;
if (test_write_fault_mode == 2)
return count ? fwrite(data, size, count - 1, stream) : 0;
return fwrite(data, size, count, stream);
}
#define fwrite test_fwrite
#include "json_numbers.c"
#undef fwrite
#endif
typedef struct {
int opened, written, closed;
long long start, end, final_position;
} TestWriteStatus;
void test_array_file(const char *path, const double *values, size_t count,
size_t stride, size_t prefix_bytes, TestWriteStatus *status) {
memset(status, 0, sizeof(*status));
FILE *stream = fopen(path, "wb");
if (!stream) return;
status->opened = 1;
for (size_t i = 0; i < prefix_bytes; i++) fputc('p', stream);
status->start = (long long)ftell(stream);
status->written = native_json_write_array(stream, values, count, stride);
status->end = (long long)ftell(stream);
if (status->written) fputs("TAIL", stream);
status->final_position = (long long)ftell(stream);
status->closed = fclose(stream) == 0;
}
void test_number_file(const char *path, double value, TestWriteStatus *status) {
memset(status, 0, sizeof(*status));
FILE *stream = fopen(path, "wb");
if (!stream) return;
status->opened = 1;
status->written = native_json_write_number(stream, value);
status->end = status->final_position = (long long)ftell(stream);
status->closed = fclose(stream) == 0;
}
/* The number writer flushes its own block to FILE, not FILE's stdio buffer.
The caller must still propagate a delayed fclose failure. */
int test_delayed_close_failure(const char *path, int *written, int *closed) {
FILE *stream = fopen(path, "wb");
if (!stream) return 0;
char buffer[4096];
if (setvbuf(stream, buffer, _IOFBF, sizeof(buffer))) { fclose(stream); return 0; }
*written = native_json_write_number(stream, 0.1);
*closed = fclose(stream) == 0;
return 1;
}
'''
class NativeJsonWriterTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
command = shlex.split(os.environ.get('CC', ''))
if not command:
compiler = shutil.which('gcc') or shutil.which('clang')
if not compiler:
raise unittest.SkipTest('A native C compiler is required')
command = [compiler]
cls.compiler = command
cls.directory = tempfile.TemporaryDirectory(prefix='native-json-writer-')
cls.addClassCleanup(cls.directory.cleanup)
cls.root = Path(cls.directory.name)
cls.library = cls.build_library('ordinary', faults=False)
cls.fault_library = cls.build_library('faults', faults=True)
cls.portable_library = cls.build_library('portable-64-bit', faults=False, only_64_bit=True)
cls.fault_mode = ctypes.c_int.in_dll(cls.fault_library, 'test_write_fault_mode')
cls.fault_calls = ctypes.c_int.in_dll(cls.fault_library, 'test_write_call_count')
@classmethod
def build_library(cls, name, *, faults, only_64_bit=False):
source = cls.root / f'{name}.c'
source.write_text(HARNESS)
library_path = cls.root / (name + ('.dll' if os.name == 'nt' else '.so'))
command = cls.compiler + ['-std=c11', '-O2', '-Wall', '-Wextra', '-Werror',
'-ffp-contract=off', '-fno-fast-math', '-shared']
if os.name != 'nt':
command.append('-fPIC')
if faults:
command.append('-DTEST_FWRITE_FAULTS')
if only_64_bit:
command.append('-DRYU_ONLY_64_BIT_OPS')
command += ['-I', str(ROOT / 'native/include'), '-I', str(ROOT / 'native/runtime'),
str(source)]
if not faults:
command.append(str(ROOT / 'native/runtime/json_numbers.c'))
command += [str(ROOT / 'native/encoding/ryu/d2s.c'), '-lm', '-o', str(library_path)]
compiled = subprocess.run(command, capture_output=True, text=True, timeout=60)
if compiled.returncode:
raise AssertionError(compiled.stderr)
library = ctypes.CDLL(str(library_path))
if os.name == 'nt':
import _ctypes
cls.addClassCleanup(_ctypes.FreeLibrary, library._handle)
library.native_json_format_double.argtypes = [ctypes.c_void_p, ctypes.c_double]
library.native_json_format_double.restype = ctypes.c_int
library.native_json_write_number.argtypes = [ctypes.c_void_p, ctypes.c_double]
library.native_json_write_number.restype = ctypes.c_int
library.native_json_write_array.argtypes = [ctypes.c_void_p, ctypes.POINTER(ctypes.c_double), ctypes.c_size_t, ctypes.c_size_t]
library.native_json_write_array.restype = ctypes.c_int
library.test_array_file.argtypes = [ctypes.c_char_p, ctypes.POINTER(ctypes.c_double),
ctypes.c_size_t, ctypes.c_size_t, ctypes.c_size_t,
ctypes.POINTER(WriteStatus)]
library.test_array_file.restype = None
library.test_number_file.argtypes = [ctypes.c_char_p, ctypes.c_double, ctypes.POINTER(WriteStatus)]
library.test_number_file.restype = None
library.test_delayed_close_failure.argtypes = [ctypes.c_char_p, ctypes.POINTER(ctypes.c_int), ctypes.POINTER(ctypes.c_int)]
library.test_delayed_close_failure.restype = ctypes.c_int
return library
def encode(self, value, library=None):
# Sentinels bracket the promised 32-byte output, with no assumption that
# the returned token is NUL-terminated.
storage = (ctypes.c_ubyte * 34)(*([0xA5] * 34))
length = (library or self.library).native_json_format_double(ctypes.byref(storage, 1), value)
self.assertEqual((storage[0], storage[33]), (0xA5, 0xA5))
self.assertGreater(length, 0)
self.assertLessEqual(length, 32)
return bytes(storage[1:1 + length])
def assert_roundtrip(self, text, expected_bits):
self.assertRegex(text, JSON_NUMBER, f'Invalid JSON token for {expected_bits:016x}')
decoded = json.loads(text)
self.assertEqual(double_bits(float(decoded)), expected_bits,
f'{expected_bits:016x} became {text!r} then {decoded!r}')
def write_array(self, values, *, count=None, stride=1, prefix=0, library=None):
array = (ctypes.c_double * len(values))(*values) if values else None
status = WriteStatus()
output = self.root / 'array.json'
(library or self.library).test_array_file(os.fsencode(output), array,
len(values) if count is None else count, stride, prefix, ctypes.byref(status))
self.assertTrue(status.opened)
return output.read_bytes(), status
def test_binary64_boundaries_and_seeded_random_values_roundtrip(self):
for bits in finite_patterns():
self.assert_roundtrip(self.encode(double_from_bits(bits)), bits)
def test_64_bit_fallback_roundtrips_the_same_binary64_corpus(self):
patterns = finite_patterns()
for bits in patterns:
self.assert_roundtrip(self.encode(double_from_bits(bits), self.portable_library), bits)
values = [double_from_bits(bits) for bits in patterns]
ordinary, ordinary_status = self.write_array(values)
portable, portable_status = self.write_array(values, library=self.portable_library)
self.assertTrue(ordinary_status.written and ordinary_status.closed)
self.assertTrue(portable_status.written and portable_status.closed)
self.assertEqual(portable, ordinary)
def test_plain_decimal_is_used_only_when_it_shortens_the_token(self):
# Tie cases deliberately keep scientific notation; far exponents must
# never be expanded to hundreds of zeroes in the 32-byte destination.
cases = ((10.0, b'10'), (12.0, b'12'), (-12.0, b'-12'),
(.1, b'0.1'), (-.1, b'-0.1'), (123.45, b'123.45'),
(100.0, b'1E2'), (.01, b'1E-2'), (1e100, b'1E100'),
(double_from_bits(1), b'5E-324'))
for library in (self.library, self.portable_library):
for value, expected in cases:
with self.subTest(value=value, expected=expected):
token = self.encode(value, library)
self.assertEqual(token, expected)
self.assert_roundtrip(token, double_bits(value))
def test_zero_sign_and_single_number_file(self):
self.assertEqual(self.encode(0.0), b'0')
self.assertEqual(self.encode(-0.0), b'-0.0')
output = self.root / 'number.json'
for bits in (0, SIGN, 1, SIGN | 1, 0x0010000000000000, 0x7fefffffffffffff):
status = WriteStatus()
self.library.test_number_file(os.fsencode(output), double_from_bits(bits), ctypes.byref(status))
self.assertTrue(status.opened and status.written and status.closed)
data = output.read_bytes()
self.assertEqual(status.end, len(data))
self.assert_roundtrip(data, bits)
def test_array_crosses_block_boundaries_without_changing_offsets(self):
cases = [([], 2), ([0.0] * 32767, BUFFER_BYTES - 1),
([-0.0] + [0.0] * 32765, BUFFER_BYTES),
([0.0] * 32768, BUFFER_BYTES + 1),
([double_from_bits(bits) for bits in finite_patterns(0)[::3]], None)]
for values, expected_length in cases:
for prefix in (0, 37, BUFFER_BYTES - 1):
with self.subTest(values=len(values), expected_length=expected_length, prefix=prefix):
data, status = self.write_array(values, prefix=prefix)
self.assertTrue(status.written and status.closed)
self.assertEqual(status.start, prefix)
self.assertEqual(data[:prefix], b'p' * prefix)
self.assertEqual(data[status.end:], b'TAIL')
self.assertEqual(status.final_position, len(data))
token = data[status.start:status.end]
if expected_length is not None:
self.assertEqual(len(token), expected_length)
decoded = json.loads(token)
self.assertEqual([double_bits(float(v)) for v in decoded], [double_bits(v) for v in values])
def test_strided_array_keeps_selected_column_order(self):
selected = [double_from_bits(bits) for bits in finite_patterns(0)[::13]]
for stride in (1, 3, 17):
with self.subTest(stride=stride):
values = [math.nan] * (len(selected) * stride)
values[::stride] = selected
data, status = self.write_array(values, count=len(selected), stride=stride)
self.assertTrue(status.written and status.closed)
decoded = json.loads(data[:status.end])
self.assertEqual([double_bits(float(v)) for v in decoded], [double_bits(v) for v in selected])
def test_invalid_file_pointer_and_array_bounds_fail_before_access(self):
value = (ctypes.c_double * 1)(.1)
self.assertEqual(self.library.native_json_write_number(None, .1), 0)
self.assertEqual(self.library.native_json_write_array(None, value, 1, 1), 0)
for values, count, stride in (([], 1, 1), ([.1], 2, 0),
([.1], 2, ctypes.c_size_t(-1).value)):
with self.subTest(count=count, stride=stride):
data, status = self.write_array(values, count=count, stride=stride)
self.assertTrue(status.opened and status.closed)
self.assertFalse(status.written)
self.assertEqual(data, b'')
# A single sample never advances its pointer; an enormous stride is safe.
data, status = self.write_array([.1], count=1, stride=ctypes.c_size_t(-1).value)
self.assertTrue(status.written and status.closed)
self.assertEqual(json.loads(data[:status.end]), [.1])
def test_nonfinite_numbers_fail_instead_of_emitting_invalid_json(self):
output = self.root / 'nonfinite.json'
for bits in (0x7ff0000000000000, 0xfff0000000000000,
0x7ff8000000000000, 0xfff8000000000001, 0x7ff0000000000001):
value = double_from_bits(bits)
with self.subTest(bits=f'{bits:016x}'):
token = ctypes.create_string_buffer(32)
self.assertEqual(self.library.native_json_format_double(token, value), 0)
status = WriteStatus()
self.library.test_number_file(os.fsencode(output), value, ctypes.byref(status))
self.assertTrue(status.opened and status.closed)
self.assertFalse(status.written)
self.assertEqual(output.read_bytes(), b'')
for finite_prefix in ([], [0.0] * (BUFFER_BYTES + 1)):
data, status = self.write_array(finite_prefix + [value])
self.assertFalse(status.written)
self.assertTrue(status.closed)
self.assertNotIn(b'NaN', data)
self.assertNotIn(b'Infinity', data)
def test_zero_and_short_writes_are_reported_without_stdio_error_flag(self):
# The injected fwrite can return short without setting FILE's error bit;
# relying only on ferror/fclose would incorrectly report success.
output = self.root / 'fault-number.json'
try:
for mode in (1, 2):
with self.subTest(mode=mode):
self.fault_mode.value = mode
self.fault_calls.value = 0
status = WriteStatus()
self.fault_library.test_number_file(os.fsencode(output), .1, ctypes.byref(status))
self.assertTrue(status.opened and status.closed)
self.assertFalse(status.written)
self.assertGreater(self.fault_calls.value, 0)
self.fault_calls.value = 0
_, status = self.write_array([1.0, 2.0], library=self.fault_library)
self.assertFalse(status.written)
self.assertTrue(status.closed)
self.fault_mode.value = 3
self.fault_calls.value = 0
data, status = self.write_array([0.0] * (BUFFER_BYTES * 2), library=self.fault_library)
self.assertFalse(status.written)
self.assertTrue(status.closed)
self.assertGreaterEqual(self.fault_calls.value, 2)
self.assertTrue(data, 'The initial successful block must survive a later write failure')
finally:
self.fault_mode.value = 0
self.fault_calls.value = 0
@unittest.skipUnless(os.name == 'posix' and Path('/dev/full').exists(), '/dev/full is needed for a real delayed I/O failure')
def test_caller_must_check_delayed_fclose_failure(self):
written, closed = ctypes.c_int(), ctypes.c_int()
self.assertEqual(self.library.test_delayed_close_failure(b'/dev/full', ctypes.byref(written), ctypes.byref(closed)), 1)
self.assertEqual(written.value, 1, 'A small buffered token should not force FILE fflush')
self.assertEqual(closed.value, 0, 'FILE close must expose the delayed device error')
if __name__ == '__main__':
unittest.main()
tests/test_native_result_transport.py
+143
View File
@@ -0,0 +1,143 @@
"""The HTTP fast path must preserve real native values and task semantics."""
from dataclasses import replace
import json
from pathlib import Path
import tempfile
from types import SimpleNamespace
import unittest
from unittest.mock import patch
from uuid import uuid4
import asyncio
from app.main import app, _register_simulation_task, _request_simulation_task_cancel, simulation_event_stream
from app.simulation.backends import simulation_config
from app.simulation.native_codegen.build import build_native
from app.simulation.native_codegen.compiler import compile_native_program
from app.simulation.native_codegen.input import load_input
from app.simulation.native_codegen.runner import execute_native
from app.simulation.native_codegen.transport import NativeSeriesJson, read_indexed_result, serialize_result_parts
from app.main import compile_system_xml_network
class AsgiClient:
"""Exercise real routing/response bodies without an optional HTTP client dependency."""
def __init__(self, application): self.application = application
def post(self, path, *, content=b'', headers=None): return self.request('POST', path, content, headers)
def get(self, path): return self.request('GET', path, b'', None)
def request(self, method, path, content, headers):
async def run():
messages = []
scope = {'type':'http', 'asgi':{'version':'3.0','spec_version':'2.4'},
'http_version':'1.1','method':method,'scheme':'http','path':path,
'raw_path':path.encode(),'query_string':b'', 'root_path':'',
'headers':[(k.lower().encode(),v.encode()) for k,v in (headers or {}).items()],
'server':('testserver',80),'client':('127.0.0.1',1234)}
async def receive(): return {'type':'http.request','body':content,'more_body':False}
async def send(message): messages.append(message)
await self.application(scope,receive,send)
status = next(m['status'] for m in messages if m['type']=='http.response.start')
body = b''.join(m.get('body',b'') for m in messages if m['type']=='http.response.body')
return SimpleNamespace(status_code=status,content=body,json=lambda:json.loads(body))
return asyncio.run(run())
class NativeResultTransportTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.temp = tempfile.TemporaryDirectory(prefix='test-native-transport-')
cls.root = Path(cls.temp.name)
cls.xml = Path('tests/fixtures/native-skill-test.xml').read_bytes().replace(b'tStop="10"', b'tStop="0.1"')
source = cls.root/'input.xml'; source.write_bytes(cls.xml)
_, document = load_input(source)
cls.config = simulation_config(document.simulation)
cls.build = build_native(compile_native_program(compile_system_xml_network(document)))
cls.normal = execute_native(cls.build, cls.config, .001, run_dir=cls.root/'ordinary')
cls.indexed = execute_native(cls.build, cls.config, .001, run_dir=cls.root/'indexed', raw_series=True)
@classmethod
def tearDownClass(cls):
cls.temp.cleanup()
def test_real_native_series_are_equal_without_large_python_parse(self):
self.assertIsInstance(self.indexed['series'], NativeSeriesJson)
self.assertEqual(self.indexed['series'].materialize(), self.normal['series'])
self.assertEqual(self.indexed['series'].sample_count, len(self.normal['series']['time']))
for key in ('final', 'finalState', 'nfev', 'acceptedSteps', 'rejectedSteps'):
self.assertEqual(self.indexed[key], self.normal[key])
self.assertGreater(len(self.indexed['series'].data), 100000)
original = json.loads
sizes = []
def small_only(value):
sizes.append(len(value))
self.assertLess(len(value), 50000, 'Raw series was decoded through Python')
return original(value)
with patch('app.simulation.native_codegen.transport.json', SimpleNamespace(loads=small_only)):
payload = read_indexed_result(self.root/'indexed/result.json', self.root/'indexed/result-index.json')
self.assertEqual(len(sizes), 2) # index and small metadata only
self.assertEqual(payload['series'].data, self.indexed['series'].data)
def test_real_http_stream_and_retained_task_get_keep_same_schema(self):
client = AsgiClient(app)
ident = 'transport-'+uuid4().hex
response = client.post('/api/system-xml/simulate-stream', content=self.xml, headers={'X-Simulation-Id':ident})
self.assertEqual(response.status_code, 200)
events = [json.loads(line) for line in response.content.splitlines()]
result = next(event['result'] for event in events if event['event'] == 'result')
self.assertTrue(result['success'])
self.assertEqual(result['simulatedUntil'], .1)
self.assertEqual(result['diagnostics']['sampleCount'], len(result['series']['time']))
# The bytes survive deletion of the worker directory and repeated task reads.
for _ in range(2):
retained = client.get('/api/system-xml/simulations/'+ident)
self.assertEqual(retained.status_code, 200)
self.assertEqual(retained.json()['result'], result)
synchronous = client.post('/api/system-xml/simulate', content=self.xml).json()
for key in ('series', 'final', 'variables', 'model', 'simulation'):
self.assertEqual(synchronous[key], result[key])
def test_cancelled_raw_stream_keeps_partial_result_and_public_status(self):
for reason, status in [('user', 'stopped'), ('stalled', 'stalled')]:
task = _register_simulation_task('transport-'+uuid4().hex)
_request_simulation_task_cancel(task, reason)
body = b''.join(part.encode() if isinstance(part, str) else part
for part in simulation_event_stream(self.xml, task=task, raw_series=True))
result = next(event['result'] for event in map(json.loads, body.splitlines()) if event['event']=='result')
self.assertEqual(result['status'], status)
self.assertTrue(result['partial'])
self.assertLess(result['simulatedUntil'], .1)
self.assertEqual(result['series']['time'][-1], result['simulatedUntil'])
self.assertEqual(AsgiClient(app).get('/api/system-xml/simulations/'+task.simulation_id).json()['result'], result)
def test_index_corruption_or_truncated_output_is_rejected(self):
directory = self.root/'corrupt'; directory.mkdir(exist_ok=True)
data = (self.root/'indexed/result.json').read_bytes()
original = json.loads((self.root/'indexed/result-index.json').read_bytes())
output = directory/'result.json'; index = directory/'index.json'
output.write_bytes(data)
for change in ({'version':2}, {'version':True}, {'version':1.0}, {'seriesStart':True}, {'seriesStart':-1}, {'seriesEnd':len(data)+1},
{'resultBytes':len(data)-1}, {'sampleCount':-1}, {'seriesStart':original['seriesStart']+1}):
with self.subTest(change=change):
index.write_text(json.dumps(original | change))
with self.assertRaises(ValueError): read_indexed_result(output,index)
index.write_text(json.dumps(original)); output.write_bytes(data[:-2])
with self.assertRaises(ValueError): read_indexed_result(output,index)
def test_json_framing_and_escaping_cannot_confuse_raw_series(self):
values = {'time':[0,1], 'odd\\"},"final":{\n温度':[1e-300,-0.0]}
raw = NativeSeriesJson(json.dumps(values,ensure_ascii=False).encode(),2)
for payload in ({'result':{'series':raw}},
{'event':'result','message':'\n"series":{},"result":null',
'result':{'series':raw,'label':'\\"雪\n','success':True}}):
encoded = b''.join(serialize_result_parts(payload))
expected = dict(payload, result=dict(payload['result'], series=values))
self.assertEqual(json.loads(encoded), expected)
self.assertEqual(json.loads(b''.join(serialize_result_parts({'event':'progress'}))), {'event':'progress'})
def test_solve_only_raw_series_is_empty_object(self):
result = execute_native(self.build, replace(self.config,t_stop=.001),.001,
run_dir=self.root/'solve-only', raw_series=True,record_samples=False)
self.assertEqual(result['series'].data,b'{}')
self.assertEqual(result['series'].sample_count,0)
if __name__ == '__main__': unittest.main()