Local evidence audit · window Jun 17 – Jul 30, 2026 · generated Jul 30, 2026
The slowdown is real and concentrated in subagent work. Provider APIs are not the main cause.
| ~/.omp/stats.db | 520,183 calls · 481,209 tool calls · 11,297 user turns |
| ~/.omp/agent/sessions | 236 projects · 796 run dirs · 5,918 jsonl |
| ~/.omp/logs | 91 files · 37.2 MB · incl. 26 gzipped |
| ~/.omp/agent/history.db | 6,593 history rows |
| live host snapshot | ps / vm_stat / sysctl / uptime |
| Prior 3 weeks | Jun 17 – Jul 7 (21 days) |
| Recent 3 weeks | Jul 8 – Jul 28 (21 days) |
| Live tail | Jul 29 – Jul 30 (logs only) |
stats.db ends Jul 28 07:10 UTC, so Jul 29–30 appear only in runtime logs and the live snapshot.task or job tool call.*.jsonl session; duration = first call start → last call end.task+job+hub+irc) per 1,000 main-session tool calls.input_tokens + cache_read_tokens per call.verified means recomputed directly from local data in this session. inference marks interpretation beyond the measurement.
Subagent task duration was stable through Jul 7, then spiked hard in the week of Jul 15 — the same week the coordination layer switched from job/irc to hub.
Full percentile distributions, prior versus recent, split by whether the task used subagents.
Each individual subagent runs about twice as long and issues about twice as many model calls. Percentiles here exclude idle time: gaps longer than 5 minutes between a child's calls are clipped, because 303 of the 317 sessions over 30 minutes were parked and later resumed rather than working continuously.
The clearest signal in the dataset: more steps per task, at the same request complexity.
Coordination calls per 1,000 main-session tool calls doubled. The mechanism changed mid-July.
task | Jun 17 06:44 → Jul 28 05:25 | 2,664 |
job | Jun 17 06:47 → Jul 17 14:08 | 8,215 |
irc | Jun 17 08:52 → Jul 17 14:09 | 9,156 |
hub | Jul 16 23:17 → Jul 28 07:01 | 6,006 |
job/irc pair stops within 15 hours of hub appearing.Fewer tasks use subagents, but heavy tasks spawn far more children. The regression lives in that tail.
Per-call latency is close to flat for the main agent. Subagent calls got moderately slower, mostly through routing mix.
gpt-5.5 to reasoning-oriented gpt-5.6-sol: subagent TTFT p50 1.66 s → 2.26 s and duration p50 5.28 s → 5.59 s. Flash subagent calls complete in 1.43 s p50. Workloads differ per model, so this compares routing cost, not model quality. inferenceTypical calls barely grew. Long sessions grew a lot, which is where compaction overhead appears.
Reliability improved. Provider failures do not explain the slowdown.
Chronic in both windows, not a new regression. The severe tail is happening right now.
ui.loop-blocked instrumentation is not new. Scanning all 91 log files including gzipped rotations yields 29,777 events from Jun 16 onward, with 14,620 inside the prior window. The earlier figure of 14,489 "recent only" came from uncompressed files alone. verifiedSpawn overhead is negligible historically and pathological today.
Measured during this diagnostic. This is not a historical trend, it is the current operating condition.
These are not part of the recent regression. They were present across the whole six weeks and are worth fixing regardless.
Same model, same agent role, whole period. Holding gpt-5.6-sol fixed removes model mix as an explanation.
read is unboundedread alone accounts for 908 M characters, 59 percent of all tool output. Several tools clamp near 51k characters with zero results above that line, but read reached 460k characters in a single result, roughly 115k tokens, about 42 percent of a 272k window. verifiedtool_calls table records timestamp, args_chars, result_chars, and is_error, but no duration. End-to-end task time therefore cannot be split into model time, tool execution, child critical path, coordination wait, and compaction. Every latency question has to be reconstructed from message timestamps, which is exactly why the idle-versus-work correction on this page was needed. verified by source reading.ui.loop-blocked events since Jun 16, present in both windows. The watchdog compares a 250 ms timer against monotonic time, which keeps advancing during system sleep, so a suspended laptop is recorded as one enormous stall. That is where the 4,379,642 ms event came from. Both the chronic stalls and the false measurement are worth fixing, and the second one is a few lines. verified by source reading.task.maxRuntimeMs is set to 30 minutes locally and is armed per runSubagent call, not across a child's whole life. A child that parks and resumes gets a fresh timer each turn, which is why session spans reach two hours with only 7.8 percent of that span in active model time. There is also no cumulative cap on children per parent run. verified by source reading.stats.db records no CPU, RSS, or swap, so today's pressure cannot be projected backward.sol handles harder slices.| Earlier claim | Verified finding |
|---|---|
14,489 ui.loop-blocked events, recent window only | 29,777 events total; 14,620 in the prior window, 3,035 recent, 11,952 on Jul 29–30 |
| Instrumentation activated in the recent window | Present from Jun 16 onward; earlier count missed gzipped rotations |
| Launch timing from 150 events, p90 345 ms | 1,064 events; p90 521 ms prior, 206 ms recent, 3,540 ms live |
| Event-loop stalls framed as a recent driver | Chronic in both windows; only the live tail is exceptional |
| Child session p90 +277.4% (11.0 → 41.6 min) | That span counted parked idle time. Work time p90 rose 9.9 → 19.1 min (+92.4%); active model time 6.4 → 12.5 min (+95.3%) |
| Subagent task p50 +90.6%, p90 +248.5% | Work time p50 +79.5%, p90 +170.1% after clipping gaps over 5 minutes |
Child sessions exceed the 30-minute maxRuntimeMs cap | The cap is per invocation, not per session; long spans are parked-and-resumed children, 7.8% active |
Group 1 is local configuration you can change today. Group 2 is upstream work in the harness source, with file targets from a read-only source audit.
| # | Setting | Now | Change to | Why, from the data |
|---|---|---|---|---|
| 1 | process hygiene | 39 omp, 20 over 24 h | close stale sessions, then restart | Live launch p99 145.7 s and swap at 35.3 / 36.0 GB. Biggest immediate win, no config edit. |
| 2 | task.softRequestBudget | 200 (force-stop 300) | 60-80 | Child calls p50 36, p90 85. The current budget never fires, so a runaway child is only stopped at 300 requests. |
| 3 | task.maxRuntimeMs | 1,800,000 | 600,000-900,000 | Per-invocation cap. 30 min is far above child work-time p95 of 1,620 s, so it rarely bounds anything. |
| 4 | task.agentModelOverrides.session-evidence-auditor | gemini-3.6-flash:low | a mid-tier reasoning model | This exact route produced two wrong log figures in this investigation. Evidence work needs care, not speed. |
| 5 | scout routing | flash:low | keep | Flash subagent duration p50 1.43 s versus 5.59 s for sol. Correct for discovery; keep reasoning models off scouting. |
| 6 | advisor.enabled | false | keep false | Advisor was 27.4 percent of all calls and 6.42 B tokens over six weeks. |
| 7 | session hygiene per task | p50 176k ctx | new session past ~200k | Same model, TTFT p50 2.22 s at 50-100k versus 6.98 s at 250k+. 32.8 percent of calls already run above 200k. |
| 8 | task.maxConcurrency | 4 | keep 4 | Concurrency is not the problem; cumulative dispatch count is. Keep the cap and limit total waves by habit until a budget exists upstream. |
| 9 | dispatch habit | p90 32.6 children | stop at ~10 per task | No duration benefit measured between 3 and 10 dispatches; past 11 the median doubles. |
Repository targets verified by reading the source. Tagged by whether they address the recent regression or a long-standing defect.
| # | Change | Target | Class | Evidence |
|---|---|---|---|---|
| 1 | Record tool-call duration; add a per-turn attribution breakdown (model, tool, child, wait, compaction) | packages/stats/src/db.ts, parser.ts, meta key tool_calls_v2 | chronic | No duration column exists today, so no latency question can be answered without reconstruction |
| 2 | Ignore suspension in the loop watchdog: compare monotonic delta against wall-clock delta, or drop samples over ~30 s | packages/tui/src/loop-watchdog.ts | chronic | 4,379,642 ms recorded as one stall; 29,777 events total pollute any trend |
| 3 | Cumulative child and wave budget per parent run, not just a concurrency semaphore | packages/coding-agent/src/task/index.ts (TaskTool.execute), semaphore at ~line 620 | regression | p90 children per run 19 → 32.6; duration doubles past 11 dispatches |
| 4 | Make task.maxRuntimeMs session-scoped as well as per invocation, or add a separate session cap | packages/coding-agent/src/task/executor.ts ~line 1118 | chronic | Timer re-arms per turn; spans reach 2 h at 7.8 percent active |
| 5 | Apply the artifact spill threshold to read as it already applies elsewhere; make the cap uniform and configurable | spill wrapper at packages/coding-agent/src/sdk.ts ~line 2519 | chronic | grep/search/hub clamp near 51k with zero exceptions; read reached 460k in one result |
| 6 | Convergence guard: detect repeated verify/repair loops and force a wrap-up before request budget exhaustion | soft budget machinery in executor.ts (SOFT_REQUEST_BUDGET, buildBudgetNotice) | regression | Week of Jul 15: median 151 main calls per subagent task, p90 209 min |
| 7 | Batch terminal, log, and session-append work off the render path; add backpressure during heavy subagent activity | packages/tui render path; native offload seam in crates/pi-natives | chronic | Stalls in both windows; live p90 3.37 s with 908 events over 5 s |
| 8 | Prefetch or warm the launch path so modelRegistry.refresh() and SessionManager.open() do not sit on the critical path | executor.ts ~lines 2583 and 2697 | regression | invoke-to-first-chat p90 521 ms historically, 3,540 ms and p99 145.7 s under pressure |
| 9 | Latency-aware routing defaults per agent role using existing catalog metadata | packages/coding-agent/src/config/model-resolver.ts, packages/catalog/src/models.json | chronic | Catalog already carries contextWindow, cost, and reasoning fields; nothing consults measured latency |
| 10 | Per-child context budget and cheaper repeat compaction; surface compaction cost in telemetry | packages/coding-agent/src/session/session-maintenance.ts (PRUNE_CACHE_WARM_SUFFIX_TOKENS at line 141) | chronic | Main p50 context 175.6k across the whole period; observed compactions at 231k-318k |
| 11 | Retain one rolled log per day per PID so warning trends survive; today stale pruning keeps only five files globally | packages/utils/src/logger.ts (pruneStaleProcessLogs, RETAINED_STALE_LOG_FILES) | chronic | Jul 12-27 retained almost no log content, making counts untrendable |
| 12 | Propagate cancellation to child processes and reap orphans when a parent dies | packages/coding-agent/src/task/executor.ts, task/index.ts | chronic | 20 omp processes older than 24 h alongside 35.3 GB swap |
gpt-5.6-sol and Opus for correctness-critical slices.hub. Dispatch, continue other work, and let completion deliver.| # | Fix | Evidence | Expected effect |
|---|---|---|---|
| 1 | Instrument task critical path: active model time, tool time, child critical path, coordination wait, compaction, host snapshot | Attribution currently requires reconstruction from timestamps | Makes regressions detectable in days, not weeks |
| 2 | Strictly event-driven subagent completion; no periodic status, wait, or log polling | Coordination density +100.5%; median 4 waits per subagent task | Removes waiting from the critical path |
| 3 | Session-level child and wave budget, not just concurrency cap | p90 fan-out 19 → 32.6; p95 36 → 45.8 | Bounds worst-case task time |
| 4 | Convergence guard on repeated verify/repair loops | Jul 15 week: 151 median calls, p90 209 min | Prevents runaway task expansion |
| 5 | Batch terminal and subagent output off the UI thread | 29,777 stalls; live p90 3.37 s, worst 73 min | Restores local responsiveness |
| 6 | Complexity-aware routing defaults per subagent role | Flash 1.43 s vs sol 5.59 s subagent p50 | Cuts latency on discovery work |
| 7 | Warn on stale process accumulation and descendant RSS | 39 processes, 35.3 GB swap, live launch p99 145.7 s | Prevents silent host degradation |
| 8 | Worker pooling | Launch p95 ≤ 521 ms historically | Low priority; does not explain minute-scale loss |