#51 - Orphaned Gate.serve() REPLs are never reaped — multi-GB leak under headless server - kahliburke/Kaimon.jl

1-Bart-1 commented

2026-06-07 12:21:15 -07:00

(Migrated from github.com)

Summary

Gate worker processes started via Gate.serve() register a session with the server but are never terminated when abandoned. Over many editor/agent sessions and projects they accumulate as idle multi-GB Julia processes. On my machine I found four orphaned serve REPLs (one project had two) plus a leaked /tmp/kaimon_test_runner child — ~20 GB of RSS held by processes that had been idle for hours to over a day.

Root cause

close_session! (src/session.jl) only sets session.state = CLOSED — it never signals the OS process, so the worker keeps running.
The only stale-session reaper (_reap_stale_sessions! in src/tui/view.jl) is driven from the TUI render loop and operates on MCP client sessions (STANDALONE_SESSIONS), not on the gate worker processes (ConnectionManager REPLConnections with .pid). A headless -m Kaimon server never ticks it.
Idle can't be measured from last_seen: health pings flow through _req_send_recv, which refreshes last_seen every cycle. There was no activity timestamp independent of pings.

Net effect: nothing ever sends a signal to an abandoned gate, even though REPLConnection already carries everything needed (pid, project_path, status, allow_restart).

Proposed fix

Branch: 1-Bart-1:fix/reap-orphaned-serve-repls · commit 7a8ec78

Reap idle gates from the ConnectionManager health loop (runs in both TUI and headless modes):

Add a ping-independent last_tool_call timestamp, stamped via _note_tool_activity! at every real tool call / eval.
_should_reap_idle_gate flags a :connected, restartable, non-paused gate idle beyond a threshold (evaluating / stalled / debug-paused / allow_restart=false extension gates are skipped).
Such gates are shut down gracefully via send_shutdown! and pruned through the existing removal path.
Threshold is a new gate_idle_reap_seconds preference, default 0 (disabled) so behaviour is opt-in.

Happy to open this as a PR.

## Summary Gate worker processes started via `Gate.serve()` register a session with the server but are never terminated when abandoned. Over many editor/agent sessions and projects they accumulate as idle multi-GB Julia processes. On my machine I found four orphaned serve REPLs (one project had two) plus a leaked `/tmp/kaimon_test_runner` child — ~20 GB of RSS held by processes that had been idle for hours to over a day. ## Root cause 1. `close_session!` (`src/session.jl`) only sets `session.state = CLOSED` — it never signals the OS process, so the worker keeps running. 2. The only stale-session reaper (`_reap_stale_sessions!` in `src/tui/view.jl`) is driven from the **TUI render loop** and operates on MCP *client* sessions (`STANDALONE_SESSIONS`), not on the gate worker processes (`ConnectionManager` `REPLConnection`s with `.pid`). A headless `-m Kaimon` server never ticks it. 3. Idle can't be measured from `last_seen`: health pings flow through `_req_send_recv`, which refreshes `last_seen` every cycle. There was no activity timestamp independent of pings. Net effect: nothing ever sends a signal to an abandoned gate, even though `REPLConnection` already carries everything needed (`pid`, `project_path`, `status`, `allow_restart`). ## Proposed fix Branch: [`1-Bart-1:fix/reap-orphaned-serve-repls`](https://github.com/kahliburke/Kaimon.jl/compare/main...1-Bart-1:Kaimon.jl:fix/reap-orphaned-serve-repls) · commit [`7a8ec78`](https://github.com/1-Bart-1/Kaimon.jl/commit/7a8ec78e9f73bce3ad90bece77cb41f93d8e1036) Reap idle gates from the `ConnectionManager` health loop (runs in both TUI and headless modes): - Add a ping-independent `last_tool_call` timestamp, stamped via `_note_tool_activity!` at every real tool call / eval. - `_should_reap_idle_gate` flags a `:connected`, restartable, non-paused gate idle beyond a threshold (evaluating / stalled / debug-paused / `allow_restart=false` extension gates are skipped). - Such gates are shut down gracefully via `send_shutdown!` and pruned through the existing removal path. - Threshold is a new `gate_idle_reap_seconds` preference, **default `0` (disabled)** so behaviour is opt-in. Happy to open this as a PR.

kahliburke commented

2026-06-08 23:04:04 -07:00

(Migrated from github.com)

@1-Bart-1 Thank you for the bug report, I will take a look. I am working on a 2.0.0 release and I may have fixed a related issue but I will make sure to incorporate any additional fixes in your branch.

🚀 1

kahliburke commented

2026-06-16 01:47:30 -07:00

(Migrated from github.com)

Thanks for the detailed report and the branch, @1-Bart-1. I looked into this for the 2.0 release.

When I instrumented it, Kaimon-spawned sessions (start_session and agent sessions) don't actually orphan. I tested both exit paths on a headless server:

Clean restart: the spawned gate is reaped by stop_all_sessions!.
kill -9 of the server: the spawned gate still dies — it's a PTY child, so it gets SIGHUP when the parent goes away.

A process check after each confirmed no spawned gate survived. So the persistent multi-GB Gate.serve() REPLs are user/editor-started gates — separate processes Kaimon connects to but doesn't own.

That persistence is intended. Sessions are meant to survive a Kaimon restart so REPL state isn't lost; a restarted Kaimon (or an agent) finds them again via socket discovery / ping and reuses them. So Kaimon doesn't force-kill gates it didn't spawn — one might hold work you still want. An opt-in idle cleanup scoped to spawned sessions only is something I'd consider as a future feature.

One item in your report is a real leak and I'll fix it: the /tmp/kaimon_test_runner child. The test-runner subprocess isn't tied into shutdown and can orphan if a run hangs or Kaimon exits mid-run, and its temp script isn't cleaned up. I'll track in-flight runs, reap them on shutdown, and remove the temp file.

Your report also prompted a wider change: the maintenance you noted was TUI-only (stale-session reaping, extension/session monitoring) now runs headless too in 2.0, along with headless analytics-DB init and periodic indexing.

One question: were the four orphaned serve REPLs started by start_session / the VS Code extension, or by hand (julia> Gate.serve())? I'll revisit this after the 2.0 release — if you can share more detail on how those REPLs were started and your setup, it'll help me track down whether there's a real issue in Kaimon's path.

Thanks again.

Thanks for the detailed report and the branch, @1-Bart-1. I looked into this for the 2.0 release. When I instrumented it, Kaimon-spawned sessions (`start_session` and agent sessions) don't actually orphan. I tested both exit paths on a headless server: - Clean restart: the spawned gate is reaped by `stop_all_sessions!`. - `kill -9` of the server: the spawned gate still dies — it's a PTY child, so it gets `SIGHUP` when the parent goes away. A process check after each confirmed no spawned gate survived. So the persistent multi-GB `Gate.serve()` REPLs are user/editor-started gates — separate processes Kaimon connects to but doesn't own. That persistence is intended. Sessions are meant to survive a Kaimon restart so REPL state isn't lost; a restarted Kaimon (or an agent) finds them again via socket discovery / `ping` and reuses them. So Kaimon doesn't force-kill gates it didn't spawn — one might hold work you still want. An opt-in idle cleanup scoped to spawned sessions only is something I'd consider as a future feature. One item in your report is a real leak and I'll fix it: the `/tmp/kaimon_test_runner` child. The test-runner subprocess isn't tied into shutdown and can orphan if a run hangs or Kaimon exits mid-run, and its temp script isn't cleaned up. I'll track in-flight runs, reap them on shutdown, and remove the temp file. Your report also prompted a wider change: the maintenance you noted was TUI-only (stale-session reaping, extension/session monitoring) now runs headless too in 2.0, along with headless analytics-DB init and periodic indexing. One question: were the four orphaned serve REPLs started by `start_session` / the VS Code extension, or by hand (`julia> Gate.serve()`)? I'll revisit this after the 2.0 release — if you can share more detail on how those REPLs were started and your setup, it'll help me track down whether there's a real issue in Kaimon's path. Thanks again.

1-Bart-1 commented

2026-06-17 15:14:57 -07:00

(Migrated from github.com)

I start the REPLs by hand (julia> Gate.serve()). Then using claude code to connect to the MCP server.

1-Bart-1 commented

2026-06-17 15:16:04 -07:00

(Migrated from github.com)

Thanks for looking into this, and explaining this. Makes sense.

kahliburke commented

2026-06-19 12:33:19 -07:00

(Migrated from github.com)

Following up now that 2.0 is further along, and given your confirmation that all four orphaned REPLs were hand-started (julia> Gate.serve()) rather than via start_session / the VS Code extension.

Decision: not taking the idle-reaper approach from the branch as-is.

The proposed fix reaps idle gates from the ConnectionManager health loop based on a gate_idle_reap_seconds threshold. The problem is it can't distinguish "abandoned" from "idle but wanted" for gates Kaimon doesn't own. A hand-started Gate.serve() REPL is a process Kaimon merely connects to — it may hold expensive in-memory state (loaded data, a half-finished computation) you want to keep across a Kaimon restart. That cross-restart persistence is intentional: a restarted Kaimon (or a fresh agent) rediscovers these gates via socket discovery / ping and reuses them. Time-based force-reaping would kill exactly the long-lived REPLs people deliberately keep running.

What did change for 2.0:

Kaimon-spawned sessions (start_session, agent sessions) are confirmed not to orphan — reaped by stop_all_sessions! on clean stop, and they die on kill -9 of the server too (PTY children get SIGHUP).
The maintenance that was TUI-only (stale-session reaping, session/extension monitoring) now runs headless as well.

So for the user-started case there isn't a Kaimon-side leak to fix per se — but I don't want to just close this, because the underlying ask ("don't let abandoned hand-started REPLs pile up") is real. Before adding anything I'd want to investigate:

An opt-in, spawned-or-explicitly-tagged-only idle cleanup that never force-kills a gate Kaimon didn't start (or that wasn't declared reapable), so the safe default is preserved.
A ping-independent activity signal — the health ping refreshes last_seen, so idleness can't be measured from it today. The branch's last_tool_call idea is reasonable here.
Whether a lighter-weight / lower-memory idle mode for serve gates is a better lever than reaping at all (overlaps with the memory-footprint discussion in #33).

Keeping this open as an investigation item. Thanks again @1-Bart-1 — the report and branch were a useful prompt even though the fix will likely land differently.

Following up now that 2.0 is further along, and given your confirmation that all four orphaned REPLs were hand-started (`julia> Gate.serve()`) rather than via `start_session` / the VS Code extension. **Decision: not taking the idle-reaper approach from the branch as-is.** The proposed fix reaps idle gates from the `ConnectionManager` health loop based on a `gate_idle_reap_seconds` threshold. The problem is it can't distinguish "abandoned" from "idle but wanted" for gates Kaimon doesn't own. A hand-started `Gate.serve()` REPL is a process Kaimon merely connects to — it may hold expensive in-memory state (loaded data, a half-finished computation) you want to keep across a Kaimon restart. That cross-restart persistence is intentional: a restarted Kaimon (or a fresh agent) rediscovers these gates via socket discovery / `ping` and reuses them. Time-based force-reaping would kill exactly the long-lived REPLs people deliberately keep running. What *did* change for 2.0: - Kaimon-**spawned** sessions (`start_session`, agent sessions) are confirmed not to orphan — reaped by `stop_all_sessions!` on clean stop, and they die on `kill -9` of the server too (PTY children get `SIGHUP`). - The maintenance that was TUI-only (stale-session reaping, session/extension monitoring) now runs headless as well. So for the user-started case there isn't a Kaimon-side leak to fix per se — but I don't want to just close this, because the underlying ask ("don't let abandoned hand-started REPLs pile up") is real. Before adding anything I'd want to investigate: - An **opt-in, spawned-or-explicitly-tagged-only** idle cleanup that never force-kills a gate Kaimon didn't start (or that wasn't declared reapable), so the safe default is preserved. - A ping-independent activity signal — the health ping refreshes `last_seen`, so idleness can't be measured from it today. The branch's `last_tool_call` idea is reasonable here. - Whether a lighter-weight / lower-memory idle mode for serve gates is a better lever than reaping at all (overlaps with the memory-footprint discussion in #33). Keeping this open as an investigation item. Thanks again @1-Bart-1 — the report and branch were a useful prompt even though the fix will likely land differently.

🚀 1

kahliburke commented

2026-07-08 22:18:32 -07:00

(Migrated from github.com)

@1-Bart-1 Please check out 2.0 which has shipped, there is more agent/session affinity which may help your use cases. Otherwise please open an issue against 2.0.

🚀 1