Stale session JSON survives MCP-shim restart, silently breaks `ex` (PUB port mismatch) #35

New issue

Closed

opened 2026-04-28 06:37:04 -07:00 by prittjam · 2 comments

prittjam commented

2026-04-28 06:37:04 -07:00

(Migrated from github.com)

Summary

When the Julia gate process is replaced (e.g. Slurm job restart that rebinds
the same REQ port), kaimon resurrects the previous session from
~/.cache/kaimon/sock/.json instead of re-handshaking. REQ-path tools
(ping, connect_tcp, cancel_eval) keep working because the new gate happens to
bind the same REQ port. PUB-path tools (ex) hang forever because the cached
stream_endpoint points at the dead gate's PUB port, which the new gate did
not reuse.

Repro

Launch a gate via kaimon-julia-slurm
(Slurm: srun --pty julia ... -e 'Kaimon.Gate.serve(mode=:tcp, port=9876)').
Connect via MCP. Confirm ex works.
scancel the Slurm job (or kill the Julia PID). Re-launch the same
kaimon-julia-slurm command -- new Julia process, same REQ port 9876.
Restart the kaimon MCP shim (or /mcp reconnect from the client).
ping shows the session as connected.
Call ex e="Threads.nthreads()" q=false
-> hangs >30s, gets promoted to a background job, never completes.

Observed

ping reports session tcp-127.0.0.1-9876 as connected, free, with PID
matching the PREVIOUS gate.
~/.cache/kaimon/sock/tcp-127.0.0.1-9876.json still contains the dead
gate's pid and stream_endpoint. Example:

{
"pid": 464140,
"stream_endpoint": "tcp://127.0.0.1:44161",
"last_pong": "2026-04-28T15:26:13.583",
...
}
last_pong keeps updating (REQ port works), so kaimon's liveness check
considers the session healthy.
ss -ltnp shows 9876 bound by the new Julia, but 44161 is NOT listening
anywhere on the host -- nc -z 127.0.0.1 44161 -> Connection refused.
The new gate picked a different PUB port and never advertised it because
kaimon never re-handshook.

Workaround

kill <kaimon-mcp-pid>
rm ~/.cache/kaimon/sock/tcp-<host>-<port>.json
# then reconnect via /mcp and call connect_tcp again

After this, kaimon re-handshakes against the live gate, gets the correct
stream_endpoint, and ex returns immediately.

Suggested fix (one or more)

On startup / reconnect, always re-handshake. Treat the on-disk JSON as
a hint (project name, last-known endpoint), not as a source of truth
for pid / stream_endpoint. Rebuild from the live pong response.
Include a gate-instance nonce in the handshake (UUID generated when
Gate.serve starts). Persist it in the session file. If a pong returns
a different nonce than the cached one, invalidate the file and
re-handshake. This survives REQ-port reuse, which PID alone does not
(and PID is also unreliable under Slurm cgroup namespacing -- host PID
and in-cgroup PID differ).
Sanity-check stream_endpoint reachability before declaring the session
"connected". A simple non-blocking connect on the PUB port at handshake
time would have surfaced this immediately as "stream port unreachable"
instead of an indefinite ex hang.
Time out promoted background ex jobs when no PUB messages have ever
been received on the session, with an error pointing at stream_endpoint
mismatch -- easier to diagnose than a silent hang.

Environment

kaimon MCP server: julia 1.12.6, julia -m Kaimon, server uptime 16h,
MCP listening on 127.0.0.1:2828
Gate: julia -t 96 --project=... -i -e 'Kaimon.Gate.serve(mode=:tcp, host="127.0.0.1", port=9876)'
under srun --pty on a 224-thread Slurm node
Client: Claude Code MCP client
Host OS: Linux 6.8.0-90-generic

Summary ------- When the Julia gate process is replaced (e.g. Slurm job restart that rebinds the same REQ port), kaimon resurrects the previous session from ~/.cache/kaimon/sock/<session>.json instead of re-handshaking. REQ-path tools (ping, connect_tcp, cancel_eval) keep working because the new gate happens to bind the same REQ port. PUB-path tools (ex) hang forever because the cached stream_endpoint points at the dead gate's PUB port, which the new gate did not reuse. Repro ----- 1. Launch a gate via `kaimon-julia-slurm` (Slurm: srun --pty julia ... -e 'Kaimon.Gate.serve(mode=:tcp, port=9876)'). Connect via MCP. Confirm `ex` works. 2. scancel the Slurm job (or kill the Julia PID). Re-launch the same `kaimon-julia-slurm` command -- new Julia process, same REQ port 9876. 3. Restart the kaimon MCP shim (or /mcp reconnect from the client). 4. `ping` shows the session as connected. 5. Call ex e="Threads.nthreads()" q=false -> hangs >30s, gets promoted to a background job, never completes. Observed -------- - `ping` reports session tcp-127.0.0.1-9876 as `connected, free`, with PID matching the PREVIOUS gate. - ~/.cache/kaimon/sock/tcp-127.0.0.1-9876.json still contains the dead gate's pid and stream_endpoint. Example: { "pid": 464140, "stream_endpoint": "tcp://127.0.0.1:44161", "last_pong": "2026-04-28T15:26:13.583", ... } - last_pong keeps updating (REQ port works), so kaimon's liveness check considers the session healthy. - `ss -ltnp` shows 9876 bound by the new Julia, but 44161 is NOT listening anywhere on the host -- `nc -z 127.0.0.1 44161` -> Connection refused. The new gate picked a different PUB port and never advertised it because kaimon never re-handshook. Workaround ---------- kill <kaimon-mcp-pid> rm ~/.cache/kaimon/sock/tcp-<host>-<port>.json # then reconnect via /mcp and call connect_tcp again After this, kaimon re-handshakes against the live gate, gets the correct stream_endpoint, and `ex` returns immediately. Suggested fix (one or more) --------------------------- 1. On startup / reconnect, always re-handshake. Treat the on-disk JSON as a hint (project name, last-known endpoint), not as a source of truth for pid / stream_endpoint. Rebuild from the live pong response. 2. Include a gate-instance nonce in the handshake (UUID generated when Gate.serve starts). Persist it in the session file. If a pong returns a different nonce than the cached one, invalidate the file and re-handshake. This survives REQ-port reuse, which PID alone does not (and PID is also unreliable under Slurm cgroup namespacing -- host PID and in-cgroup PID differ). 3. Sanity-check stream_endpoint reachability before declaring the session "connected". A simple non-blocking connect on the PUB port at handshake time would have surfaced this immediately as "stream port unreachable" instead of an indefinite `ex` hang. 4. Time out promoted background `ex` jobs when no PUB messages have ever been received on the session, with an error pointing at stream_endpoint mismatch -- easier to diagnose than a silent hang. Environment ----------- - kaimon MCP server: julia 1.12.6, `julia -m Kaimon`, server uptime 16h, MCP listening on 127.0.0.1:2828 - Gate: `julia -t 96 --project=... -i -e 'Kaimon.Gate.serve(mode=:tcp, host="127.0.0.1", port=9876)'` under `srun --pty` on a 224-thread Slurm node - Client: Claude Code MCP client - Host OS: Linux 6.8.0-90-generic

prittjam commented

2026-04-30 06:13:23 -07:00

(Migrated from github.com)

The kaimon launcher sometimes doesn't write the tcp-.json metadata file in ~/.cache/kaimon/sock/. We saw this twice this session — the new RVG kaimon ran fine but no json appeared, while old ones
(kaimon-features) had stale jsons.
The MCP server discovers sessions by reading those json files. If the json is missing or stale, MCP either has no session to address or has the wrong PID/name attached. After /mcp reconnect the cache may
still be stale.
The MCP API addresses sessions by port-derived keys (tcp-127.0.0.1-9876), not your friendly --session-id rvg. So even when MCP sees the right session it doesn't know "rvg" by name.

What I end up doing each restart: ss -tlnp | grep 987[0-9] (find which port a julia PID is listening on), then connect_tcp(host=127.0.0.1, port=) to manually register, then
ex(ses="tcp-127.0.0.1-", ...). It works reliably but is more steps than it should take.

1. The kaimon launcher sometimes doesn't write the tcp-<port>.json metadata file in ~/.cache/kaimon/sock/. We saw this twice this session — the new RVG kaimon ran fine but no json appeared, while old ones (kaimon-features) had stale jsons. 2. The MCP server discovers sessions by reading those json files. If the json is missing or stale, MCP either has no session to address or has the wrong PID/name attached. After /mcp reconnect the cache may still be stale. 3. The MCP API addresses sessions by port-derived keys (tcp-127.0.0.1-9876), not your friendly --session-id rvg. So even when MCP sees the right session it doesn't know "rvg" by name. What I end up doing each restart: ss -tlnp | grep 987[0-9] (find which port a julia PID is listening on), then connect_tcp(host=127.0.0.1, port=<port>) to manually register, then ex(ses="tcp-127.0.0.1-<port>", ...). It works reliably but is more steps than it should take.

kahliburke commented

2026-06-16 04:25:49 -07:00

(Migrated from github.com)

Fixed in 2.0 — stale session metadata no longer resurrects a dead gate PUB endpoint; the SUB socket is replaced when a gate restarts on a new PUB port, with test coverage. Shipping in 2.0.