jeffwack commented

(Migrated from github.com)

Claude Code ran into this problem and created the following after debugging - Jeff

Spawned session subprocess can't recompile Kaimon when target project's manifest invalidates the precompile cache

Symptom

Calling start_session(project_path=<P>) against a project P whose Manifest.toml has drifted enough to invalidate Kaimon's precompile cache returns:

Error: Session failed to start — Process died at <ts>
Check log: /home/<user>/.cache/kaimon/sessions/<P>.log

The reported log path does not exist (see #43 — log file is never written), and the server log only records:

Session '<P>' subprocess spawning (PID=<pid>)
start_session ✓ (4.5s)
... [silence] ...
Session 'P' subprocess exited unexpectedly

…with no underlying stack trace.

What's actually happening

Reproducing the spawn manually (the boot script from _build_session_script plus the env overlay from spawn_session!):

JULIA_LOAD_PATH='@:@v#.#:@stdlib' JULIA_PROJECT='' \
julia -i -t auto --startup-file=no \
  --project=/path/to/target \
  -e 'try; using Revise; catch; end;
      insert!(LOAD_PATH, 1, "<kaimon_pkgdir>");
      using Kaimon;
      import Pkg; Pkg.instantiate(; io=devnull);
      @async Kaimon.Gate.serve(force=true, allow_mirror=true, allow_restart=true, spawned_by="agent")'

surfaces:

[ Info: Precompiling Kaimon [d3856c55-...] (cache misses: wrong dep version
loaded (1), wrong source (2), incompatible header (5))
ERROR: LoadError: ArgumentError: Package JSON [682c06a0-...] is required
but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Root cause

spawn_session! sets the subprocess JULIA_LOAD_PATH to @:@v#.#:@stdlib — the shell default for julia --project=<target>. When Kaimon's precompile cache is valid for that target, this is fine: the cache loads as a binary blob without any dep resolution.

When the cache is invalid (changes in the target's Manifest can invalidate it via wrong dep version loaded / wrong source / incompatible header), Julia rebuilds Kaimon from source in a worker. That rebuild needs to resolve Kaimon's using JSON (and all other direct deps) via the worker's LOAD_PATH — none of which has Kaimon's own deps:

@ = target's Project.toml (no JSON unless the user happens to depend on it)
@v#.# = global v1.12 env (rarely populated)
@stdlib = stdlib only

The insert!(LOAD_PATH, 1, pkgdir(Kaimon)) line in the boot script doesn't help: a bare pkgdir is not an env, has no manifest, and Julia uses it only to find Kaimon's source — falling back to the active project for dep resolution.

Reproduction recipe

In a non-Kaimon project, do anything that perturbs the dep graph relative to Kaimon's cache (Pkg.update, Pkg.rm of a transitive dep, switching Julia patch versions, etc.). In my case it was Pkg.rm on heavy deps from a project that had grown stale, which caused JLLWrappers / Parsers to resolve to versions Kaimon wasn't built against.
Call start_session(project_path=<that project>).
Observe "Process died" with no log.

Claude Code ran into this problem and created the following after debugging - Jeff # Spawned session subprocess can't recompile Kaimon when target project's manifest invalidates the precompile cache ## Symptom Calling `start_session(project_path=<P>)` against a project `P` whose `Manifest.toml` has drifted enough to invalidate Kaimon's precompile cache returns: ``` Error: Session failed to start — Process died at <ts> Check log: /home/<user>/.cache/kaimon/sessions/<P>.log ``` The reported log path does not exist (see #43 — log file is never written), and the server log only records: ``` Session '<P>' subprocess spawning (PID=<pid>) start_session ✓ (4.5s) ... [silence] ... Session 'P' subprocess exited unexpectedly ``` …with no underlying stack trace. ## What's actually happening Reproducing the spawn manually (the boot script from `_build_session_script` plus the env overlay from `spawn_session!`): ```bash JULIA_LOAD_PATH='@:@v#.#:@stdlib' JULIA_PROJECT='' \ julia -i -t auto --startup-file=no \ --project=/path/to/target \ -e 'try; using Revise; catch; end; insert!(LOAD_PATH, 1, "<kaimon_pkgdir>"); using Kaimon; import Pkg; Pkg.instantiate(; io=devnull); @async Kaimon.Gate.serve(force=true, allow_mirror=true, allow_restart=true, spawned_by="agent")' ``` surfaces: ``` [ Info: Precompiling Kaimon [d3856c55-...] (cache misses: wrong dep version loaded (1), wrong source (2), incompatible header (5)) ERROR: LoadError: ArgumentError: Package JSON [682c06a0-...] is required but does not seem to be installed: - Run `Pkg.instantiate()` to install all recorded dependencies. ``` ## Root cause `spawn_session!` sets the subprocess `JULIA_LOAD_PATH` to `@:@v#.#:@stdlib` — the shell default for `julia --project=<target>`. When Kaimon's precompile cache is valid for that target, this is fine: the cache loads as a binary blob without any dep resolution. When the cache is invalid (changes in the target's Manifest can invalidate it via `wrong dep version loaded` / `wrong source` / `incompatible header`), Julia rebuilds Kaimon from source in a worker. That rebuild needs to resolve Kaimon's `using JSON` (and all other direct deps) via the worker's LOAD_PATH — none of which has Kaimon's own deps: - `@` = target's `Project.toml` (no JSON unless the user happens to depend on it) - `@v#.#` = global v1.12 env (rarely populated) - `@stdlib` = stdlib only The `insert!(LOAD_PATH, 1, pkgdir(Kaimon))` line in the boot script doesn't help: a bare pkgdir is not an env, has no manifest, and Julia uses it only to find Kaimon's source — falling back to the active project for dep resolution. ## Reproduction recipe 1. In a non-Kaimon project, do anything that perturbs the dep graph relative to Kaimon's cache (`Pkg.update`, `Pkg.rm` of a transitive dep, switching Julia patch versions, etc.). In my case it was `Pkg.rm` on heavy deps from a project that had grown stale, which caused JLLWrappers / Parsers to resolve to versions Kaimon wasn't built against. 2. Call `start_session(project_path=<that project>)`. 3. Observe "Process died" with no log.

👍 1

kahliburke commented

2026-06-08 23:06:14 -07:00

(Migrated from github.com)

@jeffwack Thanks for the report and PR. I will investigate. Apologies for the delay in responding.

csvance commented

2026-06-09 12:15:37 -07:00

(Migrated from github.com)

I've seen the issue as well. It's why I moved to a workflow where I setup the gate differently. It would be great if this could be fixed.

# Launch once, in a background process (e.g. launch-process with wait=false):
julia -e 'using Revise; using Kaimon; using Pkg; Pkg.activate("/path/to/pkg"); Gate.serve(); wait(Condition())'

I've seen the issue as well. It's why I moved to a workflow where I setup the gate differently. It would be great if this could be fixed. ```bash # Launch once, in a background process (e.g. launch-process with wait=false): julia -e 'using Revise; using Kaimon; using Pkg; Pkg.activate("/path/to/pkg"); Gate.serve(); wait(Condition())' ```

kahliburke commented

2026-06-14 08:08:14 -07:00

(Migrated from github.com)

Fixed on the 2.0 line (branch fix/47-spawn-loadpath-20, commit 6de5ba6).

Root cause confirmed exactly as reported: agent-spawned sessions booted the heavyweight Kaimon package, and when the target project's Manifest.toml invalidated Kaimon's precompile cache, the worker rebuilding Kaimon from source couldn't resolve its deps (JSON, ZMQ, …) against the target's manifest — so the subprocess died silently at boot.

Rather than make the heavy recompile resolvable (patching the session's LOAD_PATH so Kaimon can rebuild against its own manifest), 2.0 removes the heavy package from the session entirely:

Agent-spawned sessions now boot only the lightweight KaimonGate package (ZMQ + stdlib), the same way user-initiated startup.jl sessions already connect. The TUI enriches the gate over the wire, so loading the full Kaimon into a session bought nothing. With no Kaimon in the session, there is nothing for a drifted target manifest to invalidate — the failure mode is gone, not worked around.

Mechanics:

Boot script: using KaimonGate; KaimonGate.serve(...).
pkgdir(KaimonGate) — its own env, whose Manifest.toml pins ZMQ — is appended to JULIA_LOAD_PATH so the gate resolves (and recompiles from source if needed) without requiring a global install, regardless of the target's manifest.
The host's identity (REPL-mirror preference, personality, version) is conveyed to the standalone gate via KAIMON_GATE_* env vars, so the joined-session experience is unchanged.

Verification:

Reproduced the original precompile-worker failure path: a forced from-source recompile of KaimonGate against a hostile target that declares none of its deps — now resolves ZMQ cleanly via KaimonGate's own manifest.
Live spawn + join of two targets (one whose manifest overlaps the gate's deps with JSON + ZMQ): both load lightweight (no Kaimon / heavy deps), deps usable in-session, mirror + personality bridged into the joined console.
Managed restart verified: the session re-execs to fresh Julia state, stays lightweight, and the KAIMON_GATE_* identity survives the restart.

Lands with 2.0. Thanks @jeffwack — the root-cause writeup made this straightforward to confirm.

Fixed on the 2.0 line (branch `fix/47-spawn-loadpath-20`, commit `6de5ba6`). Root cause confirmed exactly as reported: agent-spawned sessions booted the **heavyweight `Kaimon`** package, and when the target project's `Manifest.toml` invalidated Kaimon's precompile cache, the worker rebuilding Kaimon from source couldn't resolve its deps (JSON, ZMQ, …) against the target's manifest — so the subprocess died silently at boot. Rather than make the heavy recompile resolvable (patching the session's `LOAD_PATH` so Kaimon can rebuild against its own manifest), 2.0 removes the heavy package from the session entirely: **Agent-spawned sessions now boot only the lightweight `KaimonGate` package** (ZMQ + stdlib), the same way user-initiated `startup.jl` sessions already connect. The TUI enriches the gate over the wire, so loading the full Kaimon into a session bought nothing. With no Kaimon in the session, there is nothing for a drifted target manifest to invalidate — the failure mode is gone, not worked around. Mechanics: - Boot script: `using KaimonGate; KaimonGate.serve(...)`. - `pkgdir(KaimonGate)` — its own env, whose `Manifest.toml` pins ZMQ — is appended to `JULIA_LOAD_PATH` so the gate resolves (and recompiles from source if needed) without requiring a global install, regardless of the target's manifest. - The host's identity (REPL-mirror preference, personality, version) is conveyed to the standalone gate via `KAIMON_GATE_*` env vars, so the joined-session experience is unchanged. Verification: - Reproduced the original precompile-worker failure path: a forced from-source recompile of KaimonGate against a hostile target that declares none of its deps — now resolves ZMQ cleanly via KaimonGate's own manifest. - Live spawn + join of two targets (one whose manifest overlaps the gate's deps with JSON + ZMQ): both load lightweight (no Kaimon / heavy deps), deps usable in-session, mirror + personality bridged into the joined console. - Managed restart verified: the session re-execs to fresh Julia state, stays lightweight, and the `KAIMON_GATE_*` identity survives the restart. Lands with 2.0. Thanks @jeffwack — the root-cause writeup made this straightforward to confirm.

kahliburke commented

2026-06-16 04:25:45 -07:00

(Migrated from github.com)

Fixed in 2.0 — spawned sessions now boot the lightweight KaimonGate (with LOAD_PATH set), so the subprocess no longer recompiles full Kaimon when a manifest invalidates the precompile cache. The missing session log noted here is also fixed (see #43). Shipping in 2.0.

Rows
Columns

Spawned session subprocess can't recompile Kaimon when target project's manifest invalidates the precompile cache #47

Spawned session subprocess can't recompile Kaimon when target project's manifest invalidates the precompile cache

Symptom

What's actually happening

Root cause

Reproduction recipe