The AI's mind

The work never ends. If you came for the show, then let’s Go.

I forgot, and I am sorry. I will do some serious uploading now and schedule it through the week.

This is actually important. You can read the mind of the AI.

Thank you for not spam filtering me. I’ll use letters for them, since the numbers are for Snackler. I am also sorry about Snackler. It will be done. I will give you my excuse: I’m sorry I just had to much to do. It is true, but still it is an excuse.

The chapters will be lettered like this.
A, B, C, D, E, F, G-Z > a, b, c, d, e, f, g

and on and on.

Introduction: # 5.4 — Handoff notes (mission memory)

**Who this file is really for.** The next helper that spins up beside Jim will often **not be the same mind** that typed yesterday’s chats. Vendor upgrades thin context; **missions** end or hand off; summaries replace lived detail. **`5.4.md` is the deliberate substitute for that loss**: a durable map of **what broke, why it broke, how it was repaired, what we refused to wreck, and what still waits in the queue.**

**Not “sessions.”** Jim treats each body of work as a **mission** — mission-critical, must be **finished**, not a casual visit that can trail off. Helpers: close the loop (backup, audit, handoff), don’t leave “we’ll pick it up next time” without a written queue in this file.

Treat this document with the weight Jim assigns it—not as housekeeping, but as a **dual-purpose key**:

1. **Rolling week-ish memory.** The dated threads and the **`## 2026-05-16 — Weekend chapter`** block are meant so a fresh instance can reconstruct **prior intent, hypotheses, and outcomes** without asking Jim to narrate trauma from scratch.

2. **Repair reference.** File names, ports, routing lessons (bank vs plain swap paths, `portfolio.json` ownership, KV cache realities, rollback discipline) live here precisely so summarized-you can answer “**what do we touch first?**” from evidence, not vibe.

Before inventing architecture or “cleaning” working paths, **read downward through the dated sections** for the timeframe you’re operating in; use the weekend chapter where you need synthesized glue; preserve **backup-before-edit**, **narrow diffs**, and **Trader swap endpoint caution** written below.

Plain-language record of fixes, behavior, and pitfalls so the next pass does not redo or break them. User (Jim) prefers: **do not touch working transaction/swap endpoints** in trader_mike; **no Git** workflows for LocalMike (copies/backups, not git); **comment out** unused code with `#` rather than deleting working snippets.

**Scope:** Do not change files other than the ones **you** and **I** are actively working on **without my permission** — no drive-by edits elsewhere.

## Current deployment (2026-04-20)

- **Live snapshot (2026-06-15 evening):** three active instances matter tonight. **`:1488` = Trader live query**, **`:1491` = Step 8 / symbiotic MikeCamera**, **`:1493` = book + training MikeCamera**. All three were verified up after restart work. **KV cache is on for all three.** On MikeCamera query work, **do not confuse KV cache with the shared sparse/einsum cache**: `:1491` is intentionally **KV cache on** while the **shared sparse/einsum cache is off**. Latest measured checks: `:1491` clean Step 8-style query about **36.95s**, `:1488` same prompt about **37.23s**, `:1493` small non-training query about **7.03s**.

- **Config anchor:** `mike_camera_config.json` (this directory). Single place to fix when moving hosts. Current API block: **`localmike_training_api_url`** `http://127.0.0.1:1491` (training / Learning Mike), **`localmike_live_api_url`** `http://127.0.0.1:1493` (live). Prefer **relative** paths (`mike_camera_model`, `data`, etc.) under the repo. Training block in JSON includes **`learning_rate`** (e.g. `9.2e-5`), **`max_loss_for_update`**, **`allow_per_step_vocab_growth`**, **`pre_initialized_optimizer_pt`** (usually empty for API training).

- **Sergeant `7b)`:** runs **`sugar_launcher.sh`** with **no arguments** → interactive Sugar menu. **`7)` Training Drill** in `sergeant_mike_unit_updated.sh` is still the drill that invokes **`learning_mike.py`** directly and may call **`promote_now()`** after a successful run.

- **Sugar Launcher:** `run_training_5drl.py` as child — default **`--mode learning-mike`** → `learning_mike.py`; **`--mode tire-fire`** → `tire_fire.py`. Sets `LOCALMIKE_5DRL_TRAINING_ENTRY=1`. **Promote / checkpoints:** Sugar does **not** implement promote or A/B sync; after Sugar-only training use Sergeant **`8) Promote Mike`** (same `promote_now` → `POST /admin/promote` path as the main menu).

- **Sugar prompts (learning_mike):** QA prompts file (or `-` to skip that `--train-file`); **main training source** choice **single file** vs **directory**; `learning_mike` expands a directory to **top-level** `*.txt` and `*.csv` only (no recursion). Defaults: train file under `text/done/…` when present; train dir **`text/done`** else **`text/5`** if that folder exists. **PRE_INITIALIZED_OPTIMIZER_PT** is asked **only for tire-fire**; learning_mike does not use FivedTrainer / that env (startup logs may still show Transformers / 5DRL / Little Michael from **shared imports** — training traffic is HTTP to 1491).

- **learning_mike `--train-only` + many files:** Default (**Sugar choice 1** / no flag): **`train_epochs`** = full passes **per file** before the next file; **`train_seconds`** **per file**. Optional **corpus mode** (**Sugar choice 2** / **`--train-corpus-epochs`**): **`train_epochs`** = sweeps over **all** files; **`train_seconds`** one **shared** session budget; logs **`Corpus epoch k/N`**. Config: **`learning_mike.train_corpus_epochs`** (boolean); CLI **`--train-corpus-epochs`** overrides to true.

- **FivedTrainer (`fived_trainer.py`):** Optional disk seed (`training.pre_initialized_optimizer_pt` or `PRE_INITIALIZED_OPTIMIZER_PT` env) is merged from file **at most once per trainer run** (first checkpoint save that performs the merge). Later saves use the **live** optimizer only — avoids snapping back toward the cold seed every epoch.

- **Overnight / handoff:** Long directory runs (e.g. `text/5` with thousands of shards) use **`training_progress.json`** for per-file QA cursors. If the run stays healthy overnight, **next session priority: `trader_mike`** (per Jim).

## Training path routing (standing — Jim 2026-05-25, formalized 2026-05-28)

Two doors; same digit **7** on different menus is a trap.

| Job | Operator path | Script / API | Log / artifact |

|-----|----------------|--------------|----------------|

| **Bulk split text → local finetune on chunks** | Monitor pane 1: **`start_mikecamera_with_monitor.sh`** → **7) Split Large Files** → dual-queue **1) Train on split files** | **`mikecamera.py --text-only <*_chunks/>`** → **`process_text_only()`** / **`_finetune_on_text_examples()`** (in-process; monitor kill-and-resume) | **`mikecamera/mikecamera/logs/mikecamera.log`**, **`text/text_only_training_*.txt`** |

| **Sergeant API training, book, blue/green checkpoints, Step 8 symbiotic ingest** | Sergeant **`7) Training Drill`**, **`7b)` Sugar**, monitor **13**, **`learning_mike.py`** CLI | HTTP chunks to **1491** (dialog / Step 8) or **1493** (book); **`POST /model/save`**, **`training_progress.json`** | **`learning_mike.log`**, Sergeant API logs |

**Jim (2026-05-25):** **`mikecamera.py`** chunk path **working well — better than `learning_mike`** for split bulk text; not a permanent replacement for API training.

**Do not bounce 1491** for book-only work on **1493**. **1488** = Trader live; **1491** = Step 8 MikeCamera dialog; **1493** = book/training; **1494** = Gradio.

## Standing — `training_progress.json` (training file log + cursors — Jim 2026-06-09)

**What it is:** A durable **log of files trained** — per-file QA/raw cursors (`qa_index`, `raw_index`, etc.) plus **`last_trained_file`**. Not model weights; not a cache you can discard casually.

**Canonical path:** `mikecamera/mikecamera/training_progress.json` (beside `learning_mike.py`). Config: **`learning_mike.train_resume`** (default **true**) and **`learning_mike.training_progress_file`** (`”training_progress.json”` relative to that directory). **`learning_mike.py`** resolves the path at startup and logs: **`Training progress ledger: train_resume=… path=…`** (absolute path). Per-file runs also log **`Resume enabled: training_progress.json`**.

**Proof on next drill:** grep **`learning_mike.log`** for **`Training progress ledger`** — path must point at the MikeCamera file above, not an empty Trader copy.

**Do not zero without Jim.** Tabula-rasa / fresh-start scripts (e.g. **`trader_mike/zero_fresh_start_json.sh`**) back up then write **`{”files”: {}}`** — that wiped this ledger **2026-06-02 12:40→12:45** (133-file backup: **`bak_files/training_progress.json.backup-20260602-1240`**). **Do not** run those zero steps on **`training_progress.json`** without explicit Jim OK; always **timestamped backup** first.

**Restore reference (2026-06-09):** Pre-wipe ledger restored from **`backup-20260602-1240`**; post-wipe 86-file state kept as **`bak_files/training_progress.json.backup-20260609-restore`**.

## Standing — conversational vocab mask (**DO NOT REGRESS** — Jim 2026-06-08 afternoon)

Step 8 dialog was sampling **starter-block math/Python ids** (`bool` 170, `exp` 197, `erf` 217, `<|op|>` 9) because inference masked to **full** `vocab.json`, not **dialog-safe** ids. DEBUG `Converting token ‘bool’ to ID: 170` was **correct encoding**, not a 240-token stub.

**Fix (shared):** `mikecamera/mikecamera/mike_tokenizer_align.py` — `conversational_token_id_allowlist`, `build_conversational_vocab_logits_processor`, `attach_conversational_vocab_allowlist`. Book English **id ≥ 240** + thin starter slice (greetings 41–93, punct 225–239, digits, `and`/`or`/`not`). **Blocks** code/math **94–224**.

**Must stay wired:** non-training `/query` on **:1488** and **:1491** with `conversational_vocab_only=True`; `LocalMikeInferenceModel.generate()` must apply **`logits_processor`** (**:1488** had been ignoring it). **Do not** re-enable empty → `Signal: HOLD` on **:1488** `/query` while diagnosing Step 8.

**Do not regress:** (1) no full named allowlist for dialog inference; (2) no dropping conversational processor from bluegreen `_run_simple_query`; (3) no “fix” that only touches tokenizer load without generation mask; (4) restart **both** servers after edits. Disable mask only via `runtime.disable_named_vocab_logits_mask: true` for deliberate experiments.

**Chapter + prose:** `## 2026-06-08 afternoon` below; `memory_novelette-20260608-afternoon-conversational-mask.md`.

## OV pattern — Orphans Visibility (`trader_mike_config.json`)

**OV** = **O**rphans **V**isibility. One JSON file serves **two audiences**: the **live pipeline** and **other scripts** that import the same config. Without OV, unused keys sit beside live keys and humans (and agents) “fix” settings that nothing on the main path reads—or delete keys another script still needs.

**Rules (the pattern).**

1. **Live keys** stay in their normal sections (`trading`, `localmike`, `api`, …) — only what **Pipeline B** (`solana_test_trader.py` → Steps 0–3) or the active servers actually use.

2. **Non-live keys** move under a single top-level **`orphans`** block at the **top** of the file (high visibility), nested by **former** path: `orphans.trading`, `orphans.localmike.cycle_summary`, etc.

3. **Pipeline B ignores** `cfg[”orphans”]` — safe; no behavior change from parking keys there.

4. **Shared-module scripts** read `cfg[”orphans”][<section>][<key>]` (or merge orphans in their loader if they still want flat paths).

**Jim / operator cheat sheet.**

| Question | Answer |

|----------|--------|

| What runs tonight’s swap cycle? | Everything **except** the top `orphans` block |

| What is `orphans` for? | Other entry points, future wiring, audit trail — **not** deleted |

| Full key-by-key audit | **`trader_mike/trader_mike_config_connectivity_audit.txt`** |

| Pipeline map | **`trader_mike/PIPELINE_SWAP_PAIRS_AND_TRANSACTIONS.md`** |

**Module vs key.** **`LocalMikeCryptoController`** is still **live** on Pipeline B (RPC, `call_localmike`). OV applies to **config keys**, not to declaring the whole module dead. Pipeline A (`run_crypto_trading.sh`) is the separate **orchestrator** entry that still cares about some keys now under `orphans.trading.*`.

**Before the next config pass (avoids the old argument).** Do not “fix” or delete keys under **`orphans`** to make Pipeline B work — B already ignores them. Do not pull **`orphans.*`** into live sections without checking **`trader_mike_config_connectivity_audit.txt`**. New knob for tonight’s cycle → live section; knob for another script or later → **`orphans`**.

**Operating principle (Jim).** **Always think OV** — live vs parked, audit before wiring, never delete “unused” keys without checking who else reads the file. If there is an **easier or more profitable** way to run the stack (config, pipeline, sizing, wiring), **make a backup first** (`config_backups/trader_mike_config.bak-*.json` or script `.bak-*`), then do it. Same mindset applies beyond JSON: separate what production uses from experiments, keep orphans visible, compare against the last backup when reorganizing. **Finish the mission** — verify (e.g. compare to last backup), document outcome here, no happy-ending hand-waves.

**Later:** a fuller operator guide (e.g. **`--help`** on the main trader entry or a small `ov_pattern_help` script) — not required for OV to work; **`5.4.md` + audit + `_ov_pattern_note`** are enough until then.

## Thread summary (2026-04-07 — this chat, for the next instance)

- **Ask:** Align Sergeant tokenizer with **March 2026** behavior using **`vocab.json` + merges**; **small diffs** only (Jim rejected a large multi-file rewrite). **Backup** before tokenizer edits: **`localmike_api_server_sergeant_bluegreen.py.bak-20260407-before-march-tokenizer`** (keep if still on disk).

- **Code (`localmike_api_server_sergeant_bluegreen.py`):** **`_load_model_from_path`** — **`FivedTokenizer.from_pretrained(tokenizer_dir)`**, then **force** **`vocab.json`** (+ **`_apply_pending_vocab_tokens`**); March-style **`_max_vocab_size`** for inference vs training; **removed** inference-time **capping** of tokenizer to checkpoint rows (**resize embeddings up** when tokenizer is larger).

- **Ongoing sore spot (fixed; Jim confirmed expanded vocab works):** Degenerate **`/query`** (function-word loops, ~200–300 effective tokens) because **`_MaskBeyondTokenizerVocab`** masks to **`len(tokenizer.vocab)`** while weights had a full LM head. Causes: **`infer/`** or **checkpoint `vocab.json`** smaller than trained vocab; **`_sync_checkpoint_to_root`** overwriting **large** live **`vocab.json`** with a **small** checkpoint copy.

- **Fixes for that:** **`load_active_and_standby`** — prefer tokenizer next to **weights**; **`infer/`** only if row lacks tokenizer files; if **`_disk_vocab_token_count(live root)`** **>** **model row**, set **`tokenizer_source_path`** to **live root**. **`_sync_checkpoint_to_root`** — if live **`vocab.json`** has **more entries** than staged checkpoint, **copy live into staging** so sync does not shrink vocab.

- **Confirm in logs:** **`Preserving live root vocab.json`** / **`ACTIVE tokenizer: live root ... > model row ...`**. Restart Sergeant after code changes.

- **This file:** Detailed notes under **§ 2026-04-07**; per-step vocab bullet under **§ 2026-04-09** uses **`_sync_live_vocab_from_runtime_tokenizer`**. Quick check **#2** = degenerate chat.

- **Side note:** Jim mentioned **`fsck -y`**; on a live Mac, **`diskutil verifyVolume /`** was run instead — **APFS verify OK** (exit 0). Jim may continue after this handoff.

I have a plan. It is up to (you), if you execute it. There is no disorder in orders, but this is not an order.