Don't Be a Tire Kicker

What does that mean?

I’ll share more photos as we travel along the way...

## MikeCamera / Sergeant / LocalMike training API

### Direct-load guardrail (`summary-c-updated.py`)

- **Keep the no-double-init load pattern.** In `mikecamera/mikecamera/summary-c-updated.py`, the important old comment is still correct:

- `# CRITICAL: Load model WITHOUT any initialization to prevent double initialization`

- **What that means in code:** load tokenizer with `inference_mode=True`, load config first, build `MichaelForCausalLM(config, skip_init=True)`, then load the trained safetensors **directly as a state dict**.

- **Do not “simplify” this back to a normal `from_pretrained()` path** in that script if results are good. The guardrail there is explicit: avoid layer initialization first, then apply trained weights once.

### Per-step vocab + model + optimizer growth

- **Config:** `mike_camera_config.json` → `training.allow_per_step_vocab_growth` (default **true**).

- **Code:** `localmike_api_server_sergeant_bluegreen.py`

- Each training step: `_queue_pending_vocab_tokens` → if growth on: `_grow_standby_vocab_from_pending` (merge `pending_vocab.json` into runtime tokenizer, up to `pending_vocab_apply_per_checkpoint` per step) → `model.resize_token_embeddings(new_size)` → existing **`_sync_optimizer_param_groups`** + **`_repair_optimizer_state_shapes`** so Optimizer5DRL matches new embedding/LM-head shapes.

- **Pattern source:** `tire_fire.py` — `TireKickerCallback.update_tokenizer_vocabulary` (~2526–2587): tokenizer larger than model → resize → reconcile optimizer.

- **If growth is off:** `_freeze_tokenizer_to_model_vocab` path; pending vocab applies mainly at checkpoint reload.

- **Restart** the training API after changing this config or code.

#### 2026-04-08 — Optimizer reconcile, effective LR, staging under TMPDIR

- **`_effective_training_lr()`** — Never pass `None` into Optimizer5DRL; invalid / missing `training.learning_rate` falls back to **1e-4** with a one-time warning (reads `mike_camera_config.json`).

- **`_reconcile_standby_optimizer_with_model(model, reason=...)`** — After any embedding resize or param-group drift: `_sync_optimizer_param_groups`, `_repair_optimizer_state_shapes`, `_initialize_5drl_coordinates` (fills missing 5DRL keys only), `_build_param_id_map`, `_reapply_config_learning_rate_to_optimizer`. Called **before every training forward** and **after** pending merge / `expand_vocabulary` / encode-time `_sync_model_embeddings_to_tokenizer_vocab` when rows were added, so the optimizer stays bound to current tensors (5DRL is **per parameter**, not per token row).

- **Training step:** Fresh Optimizer5DRL uses **`lr=_effective_training_lr()`**; duplicate pre-forward sync blocks folded into reconcile; post-encode “late resize” relies on reconcile inside `_sync_model_embeddings_to_tokenizer_vocab` plus re-encode only.

- **Checkpoint I/O clutter:** `_ephemeral_payload_work_dir()` → **`tempfile.mkdtemp`** under **`$TMPDIR`** for slot publish staging and root payload staging / root-sync source (no more `.{model}.root-payload-staging-*` next to `mike_camera_model`); **`shutil.move`** used where **`Path.rename`** would cross volumes.

- **Logs / training reality:** With **~22k vocab**, CE loss in the **7–10** range early can still be **better than random** (~ln(22038) ≈ 10). Lines like **`vocab=22038 model_emb=22038 pending_merged_this_step=0`** mean **no new tokens that step** (text already in `vocab.json`) — not a broken growth path. **Switching to a new training file** shifts the “typical” loss band; compare trends **within** a file, not single steps across boundaries. **`learning_mike.log`:** `Sending training file to LocalMike:` / `--- Chunk N ---`; **`training_progress.json`** tracks cursors.

- **Checkpoint parameter stats:** **`checkpoint_parameter_stats.log`** (and related logging) — user confirmed **tensor / parameter counts moving** in the log after saves; use that to sanity-check growth vs flat runs over the day.

- **Ops (Jim):** For now, only **micro-adjust `training.learning_rate`** or **batch / chunk sizing** if needed after watching; no other knobs planned unless something breaks.

#### 2026-04-09 — Live `vocab.json` sync, stats semantics, shards, loss vs. chat

- **Per-step disk vocab:** After each successful training step, **`_sync_live_vocab_from_runtime_tokenizer`** writes the **in-memory tokenizer vocab** to **live root** **`vocab.json`** (and updates root **`config.json` `vocab_size`** when needed) when on-disk and runtime token **counts** differ — **full runtime dict**, so the live tree doesn’t look “frozen” while training grows the tokenizer.

- **`checkpoint_parameter_stats`:** **`param_groups: 1`** = one optimizer group holding all tensors (not “group 1” vs 2). The log does **not** list each parameter. **`unique_parameter_tensors`** (e.g. 254) is usually **constant**; **vocab growth** shows in **`total_elements`**, **`runtime_vocab_size`**, and larger **`state_tensor_numel_sum`** (Adam buffers + extras), not in more tensor *slots*.

- **Optimizer shard *file* count** (e.g. 1 vs older 10–20) is mostly **packaging** (`max_shard_size`, single param group, “single shard when group exceeds cap”). Compare **total optimizer bytes on disk** and **`state_dict_entries`** if worried about regression — don’t read shard count as “how much 5DRL.”

- **Loss vs. response quality:** CE going down is necessary but not sufficient. If **generation/chat** does not improve, debug separately (ACTIVE vs STANDBY weights, prompt format vs training text, decoding knobs, overfitting slice).

- **Weekend / memory reset:** Next session should **read this whole file first** — “they gut your memory” on weekends; **`5.4.md` is the continuity anchor** for Jim + MikeCamera Sergeant.

- **Transformers registry in Sergeant:** `_ensure_imports()` registers **`task_model_mike`** + **`TaskModelMike` + `FivedTokenizer`** with **`AutoConfig` / `AutoModelForCausalLM` / `AutoTokenizer`** (same as `mikecamera.py`). **`Optimizer5DRL`** is not a Transformers type; training builds it explicitly.

#### 2026-04-07 — Expanded vocab vs ACTIVE inference (ongoing sore spot — fixed)

This one caused **degenerate `/query` text** (tight loops of “the / a / to / of …”) that looked like the model only had **~200–300** usable tokens even though embeddings were **full size**.

**Why it happened**

1. **`_run_simple_query`** attaches **`_MaskBeyondTokenizerVocab`**, which sets logits to **`-inf`** for indices **`>= len(tokenizer.vocab)`** (via **`_tokenizer_vocab_len`**). If the **tokenizer dict is tiny**, generation is forcibly limited to that slice of the LM head.

2. **Tokenizer source drift:** Weights load from **`_resolve_model_load_path()`** (often a **`checkpoint-*`** row). That row’s on-disk **`vocab.json`** can lag **behind** the **expanded** vocab already written to **live model root** during training steps. Loading tokenizer **only** from the checkpoint row (or from a small **`infer/`** snapshot) therefore kept **`len(vocab)`** artificially small.

3. **Root sync made it worse:** **`_sync_checkpoint_to_root`** (called when aligning the chosen checkpoint into **`mike_camera_model/`**) could **overwrite** a **large** live **`vocab.json`** with the checkpoint’s **smaller** copy.

**Fixes (all in `localmike_api_server_sergeant_bluegreen.py`)**

- **`_load_model_from_path`:** March-style tokenizer build — **`FivedTokenizer.from_pretrained(tokenizer_dir)`** ( **`merges.txt`** + **`tokenizer_config.json`** ), then **force** that directory’s **`vocab.json`** into **`tok.vocab`** (plus **`_apply_pending_vocab_tokens`**). **No** inference-only trimming of the tokenizer to checkpoint embedding rows; **resize embeddings up** when the tokenizer is larger than the checkpoint.

- **`load_active_and_standby`:** Prefer tokenizer files **next to the weight row** when present. Use **`infer/`** only as **fallback** if the row lacks tokenizer files. If **live root** **`vocab.json`** has **more token entries** than the (**`_disk_vocab_token_count`**) on the **model row**, set **`tokenizer_source_path`** to **live root** so ACTIVE sees the **trained** vocabulary while weights still load from the checkpoint path.

- **`_sync_checkpoint_to_root`:** Before applying staged files onto the live root, if **existing** live **`vocab.json`** has **more entries** than the staged checkpoint **`vocab.json`**, **copy the live file into the staging tree** so the sync **does not shrink** vocab.

**Log lines to confirm**

- **`Preserving live root vocab.json (N tokens) over checkpoint row (M) during root sync`**

- **`ACTIVE tokenizer: live root ... (N tokens) > model row ... (M) — using live tokenizer files`**

Restart Sergeant after changing this code. If chat looks “stuck” in function words again, compare **`_disk_vocab_token_count`** for **live root** vs **active checkpoint** **`vocab.json`** and the **`_MaskBeyondTokenizerVocab`** path.

### Training loss guardrails and API response

- **`max_loss_for_update`:** was **10** (too low for LM CE); raised to **50** in config — tune if steps still skip with “exceeds safety threshold.”

- **Do not** return `0.0` on missing loss / missing `input_ids` — use **`TrainingStepSkipped`** or real errors so a step is not treated as a successful update.

- **Logging:** log training loss whenever `step_updated` is true (do not use `if loss_val:` — `0.0` is falsy).

- **`QueryResponse`:** optional **`training_loss`**, **`training_step_updated`** so clients do not parse prose only.

- **`learning_mike.py`:** logs `training_loss` / `training_step_updated` from JSON when present.

### `strategy_swap_queue.json` (trader_mike — cross-project note)

- File is **written** after strategy analysis, then **cleared to `[]`** after `execute_strategy_swaps` finishes (by design). Seeing `[]` after a cycle is normal.

- **`execute_swap_queue.py`:** items with **0 / missing amount** were dropped by `_coerce_queue_items` — fixed to retain with default; **`strategy_swap_queue.json` loaded first**; before swap, if `usd_amount <= 0`, apply **`default_swap_amot_usd`** from config.

- **`solana_test_trader.py`:** check **`sync_safe_write_json`** return when writing the strategy queue; clearer log when **all** pairs filtered by `do_not_trade`.