AgentHub/docs/superpowers/specs/2026-09-29-codex-usage-design.md

# Codex parity, part C1: sessions, token usage and pricing

Issue: #40 part C. Date: 2026-09-29. Status: self-reviewed (the user asked for
autonomous work with a self-review per part). Part C2 (Codex hooks) is separate.

## Problem

Codex emits no hook AgentHub listens to, so a Codex session never reaches the
`sessions` table and its tokens cost nothing on any page. The pricing table
and the automatic pricing sync know Claude models only.

## What Codex writes (verified on 0.155, 178 real rollouts)

`$CODEX_HOME/sessions/YYYY/MM/DD/rollout-<ts>-<thread id>.jsonl`, append-only:

- **Line 1** is `session_meta`, with these payload fields:
  - `id`: the thread.
  - `session_id`: the ROOT session. It is the same as `id` except in a sub-agent thread, whose `thread_source` is `subagent`. Older CLIs omit it.
  - `cwd`, `timestamp`, `cli_version`.
  - `forked_from_id`: set when the file is a fork.
- **`turn_context.payload.model`** is the model for the lines that follow. It can change mid-session, or be a router id such as `clinepass/cline-pass/deepseek-v4-pro`.
- **`event_msg` with `payload.type == "token_count"`** carries two things:
  - `info.total_token_usage`: a cumulative total for the thread.
  - `info.last_token_usage`: the delta.
  
  `info` may be null. The same total is often emitted twice in a row.
- **Token fields:**
  - `input_tokens` includes `cached_input_tokens`.
  - `output_tokens` includes `reasoning_output_tokens`.
  - `cache_write_input_tokens` is separate.
- **A fork COPIES its parent's whole history** into its own file, `token_count` events included, with rewritten timestamps. It then carries on from the parent's cumulative total. Summing the fork file naively counts the parent twice.

## Design

1. **`sessions.tool`** is a new column: `TEXT NOT NULL DEFAULT 'claude'`, added with the additive `ALTER TABLE` idiom the `source` column used. Every existing row and every hook-created row stays `claude`.
2. **`codex_rollouts`** is a new table recording how far each file has been read:
   - Columns: `path` PK, `session_id` (root), `thread_id`, `size`, `mtime_ms`, `offset`, `state` JSON, `updated_at`.
   - `state` holds the current model, the last total seen, the per-model usage sums, and the meta.
   - A file whose size and mtime are unchanged is not read again.
   - A grown file is read from `offset` up to its last newline.
   - A shrunken file is re-read from 0.
   - Because this table lives in the DB, a session the user deleted stays deleted until Codex writes to its file again, while wiping the DB re-imports everything.
3. **`server/lib/codex-usage.js`** does the ingest:
   - **`parseRolloutChunk(text, state, skip)`** is pure. It JSON-parses only the lines it needs, picked by five needles (`NEEDLES`): `session_meta`, `turn_context`, `token_count`, and a prompt as `user_message` (older CLIs) or `UserMessage` (0.149+). Each new total adds its `last_token_usage` to the current model, mapped as:
     - input = `input − cached`
     - cache_read = `cached`
     - cache_write = `cache_write_input_tokens`
     - output = `output_tokens`
     
     OpenAI bills reasoning as output.
   - A total equal to the previous one is skipped.
   - **Forks:** when a file with `forked_from_id` is read from offset 0, the parent's file is found by the `-<id>.jsonl` name. Every total vector in the parent is skipped in the child: the copied prefix matches the parent exactly, and the child's own totals never do. If the parent's file is gone, nothing is skipped and the overcount is logged.
   - **`ingestCodexUsage({ broadcast })`:**
     - Walks every rollout (no newest-40 cap) and yields to the event loop between files.
     - For each root session touched, it sums its threads' usage per model and writes it with `stmts.setTokenUsage`. The totals are exact recomputes, so the high-water baseline of `replaceTokenUsage` would be wrong here.
     - It then inserts or updates the `sessions` row:
       - `tool='codex'`
       - `name` = the `thread_name` from `session_index.jsonl`, else the first prompt (first line, 60 chars; `user_message` or the 0.149+ `item_completed` `UserMessage`), else `Codex <8>`
       - `cwd` and `started_at` from the root meta
       - `model` = the latest model
       - `status` = `active` while any of the session's files was written in the last 2 minutes, else `completed`, with `ended_at` = the newest journal's mtime
       - metadata `{source:"codex-rollout"}`
     - It never touches a row whose `tool` is not `codex`, and never sets `transcript_path`, so the Claude transcript parsers and sweeps skip these rows.
     - It broadcasts `session_created`/`session_updated`.
     - One pass at a time. It runs at startup (in the background) and then every `DASHBOARD_CODEX_POLL_MS`, the codex lane poll's own setting (`0` turns both off).
4. **Pricing sync learns OpenAI.**
   - `familyOf` also accepts `gpt-…`, `o<digit>…` and `codex-…` ids, dropping a `-YYYY-MM-DD` suffix.
   - `toRow` takes `litellm_provider == "openai"` entries. A cache write costs `cache_creation_input_token_cost` or 0, since OpenAI has no cache-write charge by default. The fast rate is 0. The display name comes from `displayName` (`gpt-5.6-sol` → `GPT-5.6 Sol`).
   - Long-context (>272k) and priority/flex tiers are not modelled. Codex's context window stays under 272k.
   - `POST /api/settings/reset-pricing` starts a sync right after reseeding, so OpenAI rows wiped by a reset return at once.
5. **Model Pricing in Codex → AgentHub.**
   - The pricing section of `ClaudeAgentHubSettings.tsx` becomes `components/settings/PricingSettings.tsx` with a `vendor` prop. The Claude tab shows `claude-…` rows; the Codex tab shows OpenAI rows.
   - A **Show all models** switch shows every row, since one table prices every tool's sessions (a router model may serve either). A row added outside the filter turns the switch on, so it never vanishes from view.
   - Reset Defaults stays on the Claude tab only and sends `{scope: "claude"}`: it deletes and reseeds only `claude-…` rows, so OpenAI rows (and a custom router row) survive it; a reset with no body wipes every row, and OpenAI rows come back from the sync.
6. **Claude-only surfaces exclude Codex sessions.**
   - The Workspace's external sessions, which offer attach and `claude --resume`, list `tool === 'claude'` only.
   - The Sessions list and session detail show a **Codex** badge.

## Found in the browser check, fixed

- The Claude liveness reap and the transcript watchdog (`routes/hooks.js`) probed Codex sessions for a `claude` process and reaped them: both now select `tool = 'claude'` only.
- The pricing sync had already run when the first ingest wrote new models; a pass that writes sessions now triggers a sync.

## Found in the code review, fixed

- **A journal met under a new path counted twice** (CODEX_HOME moved, or a symlinked alias): there is now one `codex_rollouts` row per thread, and reading a file drops the thread's rows under other paths.
- **The Claude resume picker offered Codex sessions** (`RunSetup.tsx`), which `claude --resume` cannot open: it lists `tool === 'claude'` only.
- **A failed price-list fetch was retried on every Codex pass** (every 15 s): a failure now waits for the sync's own 15-minute check.
- **A fork's copied history arriving after the first read was counted**: the parent's totals are skipped until the fork's first own total (`state.pastFork`), on every read, not just the one from offset 0.
- **A journal written in the future** (a clock running ahead) kept its session `active`: it no longer counts as activity.
- **A large journal blocked the event loop**: one pass reads at most 16 MB of a file, and a fork's parent is streamed.
- **A model gone from a re-read journal kept its old usage row**: a Codex session's usage is rewritten whole.
- **A sub-agent file read before its root fixed the session's cwd, start and model**: the root thread's file sets them once it is read.
- **The startup cleanup completed active Codex sessions** and **a Claude lane's purge deleted a Codex lane's sessions** at the same directory: both are scoped by tool.
- **The codex lane mirror flapped** (`codex-ingest.js`, predates C1): a lane with an old journal beside the live one went running → idle → running on alternate passes, and a journal written in the current millisecond read as future. One verdict per lane now, with a second of slack.

## Found in the whole-change review, fixed

- **Hooks kept a session's status forever.** A Codex session with any hook event kept its hook-given status even after the hooks were removed or no longer trusted, so it could stay `active` for good. Hooks now own the status only while current — `hooksCurrent`: the session's newest event is at or after its newest journal write minus 2 minutes — and otherwise the journal decides again, in `settleActive` too.
- **Clear data stopped Codex (and Antigravity) sessions ever coming back.** `POST /api/settings/clear-data` kept `codex_rollouts`, so a cleared session was never re-imported, or came back half on its next write: it now empties `codex_rollouts` and `antigravity_conversations` too.
- **The per-thread lookup scanned the table**: `idx_codex_rollouts_thread` indexes `codex_rollouts(thread_id)`.
- **A busy Codex lane pushed Claude rows out of a page**: `GET /api/sessions` takes `?tool=`, applied before the limit, and the Workspace attach list and the run form's resume picker ask for `tool: "claude"`.
- **Reset Defaults wiped the OpenAI rows** a user had edited: the Claude tab's reset is scoped to `claude-…` rows (above).

Left as is: a journal rewritten to the same or a larger size is not detected (Codex only appends), and a fork without a Codex thread name is named after the prompt it copied from its parent.

## Not changed

Hook ingestion, `transcript-cache.js`, `import-history.js` and remote sync still handle Claude Code only. Codex lanes keep their idle/running mirror in `codex-ingest.js`. Session↔lane binding is untouched.

## Tests

- **Server:**
  - Chunk parser: mapping, duplicate totals, model switch, null info, partial last line.
  - Fork skip, with and without the parent file.
  - Sub-agent thread folded into the root session.
  - Incremental read after an append; a shrunken file re-read.
  - An unchanged file not re-read; a deleted session not resurrected.
  - A claude row never touched.
  - Pricing sync for an OpenAI family.
  - Reset triggers a sync.
- **Client:** `PricingSettings` vendor filter and the show-all switch; the resume picker excluding codex sessions; the Codex badge.
- **Browser:** an isolated server on a temp `CODEX_HOME` with copied real rollouts.