Research wiki 4 — Observability with Opik

Notes from a research desk I built to teach myself audio ML: get signal, find relevant data, and answer questions from a compounding LLM wiki kept as plain files under OKF.

Part 4 — after the jobs and the search layer

Part 1 is the job graph: scheduled ingest into a compounding wiki.

Part 2 is the file contract.

Part 3 is when I search those same files by meaning in a chat UI, and when I open a written page instead.

This one is not another retrieval retune. It is about whether I can see what the desk actually did.

Two things run without me watching. Agents run overnight on a schedule. Chat against the synced Knowledge Base retrieves chunks and writes answers. For a while I had no view into either. Now both report into one self-hosted Opik project, Research Pipeline Observability.

Most of this post is about the agent half, because that is where I got it wrong. My first shipping path sent some traces cleanly and silently missed almost every main session I later audited, heartbeats included.

One project, two surfaces

Surface	What I couldn’t see	How it reaches Opik now
Agent runs	Which model ran, which tools, empty vs real replies, whether main-session work showed up at all	Official OpenClaw plugin (@opik/opik-openclaw) on LLM-call hooks
Chat / RAG	What was retrieved and what the model answered	Open WebUI Pipelines filter that intercepts chat and emits traces

Both land in the same project, so an ingest run and a chat that later queried its output sit in the same trace list. I did not want two dashboards.

Hooks start things — they don’t do them is why I care. Ownership bugs and silent double-starts do not announce themselves. Traces are how I notice the next silent failure, including the one where the dashboard has traces, so I assume coverage is fine.

Blind spot 1 — agent runs

The path that looked done

For weeks, agent traces came from a gateway hook on message:sent. OpenClaw writes each agent run to a trajectory file. When a reply went out, the hook handed that file to a small importer, which shipped it to Opik and left a .done marker next to it.

That path taught me a few things the hard way:

• Configuring the SDK unconditionally reopened an interactive setup wizard on every run. Unusable from a hook.
• A freshness check on the file’s wall-clock age dropped ships whenever the event arrived late. No log.
• When a model fell back mid-run, the trajectory had two segments. The importer kept the first one, the aborted attempt, and shipped an empty trace while Telegram had the real reply.
• Ending spans and traces too early, under SDK batching, emptied them another way.

Each was a real bug. Fixing them only helped traces that did ship.

What the audit actually measured

I set up the official OpenClaw Opik plugin as a week-long parallel trial, not a cutover. The audit ended that plan early.

I counted .done markers against trajectory files for sessions whose key ends in :main — each agent’s long-lived main session, where heartbeats land along with other work. 108 files. 106 had never reached Opik. Only sessions created after my Aug 7–8 fixes: still 5 of 6 missing. So the gap was still open, not leftover from bugs I had already fixed.

That audit did not cover cron sessions, which use a different key shape (…:cron:…). And I have evidence the old path could ship a cron run: a manual paper-ingest trigger grew the project by one trace while the importer was still live. So I am not saying every cron job was invisible. I am saying almost every audited main session was.

Two failure classes shared that stack. Empty or broken traces on runs that did ship — batching, early end(), first segment wins — are one bug. Sessions that never triggered a ship at all are another. Fixing the first cannot close the second. Only changing when shipping fires does.

Tying shipping to message delivery works for interactive chat turns, because those always end in a delivered message. Heartbeats and other main-session work often don’t. The dashboard looked alive because the interactive slice was alive.

What ships now

I switched right away. Live shipping is the official @opik/opik-openclaw plugin, installed as the npm package at v0.2.17 and pointed at the same production project; the trial project was only scaffolding. It hooks at the LLM-call level (llm_input / agent_end), so it doesn’t depend on Telegram or message:sent. Cron runs and heartbeats both go through it.

I checked it on a real daily paper-ingest cron run and on a heartbeat. Both showed up in Opik right away. The old hook is disabled, not deleted. I kept the importer in case I backfill the missed main sessions.

Disabling the hook was not enough on its own. The running gateway kept the old message:sent registration in memory until I restarted it. One digest in that window shipped twice — once from the plugin, once from the importer, same run. After the restart, cron runs show only plugin traces. hooks list saying “disabled” does not mean the old path is dead.

The plugin setup had its own friction:

• The config key in the integration docs did not match the installed plugin id.
• Adding the plugin to plugins.allow silently disabled every plugin that had been running unrestricted until I listed each one.
• The setup wizard asks for an API key even for a local Opik with no auth and no key UI. I put a placeholder in the config instead of going through the wizard.

Blind spot 2 — chat and RAG

The chat from Part 3 runs on a different path. Agent traces tell me nothing about what a chat retrieval pulled.

Open WebUI’s Pipelines admin page does not light up by itself. Pipelines is a separate service. You run it, then register it under Connections as an OpenAI-compatible endpoint, and only then does Admin → Pipelines find it. Starting the container is not enough.

The Opik filter is one of the upstream example pipelines. Installed into that service, it intercepts chat, opens a trace per conversation, and sends retrieval and generation to Opik.

Its default Valve for the Opik host points at Comet’s cloud, not a self-hosted instance. Easy to miss: the filter works, and the traces go somewhere else. I overrode the host, workspace, and project_name so chat traces join the agents’ project.

Checked live: filter chat traces and agent runs share the project, and large RAG sessions sit next to ingest and digest runs.

What I am not claiming

• That Opik is the only tool for this.
• That you should copy my compose setup or plugin pin.
• That Part 3’s retrieval settings are done. Retuning from the chat traces is still open.
• That the old path never shipped cron. The hole I measured was in main sessions. I never audited cron coverage on the importer the same way.
• That every silent failure is gone. Live shipping runs through the plugin; backfilling the missed main sessions is still optional.

Pointers

• Prequel: Research wiki 3 — Search over the same files
• Related: Hooks start things — they don’t do them
• Opik: documentation
• OpenClaw → Opik plugin: comet-ml/opik-openclaw
• Open WebUI Pipelines (upstream filter examples): open-webui/pipelines