The Observation Contract

Why Memory Formation Cannot Live in One Service

Memory formation has two distinct competencies that cannot be co-located without coupling. The first is discovery: parsing raw data, typing it, extracting meaning from a source. This requires source-domain knowledge — what is an email, what is a tool call, what is a Telegram message. The second is integration: deduplication, conflict detection, schema fit, decay, writing to the entity graph. This requires knowledge of current graph state.

If discovery and integration both live in cortex-insight, insight must track graph state — and every pipeline becomes coupled to the warehouse schema. Memory loses its ability to arbitrate dedup. If both live in cortex-memory, memory must contain parsers for every source domain — the generic warehouse schema is contaminated; memory becomes heavyweight. Neither pure assignment yields a clean cut. The solution is to split across two services with an explicit typed contract at the boundary.

The Lake / Warehouse Split

This is the classic data-engineering pattern — lake for raw, append-heavy, schema-on-read storage; warehouse for structured, schema-on-write, consumer-facing truth — applied at embedded scale.

cortex-insight (lake) is append-heavy and schema-on-read. It stores raw events, runs source-specific ETL pipelines, and publishes typed Observation records. Rewrites are cheap: the lake is a staging area, not a long-term asset.

cortex-memory (warehouse) is schema-on-write and generic. It receives Observation records, consolidates them into the entity graph, and handles decay, dedup, conflict resolution, compiled truth, assembly, and actuation. Schema changes are versioned and protected because this is the long-term asset.

Two boundary rules follow directly. Insight never stores the graph. Memory never parses raw source data. These rules are enforced structurally: cross-package imports between the two services are not declared in pyproject.toml and fail at import time.

The practical consequences compound. Adding a new source means writing a new parser and a new pipeline inside cortex-insight only — cortex-memory is unchanged. The entire insight service can be rewritten without touching accumulated knowledge. Dependency asymmetry is also explicit: insight is heavy (parsers, regex, LLM calls, NATS consumers, columnar processing); memory is minimal (graph store, NATS).

The Biological Analogy: Hippocampus → Cortex

The two-phase design mirrors established neuroscience of memory consolidation. The hippocampus encodes fast and lossy — it captures the episode without resolving it against long-term knowledge. The semantic cortex consolidates slowly, integrating and deduplicating across the semantic graph over time. The prefrontal cortex provides goal-conditioned retrieval against the consolidated store.

The system maps precisely onto this model. External producers correspond to perception. cortex-insight is the hippocampal layer: fast, source-specific, episodic. cortex-memory is the semantic cortex: slow, generic, graph-aware. The memory.assemble function and the goal stack correspond to prefrontal retrieval.

Observation is the episodic memory unit: typed, encoded, not yet resolved against the graph. It sits exactly at the hippocampal boundary. This is consistent with the hippocampal memory-strength formula adopted for the memory_strength decay model: fast write, slow integration, durable retrieval.

The Observation Contract

Observation is the typed publish contract between cortex-insight and cortex-memory — the only artifact that crosses the service boundary. It is published to the NATS subject roxabi.memory.observations.publish.

class Observation(BaseModel):
    id: ULID
    source: str           # "claude-code-jsonl" | "mail" | "telegram" | …
    source_ref: str       # natural ID within the source
    timestamp: int        # epoch ms
    category: ObservationCategory  # Interaction | Finding | Decision | Artifact | …
    actors: list[ActorRef]         # entity references (unresolved)
    topic: list[str] | None        # extracted concepts
    sentiment: str | None
    payload_typed: dict            # category-specific structured payload
    correlation: dict              # trace_id, parent_obs_id, episode_id, …

Four design choices are load-bearing. First, category is the stable taxonomic anchor (ObservationCategory): Interaction, Finding, Decision, Artifact, Statement. The category set must be exhaustive and stable — it is the schema surface that cortex-memory's Retain Job dispatches on. Second, actors carries references — email, name, slug — but resolution against the graph is deliberately deferred to memory. Insight does not know whether an actor already exists in the graph. Third, payload_typed is category-specific structured content, typed by insight and consumed by the Retain Job. It is not generic freeform text. Fourth, correlation (trace_id, parent_obs_id, episode_id) enables causality tracing across the full pipeline.

Observation is explicitly not an EntityProposal. Entity resolution is memory's responsibility. Insight produces a well-typed, source-aware episodic record; memory resolves it into the durable graph.

The Retain Job: Consolidation in Memory

The Retain Job lives entirely inside cortex-memory. It is the consolidation engine: for every Observation received from the NATS subject, it executes a deterministic sequence.

First, it resolves actors against the graph — each ActorRef either maps to an existing entity or creates a new one. Second, it deduplicates against existing graph content. Third, it computes schema_fit for the incoming observation relative to the current graph state. Fourth, it detects conflicts. Fifth, it writes entities and relations. Sixth, it updates memory_strength and applies the decay model. Seventh, if the impact of the observation exceeds a threshold, it triggers regeneration of compiled_truth.

The Retain Job was initially co-located with insight's detectors. It was moved to cortex-memory because consolidation is structurally an attribute of the warehouse: it requires graph state, arbitrates identity, and controls the long-term asset. Locating it in the lake would have required insight to track graph state — exactly the coupling the split was designed to prevent.

Alternatives Rejected

All generation in insight, EntityProposal as final output. Insight must then know graph state; every pipeline couples to the warehouse schema; memory cannot arbitrate dedup. Rejected.

All generation in memory, insight as passive lake. Memory accumulates per-source parsers; the generic warehouse schema is contaminated by domain-specific structure; memory becomes heavyweight. Rejected.

Single cross-service pipeline via NATS streaming. Premature complexity. The two phases are logically distinct; naming them explicitly with a typed contract is more durable than flattening them into a single pipeline. Rejected.

No lake/warehouse split, single-repo status quo. The raw event schema is JSONL-centric and unusable for mail or Telegram. The Retain Job would couple the graph to a single source domain. The long-term graph asset lives under the analyser — inverted ownership. Rejected.

Three-way split (sources / extraction / store). Over-decomposed for the current scale. Coupling between extraction and raw data justifies co-locating them in insight. Rejected.

Contracts as Protocol SSoT

NATS subjects and Pydantic message schemas — including Observation — live in the shared roxabi-contracts package, not inside either service. Both cortex-insight and cortex-memory declare roxabi-contracts as an explicit versioned dependency in their pyproject.toml.

The rule is unconditional: never define NATS subjects or message schemas inside a service package. Cross-package imports between insight and memory are not declared — any such import fails at import time. The only valid cross-service communication channel is NATS, via contracts.

Integration tests are the only valid site for cross-service assertions. They mount both services and a local NATS broker. Unit tests within each service test that service in isolation, against the contracts package alone. This boundary is enforced at the dependency graph level, not by convention.