# memorix — complete library (all audiences)

_Generated whole-library download. See `agent-manifest.json` → `library` for discovery._

## Table of contents

- **Platform Overview** (`00-platform-overview`, developers)
- **Platform Overview** (`00-platform-overview`, builders)
- **Format for Dummies** (`01-format-for-dummies`, developers)
- **Format for Dummies** (`01-format-for-dummies`, builders)
- **Runtime Data Client** (`02-runtime-data-client`, developers)
- **Lists, Queries, Pagination** (`03-lists-queries-pagination`, developers)
- **Data Access Patterns** (`04-data-access-patterns`, developers)
- **Metadata, Agents, Abstracts & Mappings** (`05-metadata-agents-abstracts`, developers)
- **Metadata, Agents, Abstracts & Mappings** (`05-metadata-agents-abstracts`, builders)
- **Relationships & Memory** (`06-relationships-and-memory`, developers)
- **Relationships & Memory** (`06-relationships-and-memory`, builders)
- **Building Services** (`07-services`, developers)
- **Building Services** (`07-services`, builders)
- **Building Pipelines** (`08-pipelines`, developers)
- **Building Pipelines** (`08-pipelines`, builders)
- **Workflow & Studio** (`09-workflow-and-studio`, developers)
- **Workflow & Studio** (`09-workflow-and-studio`, builders)
- **Structural Intelligence** (`10-intelligence`, developers)
- **Structural Intelligence** (`10-intelligence`, builders)
- **Agent Packs & OpX** (`11-agent-packs-and-opx`, developers)
- **Agent Packs & OpX** (`11-agent-packs-and-opx`, builders)
- **Building KnowX** (`12-knowx`, developers)
- **Building KnowX** (`12-knowx`, builders)
- **Migration & Troubleshooting** (`13-migration-troubleshooting`, developers)
- **Migration & Troubleshooting** (`13-migration-troubleshooting`, builders)
- **Getting Started & Tutorials** (`14-getting-started-and-tutorials`, developers)
- **Getting Started & Tutorials** (`14-getting-started-and-tutorials`, builders)

---

# Platform Overview — developers

# Memorix Platform Overview — Developers

**Audience:** Engineers integrating another product with Memorix (HTTP API or `@x12i/*` packages). 
**Twin:** [Builders version](../builders/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md)

---

## Install (start here)

```bash
# Capability SDK — primary developer entry (pipelines + workers + authoring)
npm install @x12i/memorix-sdk

# Knowledge SDK for agents and tooling — devDependency only (not a production runtime)
npm i -D @x12i/memorix-docs
```

**Memorix Capability SDK** (`@x12i/memorix-sdk`) is the productized integration path: invoke pipelines hosted by `memorix-service` and implement the custom capabilities those pipelines call. It re-exports `@x12i/memorix-client` (canonical HTTP) and adds `/worker`, `/authoring`, and `/testing`. It does **not** re-host the pipeline runtime, Mongo, or ledger.
---

## How to read this guide

1. Start with **install** above, then the **central mental model** — every later ebook builds from it.
2. Use the **architecture** and **API families** chapters when wiring a host app.
3. Use the **package matrix** to decide HTTP vs in-process libraries.
4. Labels used throughout the library:
 - **Stored** — persisted subject content or relationship links
 - **Virtual / calculated** — composed, expanded, or projected at read time
 - **Operational** — runs, ledgers, artifacts, quarantines (not ordinary subject content)

---

## The central mental model

Memorix is a **descriptor-driven data platform**.

> **One kind of content about one stable subject** — composed by `recordId`, navigated by four-field relationships, projected by virtual abstracts.

| Idea | Meaning |
|---|---|
| Subject | The stable thing (`WI-100`, employee Alex) |
| Content | One kind of information *about* that subject (snapshot, workflow, analysis, …) |
| Metadata | Declares object types, content types, abstracts, relationships, views, services, pipelines |
| Runtime data | Stored as `memorix-record/2` documents |
| Scope | Every request names a data universe (`orgId` = tenant, e.g. fresh `sandbox*`) and effective metadata (`agentIds[]`, e.g. `opx`) — agent is **not** part of the org name |

Integrators never invent list columns or write shapes in the host. They load metadata, then read/write through the supported boundary.

Authoring follows **three sides**: **A** vocabulary (types/abstracts/relationship defs) → **B** services (only if needed) → **C** data plane + Memory + pipelines. See [Getting started — three sides](../../14-getting-started-and-tutorials/developers/BOOK.md).

**Published npm lineup:** public `@x12i/memorix-*` at **`3.0.2+`** (`memorix-service` / `memorix-opx-pack` **`3.0.3`**); Capability SDK **`@x12i/memorix-sdk@1.0.0+`**. Install with `^` ranges (or omit and take `latest`).

---

## 1. What Memorix is

### The problem it solves

Host products need one place to:

- store many kinds of content about the same subject without flattening them into one blob,
- relate different subjects without copying neighbor payloads,
- project unified “views” (people, knowable entities) without separate databases,
- ingest external sources faithfully, then promote them through pipelines,
- operate runs, reviews, and intelligence proposals with human control.

Memorix answers that with one envelope (`memorix-record/2`), one HTTP service, and one Studio UI.

### One service + one Studio

| Surface | Role |
|---|---|
| `@x12i/memorix-service` | Sole HTTP API (`GET /api/openapi.json`) |
| `@x12i/memorix-studio` | Sole UI (Design · Operate · Explore) — service-only client |
| `@x12i/memorix-*` (16 survivors) | Installable libraries behind the service |
| `@x12i/memorix-opx-pack` | Permanent opx acceptance pack + fixture source |

There is no second product API and no alternate operator UI. Host apps that need a human surface either embed Studio patterns or deep-link operators into Studio.

### Who uses what

| You are… | Primary surface | Install packages? |
|---|---|---|
| Remote product / integrator | `@x12i/memorix-sdk` → `memorix-service` HTTP | **Yes — `@x12i/memorix-sdk`** (or raw HTTP) |
| Called service / worker | `@x12i/memorix-sdk/worker` | **Yes — `@x12i/memorix-sdk`** |
| Operator / PM / builder | `memorix-studio` | Run from monorepo; not an app dependency |
| Platform engineer | Monorepo + in-process libraries | Yes, inside trusted boundary |
| Acceptance / demo | opx-pack + dedicated ops org | `memorix-opx-pack` locally |
### Four tiers

| Tier | Role |
|---|---|
| **1 — Platform** | Engines + service + Studio (`@x12i/memorix-*`) |
| **2 — Instance metadata** | What *this* org/agents know how to run |
| **3 — Instance data** | Records, links, operational ledgers |
| **4 — Packs** | Fixtures, connectors, sample sources (e.g. opx-pack) |

Product vocabulary belongs in **tier 2/4**, never hard-coded into tier-1 engines.

---

## 2. Architecture

```mermaid
flowchart TB
 YourApp[Your solution] --> Service[memorix-service]
 Studio[memorix-studio] --> Service
 Service --> MD[metadata]
 Service --> Data[memorix-data]
 Service --> Rel[relationships]
 Service --> Mem[memory]
 Service --> Pipe[pipeline]
 Service --> Intel[intelligence]
 Service --> Wf[workflow]
 Data --> Mongo[(Mongo)]
```

See also [_shared/diagrams/platform-planes.md](../../_shared/diagrams/platform-planes.md) and [diagrams/surfaces.md](../diagrams/surfaces.md).

### How packages connect

```mermaid
flowchart TB
 App[Your solution] --> Svc[memorix-service]
 Studio[memorix-studio] --> Svc
 Svc --> Meta[metadata + runtime]
 Svc --> DataPlane[memorix-data]
 Svc --> RelEng[relationships]
 Svc --> MemEng[memory]
 Svc --> PipeEng[pipeline]
 Svc --> IntelEng[intelligence]
 Fmt[format] --> DataPlane
 Map[mapping] --> PipeEng
 DataPlane --> Mongo[(Mongo)]
 RelEng --> Store[relationship-store]
 PipeEng --> PipeRt[pipeline-runtime]
 MemEng --> DataPlane
 PipeEng --> DataPlane
 PipeEng --> RelEng
 IntelEng -.->|proposes| Meta
```

### Request lifecycle

1. Client sends `x-memorix-org-id` + `x-memorix-agent-ids`.
2. Service resolves **effective metadata** for that agent set.
3. The requested engine (data, relationships, memory, pipelines, …) runs against scoped Mongo.
4. Response may include composition and/or relationship expansion (virtual at read time).

### Ingest path (opx journey)

```text
External source (opx-source)
 → Memory pull lands exact raw payload (*-raw)
 → Pipeline maps / validates / writes subject content
 → Relationship engine materializes navigation links
 → Explore / host apps see composition + links (never copied payloads)
```

Example subjects in the permanent narrative: employee **Alex**, work item **WI-100**, procedure **PROC-2**. See [_shared/opx-scenario.md](../../_shared/opx-scenario.md).

### Trusted writers

| Concern | Writer |
|---|---|
| Subject content | `memorix-data` (via service HTTP) |
| Links | `memorix-relationships` only |
| Raw landing | `memorix-memory` → data plane |
| Metadata install | Explicit metadata-runtime / HTTP — **not** intelligence |

---

## 3. Scope on every request

Two orthogonal headers. Singular `agentId` is **rejected**.

```http
x-memorix-org-id: sandbox<suffix>
x-memorix-agent-ids: opx
```

| Header | Selects |
|---|---|
| `orgId` | **Data** universe (records, links, operational ledgers) |
| `agentIds[]` | **Effective metadata** (multi-agent inheritance) |

```mermaid
flowchart LR
 Req[Request] --> Org[orgId → data DB]
 Req --> Ag[agentIds → effective metadata]
```

DB naming is owned by `memorix-db-router`. Do not invent database names from `MONGO_URI` path segments in host apps.

**Acceptance rule:** always use a fresh `sandbox*` organization with `agentIds: ["opx"]`. Never use live/legacy customer orgs (`fieldstone`, `neo`, …) for drills.

Studio sets the same scope in the scope bar:

![Scope bar](../screenshots/01-isolated-workspace-scope.png)

---

## 4. API families

Base: running `memorix-service`. OpenAPI: `GET /api/openapi.json`.

| Prefix | Purpose |
|---|---|
| `/api/metadata` | Agents, types, abstracts, install/publish, effective resolution |
| `/api/data` | Content get/list/compose, abstract get/list/patch, writes |
| `/api/relationships` | Verify/rebuild/query (no ad-hoc link PUT/DELETE) |
| `/api/memory` | Pulls, cursors, quarantine (`credentialRef` only for secrets) |
| `/api/pipelines` | Definitions, runs, retries, artifacts |
| `/api/intelligence` | Profiles and proposals (never auto-installs) |
| `/api/workflow` | Workflow sibling content + assignments |
| `/api/operations` | Operational health / ops helpers |

### Writes

Require scope + revision precondition (`absent` or exact `revision`). Stale revision → conflict (`409`).

### Prefer HTTP over libraries

Remote hosts should call **service HTTP**. Install `@x12i/memorix-data` / relationships / pipeline only inside the same trusted process boundary as the platform. Never write domain records with raw Mongo drivers.

Deep dive: [Runtime Data Client](../../02-runtime-data-client/developers/BOOK.md) · [Data Access Patterns](../../04-data-access-patterns/developers/BOOK.md).

---

## 5. Packages (when to use what)

All of the packages below publish as **`@x12i/memorix-*@3.0.2+`** (Capability SDK `@x12i/memorix-sdk@1.0.0+`). HTTP compliance kits (`ports-manager`, `api-live-view`, `core-service`) are separate `@x12i/*` packages.

| Package | Integrator use |
|---|---|
| **`memorix-sdk`** | **Primary developer edge** — `/client` (pipelines + Explore + checkpoints + raw + `commitPage` + grants), `/worker`, `/authoring`, `/testing` |
| `memorix-client` | Canonical HTTP foundation (re-exported by SDK `/client`); keep for existing installs |
| `memorix-format` | Validate / migrate `memorix-record/2` offline |
| `memorix-metadata` | Declarative contracts + abstract unions |
| `memorix-metadata-runtime` | Install / effective metadata client (trusted boundary) |
| `memorix-mapping` | Deterministic mapping AST execution |
| `memorix-data` | Read/write/compose/abstract (preferred library behind service) |
| `memorix-relationship-store` | Link storage contract (engine-internal; not for ad-hoc writes) |
| `memorix-relationships` | Trusted relationship engine |
| `memorix-memory` | Source pull → `*-raw` landing |
| `memorix-pipeline` | Pipeline definitions and execution |
| `memorix-pipeline-runtime` | Run/attempt/artifact ledgers |
| `memorix-intelligence` | Structural profiling + proposals |
| `memorix-service` | HTTP composition root |
| `memorix-studio` | Product UI |
| `memorix-opx-pack` | Ops metadata + fixtures pattern |
| `memorix-db-router` | Org/agent DB naming |
| `memorix-mongo` | Mongo helpers used by data plane |

### Decision tree — HTTP vs package

```text
Remote host / another product?
 └─ Yes → npm i @x12i/memorix-sdk (or call memorix-service HTTP)
Implement a called service / worker?
 └─ Yes → @x12i/memorix-sdk/worker (+ core-service compliance)
Same trusted process as the platform?
 └─ Yes → may use memorix-data / metadata-runtime / engines in-process
Need to mutate domain content?
 └─ Always through data plane (service or memorix-data) — never raw Mongo
Need to mutate relationship links?
 └─ Only via relationship engine APIs (verify/rebuild/materialize) — never ad-hoc PUT/DELETE
```
---

## 6. The opx mental path end to end

1. **Scope** — create/select fresh `sandbox*`, set `agentIds: ["opx"]`.
2. **Resolve metadata** — install opx pack; load types, views, pipelines for that scope.
3. **Write / read** — data plane (`/api/data`) for subject content.
4. **Ingest** — Memory lands exact raw → pipelines promote → relationships link.
5. **Operate** — verify/rebuild links, inspect runs, install proposed metadata (**human** confirm).

Composition example: request all content about `WI-100` and receive siblings (snapshot, workflow, analyses) without flattening them into one document.

![Composed subject](../screenshots/10-composed-subject.png)

Agent inheritance (Design) selects which metadata definitions contribute to effective resolution:

![Agents inheritance](../screenshots/02-agents-inheritance.png)

---

## 7. Local stack (quick start)

From the monorepo after workspace install:

```bash
node scripts/dev-stack.mjs
# opens browser tabs: stack status, /_live views, Studio
# use --no-open to skip browser tabs
```

| Surface | Default URL |
|---|---|
| Studio | http://127.0.0.1:5101/ |
| Stack status | http://127.0.0.1:5103/ (hub — not Studio; links to each `/_live`) |
| Service live view | http://127.0.0.1:5100/_live (`@x12i/api-live-view` via `@x12i/core-service`) |
| Service OpenAPI | http://127.0.0.1:5100/api/openapi.json |
| Service health | http://127.0.0.1:5100/health |
| Opx-source live / health | http://127.0.0.1:5102/_live · `/health` |

Ports: Memorix zone **5100–5119** (even = API, odd = UI) from [`@x12i/ports-manager`](../../ports-manager/). Avoid **5000/7000** (macOS AirPlay). Quarantine **5000–5099**. How-to for new HTTP workers: [Building Services — HTTP process compliance](../../07-services/developers/BOOK.md#http-process-compliance-local-services). Remote data clients call `/api/*` only — do not install ports kits.

```bash
curl -sS \
 -H 'x-memorix-org-id: sandbox-demo' \
 -H 'x-memorix-agent-ids: opx' \
 http://127.0.0.1:5100/api/openapi.json
```

What to install in *your* app:

```bash
# Default remote host: nothing required for HTTP
npm install @x12i/memorix-format@^3.0.1 # optional offline validate/migrate

# Trusted platform process only:
npm install @x12i/memorix-data@^3.0.1 @x12i/memorix-metadata-runtime@^3.0.1

# New HTTP services in x12i/exellix (bind a port):
npm install @x12i/core-service
```

---

## 8. Hard product rules (integrator checklist)

These invariants apply to every integration, acceptance example, and ebook:

1. Acceptance orgs are fresh `sandbox*` with `agentIds: ["opx"]`.
2. Permanent narrative is non-cyber **ops** (employees, work-items, procedures).
3. Relationships are four-field links only — no root `associated*` storage, no copied payloads.
4. Abstracts are virtual projections — no abstract collections / abstract `recordId`.
5. KnowX is one content type + `data.kind` (`NODE` | `EDGE`) + one abstract `KnowXEntity`.
6. Provenance (`docHash` + `SourceSpan`) is mandatory for KnowX; `epistemic.state` ≠ extraction confidence.
7. Deterministic fixtures only for acceptance — no live AI required.
8. Mutations go through the data plane (`memorix-service` / `@x12i/memorix-data`), never direct Mongo domain writes.

Full text: [HARD-RULES.md](../../HARD-RULES.md).

---

## 9. Forbidden (antipatterns)

- Direct Mongo mutation of domain content
- Copied `associated*` payloads on records
- Abstract collections / inventing an abstract `recordId` to write to
- Treating Studio as optional when operators need a UI — Studio is the sole UI
- Singular `agentId` headers
- Using live customer orgs as acceptance workspaces
- Letting intelligence auto-install metadata
- Ad-hoc relationship link PUT/DELETE outside the engine

---

## 10. Decision tree — which ebook next?

```text
Need the record envelope?
 → Format for Dummies
Need to call the API correctly?
 → Runtime Data Client
Need list/cursor/bulk semantics?
 → Lists, Queries, Pagination
Need “which call for this job?”
 → Data Access Patterns
Need agents / OT / CT / abstracts / mappings?
 → Metadata, Agents, Abstracts & Mappings
Need links or source landing?
 → Relationships & Memory
Need custom code / AI jobs?
 → Building Services
Need ordered multi-step work?
 → Building Pipelines
Need Studio / workflow siblings?
 → Workflow & Studio
Need profiling proposals?
 → Structural Intelligence
Need fixtures / harness?
 → Agent Packs & OpX
Building knowledge claims?
 → Building KnowX
Migrating or stuck?
 → Migration & Troubleshooting
```

### Recommended reading order

1. [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md)
2. [Runtime Data Client](../../02-runtime-data-client/developers/BOOK.md)
3. [Data Access Patterns](../../04-data-access-patterns/developers/BOOK.md)
4. Then topic books as your product needs them

### Glossary

Shared vocabulary: [_shared/glossary.md](../../_shared/glossary.md).

### Twin for Studio audiences

Builders who design and operate without API detail: [Builders version](../builders/BOOK.md).

---

# Platform Overview — builders

# Memorix Platform Overview — Builders

**Audience:** Power users, product managers, and authors who work in Memorix Studio. 
**Twin:** [Developers version](../developers/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md)

---

## How to read this guide

1. Start with **what you are looking at** — Studio is the only product UI.
2. Learn the **three Studio areas** and the **ops data path**.
3. Use the **words** table until they feel natural; then follow the reading order into deeper books.
4. Always work in an acceptance workspace (`sandbox*` + agent `opx`) for drills.

---

## 1. What you are looking at

Memorix stores structured records about subjects (people, work items, procedures). **Metadata** describes *what kinds of things exist* and *how they relate*. You design that metadata and operate runs in **Studio** — the only product UI.

Studio always talks to **one** backend: Memorix Service. You set organization + agent scope in the scope bar. Host-app engineers install **`@x12i/memorix-sdk`** (Capability SDK) against public `@x12i/memorix-*@3.0.2+`; as a builder you usually just open Studio against a running service.

![Scope bar](../screenshots/01-isolated-workspace-scope.png)

### The central idea (builder wording)

> **One kind of content about one stable subject.**

Same subject → same id. Different kinds of information → siblings on that subject (**composition**). Links between different subjects → **relationships** (never copied into each other). Unified lists like “People” → **abstracts** (virtual views, not separate databases).

---

## 2. Three Studio areas

```mermaid
flowchart LR
 Design[Design] --> Operate[Operate]
 Operate --> Explore[Explore]
 Design -.->|metadata install| Operate
 Explore -.->|inspect results| Design
```

| Area | You do this |
|---|---|
| **Design** | Agents, sources, object/content types, abstracts, relationships, mappings, views, services, pipelines, schedules, intelligence proposals, metadata install/publish |
| **Operate** | Runs, attempts, schedules, failures, artifacts, memory pulls, relationship verify/rebuild, intelligence runs |
| **Explore** | Concrete and abstract records, composition, relationships, workflow, analysis, decisions |

Operator screens use lists, forms, and confirmations. Raw JSON lives under expandable technical details — you should not need it for ordinary review.

### Scope bar (always)

Before any meaningful work:

1. Choose a **fresh** `sandbox*` organization (never a live customer org).
2. Set agent to **`opx`** (or the agent set your pack documents).
3. Confirm you are looking at the expected Design / Operate / Explore context.

---

## 3. How data moves (builder view)

```text
External source (opx-source)
 → Memory pull lands exact raw payload
 → Pipeline maps / validates / writes subject content
 → Relationship engine materializes navigation links
 → Explore shows composition + links (never copied payloads)
```

![Composed subject](../screenshots/10-composed-subject.png)

### What “exact raw” means

Memory keeps the source bytes (or structured raw) so you can prove what arrived. Pipelines *promote* that into subject content (snapshots, workflow, KnowX, …). Relationships then become the fast way to navigate between subjects in Explore.

### Agents and inheritance

In Design → Agents you see how packs and child agents contribute definitions. Effective metadata for your scope is the resolved union — not a single flat file you edit blindly.

![Agents inheritance](../screenshots/02-agents-inheritance.png)

---

## 4. Words you need

| Word | Meaning |
|---|---|
| Subject | The thing (`WI-100`) |
| Content | One kind of info about it (snapshot, workflow, …) |
| Composition | Several content kinds on the **same** subject |
| Relationship | Link between **different** subjects |
| Abstract | A unified *view* of several concrete types — not a separate database |
| Revision | Version number used to prevent overwriting someone else's edit |
| Memory | Where exact source material lands before promotion |
| Pipeline | Ordered, retryable steps that transform and write |
| Service | A named capability pipelines (or operations) can invoke |
| Intelligence | Profiles and *proposals* — never auto-installs metadata |
| KnowX | Knowledge claims as content (`NODE` / `EDGE`) with provenance |

Full glossary: [_shared/glossary.md](../../_shared/glossary.md).

---

## 5. A short opx tour (what success looks like)

Use subjects from the permanent narrative: employee **Alex**, work item **WI-100**, procedure **PROC-2**.

1. **Design** — confirm opx metadata is installed for your `sandbox*` org.
2. **Operate** — run a Memory pull from opx-source; inspect landing of raw material.
3. **Operate** — start the relevant pipeline; watch attempts and artifacts.
4. **Explore** — open `WI-100` (or Alex); confirm composition siblings appear.
5. **Explore** — traverse relationship links; confirm you see navigation, not pasted neighbor fields.
6. **Design / Operate** — if intelligence proposed metadata, review and **explicitly** install or reject — never expect auto-install.

---

## 6. Rules builders enforce

- Fresh `sandbox*` + agent `opx` for acceptance and drills.
- No live customer orgs as playgrounds.
- Relationships never copy the other record’s payload into this one.
- Abstracts are views — you always write to a **concrete** type.
- Intelligence proposes; humans install.
- Deterministic fixtures for acceptance — do not require live AI to prove the platform.
- When something fails, capture: org, agents, subject id, content type, revision, run/attempt id — then escalate with that packet.

Hard rules (full): [HARD-RULES.md](../../HARD-RULES.md).

---

## 7. What not to expect

- A second “admin UI” besides Studio
- Nested “associated employee” blobs on records
- An abstract `recordId` you can write to directly
- Mongo `_id` as business identity — business identity is `recordId`
- Pipelines that silently invent relationship links outside the engine
- Host apps that bypass the service with direct database edits

---

## 8. Recommended reading order

1. This overview
2. [Format for Dummies (builders)](../../01-format-for-dummies/builders/BOOK.md)
3. [Workflow & Studio](../../09-workflow-and-studio/builders/BOOK.md)
4. [Metadata, agents & abstracts](../../05-metadata-agents-abstracts/builders/BOOK.md)
5. [Relationships & Memory](../../06-relationships-and-memory/builders/BOOK.md)
6. [Pipelines](../../08-pipelines/builders/BOOK.md) / [Services](../../07-services/builders/BOOK.md) as needed
7. [KnowX](../../12-knowx/builders/BOOK.md) when reviewing knowledge claims
8. [Intelligence](../../10-intelligence/builders/BOOK.md) · [Agent packs & ops](../../11-agent-packs-and-opx/builders/BOOK.md) · [Migration & troubleshooting](../../13-migration-troubleshooting/builders/BOOK.md)

API-level detail → [developers twin](../developers/BOOK.md).

---

## 9. Acceptance workspace

Always use a fresh `sandbox*` organization with agent `opx`. Never use live customer orgs for drills. See [_shared/opx-scenario.md](../../_shared/opx-scenario.md).

---

# Format for Dummies — developers

# Memorix Format for Dummies

**Audience:** Developers integrating solutions that read/write `memorix-record/2`. 
**Twin:** [Builders version](../builders/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense product documentation for the frozen `memorix-record/2` envelope.

---

## How to read this guide

1. Start with the **central explanation** below — everything else builds from it.
2. Work chapters in order; later chapters assume earlier ones.
3. Labels used throughout:
 - **Stored** — persisted subject content or relationship links
 - **Virtual / calculated** — composed, expanded, or projected at read time
 - **Operational** — runs, ledgers, artifacts, quarantines (not ordinary subject content)
4. Mongo storage keys and `_id` appear only as *internal* concerns — never as business identity.

---

## The central explanation

A Memorix record is:

> **One kind of content about one stable subject.**

| Field | Question it answers |
|---|---|
| `recordId` | Which subject is this about? |
| `objectType` | What concrete kind of subject is it? |
| `contentType` | What kind of content about the subject is this? |
| `dataCategory` | Which data universe stores it? |
| `revision` | Which current version of this content instance is this? |
| `.data` | What is this content’s actual payload? |

Everything else — composition, discriminators, relationships, Memory, abstracts, routing — builds from that model.

---

## 1. Memorix in five minutes

### What the format solves

Memorix needs one envelope that can hold:

- an employee snapshot,
- a work-item workflow,
- a raw connector payload,
- an analysis,
- a decision,

…without inventing a separate document system for each, and without copying related records into each other.

### Subject vs content about a subject

- **Subject** — the stable thing you talk about (`WI-100`, employee Alex).
- **Content** — one kind of information *about* that subject (snapshot, workflow, analysis…).

Same subject → same `recordId`. Different kinds of content → different `contentType` (siblings).

### Why every record is self-describing

You can open one JSON document and know: format version, subject, object type, content type, category, revision, human title/identifiers, payload, and system state — without joining external catalogs for the basics.

### Why copied associated payloads caused problems

Older shapes nested full related records (names, titles, nested payloads) under `associated*`. That:

- duplicated data that went stale,
- mixed navigation with evidence,
- made writes unsafe and migrations lossy.

**New rule:** links remember endpoints and meaning; they never copy target payloads.

### The new path

```text
External payload
 → raw content
 → snapshot / derived content
 → composition or relationships
 → retrieval / expansion
```

### One tiny complete record

```json
{
 "format": "memorix-record/2",
 "recordId": "wi-100",
 "revision": 1,
 "objectType": "work-items",
 "contentType": "snapshots",
 "dataCategory": "entity",
 "concept": {
 "title": "Prepare quarterly operations review",
 "identifiers": {
 "primary": {
 "kind": "workItemId",
 "value": "WI-100",
 "issuer": "opx-source"
 }
 }
 },
 "data": {
 "status": "in-progress",
 "priority": "normal"
 },
 "_system": {
 "state": "active",
 "createdAt": "2026-01-10T10:00:00.000Z",
 "modifiedAt": "2026-01-10T10:00:00.000Z"
 }
}
```

**Why this shape:** one subject (`wi-100`), one content kind (`snapshots`), one payload under `.data`, no empty placeholder buckets.

---

## 2. Anatomy of `memorix-record/2`

```json
{
 "format": "memorix-record/2",
 "recordId": "WI-100",
 "revision": 1,
 "objectType": "work-items",
 "contentType": "snapshots",
 "dataCategory": "entity",
 "concept": {},
 "data": {},
 "_system": {}
}
```

*(The empty `{}` above are illustration only — real records must not ship empty optional buckets.)*

| Root | Meaning |
|---|---|
| `format` | Must be exactly `"memorix-record/2"`. |
| `recordId` | Stable subject identity (org-wide). |
| `revision` | Integer ≥ 1 for *this* content instance; starts at 1. |
| `objectType` | Concrete subject type (`work-items`, `employees`, …). |
| `contentType` | Kind of content (`snapshots`, `raw`, `workflow`, …). |
| `dataCategory` | Data universe: `entity` \| `event` \| `knowledge`. |
| `concept` | Stable identifying / business-facing info (title + identifiers). |
| `data` | Content-type-specific payload. |
| `_system` | Machine state, timestamps, optional lifecycle & provenance. |

**Also allowed (narrowly):**

- Snapshot-like types may carry non-empty `analytics`, `enrichment`, `insights`.
- Storage-only `_id` may exist; format law ignores it as business identity.
- Format validation still recognizes lightweight root `associated*` refs (compatibility). Production V2 writers reject *new* root `associated*` writes — authoritative links live in central `relationships` collections.

**Rules you will feel every day:**

- Unknown root properties are **rejected**.
- Empty optional buckets are **omitted** — never emit meaningless `{}`.
- Only `_system` and `_id` underscore roots are allowed on the record.

---

## 3. Subject identity and `recordId`

This is one of the most important chapters.

### What `recordId` is

- The stable identity of the **conceptual subject**.
- Unique across the organization (for example `sandboxabc123`).
- Shared by every content sibling about that subject.

### What `recordId` is not

- Not a Mongo document `_id`.
- Not a revision id.
- Not a “snapshot id”.
- Not interchangeable with external provider ids (those live under `concept.identifiers`).

### Record directory

The operational **record directory** reserves the identity:

- First content write reserves `(recordId → objectType + dataCategory)`.
- Same ID with a conflicting object type or category fails.
- Removing content does **not** free the identity for reuse.
- Directory stores availability (`contentTypes[]`) — never payloads or relationships.

### Merge behavior (intent)

One survivor ID; references move to it. IDs must not be reused after removal. (Merge/alias execution may be deferred; the identity rule still holds.)

### Mongo `_id`

Internal storage addressing may use `recordId` for singletons or a deterministic storage key for discriminated instances. That key is **not** the business subject id. Stored documents keep root `recordId` = subject so siblings stay queryable by subject.

---

## 4. Object type, content type, and data category

Explain the three separately — they answer different questions.

| Field | Meaning | Example |
|---|---|---|
| `objectType` | Concrete subject type | `work-items` |
| `contentType` | Content about that subject | `snapshots` |
| `dataCategory` | Database category | `entity` |

### Common data categories

| Category | Typical subjects |
|---|---|
| `entity` | employees, work-items, contractors |
| `event` | activity-events |
| `knowledge` | procedures, documents |

Operational runs, ledgers, artifacts, and quarantines belong to **operational** storage. They are not ordinary subject content and are not written as `memorix-record/2` subject envelopes.

### Common content types

`raw` · `snapshots` · `analysis` · `decisions` · `discovery` · `workflow` · `context`

Metadata defines which object/content types exist for an agent pack. This list is **not** permanently closed — profiles come from effective metadata (with built-in defaults for well-known types).

Physical collection name: `<objectType>-<contentType>` 
Example: `work-items-snapshots` in database `sandboxabc123_entity`.

---

## 5. The `concept` bucket

`concept` holds stable identifying / business-facing information — **not** the whole content payload.

### What belongs here

- Human-readable `title`
- `identifiers.primary` — `{ kind, value, issuer?, normalizedValue? }`
- Optional `identifiers.alternates` (omit when empty)

### Why a single legacy `_extIdentifier` was insufficient

Present/V1 used one nested machine key. Real subjects have primary + alternate issuers, kinds, and values. V2 models that explicitly.

### `recordId` vs external identifiers

| | `recordId` | External identifiers |
|---|---|---|
| Role | Memorix subject join key | Provider / business ids |
| Stability | Org identity | May have several issuers |
| Location | Root | `concept.identifiers` |

### Example

```json
{
 "title": "Prepare quarterly review",
 "identifiers": {
 "primary": {
 "kind": "workItemId",
 "value": "WI-100",
 "issuer": "opx-source"
 }
 }
}
```

*(Omit `alternates` when empty — do not emit `[]`.)*

---

## 6. The `.data` bucket

### Rule

The content-type-specific payload goes here. Snapshots and workflow may have completely different `.data` shapes.

### Raw content

- Preserves the **provider payload exactly** under `.data`.
- Provider fields must not be renamed, enriched, or decorated inside raw `.data`.
- System-calculated hashes and provenance do **not** belong in raw `.data`.
- A provider-owned field named `contentHash` remains unchanged (it is provider data).

### Comparison — raw vs snapshot (same subject)

**Raw** (`work-items-raw`) — exact provider shape:

```json
"data": {
 "id": "WI-100",
 "title": "Prepare quarterly operations review",
 "status": "in-progress",
 "priority": "normal",
 "assigneeId": "E-10",
 "procedureIds": ["PROC-1"],
 "dependsOnIds": ["WI-90"],
 "updatedAt": "2026-01-15T12:00:00.000Z"
}
```

**Snapshot** (`work-items-snapshots`) — product-normalized readable view:

```json
"data": {
 "status": "in-progress",
 "priority": "normal"
}
```

Same `recordId` (`wi-100`). Different content types. Different `.data` contracts.

Optional snapshot buckets (`analytics`, `enrichment`, `insights`) may appear only on snapshot-like types, and only when non-empty.

---

## 7. `_system`, timestamps, and provenance

### Fields

| Field | Role |
|---|---|
| `state` | `active` \| `archived` \| `superseded` |
| `createdAt` | When this content instance was created (ingestion / write time) |
| `modifiedAt` | When this content instance last changed |
| `observedAt` | Optional provider observation time |
| `lifecycle` | Optional array of `{ process, processedAt }` — omit when empty |
| `provenance` | Optional machine provenance — omit when empty |

### Distinctions that matter

| Pair | Difference |
|---|---|
| Revision vs lifecycle | Revision bumps on every successful mutation. Lifecycle appends only for meaningful process/state events. A revision may change with **no** lifecycle entry. |
| `modifiedAt` vs `observedAt` | Write/ingestion time vs when the provider observed the fact. |
| Provenance vs business data | Provenance is machine evidence (`_system`). Business facts stay in `.data` / `concept`. |
| Pull/run/attempt history | Belongs in **operational ledgers**, not on subject records. |

**Forbidden on subject records:** `_graphRuns`, `_jobTypeRuns`, and similar run histories.

**Relationship links never store provenance** — evidence lives on runs / KnowX-style facets / operational artifacts as designed, not on the four-field link.

---

## 8. Revision and safe writes

### Rules

- Every **content instance** has its own revision.
- Creation begins at revision **1**.
- Successful modification increments it **once**.
- Sibling content types do **not** share one revision.
- Discriminated instances also have **independent** revisions.
- Blind overwrites are forbidden.
- Stale writes fail **without** mutation.

### Preconditions

| Operation | Precondition |
|---|---|
| `add` | `{ kind: "absent" }` |
| `patch` / `replace` / `remove` | `{ kind: "revision", revision: N }` exact |
| `upsert` | absent **or** exact revision |

### Two writers / stale revision

1. Writer A reads `wi-100` snapshots at revision `3`.
2. Writer B patches successfully → revision becomes `4`.
3. Writer A patches with `expectedRevision: 3` → **rejected**; record unchanged at `4`.

**Why:** optimistic concurrency without silent last-write-wins.

Helpers in the format package (`checkExpectedRevision`, `nextRevision`, `applySuccessfulMutation`) document the pure mutation rules; the data-plane writer enforces them on storage.

---

## 9. Composition: same subject, different content

### Core rule

> **Same `recordId` means composition, not a relationship.**

### Ops journey example

All of these are sibling content about `wi-100`:

| Content | Instance |
|---|---|
| Snapshot | `wi-100` + `snapshots` |
| Workflow | `wi-100` + `workflow` |
| Analysis | `wi-100` + `analysis` + `capacity` |
| Analysis | `wi-100` + `analysis` + `delivery-risk` |
| Decision | `wi-100` + `decisions` + `priority` |

### Read behavior

- Default reads (`getContent`) return **one** requested content instance.
- Composition must be **explicitly** requested (`composeSubject`).
- Composed results contain **arrays by content type** (even for singletons).
- Siblings are **never** flattened into the snapshot.
- Workflow and context are ordinary sibling content.
- Composition **never** creates relationship definitions or links.

### Composition response shape (virtual)

```json
{
 "recordId": "wi-100",
 "objectType": "work-items",
 "dataCategory": "entity",
 "contents": {
 "snapshots": [ /* V2 */ ],
 "analysis": [ /* zero or more */ ],
 "decisions": [ /* zero or more */ ],
 "workflow": [ /* V2 */ ]
 },
 "warnings": []
}
```

---

## 10. Discriminated content instances

### Why some content types have more than one instance

Analysis, decisions, and discovery can exist many times per subject (capacity vs delivery-risk; priority vs schedule; …).

### Built-in discriminator paths (defaults)

| Content type | Path | Default |
|---|---|---|
| `analysis` | `data.analysisType` | `"default"` |
| `decisions` | `data.decisionType` | `"default"` |
| `discovery` | `data.discoveryType` | `"default"` |

Metadata defines path / requiredness / default / allowed values. Defaults are applied and **persisted** before addressing so the record stays self-describing.

### Addressing

Same `recordId` + `contentType` + discriminator value → one instance.

- Discriminator values are **immutable** for an existing instance.
- Changing the discriminator requires **remove + add**.
- Each instance has its own revision.
- Mongo’s deterministic storage key is internal; `recordId` remains the subject.

### Two analyses on one work item

```json
{
 "recordId": "wi-100",
 "contentType": "analysis",
 "data": { "analysisType": "delivery-risk", "risk": "low" }
}
```

```json
{
 "recordId": "wi-100",
 "contentType": "analysis",
 "data": { "analysisType": "capacity", "load": "nominal" }
}
```

Validation: missing required discriminator → `V2-DISCRIMINATOR`. Missing with default → warning `V2-DISCRIMINATOR-DEFAULT` (writers apply default).

---

## 11. Raw content and Memory

### Landing

Every connector item lands in `<objectType>-raw`. Original payload is stored under `.data`.

### Memory modes

| Mode | Retention | Landing |
|---|---|---|
| `knowledge` | Persistent, reusable, content-hash dedup | `raw` |
| `context` | Temporary / work-scoped (TTL or discard on work completion) | `raw` |

**Memory mode does not determine `dataCategory`.** Category comes from source route metadata.

### Important behaviors

- Identical knowledge content → **no** revision increment (dedup recorded on operational ledger).
- Changed content → update same raw instance; revision +1.
- Context expiry removes content safely via writer `remove`.
- Raw is **not** automatically promoted.
- Downstream work explicitly chooses: promote · enrich · relate · retain · discard.

Memory’s calculated hash lives under `_system.provenance.memory.contentHash` — never by mutating provider `.data`.

---

## 12. Relationships are not record content

### Core rule

> **Different `recordId` values are connected using relationship links.**

### Four-field stored link

```json
{
 "relationshipId": "…",
 "relationshipDefinitionId": "people-assigned-to-work-items",
 "fromRecordId": "employee-10",
 "toRecordId": "wi-100"
}
```

These are the **only** persisted link fields.

| Concept | Role |
|---|---|
| Relationship **definition** | Metadata (meaning, directionality, path) |
| Relationship **link** | Data (four fields) |

### Rules

- Links do not contain copied names, titles, or target records.
- Links live in centralized per-category `relationships` collections (`sandboxabc123_entity.relationships`, etc.).
- Forward and inverse traversal query the **same** physical link.
- Expansion loads **current** target content when requested.
- Missing targets appear as `unresolved` (calculated) — the link is not deleted.
- Updating the target changes expansion without changing the link.
- Relationship history / provenance belongs in operational runs.

### Compatibility nuance

| Shape | Status |
|---|---|
| Lightweight root `associated*` | Format-valid for validation/migration; **not** authoritative storage |
| Production V2 writers | Reject new root `associated*` writes |
| Sibling `contentType: "relationships"` | Format-valid profile only; **not** authoritative storage — writers reject it |
| Nested `.data.associated*` | Valid when it is untouched **provider raw** data |

Authoritative storage is the central per-category `relationships` collection (not root `associated*`, not a `contentType: "relationships"` sibling).

---

## 13. Abstracts are metadata, not records

### People example (ops)

People = union of `employees` ∪ `contractors`.

- There is **no** `people` collection.
- There is **no** abstract `recordId`.
- Concrete records retain their object type and record ID.
- Abstract **reads** project concrete records through bindings.
- Abstract **writes** reverse-map to one concrete content instance.
- Multi-target or non-reversible abstract writes fail.

**Why:** readers must not assume every named concept becomes a collection.

---

## 14. Scope is outside the record

```json
{
 "orgId": "sandboxabc123",
 "agentIds": ["opx"]
}
```

| Field | Selects |
|---|---|
| `orgId` | Entire data universe (all category + operational DBs) |
| `agentIds[]` | Effective metadata only |

Scope is supplied with operations; it is **not** duplicated into every record or link.

Agents never hide organization records. A caller using several agents still sees the entire organization’s data. Scope is not record-level authorization.

---

## 15. Physical routing

Explain simply — without treating Mongo as business identity.

```text
Database: <orgId>_<dataCategory>
Collection: <objectType>-<contentType>
```

### Examples (acceptance org `sandboxabc123`)

| Address | Meaning |
|---|---|
| `sandboxabc123_entity.work-items-snapshots` | Work-item snapshot content |
| `sandboxabc123_entity.work-items-analysis` | Work-item analyses |
| `sandboxabc123_knowledge.procedures-raw` | Procedure raw landings |
| `sandboxabc123_event.activity-events-snapshots` | Activity event snapshots |
| `sandboxabc123_entity.relationships` | Reserved central relationship links |
| `sandboxabc123_operational.record-directory` | Identity reservation |

Also: operational databases hold run ledgers, Memory pull history, quarantines, captured job context.

**Why physical shape ≠ business identity:** you address subjects by `recordId` + content addressing; routers derive DB/collection from metadata and scope.

---

## 16. Reading records

| Read | Behavior |
|---|---|
| Exact content | One V2 instance (`getContent`) |
| Discriminator-specific | Requires value or applicable default |
| List / filter / sort / cursor | Safe query grammar; opaque cursors (`mrxq1.…`) |
| Explicit composition | `composeSubject` — arrays by content type |
| Abstract union | e.g. People — concrete ids retained |
| Optional relationship refs | Virtual wrapper; stored V2 unchanged |
| Optional target expansion | Loads current targets |
| Independent relationship pagination | Separate from content pagination |
| Depth | Default **1**; deeper rejected |
| Related-property queries | Bounded two-phase plan required |
| Current vs historical | Current content vs captured historical **job context** (operational artifact) |

Default `getContent` / compose / abstract / list return **no** relationships section unless `options.relationships.include === true`.

---

## 17. What is stored versus calculated

| Item | Stored? |
|---|---|
| V2 content record | **Yes** |
| Concrete relationship link (4 fields) | **Yes** |
| Copied related payload | **No** |
| Composition result | **No** (virtual) |
| Expanded relationship target | **No** (virtual) |
| Abstract record | **No** (projection) |
| Effective inherited metadata | Calculated from installed metadata |
| Pipeline run ledger | Operational storage |
| Captured job context | Operational artifact |
| Intelligence proposal | Operational artifact |
| Memory pull history | Operational storage |

---

## 18. Validation and common errors

Plain-language categories. Each shows bad input → code → corrected direction.

### Missing / wrong format

- **Bad:** `"format": "memorix-record/1"` or omitted 
- **Code:** `V2-FORMAT` 
- **Fix:** `"format": "memorix-record/2"`

### Unknown root

- **Bad:** `"notes": "…"` at root 
- **Code:** `V2-ROOT` 
- **Fix:** Put business notes in `.data` (or a defined sibling content type)

### Missing required envelope field

- **Bad:** no `recordId` / `concept.title` / `_system.state` 
- **Codes:** `V2-RECORD-ID`, `V2-CONCEPT-TITLE`, `V2-STATE`, … 
- **Fix:** Complete the envelope

### Conflicting record identity

- **Bad:** Reserve `wi-100` as `employees` after it is `work-items` 
- **Codes:** `DIRECTORY_CONFLICT_OBJECT_TYPE` / `DIRECTORY_CONFLICT_DATA_CATEGORY` 
- **Fix:** New subject → new `recordId`

### Missing discriminator

- **Bad:** analysis without `data.analysisType` when required and no default 
- **Code:** `V2-DISCRIMINATOR` (or writer `WRITE_DISCRIMINATOR_*`) 
- **Fix:** Supply discriminator or rely on persisted metadata default

### Immutable discriminator changed

- **Bad:** Patch `analysisType` from `capacity` to `delivery-risk` 
- **Code:** `WRITE_DISCRIMINATOR_IMMUTABLE` 
- **Fix:** `remove` old instance + `add` new

### Copied root association

- **Bad:** `associatedPeople: [{ recordId, title, data: {…} }]` 
- **Code:** `V2-ASSOC-COPIED` 
- **Fix:** Four-field central link (or lightweight ref only for format/migration tools)

### Relationship sibling write (production)

- **Bad:** Writer invents `contentType: "relationships"` as authoritative storage 
- **Fix:** Use central `relationships` collection

### Stale revision

- **Bad:** Patch with `expectedRevision: 3` when current is `4` 
- **Code:** `WRITE_STALE_REVISION` (also `WRITE_NOT_ABSENT` / `WRITE_NOT_FOUND` for wrong precondition kind) 
- **Fix:** Re-read; retry with current revision

### Production writer rejects (relationship storage)

- **Bad:** New root `associated*` or `contentType: "relationships"` sibling as authoritative storage 
- **Codes:** `WRITE_G4_ASSOCIATED_ROOT`, `WRITE_G4_RELATIONSHIP_SIBLING`, `WRITE_G4_COPIED_PAYLOAD` 
- **Fix:** Central category `relationships` collection

### Invalid lifecycle / unsafe query / abstract write not reversible / missing content

| Situation | Typical codes |
|---|---|
| Unsafe Mongo-shaped query | `READ_QUERY_INVALID`, `READ_QUERY_INJECTION`, `READ_QUERY_FIELD_FORBIDDEN` |
| Abstract write multi-target | `ABSTRACT_WRITE_MULTI_TARGET` |
| Non-reversible property | `ABSTRACT_WRITE_NON_REVERSIBLE` |
| Reserved-empty subject | `READ_UNAVAILABLE` |
| Missing content instance | `READ_NOT_FOUND` / writer `WRITE_NOT_FOUND` |
| Relationship depth / unbounded related scan | `READ_RELATIONSHIP_DEPTH`, `READ_RELATIONSHIP_UNBOUNDED` |

---

## 19. Migration from the present / V1 format

### API

`migratePresentToV2` → `{ ok, v2, extractedRelationships, warnings }`

| Output | Meaning |
|---|---|
| `v2` | Clean V2 envelope — **no** copied associated payloads |
| `extractedRelationships` | Out-of-band candidates — **not** auto-persisted |
| `warnings` | Timestamp defaults, dropped roots, quarantine notes, … |

### Behavior highlights

- Copied related payload is removed (`droppedPayloadKeys`).
- Resolvable `recordId` links preserved out-of-band (`mapped` / pending metadata).
- Unresolvable associations quarantined or dropped with status.
- `_extIdentifier` → primary identifier (`kind` + `value`).
- Existing timestamps mapped deliberately; missing ones default with warnings.
- Subject `_graphRuns` / similar dropped (ops ledger owns history).
- Migration does **not** invent a second relationship store — links belong in the central category `relationships` collection.
- Present format remains **migration input**, not target truth.

Validate the result with `validateMemorixRecordV2(result.v2)`.

---

## 20. Complete example journey (ops / `sandbox*`)

Follow one story — not unrelated fragments.

1. **Pull employee raw** — connector lands `employees-raw` for Alex (`employee-10`); `.data` equals provider payload.
2. **Store employee snapshot** — promote to `employees-snapshots` (role, email).
3. **Store work-item snapshot / workflow / analyses** — `wi-100` siblings: snapshots, workflow, `analysis:capacity`, `analysis:delivery-risk`, `decisions:priority`.
4. **Compose the work item** — `composeSubject` returns arrays by content type; no flattening.
5. **View employee/contractor through People** — abstract union; concrete `objectType` retained.
6. **Materialize employee→work-item relationship** — four-field link in `sandboxabc123_entity.relationships`.
7. **Expand current employee content** — relationship include + expand loads current snapshot.
8. **Update employee** — patch increments employee snapshot revision; expansion shows new data; **link unchanged**.
9. **Stale write rejection** — second writer with old revision fails; no mutation.
10. **Exact run context** — operational artifact retains the older revision for the job that ran then (historical context ≠ current subject content).

---

## 21. Quick decision tree

```text
Is this about the same subject?
├─ Yes → same recordId, separate content / composition
└─ No → relationship link

Can there be multiple instances of this content type?
├─ Yes → metadata discriminator
└─ No → singleton content

Did an external connector provide it?
└─ Land it in raw .data first

Is it execution history or evidence of a run?
└─ Operational store, not the subject record

Is it an abstract concept (e.g. People)?
└─ Metadata projection, not another record/collection
```

---

## 22. Glossary and cheat sheets

| Term | Definition |
|---|---|
| **Subject** | The stable thing being described |
| **Record** | One V2 content envelope about a subject |
| **Content instance** | One addressable content document (singleton or discriminated) |
| **recordId** | Org-wide subject identity |
| **Object type** | Concrete subject type (`work-items`) |
| **Content type** | Kind of content (`snapshots`) |
| **Data category** | `entity` / `event` / `knowledge` (routing universe) |
| **Concept** | Title + identifiers (business-facing identity) |
| **Discriminator** | Field that distinguishes multi-instance content |
| **Composition** | Same `recordId`, multiple content types/instances |
| **Relationship definition** | Metadata meaning of a link |
| **Relationship link** | Stored four-field edge |
| **Abstract type** | Named union/projection over concrete types |
| **Binding** | Maps abstract properties ↔ concrete paths |
| **Revision** | Per-instance optimistic concurrency counter |
| **Lifecycle** | Explicit process/state event log |
| **Provenance** | Machine origin evidence on `_system` |
| **Operational ledger** | Run/pull/attempt history storage |
| **DataRef** | Reference to content without copying payload |
| **Raw** | Exact provider landing content |
| **Snapshot** | Readable product view of a subject |
| **Expansion** | Loading current targets for links at read time |

---

## Appendix A — Example inventory (validated goldens)

Ship / validate these shapes in CI (package examples under `memorix-format/examples/v2/` plus journey fixtures):

| Example | Typical path / note |
|---|---|
| Entity snapshot | `work-items-snapshots.json`, `employees-snapshots.json` |
| Event snapshot | `activity-events-snapshots.json` |
| Knowledge snapshot | `procedures-knowledge.json` (knowledge category) |
| Exact raw provider payload | `work-items-raw.json` |
| Workflow sibling | `work-items-workflow.json` |
| Context sibling | `work-items-context.json` |
| Analysis discriminator | `work-items-analysis.json` (+ second type in journey) |
| Decision discriminator | `work-items-decisions.json` |
| Discovery discriminator | `employees-discovery.json` |
| Lifecycle transition | `work-items-snapshots-lifecycle.json` |
| Multiple external identifiers | journey fixture (primary + alternates) |
| Composition response | virtual — reader contract shape |
| Four-field relationship link | relationship-store fixtures |
| Virtual relationship expansion | reader relationship options |
| Unresolved target | calculated `targetStatus: "unresolved"` |
| Abstract People projection | abstract read contract |
| Migration I/O / extracted relationships | `migratePresentToV2` goldens |
| Invalid copied association | rejected by `V2-ASSOC-COPIED` |
| Invalid stale revision | writer precondition failure |
| Invalid missing discriminator | `V2-DISCRIMINATOR` |
| Lightweight refs (compat) | `work-items-snapshots-lightweight-refs.json` |
| Relationships sibling (non-authoritative format example only) | `work-items-relationships-candidate.json` — not a write target |

---

## Appendix B — Source map (where truth lives)

| Topic | Authoritative sources |
|---|---|
| Envelope / validation | `memorix-format/src/v2/*`, `docs/memorix-record-v2.md` |
| Profiles / discriminators | `memorix-format/src/v2/profiles.ts` |
| Revision helpers | `memorix-format/src/v2/revision.ts` |
| Migration | `memorix-format/src/v2/migrate-present.ts`, `docs/migrate-present-to-v2.md` |
| Writes | `CONTENT-WRITE-CONTRACT.md` |
| Reads / composition / expansion | `CONTENT-READ-CONTRACT.md` |
| Directory | `RECORD-DIRECTORY-CONTRACT.md` |
| Routing | `DATA-ROUTING-CONTRACT.md` |
| Scope | `MEMORIX-SCOPE-CONTRACT.md` |
| Abstracts | `ABSTRACT-WRITE-CONTRACT.md`, read contract |
| Relationships | `RELATIONSHIP-STORE-CONTRACT.md`, `RELATIONSHIP-STORAGE-DECISION.md` |
| Raw / Memory | `SOURCE-TO-RAW-CONTRACT.md`, `MEMORY-PULL-CONTRACT.md` |

When docs conflict with runtime behavior, prefer **implemented contracts and package tests** and list the contradiction rather than silently picking a side.

### Known dual truths (do not silently merge)

| Tension | Canonical wording |
|---|---|
| Present `meta/envelope.json` / `_extIdentifier` vs V2 `concept.identifiers` | Present meta is migration/legacy only; V2 law is `memorix-record/2` |
| Lightweight `associated*` format-valid vs production writer reject | Format validation may accept legacy shapes for migration tools; **authoritative storage is the central `relationships` collection**; production writers reject new root `associated*` |
| Sibling `contentType: "relationships"` golden | Format-valid profile example only — not production storage |
| `contentType: "knowledge"` (buckets) vs `dataCategory: "knowledge"` | Different knobs; ops procedures typically use `dataCategory: "knowledge"` + `contentType: "snapshots"` |
| Acceptance org naming | Product acceptance uses fresh `sandbox*` orgs with `agentIds: ["opx"]`. Never use live or legacy customer org names as the acceptance organization. |

---

## Writing rules this ebook follows

- Mental model before schemas.
- Ops journey throughout (`sandbox*` + `agentIds: ["opx"]` for acceptance); no cyber scenario.
- Stored / virtual / operational labeled clearly.
- Mongo details never promoted to business concepts.
- Legacy package names only in migration context.
- “Why” beside important rules.
- One running scenario.

*End of Memorix Format for Dummies.*

---

# Format for Dummies — builders

# Memorix Format for Dummies — Builders

**Audience:** Studio builders and product managers who need the record mental model without API detail. 
**Twin:** [Developers version](../developers/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense Studio guide. Deterministic fixtures only — **no live AI required**.

---

## How to read this guide

1. Start with the **central idea** — one kind of content about one stable subject.
2. Match the **screenshots** to what you see in Explore.
3. Learn **composition vs relationship** and **abstracts** before editing anything.
4. Use the **ops walkthrough** with Alex / WI-100 as your acceptance drill.
5. Envelope / CLI depth → [developers twin](../developers/BOOK.md).

---

## The central idea

A Memorix record is:

> **One kind of content about one stable subject.**

| Field you see | Question it answers |
|---|---|
| Subject id (`recordId`) | Which thing is this about? |
| Object type | What kind of thing (work-item, employee, …)? |
| Content type | What kind of information (snapshot, workflow, analysis, …)? |
| Revision | Which edit version am I looking at? |

Same subject → same id. Different kinds of information → siblings (**composition**). Links between different subjects → **relationships** (never copied into each other).

```text
WI-100 (one subject id)
 ├─ snapshots ← “what the work item looks like”
 ├─ workflow ← “who is reviewing / what state”
 ├─ analysis ← “one analysis instance” (may have siblings)
 └─ … other kinds

Alex (different subject id)
 └─ linked to WI-100 via a relationship — not pasted into WI-100’s snapshot
```

Always work in a fresh **`sandbox*`** org with agent **`opx`** when drilling this model.

---

## 1. What you see in Explore

### A single content document

![Raw record](../screenshots/09-raw-record.png)

One content document about one subject — for example a work-item snapshot. You can open technical details for JSON, but ordinary review should stay on forms and status.

### Composition (siblings)

![Composed subject](../screenshots/10-composed-subject.png)

Ask Explore for “everything about WI-100” and Studio shows **siblings** — snapshot, workflow, analyses — without flattening them into one blob. Same subject id; different content types.

### Discriminators (many instances)

![Discriminators](../screenshots/12-analysis-decision-discriminators.png)

Some content types allow **many instances** on the same subject (for example several analyses or decisions). A **discriminator** labels which instance you are looking at so saves do not overwrite the wrong sibling.

| Idea | Builder meaning |
|---|---|
| One snapshot | Usually one current business picture |
| Many analyses | Each needs a clear label (discriminator) |
| Workflow | Process / assignment state — ordinary sibling, not a separate product |

---

## 2. Composition vs relationship

```text
Same recordId → composition (siblings on one Explore page)
Different recordIds → relationship (link + expansion)
```

| Composition | Relationship |
|---|---|
| Snapshot + workflow on WI-100 | Alex **assigned to** WI-100 |
| Same subject id | Two subject ids |
| Siblings listed together | Four-field link; expansion shows current targets |

Relationships remember *who links to whom* and *why*. They do **not** paste the other record’s fields into this one. If you see a nested “associated employee payload” inside a work-item snapshot, that content is wrong — screenshot and escalate.

---

## 3. Abstracts (unified views)

An abstract type (for example **People**) is a **virtual union** of concrete types (employees + contractors). There is no separate “people database.”

| You can | You cannot |
|---|---|
| List People in Explore | Write to an abstract `recordId` as if it were concrete |
| Use abstracts in relationship *definitions* | Expect a physical abstract collection |
| Drill into a concrete employee/contractor | Treat Mongo `_id` as the business id |

Writers always target a **concrete** object type. Business identity is always `recordId`.

---

## 4. Revision (why saves sometimes conflict)

Every content instance has a **revision**. When you save, Studio sends the revision it believes is current. If someone else saved first, you get a **stale conflict** — refresh, re-apply your intent, save again.

That is protection, not a bug. Do not ask for a Mongo force-write.

---

## 5. Words you need in Studio

| Word | Meaning |
|---|---|
| Subject | The thing (`WI-100`, Alex) |
| Content | One kind of info about it |
| Composition | Several content kinds on the **same** subject |
| Relationship | Link between **different** subjects |
| Abstract | Unified *view* of several concrete types |
| Discriminator | Label for which instance among many |
| Revision | Version used to prevent silent overwrite |
| Memory | Exact source material before promotion |
| Workflow | Process/assignment sibling content |

Full glossary: [_shared/glossary.md](../../_shared/glossary.md).

---

## 6. Short ops walkthrough

Subjects: employee **Alex**, work item **WI-100**, procedure **PROC-2**.

1. Scope bar → fresh `sandbox*` + agent `opx`.
2. Explore → open a single raw/content record (screenshot 09).
3. Explore → open WI-100 **composition** — confirm siblings (screenshot 10).
4. Explore → open analyses/decisions — note discriminators (screenshot 12).
5. Explore → traverse a relationship from Alex or WI-100 — confirm expansion, not copied payloads.
6. Optional: edit workflow on WI-100; if you hit a conflict, refresh and retry.

---

## 7. Decision tree

```text
Am I looking at one subject with several panels?
└─ Composition (siblings)

Am I jumping from one subject to another?
└─ Relationship (link)

Is this a unified “People” list?
└─ Abstract view — drill into concrete types to edit

Did save conflict?
└─ Refresh → re-apply → save (revision)

Do I see nested associated blobs?
└─ Wrong — escalate with screenshot
```

---

## 8. What not to expect

- Nested “associated employee payload” blocks on records
- An abstract `recordId` you can write to directly
- Mongo `_id` as business identity — business identity is `recordId`
- A second product UI besides Studio for ordinary operator work
- Intelligence that silently changes types (proposals need explicit install)

---

## 9. Anti-patterns

| Anti-pattern | Why it hurts | Do this instead |
|---|---|---|
| Treat composition as “one big JSON blob” | Loses revisions / siblings | Keep content types separate |
| Copy neighbor fields into this record | Stale data / wrong model | Relationships + expansion |
| Ignore discriminators on analyses | Overwrite the wrong instance | Pick the labeled instance |
| Practice on live customer orgs | Contamination | Fresh `sandbox*` + `opx` |
| Raw JSON as the only review surface | Miss conflicts and evidence | Forms + status first |

---

## 10. Next

| Track | Link |
|---|---|
| Envelope / validate / migrate | [Developers twin](../developers/BOOK.md) |
| Studio Design / Operate / Explore | [Workflow & Studio builders](../../09-workflow-and-studio/builders/BOOK.md) |
| Relationships & Memory | [Relationships & Memory builders](../../06-relationships-and-memory/builders/BOOK.md) |
| Metadata & abstracts | [Metadata builders](../../05-metadata-agents-abstracts/builders/BOOK.md) |
| Conflicts & escalate | [Migration & troubleshooting builders](../../13-migration-troubleshooting/builders/BOOK.md) |

---

*End of Memorix Format for Dummies — Builders.*

---

# Runtime Data Client — developers

# Memorix Runtime Data Client Guide

**Audience:** Application, connector, agent-runtime, SDK, and script authors that call Memorix through the supported service boundary. 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Packages:** `@x12i/memorix-service` (sole HTTP composition root) · `@x12i/memorix-data` (in-process data plane only) · Studio (sole product UI). 
**Prerequisites:** [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md) · [Platform Overview](../../00-platform-overview/developers/BOOK.md). 
**Status:** Present-tense product documentation for the frozen client runtime boundary. Developers only — there is no builders twin for this book.

---

## How to read this guide

1. Start with the **central mental model** below — every later chapter assumes it.
2. Work chapters in order when you are learning; jump by task when you already know the envelope.
3. Labels used throughout:
 - **Stored** — persisted subject content or four-field relationship links
 - **Virtual / calculated** — composition, abstract projection, relationship expansion at read time
 - **Operational** — pipeline runs, Memory pulls, quarantine, artifacts, verify/rebuild reports
4. OpenAPI is authoritative for path names and schemas: `GET /api/openapi.json` on a running `memorix-service`.
5. This book teaches **how clients discover, read, write, compose, query, and handle failures**. It does not re-teach format law — that lives in Format for Dummies.

**Companion distinction:** Format for Dummies answers “what does this record mean?” This guide answers “how do I safely get and change it through the service?”

---

## The central mental model

A client never “talks to Mongo.” A client talks to **Memorix Service**, which applies scope, metadata, revision preconditions, and routing through `@x12i/memorix-data`.

Clients work with **subjects** and **content instances**:

| Idea | What the client uses |
|---|---|
| Subject | Stable `recordId` (e.g. Alex, `wi-100`, `proc-2`) |
| Concrete type | `objectType` (`employees`, `work-items`, `procedures`) |
| Kind of content | `contentType` (`snapshots`, `workflow`, `raw`, `analysis`, …) |
| Which instance | Optional discriminator (`analysis:capacity`, `decisions:priority`) |
| Version of that instance | Integer `revision` (≥ 1) |
| Same subject, other content | Composition (siblings share `recordId`) |
| Different subjects | Relationships (four-field links, never copied payloads) |
| Named union (People) | Abstract projection — virtual; no abstract collection |
| Run history / artifacts | Operational stores — not subject siblings |

```text
Client (app / agent / script / SDK)
 │ x-memorix-org-id + x-memorix-agent-ids
 ▼
memorix-service (sole HTTP API)
 │
 ▼
@x12i/memorix-data (trusted in-process only)
 │
 ▼
routed stores (domain content, relationships, Memory, ops ledgers)
```

**Studio** is the sole product UI over the same scope and APIs. Remote hosts should prefer HTTP; do not install internal packages just to bypass the service.

Authoritative runtime boundary: `memorix-service` OpenAPI · `orgId` + `agentIds[]` · data-plane behavior behind the service · metadata / relationship / Memory / pipeline contracts **as exposed to clients**.

---

## 1. Client mental model in practice

### Subjects and content instances

- **Alex** is one employee subject: one `recordId`, many possible siblings (`snapshots`, `workflow`, `knowx`, …).
- **WI-100** is one work-item subject: snapshot, workflow, analyses, and decisions are separate content instances.
- **PROC-2** is one procedure subject; revising its knowledge snapshot bumps *that* instance’s revision, not “the procedure’s global revision.”

Revision belongs to **one content instance**, not to the subject as a whole. Two analyses on `wi-100` each have their own revision counter.

### What clients never do

| Never | Why |
|---|---|
| Create “abstract records” or abstract collections | Abstracts are virtual projections |
| Persist relationship expansions as source-of-truth | Expansion is a current view at read time |
| Address by Mongo `_id` or invent collection names | Business identity is `recordId` + type address |
| Infer row-level auth from `agentIds` | Agents select **metadata**, not which org rows you may see |
| Write domain content with a Mongo driver | Mutations go through the data plane only |

### Decision tree — what am I addressing?

```text
Do I need one content instance?
├─ Yes, singleton (snapshots, workflow, …)
│ → (objectType, contentType, recordId)
└─ Yes, multi-instance (analysis, decisions, …)
 → (objectType, contentType, recordId, discriminator)

Do I need several siblings of one subject?
└─ Explicit compose — response keeps arrays per contentType

Do I need a union across concrete types (People)?
└─ Abstract get/list — retain concrete objectType on each hit

Do I need neighbors of a different recordId?
└─ Relationship query / expand — four-field links + current targets
```

### May / must-not

- **You may** call service HTTP from any trusted host with scope headers.
- **You may** use `@x12i/memorix-data` **inside** the platform process that already hosts the service.
- **You must not** treat package internals, adapters, or Mongo keys as the public client contract.
- **You must not** use live/legacy orgs (`fieldstone`, `neo`, …) for acceptance — use fresh `sandbox*` with `agentIds: ["opx"]`.

---

## 2. Connection and scope

### Canonical client configuration

Prefer the Capability SDK `@x12i/memorix-sdk/client` (re-exports `@x12i/memorix-client`: Explore, pipelines, Source checkpoints, raw Memory, `commitPage`, and `fromInvokeContext` for called services). The configuration shape is:

```ts
import { createMemorixClient, fromInvokeContext } from "@x12i/memorix-sdk/client";

type MemorixClientConfig = {
 baseUrl: string; // e.g. http://127.0.0.1:5100
 orgId: string; // exactly one — entire org data universe
 agentIds: string[]; // one or more unique ids — effective metadata only
};

const client = createMemorixClient({
  baseUrl: "http://127.0.0.1:5100",
  orgId: "sandbox…",
  agentIds: ["opx"],
});

await client.explore.getContent({
  objectType: "employees",
  contentType: "snapshots",
  id: "E1",
});
await client.checkpoints.list("opx-source");

const run = await client.pipelines.start("knowx-associate", { recordId: "PROC-2" });
await client.pipelines.wait(run.runId);

// Called-service path: Memorix injects `memorix` (+ HMAC grant) on invoke
const mx = fromInvokeContext(req.memorix);
await mx.explore.listContent({ objectType: "assertions", contentType: "knowx", limit: 200 });
// memorixAccess may include "explore" | "checkpoints" | "raw"
// mx.memory.commitPage({ … }) — Model B atomic land + checkpoint
```

HTTP form (every `/api/*` call):

```http
x-memorix-org-id: sandbox<suffix>
x-memorix-agent-ids: opx
x-memorix-invocation-grant: …   # when present from fromInvokeContext
```

Rules:

1. Exactly **one** `orgId` — selects the organization data universe.
2. One or more unique **`agentIds`** — selects **effective metadata** only.
3. Canonical agent ordering is applied by the service; send a stable unique list.
4. There is **no** singular `agentId` in supported APIs — singular headers are rejected (`400`).
5. `agentIds` never filter which organization rows you can read; they do not replace authorization.
6. Deployment authentication (mTLS, gateway tokens, …) is **separate** from Memorix data/metadata scope.
7. Callback clients: server-issued **grant** enforces `memorixAccess`; mutating `access[]` cannot escalate privileges.

### Where scope is supplied

| Channel | Role |
|---|---|
| HTTP headers | Canonical for service callers |
| SDK client config | Maps to the same headers / scope object |
| Query / body | Must not invent a *different* scope semantics per route |

There is **one** scope rule for the platform: org selects data universe; agents select effective metadata. Studio’s scope bar sets the same pair.

### Example — configure then probe OpenAPI

```bash
ORG=sandboxabc123
AGENTS=ops
BASE=http://127.0.0.1:5100

curl -sS \
 -H "x-memorix-org-id: $ORG" \
 -H "x-memorix-agent-ids: $AGENTS" \
 "$BASE/api/openapi.json" | head
```

```ts
const client = {
 baseUrl: process.env.MEMORIX_BASE_URL!,
 orgId: "sandboxabc123",
 agentIds: ["opx"],
};

function scopeHeaders(c: typeof client): HeadersInit {
 return {
 "x-memorix-org-id": c.orgId,
 "x-memorix-agent-ids": c.agentIds.join(","),
 };
}
```

### Process debug vs product API

| Surface | Purpose | Client installs kits? |
|---|---|---|
| `/api/*` (OpenAPI) | Product contract — data, metadata, Memory, pipelines, … | N/A — call HTTP |
| `GET /health` | Process liveness (local stack) | **No** — do not install `@x12i/core-service` / ports kits |
| `GET /_live` | In-process request rings for the bound service | **No** — debug only; not a domain API |

Local URLs and the Memorix zone (**5100–5119**) are oriented in [Platform Overview](../../00-platform-overview/developers/BOOK.md). Authors of **new HTTP processes** (workers Memorix invokes) follow HTTP compliance in [Building Services](../../07-services/developers/BOOK.md). Remote data clients only need `baseUrl` + scope headers.

### Antipatterns

- Omitting agent headers and hoping defaults apply.
- Sending `agentId` (singular) “because Studio used to.”
- Using `agentIds` as a substitute for per-record ACL.
- Pointing a host app at Mongo “just for reads.”
- Treating `/_live` or `/health` as substitutes for `/api/data` or `/api/data-journey`.

---

## 3. Metadata discovery for runtime clients

Before constructing reads and writes, discover **effective** metadata for your scope.

Clients typically need:

- Effective object types and content types
- Data categories and routing implications
- Discriminator metadata (path, required/default)
- Property / schema / view descriptors for UI and validation
- Lifecycle rules where exposed
- Abstract types and bindings
- Relationship definitions
- Mapping / write capabilities relevant to your job

### Raw vs effective

| Kind | Meaning for clients |
|---|---|
| Raw pack metadata | What was installed/published |
| **Effective** metadata | What this `orgId` + `agentIds[]` resolves to *now* |

Prefer effective endpoints for runtime addressing. Inheritance provenance and multi-agent conflict resolution are server concerns; clients should treat conflicts as metadata errors to surface, not silently pick a winner.

### Caching guidance

- **You may** cache effective metadata for the duration of one user operation or short request batch.
- **You must** invalidate when the org’s metadata fingerprint/checksum changes, or after install/publish.
- **You must not** hardcode Mongo collection names derived from types.
- On metadata validation errors, fail closed — do not guess addresses.

### Practical flow

1. `GET /api/openapi.json` — learn operation IDs and schemas.
2. `GET /api/metadata/...` — resolve kinds, object/content types, abstracts, views (exact paths in OpenAPI).
3. Confirm discriminator defaults before writing multi-instance content.
4. Drive list columns and forms from view/property descriptors — do not invent host-side schemas.

### OpX example

For `sandboxabc123` + `opx`, discover that `work-items` exposes `snapshots`, `workflow`, `analysis`, `decisions`; that `People` is an abstract over `employees` ∪ `contractors`; that relationship definition `employee-owns-work-item` exists. Only then address `wi-100` or Alex.

```bash
curl -sS \
 -H "x-memorix-org-id: $ORG" \
 -H "x-memorix-agent-ids: $AGENTS" \
 "$BASE/api/metadata/kinds" | head
```

---

## 4. Exact content reads

Exact reads retrieve **one** content instance. They do **not** automatically compose siblings or expand relationships.

### Addressing

| Field | When |
|---|---|
| `recordId` / `id` | Always for get-by-id |
| `objectType` | Required for concrete content |
| `contentType` | Required for concrete content |
| Discriminator | When the content profile is multi-instance |
| Projection / view | When the API exposes a view name |
| Scope | Always |

### What you can retrieve

- Singleton content (e.g. Alex’s employee snapshot)
- Discriminated content (e.g. `wi-100` analysis `capacity`)
- Missing content → not-found / unavailable semantics (see errors)
- Reserved-but-empty subject → identity reserved, content not yet present
- Raw, snapshot, workflow, context, analysis, decision, discovery — as metadata allows

### Typical response envelope fields

`format`, `recordId`, `revision`, `objectType`, `contentType`, `dataCategory`, optional discriminator, `concept`, `data`, `_system`, plus request/correlation id on the HTTP layer.

### Example — read WI-100 snapshot

```bash
curl -sS \
 -H "x-memorix-org-id: $ORG" \
 -H "x-memorix-agent-ids: $AGENTS" \
 "$BASE/api/data/content?objectType=work-items&contentType=snapshots&id=wi-100"
```

```ts
async function getSnapshot(recordId: string) {
 const url = new URL("/api/data/content", client.baseUrl);
 url.searchParams.set("objectType", "work-items");
 url.searchParams.set("contentType", "snapshots");
 url.searchParams.set("id", recordId);
 const res = await fetch(url, { headers: scopeHeaders(client) });
 if (!res.ok) throw new Error(await res.text());
 return res.json(); // memorix-record/2
}
```

### May / must-not

- **You may** read one instance and later compose if the UI needs siblings.
- **You must not** assume a get returns related employees “because the work item mentions them.”
- **You must not** treat absence of a sibling as an error unless your product requires that sibling.

---

## 5. Listing and querying

Lists return pages of content (or operational items). Prefer the accepted query grammar — never raw Mongo operators.

### Safe query surface (conceptual)

| Concern | Client rule |
|---|---|
| Field paths | Only allowlisted paths (e.g. `data.status`, system timestamps as documented) |
| Operators | Service-defined; injection-shaped Mongo ops rejected |
| Sort | Documented fields + direction; server supplies stable tie-breaker |
| Limit | Required/capped (content lists commonly max 200) |
| Cursor | Opaque; bound to original query/sort/shape |
| Empty page | Valid terminal or sparse page — honor `pageInfo` |
| Cursor mismatch / expiry | Restart pagination from the beginning |

### Explain

- Cursors are **opaque** — do not decode, edit, or reuse across different queries.
- Changing filter/sort/shape requires a **new** first page.
- Unbounded scans are rejected where the service enforces bounds.
- Raw Mongo operators are rejected; Mongo injection is never supported.

### Examples (ops)

**List work-item snapshots**

```bash
curl -sS \
 -H "x-memorix-org-id: $ORG" \
 -H "x-memorix-agent-ids: $AGENTS" \
 "$BASE/api/data/content?objectType=work-items&contentType=snapshots&limit=50"
```

**Filter / sort / page** (shape follows OpenAPI — illustrative):

- Filter `data.status=in-progress` for open work.
- Sort by `_system.modifiedAt` descending with server tie-breaker.
- Pass `cursor` from `pageInfo.nextCursor` until absent.
- List only `analysis` with a specific discriminator when the API exposes it.
- On cursor mismatch → start over; do not “fix” the cursor string.

```text
GET ...&limit=50&cursor=<opaque>
→ { records|items, pageInfo: { nextCursor } }
```

Deeper pagination law lives in [Lists, queries, pagination](../../03-lists-queries-pagination/developers/BOOK.md).

### Antipatterns

- Client-side `find({ $where: … })` mental models against Memorix.
- Parallel “page 1” fetches with different sorts glued into one UI table.
- Assuming insertion order without a documented sort.

---

## 6. Explicit composition

Composition is **requested**. Same `recordId`, multiple content types/instances.

### Shape rules

- Response uses **arrays per content type** (even singleton types appear as arrays in compose).
- Siblings are **not** flattened into one mega-document.
- Clients may select which content types to include.
- Workflow / context / analysis / decisions appear only when selected.
- Composition carries **no** relationship definitions — neighbors stay relationship APIs.

### OpX examples

| Request | Result idea |
|---|---|
| Compose `wi-100` with `snapshots` + `workflow` | One snapshot array entry + workflow array entry |
| Include two analyses | `analysis: [capacity, delivery-risk]` (names illustrative) |
| Include three decisions | Arrays under decisions; independent revisions |
| Optional sibling missing | Empty array or omitted per contract — not a fake null record |

```text
composeSubject(wi-100)
 snapshots[] revision N
 workflow[] revision M
 analysis[] each own revision
 decisions[] each own revision
```

```bash
curl -sS -X POST \
 -H "content-type: application/json" \
 -H "x-memorix-org-id: $ORG" \
 -H "x-memorix-agent-ids: $AGENTS" \
 -d '{"recordId":"wi-100","explicit":true,"contentTypes":["snapshots","workflow","analysis","decisions"]}' \
 "$BASE/api/data/compose"
```

OpenAPI operation: `dataCompose`. `explicit: true` is required. Response keeps **arrays per content type**; missing siblings are empty/omitted arrays (plus warnings when the contract exposes them) — not invented singleton objects.

### May / must-not

- **You may** compose for a dossier / Studio Explore-style view.
- **You must not** invent a single “subject revision” by taking `max(sibling revisions)`.
- **You must not** flatten arrays into one object keyed only by content type when multiple discriminators exist.
- **You must not** call compose as `GET` with query params — the service accepts **`POST /api/data/compose`** only.

---

## 7. Abstract reads

Abstracts are **virtual projections** over concrete records. There is no `People` collection and no abstract `recordId`.

### Operations

- **Get** one abstract projection by concrete identity under an abstract type.
- **List** an abstract union (e.g. People = employees ∪ contractors).
- Each hit retains concrete `recordId` + `objectType`.
- Direct **pushdown** filters run when the service can map them; unsupported non-pushdown queries fail clearly.
- Cross-collection pagination is deterministic when exposed — trust the cursor, do not reimplement merges.
- Abstract binding/provenance appears where the API exposes it.

### OpX example — list People

```bash
curl -sS \
 -H "x-memorix-org-id: $ORG" \
 -H "x-memorix-agent-ids: $AGENTS" \
 "$BASE/api/data/abstract?abstractType=People&limit=50"
```

Alex may appear as `objectType: "employees"`. A contractor peer appears as `objectType: "contractors"`. Both are People in the UI; neither is stored under a People collection.

### May / must-not

- **You may** bind UI forms to abstract property names when bindings exist.
- **You must not** construct abstract storage paths or invent abstract record ids.
- **You must not** write “into People” except via supported abstract reverse-write (`patchAbstract`) that targets one concrete member.

---

## 8. Relationship reads

Default content reads contain **no** relationships. Expansion is opt-in.

### Capabilities

- Explicit relationship include / `expandRelationships`
- Forward and inverse traversal as exposed
- Filter by relationship definition
- Independent relationship pagination
- Selected target content types
- Abstract target projection where supported
- Depth-one default / hard limits
- Unresolved targets surfaced (do not invent data)
- Related-property filter/sort only when bounded
- Diagnostics when a query would be unbounded

### Virtual wrapper

```json
{
 "source": {
 "format": "memorix-record/2",
 "recordId": "employee-10",
 "objectType": "employees",
 "contentType": "snapshots"
 },
 "relationships": {
 "items": [
 {
 "definition": "employee-owns-work-item",
 "fromRecordId": "employee-10",
 "toRecordId": "wi-100",
 "targetStatus": "resolved"
 }
 ],
 "pageInfo": {}
 }
}
```

Clarify:

- Target content is **current at read time**.
- Target content is **never copied onto the source** as authoritative storage.
- Updating Alex’s snapshot changes future expansions; the **stored link remains four fields**.
- Clients **must not** persist the expansion wrapper as the system of record.

### Four-field links only

Authoritative storage is the central relationships collection: definition + endpoints (+ allowed link metadata). No root `associated*`, no copied neighbor payloads, no relationship sibling content as product storage.

### Unresolved targets

If `wi-100` was removed or never landed, expansion may return `targetStatus: "unresolved"`. Treat that as readable state — decide whether to retry later or show a gap in the UI.

### Ops story

Alex owns WI-100. Get Alex’s snapshot with relationship include for `employee-owns-work-item`. You see a four-field link and optionally the current WI-100 target. Patch Alex’s title → link unchanged → next expansion shows the new title on the source and current WI-100 on the target.

---

## 9. Concrete writes

Supported write operations through the data plane:

| Op | Intent |
|---|---|
| `add` | Create only if absent |
| `upsert` | Create if absent **or** update at exact revision |
| `patch` | Partial update at exact revision |
| `replace` | Full content replace at exact revision |
| `remove` | Delete instance at exact revision |

Every write requires:

- Memorix scope
- Concrete content address (`objectType`, `contentType`, `recordId`, discriminator if needed)
- Explicit precondition
- Immutable envelope fields respected

### Preconditions

```json
{ "kind": "absent" }
```

```json
{ "kind": "revision", "revision": 3 }
```

| Op | Precondition |
|---|---|
| `add` | `absent` |
| `patch` / `replace` / `remove` | Exact `revision` |
| `upsert` | `absent` **or** exact `revision` |

Explain:

- Blind overwrite is **forbidden**.
- Successful create produces **revision 1**.
- Successful update increments revision **once**.
- Stale write performs **no mutation** (conflict).
- Failure atomicity: a rejected write leaves the prior revision unchanged.

### Immutable / ownership reminders

- Do not patch `format`, subject identity, or discriminator in place.
- Lifecycle and server timestamps follow ownership rules (chapter 13).
- Directory identity conflicts stop the write — `recordId` already bound to a different object type/category.

### Example — add then patch WI-100

```ts
// Create snapshot (absent)
await writeContent({
 op: "add",
 objectType: "work-items",
 contentType: "snapshots",
 recordId: "wi-100",
 precondition: { kind: "absent" },
 record: {
 format: "memorix-record/2",
 recordId: "wi-100",
 objectType: "work-items",
 contentType: "snapshots",
 dataCategory: "entity",
 concept: { title: "Prepare quarterly operations review" },
 data: { status: "in-progress", priority: "normal" },
 },
});

// Later: patch with exact revision from prior read
await writeContent({
 op: "patch",
 objectType: "work-items",
 contentType: "snapshots",
 recordId: "wi-100",
 precondition: { kind: "revision", revision: 1 },
 patch: [{ op: "set", path: "data.status", value: "blocked" }],
});
```

Wire the same intent through `/api/data` per OpenAPI — never via Mongo `updateOne`.

### Retry behavior (preview)

Network unknown after write → re-read before sending another write. Stale revision → reconcile; do not convert to blind upsert. See chapters 19–20.

---

## 10. Patch language

Patches are structured operations with path rules — not free-form JSON Merge Patch against the whole envelope unless the contract says so.

### Typical operations

| Op | Use |
|---|---|
| `set` | Set a value at an allowed path |
| `unset` / remove | Clear an allowed path where supported |

### Path rules (client view)

- Prefer paths under `data`, and allowed `concept` / `_system` subsets per contract.
- Envelope fields (`format`, `recordId`, `objectType`, `contentType`, …) are immutable via patch.
- Discriminator values are immutable via patch/replace.
- Lifecycle modifications follow metadata rules — illegal transitions fail.
- Concept / data / system ownership is contract-defined; do not invent cross-bucket ownership.
- Conflicting ops in one request fail validation.
- Invalid or unsafe paths fail before mutation.
- Validation runs **after** patch application against V2 + profiles.

### Valid vs invalid (illustrative)

**Valid**

- Set `data.priority` on WI-100 snapshot.
- Unset an optional `data` field allowed by schema.
- Set `concept.title` when the profile allows title edits.

**Invalid**

- Set `revision` or `format`.
- Set discriminator in place.
- Set root `associatedEmployees` or any `associated*`.
- Patch Mongo `_id`.
- Invent `/../../` or injection-shaped paths.
- Conflicting `set` + `unset` on the same path in one request.

### After patch

Re-read if your client needs the server-normalized record (defaults, timestamps, omitted empty buckets). Do not assume the patch document equals the stored document byte-for-byte.

---

## 11. Discriminated content writes

Multi-instance content is addressed by:

```ts
{
 recordId: string;
 contentType: string;
 discriminatorPath: string; // from content profile metadata
 discriminatorValue: string; // e.g. "capacity"
}
```

### Rules

1. Discriminator metadata comes from the **content profile**.
2. Default (if any) is applied **before** addressing and is **persisted**.
3. Missing required discriminator fails **before** reservation.
4. Explicit default and omitted-with-default identify the **same** instance.
5. Discriminator **cannot** be changed with patch/replace — remove then add.
6. Each discriminator instance has an **independent** revision.
7. Internal storage key is **not** a client identifier.

### OpX examples

| Case | Behavior |
|---|---|
| Two analyses on `wi-100` (`capacity`, `delivery-risk`) | Two instances, two revisions |
| Different decision types | Separate instances under `decisions` |
| Duplicate discriminator add | Conflict / duplicate — read existing |
| Missing required discriminator | Fail before write |
| Patch that changes discriminator | Immutable failure |

```bash
curl -sS \
 -H "x-memorix-org-id: $ORG" \
 -H "x-memorix-agent-ids: $AGENTS" \
 "$BASE/api/data/content?objectType=work-items&contentType=analysis&id=wi-100&discriminator=capacity"
```

```ts
await writeContent({
 op: "add",
 objectType: "work-items",
 contentType: "analysis",
 recordId: "wi-100",
 discriminatorValue: "capacity",
 precondition: { kind: "absent" },
 record: {
 format: "memorix-record/2",
 recordId: "wi-100",
 objectType: "work-items",
 contentType: "analysis",
 dataCategory: "entity",
 data: { kind: "capacity", score: 0.72 },
 },
});
```

---

## 12. Abstract writes

Supported flow: **`patchAbstract`** (name per OpenAPI / data plane).

1. Abstract type (e.g. `People`)
2. Concrete `recordId` (Alex)
3. Expected **concrete** revision
4. Abstract-shaped changes
5. Binding resolution → generated concrete update plan
6. **One** concrete mutation

### Failures (fail closed)

| Failure | Meaning |
|---|---|
| Concrete record not a union member | Alex’s type not in People |
| Unknown abstract property | No binding |
| Non-reversible binding | Cannot map write safely |
| Multi-target binding | Would touch more than one concrete path |
| Conflicting reverse paths | Ambiguous plan |
| Stale concrete revision | Refetch and retry |
| Invalid generated path | Plan rejected |

Abstract writes **never** create abstract records. They only reverse-map onto concrete content.

### OpX example — patch People.title

```ts
await patchAbstract({
 abstractType: "People",
 recordId: "employee-10", // Alex
 expectedRevision: 4,
 changes: [{ op: "set", path: "title", value: "Operations lead" }],
});
// → one concrete employees snapshot patch
```

---

## 13. Lifecycle and timestamps

### Client responsibilities

| Concern | Guidance |
|---|---|
| `state` / lifecycle entries | Follow metadata lifecycle rules; do not invent illegal transitions |
| `createdAt` / `modifiedAt` | Prefer server ownership; do not fake them to “win” sorts |
| `observedAt` | May come from the external source when modeling observation time |
| Provenance | Supply when the content type requires it (e.g. KnowX); never put secrets in provenance |
| Revision vs lifecycle | Different knobs — revision is concurrency; lifecycle is process history |

Operational **run history** does not belong on subject records. Pipeline/Memory ledgers own execution evidence.

### May / must-not

- **You may** send source-observed timestamps when the contract allows.
- **You must not** stamp `modifiedAt` client-side to bypass optimistic concurrency.
- **You must not** embed pipeline attempt logs inside `wi-100` snapshot `.data` as a substitute for ops ledgers.

### OpX example

PROC-2 knowledge snapshot transitions through an allowed lifecycle entry when a procedure revision is published. The content instance revision increments once for the successful write; lifecycle records the process event separately.

---

## 14. Raw data clients and Memory

Preferred source-ingestion path:

1. Connector client uses **Memory** APIs.
2. Every accepted source item lands as **raw** content.
3. Original payload remains under `.data`.
4. Source identity / routing comes from metadata.
5. Context vs knowledge differ by retention.
6. Identical knowledge payloads may deduplicate.
7. Context can expire / discard per policy.
8. Malformed items go to **quarantine**.
9. Cursor / checkpoint behavior is Memory-owned.
10. Connector is **read-only** toward the external source.

### Secrets

Memory APIs accept **`credentialRef` only** — never raw secrets in request bodies or landed payloads.

### Direct raw writes

If exposed at all, they must obey the same writer/precondition rules and must **not** bypass Memory source provenance for connector-shaped ingestion.

### OpX example

Pull employees from localhost `opx-source` into org `sandboxabc123`. Alex’s provider JSON lands under raw `.data`. A later pipeline promotes a snapshot. Do not `insertOne` into a guessed `employees_raw` collection.

```bash
# Memory pull / quarantine paths — see OpenAPI /api/memory
curl -sS \
 -H "x-memorix-org-id: $ORG" \
 -H "x-memorix-agent-ids: $AGENTS" \
 "$BASE/api/memory/..."
```

---

## 15. Relationship mutation boundary

Ordinary clients **cannot** manually create arbitrary relationship links.

| Allowed | Forbidden |
|---|---|
| Trigger authorized discover / verify / rebuild when exposed | Product `PUT`/`DELETE` of individual links (`405` on ad-hoc mutation) |
| Rebuild **one** relationship definition | Assume metadata publish auto-rebuilds all links |
| Read / expand four-field links | Copied associated payloads |
| | Relationship sibling writes as authoritative storage |
| | Root `associated*` writes |

Materialization belongs to `@x12i/memorix-relationships` / pipelines — not to host apps inventing edges.

### Ops story

After Alex and WI-100 snapshots exist, a discover/materialize step creates a four-field `employee-owns-work-item` link. Updating Alex’s title does **not** rewrite the link. Rebuilding that definition refreshes links for that definition only. Metadata install alone does not magically rebuild.

```bash
# Verify / rebuild — not ad-hoc link CRUD
curl -sS \
 -H "x-memorix-org-id: $ORG" \
 -H "x-memorix-agent-ids: $AGENTS" \
 "$BASE/api/relationships/..."
```

---

## 16. Workflow clients

Workflow is **ordinary sibling content** on the subject.

- Read/write through the data API (and `/api/workflow` as a specialized view of the same rules).
- Exact revision required for updates.
- Explicit composition when the UI also needs snapshot/analysis.
- No separate “workflow database” semantics.
- Studio workflow surfaces are views over canonical content operations.

### OpX example — edit WI-100 workflow

1. GET workflow sibling for `wi-100` → note `revision`.
2. PATCH assignee / step with that revision.
3. On conflict, re-read, merge, retry.
4. Compose with snapshot when the page shows both status and assignment.

```ts
async function editWorkflow(recordId: string, nextAssignee: string) {
 const current = await getContent({
 objectType: "work-items",
 contentType: "workflow",
 recordId,
 });
 await writeContent({
 op: "patch",
 objectType: "work-items",
 contentType: "workflow",
 recordId,
 precondition: { kind: "revision", revision: current.revision },
 patch: [{ op: "set", path: "data.assignee", value: nextAssignee }],
 });
}
```

See also [Workflow & Studio](../../09-workflow-and-studio/developers/BOOK.md).

---

## 17. Operational clients

Operational artifacts are **not** subject records.

Clients may:

- Start pipeline runs; inspect runs and step attempts
- Retry failed steps; resume runs
- Retrieve DataRef / artifacts
- Inspect captured run context
- Inspect Memory pulls and quarantine
- Inspect relationship verify/rebuild reports
- Inspect intelligence **proposals** (never auto-install)

Route families: `/api/pipelines` · `/api/memory` · `/api/relationships` · `/api/intelligence` · `/api/operations` · `/api/data-journey`.

### Data journey (evidence / Explore)

Read-only family for evidence cones and expected-vs-filled coverage (views as **slots only** — descriptors never invent available cells):

| Method | Path | Use |
|---|---|---|
| `GET` | `/api/data-journey/object-types` | Selector from effective metadata |
| `GET` | `/api/data-journey/journeys` | Real runs (+ fixture option when exposed) |
| `POST` | `/api/data-journey/matrix/query` | Sparse DB/SVC matrix from evidence |
| `POST` | `/api/data-journey/coverage` | Expected (views) − filled (evidence) |
| `GET` | `/api/data-journey/expected-slots` | Descriptor slots only |
| `GET` | `/api/data-journey/evidence/:ref` | Cell drill-down (values opt-in) |

Full Memory → pipeline → client-read narrative: [docs/runtime-data-journey.md](../../../docs/runtime-data-journey.md) (Act V). Package notes: `@x12i/memorix-data-journey` — remote apps call HTTP only.

### Partial failure

Do not blindly re-run an entire pipeline when one step failed — use the retry/resume operations for the failed step/run.

### Intelligence

Proposals are suggestions. Humans (or explicit install APIs) install metadata. Agents must not treat a proposal as already-published truth.

### OpX example

1. Memory pull lands Alex raw from opx-source.
2. Pipeline run promotes snapshot; one step fails validation.
3. Retry **that** step; retrieve artifact for the failed attempt.
4. Read the promoted (or still-raw) content via `/api/data` — do not confuse the artifact with the subject record.
5. Optionally query `/api/data-journey/coverage` for the same org/agents to see expected slots vs filled evidence.

---

## 18. Error catalog (client-facing)

For each error, clients should know: code, HTTP status, whether retryable, whether to refetch, whether the same body may be repeated.

| Situation | Typical status / code class | Client action |
|---|---|---|
| Invalid scope / singular `agentId` | `400` | Fix headers; do not retry unchanged |
| Unknown object/content type | `400` / validation | Rediscover metadata |
| Missing content | `404` / `READ_NOT_FOUND` | Create with `absent` or show empty UI |
| Directory identity conflict | conflict / validation | Stop — `recordId` already bound differently |
| Missing discriminator | validation | Supply required discriminator |
| Immutable discriminator | validation | Remove then add; do not patch |
| Duplicate add | conflict | Read existing content |
| Stale revision | `409` | Refetch, reconcile, retry with new revision |
| Absent-precondition conflict | conflict | Content already exists — switch to revision write |
| Immutable envelope | validation | Remove illegal paths from patch |
| Invalid lifecycle | validation | Follow allowed transitions |
| Validation failure | `400` | Fix body against schema/profile |
| Unsafe query / injection-shaped filter | `400` | Rewrite with allowlisted grammar |
| Cursor mismatch | `400` | Restart list from page 1 |
| Unsupported abstract pushdown | `400` | Narrow query or query concrete types |
| Abstract non-reversible / non-member | `400` | Fix abstract write inputs |
| Relationship query unbounded | `400` | Add bounds / change traversal |
| Unresolved target | readable state | Show gap; optional later retry |
| Missing canonical runtime port | platform | Misconfig — not a client retry loop |
| Artifact not found / wrong org | `404` | Fix org or artifact id |
| Pipeline partial failure | run status | Retry failed step/run op |
| Malformed Memory item | quarantine | Inspect quarantine; fix source mapping |
| Durable route `503` in `dev` without Mongo | `503` | Use `prod` + `MONGO_URI` or accept in-memory limits |

Studio and HTTP both surface these classes; decode the structured error body when present.

### Example — stale write

```text
PATCH wi-100 snapshots expectedRevision=3
→ 409 WRITE_STALE_REVISION (current is 4)
→ GET again → merge → PATCH expectedRevision=4
```

---

## 19. Retry and concurrency guidance

| Failure | Retry unchanged? | Required client action |
|---|---|---|
| Network failure before response | Maybe | Read current state / idempotency result before rewriting |
| Stale revision | **No** | Refetch, reconcile, retry with new revision |
| Duplicate add | **No** | Read existing content |
| Temporary service failure | Yes where operation is idempotent | Preserve request id / precondition |
| Cursor mismatch | **No** | Restart query |
| Partial pipeline failure | No full blind rerun | Retry failed step/run operation |
| Unresolved relationship target | Readable state | Decide whether to retry later |

**Do not** “fix” stale conflicts by converting exact-revision writes into blind upserts.

### Safe concurrent edit recipe (Alex title)

1. GET employee snapshot → `revision = N`.
2. PATCH with `expectedRevision: N`.
3. On `409` stale: GET again → merge fields → PATCH with new revision.
4. Never loop the original patch body forever.

### Bounded backoff

For transient `503` / network blips on **idempotent** reads or carefully preconditioned writes: exponential backoff with a hard cap. For non-idempotent uncertainty: always re-read first.

---

## 20. Idempotency

| Mechanism | Role |
|---|---|
| Deterministic record/content addressing | Same address → same instance |
| `absent` / revision preconditions | Make create/update safe under retry *when* preconditions still hold |
| Relationship deterministic IDs | Rebuilds converge |
| Memory content hashes | Knowledge dedupe / integrity |
| Pipeline at-least-once | Steps must be safe or checkpointed |
| Request correlation / idempotency keys | Use when the API exposes them |

### Which ops are safe to repeat?

- **Safe-ish:** reads; verify; get artifact by id; Memory pull with durable cursors (per contract).
- **Require reconciliation:** any write after unknown network outcome; stale conflicts; duplicate adds.

Prefer idempotency keys for side-effecting pipeline steps derived from `(scope, subject, step, inputRevision)` when the service exposes them.

---

## 21. Compatibility and versioning

| Knob | Client expectation |
|---|---|
| Record format | `memorix-record/2` |
| HTTP API | Versioned via OpenAPI on the running service |
| SDKs | Generate from **accepted** OpenAPI; hand conveniences must still pass contract tests |
| Request content types | As declared in OpenAPI |
| Deprecated routes | Documented; do not build new clients on them |
| Lightweight `associated*` | Migration/compat only — production writers reject new root associated writes |
| Present / V1 | Migration input, not target truth |
| Unknown fields | Envelope rejects unknown roots; follow schema policy inside `.data` |
| Feature discovery | Prefer OpenAPI + metadata capabilities over guessing |

Breaking changes appear as OpenAPI / gate changes — detect contradictions against accepted implementation rather than documenting guesses.

---

## 22. Client anti-patterns

Explicitly avoid:

1. Direct Mongo access for domain reads/writes 
2. Using Mongo `_id` as subject identity 
3. Constructing collection names from type strings 
4. Storing expansion results as authoritative records 
5. Copying related payloads onto sources 
6. Writing root `associated*` 
7. Writing relationship sibling content as SoT 
8. Creating abstract records / abstract collections 
9. Flattening composition incorrectly 
10. Assuming one revision per subject 
11. Blind writes without preconditions 
12. Changing discriminator in place 
13. Treating `agentIds` as data authorization filters 
14. Placing credentials in data or provenance 
15. Retrying stale writes unchanged 
16. Depending on obsolete Explorer / Hippox / associator APIs 
17. Using `fieldstone` / `neo` orgs as acceptance fixtures 
18. Letting intelligence proposals auto-install 
19. Decoding or forging opaque cursors 
20. Confusing operational ledgers with subject content 

---

## 23. Worked examples (ops / `sandbox*`)

All examples assume:

```bash
ORG=sandboxabc123
AGENTS=ops
BASE=http://127.0.0.1:5100
HDR=(-H "x-memorix-org-id: $ORG" -H "x-memorix-agent-ids: $AGENTS")
```

### 1. Configure a client

See chapter 2 — `baseUrl` + `orgId` + `agentIds: ["opx"]`.

### 2. Discover effective metadata

```bash
curl -sS "${HDR[@]}" "$BASE/api/metadata/kinds" | head
```

### 3. Read a snapshot (WI-100)

```bash
curl -sS "${HDR[@]}" \
 "$BASE/api/data/content?objectType=work-items&contentType=snapshots&id=wi-100"
```

### 4. Read discriminated analysis

```bash
curl -sS "${HDR[@]}" \
 "$BASE/api/data/content?objectType=work-items&contentType=analysis&id=wi-100&discriminator=capacity"
```

### 5. List / filter / sort / page

```bash
curl -sS "${HDR[@]}" \
 "$BASE/api/data/content?objectType=work-items&contentType=snapshots&limit=50"
# then: &cursor="$NEXT" until pageInfo.nextCursor is empty
```

### 6. Compose a subject

```bash
curl -sS -X POST "${HDR[@]}" \
 -H "content-type: application/json" \
 -d '{"recordId":"wi-100","explicit":true,"contentTypes":["snapshots","workflow","analysis","decisions"]}' \
 "$BASE/api/data/compose"
```

### 7. List People

```bash
curl -sS "${HDR[@]}" \
 "$BASE/api/data/abstract?abstractType=People&limit=50"
```

### 8. Patch People.title (Alex)

Use abstract reverse-write with concrete revision from GET (chapter 12).

### 9. Add with `absent`

Create PROC-2 knowledge/snapshot content only if missing — precondition `absent` → revision `1`.

### 10. Patch with exact revision

Read WI-100 snapshot → patch `data.status` with that revision.

### 11. Handle stale revision

Two editors: first succeeds (`1→2`); second with `expectedRevision: 1` gets conflict → re-read → retry at `2`.

### 12. Read relationships

Query links for Alex / WI-100 via `/api/relationships` or content get with include.

### 13. Expand a target

Request expansion; render current WI-100 snapshot from the virtual wrapper — do not save it onto Alex.

### 14. Handle unresolved target

If `targetStatus: "unresolved"`, show a gap in the UI; optionally retry later after materialization.

### 15. Run Memory pull

Start/inspect Memory pull with `credentialRef` only; inspect quarantine on malformed items.

### 16. Run / retry pipeline

Start a run; on partial failure retry the failed step — not a blind full rerun.

### 17. Retrieve artifact

GET artifact by id for the **same** org; treat as operational evidence, not subject content.

### 18. Edit workflow

PATCH `wi-100` workflow sibling with exact revision (chapter 16).

### 19. Review intelligence proposal

List proposals; display diff; **do not** auto-install — human install path only.

```ts
async function memorixFetch(path: string, init: RequestInit = {}) {
 const res = await fetch(new URL(path, client.baseUrl), {
 ...init,
 headers: { ...scopeHeaders(client), ...(init.headers || {}) },
 });
 if (!res.ok) {
 const body = await res.text();
 throw Object.assign(new Error(body), { status: res.status });
 }
 return res.status === 204 ? null : res.json();
}
```

Exact operation IDs and bodies: running OpenAPI + package contract tests. Examples in CI should call the accepted service, not a mocked Mongo.

---

## 24. Client conformance notes

Any official client or SDK should prove:

- [ ] Scope serialization (`orgId` + plural `agentIds`)
- [ ] Rejection path for singular `agentId`
- [ ] `memorix-record/2` format-version handling
- [ ] Exact and discriminated addressing
- [ ] Revision preconditions on writes
- [ ] Stale conflict behavior (no mutation)
- [ ] Cursor opacity (no client decode)
- [ ] Composition shape (arrays per content type)
- [ ] Abstract union identity (concrete types retained)
- [ ] Relationship virtual wrapper semantics
- [ ] Unresolved relationship handling
- [ ] Structured error decoding
- [ ] Artifact retrieval scoped to org
- [ ] No secret logging
- [ ] No direct database assumptions

Generate SDKs only from the **accepted** OpenAPI contract. Hand-written conveniences must still pass the generated-contract and service integration suites.

### Source-of-truth rule

For client-facing claims, prefer this evidence chain:

1. OpenAPI path / operation ID 
2. Request/response schema component 
3. Service handler 
4. Canonical `@x12i/memorix-data` (or sibling package) call 
5. Contract / integration test 
6. Accepted gate 

If documentation conflicts with OpenAPI or accepted implementation, **list the contradiction and stop** — do not document guessed behavior.

OpenAPI: `GET /api/openapi.json` is the contract of record for operation IDs and schemas on a given deployment.

---

## 25. Quick decision tree (access method)

```text
Need one instance?
└─ GET /api/data/content (exact address)

Need siblings of one subject?
└─ POST /api/data/compose (explicit:true)

Need evidence / expected-vs-filled coverage?
└─ /api/data-journey/* (read-only; not subject content)

Need union across types?
└─ Abstract get/list (virtual)

Need neighbors?
└─ Relationships query / expand (four-field + current targets)

Need to change a field?
└─ patch / upsert with precondition — never blind overwrite

Need to land vendor JSON?
└─ Memory pull (+ credentialRef) — not ad-hoc raw inserts

Need to change metadata?
└─ Install/publish path — intelligence proposes only

Remote host?
└─ memorix-service HTTP only

UI?
└─ Studio (sole product UI) over the same APIs
```

```text
Need what a record means?
 → Format for Dummies
Need list/cursor/bulk depth?
 → Lists, Queries, Pagination
Need “which call for this product job?”
 → Data Access Patterns
Need links or source landing?
 → Relationships & Memory
Need ordered multi-step work?
 → Pipelines / Services
```

Task-oriented recipes continue in [Access patterns](../../04-data-access-patterns/developers/BOOK.md).

---

## 26. Glossary (client lens)

| Term | Client meaning |
|---|---|
| **Scope** | `orgId` + `agentIds[]` on every call |
| **Content address** | objectType + contentType + recordId (+ discriminator) |
| **Precondition** | `absent` or exact `revision` |
| **Compose** | Virtual multi-sibling read for one `recordId` |
| **Abstract** | Virtual union/projection — no storage collection |
| **Expansion** | Virtual current targets for four-field links |
| **Data plane** | Service `/api/data` + `@x12i/memorix-data` writer/reader |
| **Operational** | Runs, pulls, artifacts — not domain siblings |
| **credentialRef** | Only allowed secret handle on Memory APIs |

Full format glossary: [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md). Shared terms: [_shared/glossary.md](../../_shared/glossary.md).

---

## May / must-not (summary)

| May | Must not |
|---|---|
| Call `memorix-service` HTTP | Write domain data with Mongo drivers |
| Use `@x12i/memorix-data` in trusted process | Use singular `agentId` |
| Trigger verify/rebuild | Ad-hoc link PUT/DELETE |
| Cache metadata briefly | Hardcode abstract collections |
| Compose / expand explicitly | Persist expansions as SoT |
| Patch with exact revision | Blind overwrite / stale retry unchanged |
| Use `credentialRef` | Put secrets in bodies or provenance |
| Accept on fresh `sandbox*` + `opx` | Use `fieldstone` / `neo` as acceptance |

---

## Next reading

- [Lists, queries, pagination](../../03-lists-queries-pagination/developers/BOOK.md) — cursors, limits, bulk habits 
- [Access patterns decision guide](../../04-data-access-patterns/developers/BOOK.md) — which API for which job 
- [Metadata, agents, abstracts](../../05-metadata-agents-abstracts/developers/BOOK.md) — effective metadata depth 
- [Relationships and Memory](../../06-relationships-and-memory/developers/BOOK.md) — links and ingestion 
- [Services](../../07-services/developers/BOOK.md) — composition root and Studio 
- [Migration & troubleshooting](../../13-migration-troubleshooting/developers/BOOK.md) — conflicts and recovery 

HTTP overview: [`documentation/http-api.md`](../../../documentation/http-api.md) · Hands-on: [`documentation/tutorials/02-first-read-write.md`](../../../documentation/tutorials/02-first-read-write.md).

---

## Writing rules this ebook follows

- Mental model and scope before endpoints.
- Ops journey throughout (`Alex`, `WI-100`, `PROC-2`; fresh `sandbox*` + `agentIds: ["opx"]`); never `fieldstone` / `neo` as acceptance.
- Stored / virtual / operational labeled clearly.
- Sole HTTP = `memorix-service`; sole product UI = Studio; no direct Mongo; four-field links; virtual abstracts.
- Authoring-brief companion JSON catalogs are **not** shipped — error tables, examples, and conformance live in this prose.
- “Why” beside important rules; OpenAPI remains path/schema authority.

*End of Memorix Runtime Data Client Guide (Developers).*

---

# Lists, Queries, Pagination — developers

# Memorix Runtime Lists, Queries, Pagination & Bulk Access

**Audience:** Developers whose clients retrieve or process **more than one** Memorix resource — content pages, abstract unions, relationship traversals, metadata catalogs, Memory / pipeline / intelligence operational lists, batch reads, and exports. 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Complements:** [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md) · [Runtime Data Client](../../02-runtime-data-client/developers/BOOK.md) · [Access patterns](../../04-data-access-patterns/developers/BOOK.md) 
**Status:** Present-tense product documentation for the frozen list / query / pagination boundary.

---

## How to read this guide

1. Start with the **central mental model** — every later chapter is a specialization of “more than one thing, still through one service.”
2. Work chapters in order. Filters, cursors, and guarantees assume you already know the response envelope.
3. Labels used throughout:
 - **Stored** — persisted subject content or four-field relationship links
 - **Virtual / calculated** — composition, abstract projections, relationship expansion (computed at read time)
 - **Operational** — runs, attempts, quarantine, artifacts, rebuild diagnostics (not ordinary subject content)
4. Do not invent operators. If OpenAPI, route handlers, and `@x12i/memorix-data` disagree, treat the mismatch as a review issue — never “paper over” it in a client.
5. Mongo `_id`, storage keys, and collection names are **internal**. Clients address `(objectType, contentType, recordId)` or `abstractType` + `recordId`.

---

## The central mental model

A **list** in Memorix is any client-visible operation that returns or processes **more than one** resource through **`memorix-service` HTTP**.

```text
One service (memorix-service)
 ├─ Concrete content pages → records[] + pageInfo
 ├─ Abstract union pages → projected items + pageInfo (no abstract collection)
 ├─ Relationship pages → four-field links (+ optional virtual targets)
 ├─ Metadata catalogs → effective definitions for agentIds[]
 └─ Operational pages → runs / quarantine / artifacts (separate store)
```

Three laws never change:

1. **Sole service HTTP** for remote hosts. Prefer `GET /api/openapi.json` and scoped routes. Use `@x12i/memorix-data` only inside the trusted platform process. **Never** write domain content with raw Mongo drivers.
2. **Relationships are four-field links.** Expansion loads *current* targets as a virtual section — it does not copy payloads onto the source and does not invent root `associated*` storage.
3. **Abstracts are virtual.** `People` is employees ∪ contractors via metadata bindings. There is no People collection and no abstract `recordId` namespace.

Ops running example used throughout:

| Subject | `recordId` (typical) | Role |
|---|---|---|
| Employee **Alex** | `employee-10` | Snapshot / workflow siblings; People member |
| Work item **WI-100** | `wi-100` | Snapshots, workflow, analyses; relationships to Alex / PROC-2 |
| Procedure **PROC-2** | procedure subject id | May revise while lists are in flight |

Scope on every call:

```http
x-memorix-org-id: sandbox<suffix>
x-memorix-agent-ids: opx
```

Singular `agentId` headers are rejected. Never use live or legacy customer orgs (`fieldstone`, `neo`, …) as acceptance.

---

## 1. What “list” means here

Any of the following is a list concern, even when the route name does not say “list”:

| Family | Examples |
|---|---|
| **Data** | Concrete content pages; abstract union pages; composed sibling arrays; discriminated instance sets |
| **Relationships** | Forward / inverse / by-definition / for-record; related-property results; expansion pages; verify/rebuild diagnostics |
| **Metadata** | Agents, object/content/abstract types, bindings, relationship definitions, mappings, views, sources, pipelines, schedules |
| **Memory** | Sources; pull runs; checkpoints; quarantine pages |
| **Pipelines / operations** | Definitions; runs; steps; attempts; partial-batch results; artifacts / DataRefs |
| **Intelligence** | Profiles; proposals; comparisons; recorded adapter artifacts |

A **get** is not a list: `id` / `recordId` switches content and abstract routes from page mode to single-item mode. An **empty page** is a successful list (`records: []`, `nextCursor: null`). A **missing single resource** is a not-found error — different contract.

---

## 2. Endpoint inventory (client-visible)

Discover live shapes from OpenAPI. The inventory below is the product map clients must plan against.

### 2.1 Metadata lists

| Route (typical) | Returns | Pagination |
|---|---|---|
| `/api/metadata/...` kinds / packs / effective definitions | Agents, raw vs effective metadata, object types, content types, abstract types, bindings, relationship definitions, mappings, views, sources, services, pipelines, schedules | Usually full catalogs or filtered by kind — not content cursors |
| Validation / inheritance diagnostics | Conflicts, provenance paths | Diagnostic envelopes |

Metadata order is deterministic by kind / id. Multi-agent resolution uses `agentIds[]`. A conflict is an **error or diagnostic**, never a silent “last agent wins.”

### 2.2 Data lists

| Operation ID (OpenAPI) | Route | Unit returned |
|---|---|---|
| `dataGetContent` | `GET /api/data/content` | Without `id`: page of concrete V2 records. With `id`: one record |
| `dataGetAbstract` | `GET /api/data/abstract` | Without `id`: abstract projections. With `id`: one projection |
| Compose | `POST /api/data/compose` | One subject’s sibling content as **arrays by content type** (`explicit:true`) |

Concrete addressing always needs `objectType` + `contentType`. Abstract addressing needs `abstractType`. Writes are separate (`POST /api/data/content`, `POST /api/data/abstract/patch`) and are **not** “list writes.”

### 2.3 Relationship lists

| Route | Returns |
|---|---|
| `GET /api/relationships/definitions` | Relationship definitions + compiled plans |
| `GET /api/relationships/query` | Links for a `recordId` (forward / inverse), optional expand |
| `GET /api/relationships/related-property` | Bounded two-phase related-property matches |
| `POST .../discover|verify|rebuild|cleanup` | Operational run-style results (not domain pages) |
| `/api/relationships/links` | **405** — ad-hoc link mutation is not exposed |

### 2.4 Memory / pipeline / intelligence / workflow / data-journey

| Family | List-like surfaces |
|---|---|
| Memory | `GET /api/memory/sources`; `GET /api/memory/runs` (runs / quarantine / checkpoints); pull results with page counters |
| Pipelines | Definitions; run get; retry / resume; artifact / DataRef references |
| Intelligence | Profiles; proposal id lists; comparisons — never auto-install |
| Workflow | Workflow sibling content (still concrete content addressing) |
| Data journey | `GET /api/data-journey/object-types`; `GET /api/data-journey/journeys`; matrix/coverage POSTs; evidence by ref — **read-only**, not domain pages |

Operational list rows are **not** domain subject records. Do not treat a pull run id as a `recordId` for content APIs. Data-journey descriptors never invent available cells — views appear as expected **slots** only.

### 2.5 Per-operation fields to expect

For every list operation, clients should know:

- route + operation ID 
- resource returned 
- scope requirements (`orgId` + `agentIds[]`) 
- allowed filters / sorts (if any) 
- default sort + pagination method 
- default / max page size 
- response envelope (`records` vs `items` vs `links`) 
- whether totals exist 
- expansion / projection options 
- error codes and performance bounds 

If two endpoints differ without a deliberate reason, record it — do not invent a fake common envelope in application code.

---

## 3. Common list response / `pageInfo` contract

### 3.1 Concrete content page (HTTP)

```http
GET /api/data/content?objectType=work-items&contentType=snapshots&limit=50&cursor=<opaque>
x-memorix-org-id: sandboxabc123
x-memorix-agent-ids: opx
```

```json
{
 "scope": { "orgId": "sandboxabc123", "agentIds": ["opx"] },
 "addressing": {
 "orgId": "sandboxabc123",
 "objectType": "work-items",
 "contentType": "snapshots",
 "id": null
 },
 "records": [ /* memorix-record/2 … */ ],
 "pageInfo": { "nextCursor": "mrxq1...." },
 "unresolvedRelationships": [],
 "currentRevision": null
}
```

### 3.2 Abstract page (HTTP)

```json
{
 "scope": { "orgId": "sandboxabc123", "agentIds": ["opx"] },
 "abstractType": "People",
 "records": [
 {
 "abstractType": "People",
 "recordId": "employee-10",
 "objectType": "employees",
 "dataCategory": "entity",
 "contentType": "snapshots",
 "properties": { }
 }
 ],
 "pageInfo": { "nextCursor": null }
}
```

### 3.3 Package-level result

Inside `@x12i/memorix-data`, list results use `items` + optional `nextCursor`. HTTP maps that to `records` + `pageInfo.nextCursor` (null when exhausted). Relationship sections expose `items` + `pageInfo: { limit, nextCursor? }`.

### 3.4 Clarifications

| Situation | Contract |
|---|---|
| Empty list | HTTP 200, empty array, `nextCursor` null/omitted |
| Missing single get | Not-found class error — not an empty list |
| Totals | **May be absent** even when pagination works — do not require `totalCount` |
| Relationship pagination | Independent from source-content pagination |
| Expanded targets | Virtual; never part of stored source |
| Diagnostics | Related-property and some operational lists may include plan diagnostics |

`PageInfo` (OpenAPI): `nextCursor` (string \| null) and optionally `limit`. Do not assume `hasMore`, query-plan dumps, or correlation ids unless the accepted schema includes them.

---

## 4. Exact query grammar

Public filters are **SafeFilter** objects — not Mongo operators.

### 4.1 Supported operators

| `op` | Value shape | Meaning |
|---|---|---|
| `eq` | scalar / comparable | Equality |
| `ne` | scalar / comparable | Inequality |
| `gt` / `gte` / `lt` / `lte` | comparable | Ordered comparison |
| `in` | **array** | Membership |
| `nin` | **array** | Non-membership |
| `exists` | **boolean** | Property presence in storage sense |

Multiple filters combine with **AND**. There is no public nested OR tree, negation wrapper, regex, full-text, fuzzy, or vector operator in the safe grammar.

```json
[
 { "field": "data.status", "op": "eq", "value": "in-progress" },
 { "field": "data.priority", "op": "in", "value": ["normal", "high"] }
]
```

### 4.2 Sort

```json
[{ "field": "recordId", "direction": "asc" }]
```

`direction` is only `asc` | `desc`. Multi-field sort is supported in the package grammar. The implementation always adds an **internal** stable tie-breaker (storage key / `_id`) — clients must not send or depend on Mongo `_id` as a public sort field.

### 4.3 Type safety (no silent casting)

Prefer the safe rule:

- `"10"` is **not** automatically the number `10`
- ISO date strings are not silently reinterpreted unless the field’s schema / operator says so
- Invalid mixed-type comparisons fail clearly (`READ_QUERY_INVALID` or validation)

### 4.4 Unsupported (do not document as features)

- Raw Mongo `$where`, `$regex`, `$expr`, aggregation pipelines 
- Client-supplied `$` operators or field names 
- Unbounded collection scans presented as “search” 
- Implicit OR across unrelated object types without an abstract plan 

Invalid / injected fields → `READ_QUERY_INJECTION`. Unknown/disallowed paths → `READ_QUERY_FIELD_FORBIDDEN`. Bad shapes → `READ_QUERY_INVALID`.

### 4.5 HTTP vs package surface

OpenAPI documents optional `filter` / `sort` query strings (SafeFilter / SafeSort JSON) on content lists. The canonical compilation lives in `@x12i/memorix-data` (`compileFiltersToMongo` / `compileSortToMongo`). Always verify the **running** OpenAPI + handler wiring for your service build; do not invent a second grammar in the client.

---

## 5. Field-path rules

### 5.1 What clients may query

Allowed paths come from **effective metadata** (`getAllowedQueryPaths` for concrete OT/CT; abstract field allow-lists for unions):

| Path family | Typical use |
|---|---|
| Envelope | `recordId`, `revision`, `objectType`, `contentType`, `dataCategory` (when allowed) |
| `concept.*` | Titles / identifiers clients are allowed to filter |
| `data.*` | Payload fields (e.g. `data.status` on WI-100 snapshots) |
| Selected `_system.*` | Only when metadata explicitly allows |
| Discriminator paths | Multi-instance content — public discriminator **value**, never internal storage key |
| Abstract properties | Names from the abstract type, mapped via bindings |
| Operational fields | Run/status/time fields on operational endpoints only |

### 5.2 Syntax and safety

- Dot paths: `data.status`, `concept.title`
- Nested objects: walk with dots; arrays are not a free-form “any element” query language unless an allowed path says so
- Rejected: empty field, leading/trailing `.`, `..`, NUL, `$`, field `_id`, any `$…` injection
- Unknown paths: forbidden unless a `data.*`-style wildcard is configured in metadata
- Prototype-pollution style paths and Mongo operator smuggling are injection errors

**Clients never query internal Mongo fields** unless an accepted public field represents the same concept.

---

## 6. Missing, null, empty, and absent

Filters must not silently treat these as one condition.

| State | Meaning |
|---|---|
| Property **absent** | Path does not exist on the document |
| Property **null** | Path exists with JSON `null` |
| Empty string / object / array | Present values — still “exists” for `exists:true` |
| Content instance missing | No row for that address — not a filter match on another sibling |
| Subject reserved but empty | Directory may know the subject while some content types are missing |
| Unresolved relationship target | Link exists; target content cannot be loaded — `targetStatus: "unresolved"` |

Operator guidance:

| Operator | Absent field | Explicit `null` |
|---|---|---|
| `eq` / comparisons | Typically no match | Matches only if compared to `null` (do not rely on DB-specific null quirks) |
| `exists: true` | No match | Match |
| `exists: false` | Match | No match (property present) |
| `in` / `nin` | No match for `in` unless values include the stored value | As equality against array members |

When writing filters for Alex’s snapshot (`data.role`) or WI-100 (`data.status`), decide whether you mean “field missing,” “field null,” or “field equal to empty string” — encode that explicitly.

---

## 7. Sorting

### 7.1 Rules

- Sort only on **allowed** paths 
- `asc` / `desc` only 
- Default sort is server-defined per collection when the client omits `sort` — **do not assume insertion order** 
- Multi-field sort: supported in the package grammar; each paginated sort needs a deterministic tie-breaker 
- Tie-breaker may be encoded **inside** the opaque cursor; clients must not depend on internal storage keys 
- Null / missing ordering follows storage engine order for that path — do not invent a client-side nulls-first policy 
- String collation is UTF-8 / server default unless metadata documents otherwise 
- Numeric and ISO-timestamp fields sort as stored types — no silent string/number coercion 

### 7.2 Abstract and related sorts

- Abstract union sorts must be pushdown-safe on bound properties or be rejected (`READ_QUERY_NOT_PUSHDOWN`)
- Related-target-property sorts belong to the related-property plan — not to “sort the source list by expanded JSON”

---

## 8. Cursor contract

Cursors are opaque tokens. Product encoding uses a versioned prefix (e.g. `mrxq1.…`) wrapping a payload that includes:

- cursor version + kind (`concrete` | `abstract`) 
- **query shape hash** bound to filters + sort + addressing 
- continuation position (sort values + internal storage key) 

### 8.1 Public rules

| Rule | Why |
|---|---|
| Opaque | Clients never parse or mint cursors |
| Bound to query shape | Changing filters/sort/addressing → `READ_CURSOR_MISMATCH` |
| Bound to kind | Concrete cursor ≠ abstract cursor |
| No credentials | Cursor must not carry secrets or unrestricted payloads |
| Forward continuation | Resume with the **same** query + `nextCursor` |
| Exhausted | `nextCursor` null/omitted → done |

### 8.2 Error matrix

| Condition | Code (package) | Client action |
|---|---|---|
| Malformed / wrong prefix | `READ_CURSOR_INVALID` | Restart query from the beginning |
| Unsupported version | `READ_CURSOR_INVALID` | Restart; upgrade client expectations |
| Shape / kind mismatch | `READ_CURSOR_MISMATCH` | Restart with one consistent query |
| Cursor from another endpoint | mismatch / invalid | Do not reuse across routes |
| Referring to removed data | Continue or shorten pages | Treat as concurrent mutation (ch. 20) |

**When the query shape changes, restart.** Do not “fix” a cursor by editing it.

---

## 9. Pagination guarantees

### 9.1 What is guaranteed

- Forward cursor pagination with stable ordering **for a fixed sort + tie-breaker** under quiet data 
- Page size clamped to an accepted max (content lists: default **50**, max **200**) 
- Empty final page signal via null/omitted `nextCursor` 
- Org isolation: another org’s WI-100 never appears in your page 

### 9.2 What is **not** promised

| Question | Honest answer |
|---|---|
| Snapshot-isolated pages? | **No** — do not promise snapshot isolation unless a read token exists |
| Concurrent inserts? | May appear on later pages |
| Concurrent deletes? | May shorten later pages |
| Updates to sort fields? | May move an item across boundaries (rare duplicate/gap under change) |
| Reverse pagination? | Not a public contract |
| Unlimited cursor TTL? | Do not retain cursors indefinitely; restart if mismatch/expiry occurs |
| Metadata change mid-abstract-iteration? | Effective union may change with `agentIds[]` / pack updates — restart |

### 9.3 Safe client guidance

**Stable data:** paginate once; process; optionally re-list with a watermark filter (`_system.modifiedAt` / business field) if metadata allows.

**Concurrent data:** design handlers idempotent; prefer keys (`recordId` + `contentType` + discriminator) for side effects; for exports use an operational job (ch. 16) instead of a hours-long cursor walk.

---

## 10. Page-size behavior

| Input | Behavior |
|---|---|
| Omitted | Default **50** (content / relationship readers) |
| `1..200` | Honored |
| `< 1` | Clamped up to 1 in package readers |
| `> 200` | Clamped to **200** (content / related scans) |
| Unbounded / “all” | **Not supported** |

Resource-specific notes:

- Abstract HTTP list may use a fixed default page in some builds — always pass/observe `pageInfo` 
- Expanded-relationship pages use their own `limit` (default 50, max 200) 
- Related-property `relatedScanLimit` is a **hard bound**; exceeding it rejects (`READ_RELATIONSHIP_UNBOUNDED`) 
- Operational logs / artifacts may use smaller practical limits — check OpenAPI 

Clients must not request an unbounded page. Large exports are jobs, not `limit=999999`.

---

## 11. Concrete content lists

### 11.1 Unit returned

Each list item is one **content instance** — a full `memorix-record/2` for a fixed `(objectType, contentType)` (and discriminator when required).

Independent content instances have **independent revisions**. Listing `work-items` + `snapshots` does not return workflow siblings; compose or separate lists for that.

### 11.2 Typical ops queries

| Intent | How |
|---|---|
| All WI snapshots | `objectType=work-items&contentType=snapshots` |
| Raw landing only | `contentType=raw` |
| One analysis type | Discriminated content + discriminator filter/value |
| Decisions / workflow / context | Their content types |
| Modified after time | Filter on allowed `_system.modifiedAt` / business field if permitted |
| Lifecycle state | Filter on allowed lifecycle / `_system.state` paths |

```ts
// Conceptual package iteration — preserve exact query between pages
let cursor: string | undefined;
do {
 const page = await reader.listContent(scope, {
 objectType: "work-items",
 contentType: "snapshots",
 filters: [{ field: "data.status", op: "eq", value: "in-progress" }],
 sort: [{ field: "recordId", direction: "asc" }],
 limit: 50,
 cursor,
 });
 for (const item of page.items) {
 await handle(item); // e.g. WI-100 snapshot
 }
 cursor = page.nextCursor;
} while (cursor);
```

HTTP equivalent: honor `records` + `pageInfo.nextCursor` until null.

---

## 12. Subject and composition lists

### 12.1 Composition is not a silent list join

```http
POST /api/data/compose
```

```json
{
 "recordId": "wi-100",
 "explicit": true,
 "contentTypes": ["snapshots", "workflow", "raw"]
}
```

Rules:

- `explicit: true` is required 
- Response keeps **arrays per content type** (including singletons) 
- Missing siblings are absent arrays / warnings — not invented empties pretending to be writes 
- Composition does **not** mutate storage 
- There is no “compose every subject in the org” list API 

### 12.2 Batch composition

If a batch-compose endpoint is not in OpenAPI, **do not** invent unbounded N+1 compose loops in UI code. Prefer:

1. Paginate the driving content list (e.g. open work-items) 
2. Compose **on demand** for the visible subject 
3. Or run a pipeline that materializes a view/artifact 

---

## 13. Discriminated-instance lists

Multi-instance content types (analyses, decisions, discoveries) use a **public discriminator value**.

| Concern | Rule |
|---|---|
| List every instance of a CT | Concrete list for that `contentType` |
| Filter / sort by discriminator | Use the public field path from metadata |
| Duplicate discriminator | Writer / profile rules prevent illegal duplicates |
| Revisions | Each discriminated instance has its own revision 
| Singletons vs multi | Compose still returns **arrays**; get-by-id may require discriminator |

Never use internal storage keys as discriminator values in filters.

---

## 14. Abstract-union lists

### 14.1 People is not a collection

```text
People = employees ∪ contractors
 (metadata bindings + mappings)
```

- No People Mongo collection 
- No abstract `recordId` namespace — Alex remains `employee-10` with `objectType: "employees"` 
- Each result retains concrete identity 
- Member collections are queried independently and **merged deterministically** 
- Duplicate `recordId` values must not appear twice in one union page stream 
- `agentIds[]` may change the **effective** union; org data visibility stays org-scoped 

### 14.2 Pushdown

| Binding kind | Filter/sort |
|---|---|
| Direct property bindings | Prefer full pushdown |
| Mapping-derived / constants / templates / transforms | May be non-pushdown |
| Non-pushdown plan | Rejected as `READ_QUERY_NOT_PUSHDOWN` (or equivalent) — **not** a silent full scan |

For every abstract query shape, the platform classifies: fully pushed down · bounded post-process · rejected. Clients treat rejection as “refine the query,” not “retry harder.”

### 14.3 OpX example

List People, then open Alex’s concrete employee snapshot and contractor peers without ever writing a `People` row:

```http
GET /api/data/abstract?abstractType=People&limit=50
GET /api/data/content?objectType=employees&contentType=snapshots&id=employee-10
```

---

## 15. Relationship lists

### 15.1 Traversal forms

| Form | Meaning |
|---|---|
| Forward | From Alex → assigned work items |
| Inverse | Into WI-100 ← assignees |
| For-record | All links touching a record |
| By-definition | All links for one `relationshipDefinitionId` |
| Self / undirected | One physical link may appear from either traversal direction — not a duplicate store row |

### 15.2 Item shape (read model)

```ts
{
 relationshipId: string;
 relationshipDefinitionId: string;
 fromRecordId: string;
 toRecordId: string;
 relatedRecordId: string;
 traversal: "forward" | "inverse";
 path?: string;
 targetStatus: "available" | "unresolved";
 target?: /* virtual concrete expansion */;
 abstractTarget?: /* virtual abstract expansion */;
}
```

Clarifications:

- Inverse traversal ≠ duplicate link storage 
- Relationship pagination is independent of content pagination 
- Reading relationships does not bump source content revision 
- Ad-hoc `PUT/DELETE` on `/api/relationships/links` → **405**

### 15.3 Expansion lists

Options (package): `include: true`, `expandContentTypes`, `expandAbstractTypes`, `depth` (depth > 1 rejected → `READ_RELATIONSHIP_DEPTH`), `limit`, `cursor`.

| Mode | Behavior |
|---|---|
| References only | Links without target payloads |
| Expand selected CTs | Load current target content via unified reader |
| Abstract target expansion | Project via abstract reader |
| Unresolved | First-class `targetStatus: "unresolved"` |
| Fan-out / depth | Bounded; excess → error |

Expansion is **per get / per relationship page**, not a silent list-side join that rewrites stored sources.

### 15.4 Related-property filtering

Two-phase **bounded** plan:

1. Query candidate **target** records (hard `relatedScanLimit`) 
2. Match relationships for those targets against the subject 

Diagnostics include plan type, scan limit, candidates matched, relationships matched, rejection reason. Unbounded related-property queries are **rejected** (`READ_RELATIONSHIP_UNBOUNDED`), never silently executed.

OpX example: “work items assigned to people whose `data.role` exists” is a related-property query — not a client-side Cartesian join.

---

## 16. Metadata lists

| Mode | Behavior |
|---|---|
| Raw metadata for one agent | Pack-local definitions |
| Effective / inherited | Child overrides + inheritance path |
| Multi-agent combined | Deterministic merge for `agentIds[]` |
| Conflicts | Surfaced — not ordered away by array position |
| Ordering | Stable by kind / id |

Use metadata lists to learn allowed query paths, abstract members, and relationship definitions **before** building filters. Do not hardcode Mongo collection names from Studio screenshots.

---

## 17. Operational lists

| Surface | Filters / order (typical) |
|---|---|
| Memory pulls / runs | Time, sourceId, status |
| Quarantine | Pull id, reason, time |
| Pipeline runs / steps / attempts | runId, status, retry linkage |
| Artifacts / DataRefs | runId; respect retention / expiry |
| Relationship verify/rebuild | definitionId; confirmation flags on mutating ops |
| Intelligence proposals | Profile / proposal ids — never auto-install |
| Data journey | Object-types / journeys selectors; matrix & coverage results; evidence cells — not subject pages |

Rules:

- Cross-org isolation still applies 
- Large logs are artifacts, not inline list payloads forever 
- Deleted/expired artifacts → not-found / expired errors 
- Operational rows are evidence of work — they are not substitutes for subject content lists 
- `/api/data-journey/*` is read-only evidence/coverage — never invent cells the descriptors do not expose 
- Process debug (`/health`, `/_live`) is **not** a list API — see [Platform Overview](../../00-platform-overview/developers/BOOK.md); remote clients do not install ports kits
---

## 18. Counts, search, and views

### 18.1 Counts

| Capability | Status |
|---|---|
| Page item count | Always available as `records.length` / `items.length` |
| Exact total | **Do not assume** — often absent |
| Estimated total | Only if OpenAPI documents it |
| Facets / group-by / distinct / time buckets | Not a general public analytics API unless a specific endpoint says so |

Do not calculate expensive totals in a hot request path by walking every cursor page “just to show a number.”

### 18.2 Search

| Kind | Public contract |
|---|---|
| Exact SafeFilter | Yes |
| Prefix / full-text / fuzzy / vector / regex | **Not** general public features — do not document Mongo regex as product search |

If a specialized search endpoint appears in OpenAPI later, follow that operation — do not invent one.

### 18.3 Views and projections

- Metadata-defined views select presentation fields 
- Server-owned identity / revision fields remain visible 
- Projection does not mutate storage and does not create new content records 
- Projections must not expose credentials or internal storage fields 

---

## 19. Batch reads and batch writes

### 19.1 Batch reads

Inventory what OpenAPI actually offers (batch get by addresses, artifact resolution, etc.). For each supported batch:

- maximum batch size 
- ordering / duplicate input handling 
- per-item success vs fail-fast 
- cross-object-type / cross-category rules 
- cross-org prohibition 
- response-size limits 

If no batch-get exists, the accepted pattern is **bounded parallel gets** with a client-side concurrency cap — still through HTTP — not a Mongo `$in` on `_id`.

### 19.2 Batch writes

Public single-record writes: `add` | `upsert` | `patch` | `replace` | `remove` with `absent` or exact `revision`.

| Question | Answer |
|---|---|
| General multi-row atomic batch write? | **Do not assume** |
| Partial success? | Belongs to **pipeline** semantics |
| Global atomicity across OT/CT? | Not implied |
| Ad-hoc bulk Mongo endpoint? | Forbidden |

Direct clients that need bulk mutation use the **pipeline runtime** (bounded steps, retries, artifacts), not a home-grown multi-write loop without preconditions.

---

## 20. Large exports, streaming, and subscriptions

### 20.1 Export pattern

Prefer:

1. Start an asynchronous operation / pipeline run 
2. Internally read bounded pages 
3. Write a durable artifact / DataRef 
4. Poll operation status 
5. Download/stream the artifact through the service 
6. Honor expiry / retention metadata 

Do **not** support as product behavior: `limit=unbounded`, direct DB dumps, unrestricted browser memory loading, cursor bypass, or collection-name access.

### 20.2 Streaming / realtime

Document only what exists. Canonical observation is often **polling** run status with backoff. Do not invent WebSocket / SSE guarantees. Recommended client pattern: exponential backoff, honor terminal statuses, store an opaque cursor/`since` only if the endpoint defines one.

---

## 21. Client-side iteration patterns

### 21.1 Safe page loop (HTTP)

```ts
async function iterateWorkItemSnapshots(baseUrl: string, headers: HeadersInit) {
 let cursor: string | null = null;
 const filter = encodeURIComponent(
 JSON.stringify([{ field: "data.status", op: "eq", value: "in-progress" }]),
 );
 const sort = encodeURIComponent(
 JSON.stringify([{ field: "recordId", direction: "asc" }]),
 );

 for (;;) {
 const qs = new URLSearchParams({
 objectType: "work-items",
 contentType: "snapshots",
 limit: "50",
 });
 // Pass filter/sort only when your service build wires them (see OpenAPI).
 qs.set("filter", filter);
 qs.set("sort", sort);
 if (cursor) qs.set("cursor", cursor);

 const res = await fetch(`${baseUrl}/api/data/content?${qs}`, { headers });
 if (!res.ok) throw new Error(await res.text());
 const body = await res.json();

 for (const record of body.records ?? []) {
 await handle(record); // idempotent on recordId+revision
 }

 cursor = body.pageInfo?.nextCursor ?? null;
 if (!cursor) break;
 }
}
```

### 21.2 Checklist

- Preserve the **exact** query between pages 
- On process restart, reuse stored cursor **only** with the same shape; on mismatch, restart 
- Avoid duplicate side effects (upsert by business key / revision) 
- Iterate abstract unions with abstract cursors — do not merge concrete pages yourself unless you reimplement the merge contract 
- Paginate relationships independently from content 
- Prefer export jobs for huge extracts 

---

## 22. Caching

| Topic | Guidance |
|---|---|
| List pages | Generally treat as **non-authoritative** caches; short TTL if any |
| Cursor pages | Do not cache page N as forever-true |
| Metadata | Cache with checksum / pack version invalidation |
| ETag / If-None-Match | Only if OpenAPI documents it |
| Relationship expansions | Become stale when targets change — re-expand when correctness matters |
| Operational lists | Prefer fresh reads; runs grow while you watch |

Clients must not assume a cached expansion of Alex→WI-100 is current forever.

---

## 23. Errors and recovery

| Code / class | HTTP (typical) | Retry? | Action |
|---|---|---|---|
| Validation / `READ_QUERY_INVALID` | 400 | No | Fix filter/sort body |
| `READ_QUERY_INJECTION` | 400 | No | Remove illegal paths/operators |
| `READ_QUERY_FIELD_FORBIDDEN` | 400 | No | Use metadata-allowed paths |
| `READ_QUERY_NOT_PUSHDOWN` | 400 | No | Narrow abstract filters/sorts |
| `READ_CURSOR_INVALID` | 400 | No | Restart pagination |
| `READ_CURSOR_MISMATCH` | 400 | No | Restart with consistent query |
| `READ_RELATIONSHIP_DEPTH` | 400 | No | Keep depth at 1 |
| `READ_RELATIONSHIP_UNBOUNDED` | 400 | No | Lower scan / add filters |
| `READ_NOT_FOUND` | 404 | No | Distinct from empty list |
| Revision conflict | 409 | After re-read | Merge; retry write |
| Metadata conflict | 4xx | No | Fix agent pack / ids |
| Rate limit / overload | 429 / 503 | Yes, backoff | Cap parallelism |

Partial batch failure → inspect per-item results; do not blindly retry successful items.

---

## 24. Performance contract

| Limit | Typical accepted value |
|---|---|
| Default page size | 50 |
| Max page size (content / rel) | 200 |
| Related-property scan | Explicit bound ≤ 200 (route clamps) |
| Expansion depth | 1 |
| Filter / sort fields | Keep small; respect metadata allow-lists |
| Batch size | Per OpenAPI — never unbounded |
| Request timeouts | Client-defined; prefer smaller pages under load |

Distinguish: **hard contractual maximum** · **deployment configurable maximum** · **current default**. Prefer diagnostics over silent truncation.

---

## 25. Consistency and concurrent mutation

| Scenario | What may happen | Safe client response |
|---|---|---|
| Insert between pages | New WI appears later | Accept or re-scan with watermark |
| Delete between pages | Shorter later page | Idempotent skips |
| Sort field updated | Item may move | Idempotent processing |
| Target updated during expansion | New target payload | Re-expand when needed; link unchanged |
| Abstract membership metadata changes | Union membership shifts | Restart abstract iteration |
| Relationship rebuild mid-pagination | Link set changes | Restart relationship cursor |
| Workflow revision while listing | Sibling revisions diverge | Address by content type + revision |
| Operational run grows steps | Later pages see new steps | Expected for live runs |

Re-read after write before assuming list freshness in UIs.

---

## 26. Decision trees

### 26.1 Which list API?

```text
Need more than one resource?
├─ Concrete OT+CT pages → GET /api/data/content
├─ Union across concrete types → GET /api/data/abstract
├─ Siblings of one subject → POST /api/data/compose (explicit)
├─ Links / expansions → /api/relationships/query (+ related-property)
├─ Type catalogs → /api/metadata/...
├─ Evidence / coverage → /api/data-journey/*
└─ Runs / quarantine / artifacts → Memory / pipelines / intelligence routes
```

### 26.2 Filter vs abstract vs related-property

```text
Filter on fields of the listed records?
└─ SafeFilter on that list

Filter on fields of *related* targets?
└─ related-property (bounded) — not client Cartesian join

Need employees ∪ contractors as one stream?
└─ abstract People — not a People collection
```

### 26.3 Pagination vs export

```text
Interactive UI / modest backfill?
└─ Cursor pages (≤200)

Huge extract / overnight job?
└─ Pipeline / export artifact — not unbounded list
```

---

## 27. Antipatterns

| Antipattern | Do this instead |
|---|---|
| Parse / forge cursors | Treat as opaque; restart on mismatch |
| `limit` enormous to “get all” | Page or export job |
| Client-merge by copying target `.data` onto source | Relationship expand / compose |
| Root `associated*` storage | Four-field links only |
| Abstract collection writes | Concrete writes or `patchAbstract` |
| Direct Mongo domain reads/writes | `memorix-service` / `@x12i/memorix-data` |
| Silent N+1 compose for entire org | Paginate + on-demand compose / pipeline |
| Unbounded related-property scan | Bound + refine filters |
| Assume totals always exist | Use page length; request analytics only where documented |
| Cache expansions forever | Re-expand when correctness matters |
| Use live customer orgs for acceptance | Fresh `sandbox*` + `agentIds: ["opx"]` |
| Ad-hoc multi-write without preconditions | Pipeline batch semantics + revisions |

---

## 28. SDK conveniences (when present)

Official helpers should expose:

- `listPage` / `iterateAll` (cancellation-aware, never silent unbounded) 
- `batchGet` with partial-failure visibility 
- Typed filter / sort builders matching SafeFilter / SafeSort 
- Opaque cursor wrappers 
- Abstract / relationship / operational iterators 
- Export helpers 

SDKs must preserve cursor opacity and query shape, surface diagnostics, never silently retry stale writes, and never hide partial failures.

---

## 29. Conformance matrix (notes)

For **every** list endpoint, acceptance should cover:

| Case | Expectation |
|---|---|
| Empty result | 200 + empty array + null cursor |
| One page / multi page / exact boundary | Stable order; final null cursor |
| Equal sort values | Tie-breaker prevents reshuffle under quiet data |
| Malformed / mismatched cursor | Explicit error; restart |
| Max / invalid limit | Clamp or validate per contract |
| Null vs missing filters | Distinct behavior |
| Org isolation | No cross-org leakage |
| Multi-agent metadata | Effective union/defs; conflicts visible |
| No Mongo injection | `$` / `_id` / operator smuggling rejected |
| Discriminated / abstract merge | Deterministic; no dup `recordId` in union stream |
| Forward/inverse relationships | Independent pagination; unresolved first-class |
| Related-property bounds | Reject unbounded |
| Operational growth | Later pages may include new rows |
| Batch partial success | Visible per item |
| Large export | Artifact path, not unlimited list |

Classify every doc claim as: guaranteed public contract · current default · configurable limit · implementation detail · unsupported. Never present an implementation detail as a client guarantee.

---

## 30. Source-of-truth rule

Each claim should be traceable to:

1. Accepted OpenAPI operation (`GET /api/openapi.json`) 
2. Public schema (`PageInfo`, `SafeFilter`, record envelopes) 
3. Route handler (`memorix-service/src/routes/*`) 
4. Canonical package (`@x12i/memorix-data` query/reader/relationships) 
5. Tests / gate decisions 

When docs disagree with runtime, prefer **implemented contracts and tests** and file the contradiction — do not silently pick a side in application code.

---

## 31. Quick opx journey (lists edition)

1. Install opx pack into fresh `sandbox*` with `agentIds: ["opx"]`. 
2. List `work-items` / `snapshots` until WI-100 appears; follow `nextCursor`. 
3. Compose WI-100 with `explicit:true` — siblings stay arrays. 
4. List abstract `People`; find Alex; note concrete `objectType: employees`. 
5. Query relationships for Alex (forward) and WI-100 (inverse); expand snapshots. 
6. Related-property: find assignments whose target snapshots match a bounded filter. 
7. Patch Alex’s snapshot; re-expand — link ids unchanged, target payload current. 
8. Start a pipeline export instead of holding an overnight cursor. 

---

## Next

- [Access patterns decision guide](../../04-data-access-patterns/developers/BOOK.md) — which read shape to pick 
- [Metadata, agents, abstracts](../../05-metadata-agents-abstracts/developers/BOOK.md) — bindings and effective catalogs 
- [Relationships & Memory](../../06-relationships-and-memory/developers/BOOK.md) — four-field links and pulls 
- [Pipelines](../../08-pipelines/developers/BOOK.md) — bulk mutation and export jobs 
- [Migration & troubleshooting](../../13-migration-troubleshooting/developers/BOOK.md) — conflicts and recovery 

*End of Memorix Runtime Lists, Queries, Pagination & Bulk Access (Developers).*

---

# Data Access Patterns — developers

# Memorix Runtime Data Access Patterns

**Audience:** Application, agent, connector, pipeline-step, Studio/UI, and operator-script developers choosing *which* runtime access method to use for a job. 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Complements:** [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md) · [Runtime Data Client](../../02-runtime-data-client/developers/BOOK.md) · [Lists, queries, pagination](../../03-lists-queries-pagination/developers/BOOK.md) 
**Status:** Present-tense decision companion — not an OpenAPI dump. Prefer implemented contracts and package tests when wording conflicts with older notes.

---

## How to read this guide

1. Start with the **central decision model** — every later chapter is a specialization of it.
2. Use the **master decision matrix** when you already know the goal and need a one-glance answer.
3. Use numbered chapters when you need the *why*, the opx narrative, and the conflict/retry story.
4. Use the **cookbooks** (investigation / correction / large-data) for bounded recipes you can copy.
5. Use the **decision tree** and **anti-patterns** when you are about to invent a shortcut.
6. Labels used throughout:
 - **Stored** — persisted subject content or relationship links
 - **Virtual / calculated** — composed, expanded, or projected at read time
 - **Operational** — runs, ledgers, artifacts, quarantines (not ordinary subject content)
7. Classification labels for every pattern:
 - **Best** — preferred production path
 - **Acceptable** — supported, sometimes ad-hoc, with clear bounds
 - **Avoid** — works once, fails at scale or under concurrency
 - **Forbidden** — bypasses Memorix law (Mongo, copied payloads, abstract collections, …)

This guide answers one question:

> Given a runtime task, which access method should a client use — and what is best, acceptable ad-hoc, or forbidden?

---

## The central decision model

Memorix exposes many ways to touch data. They are not interchangeable.

> **Address what you already know. Discover what you do not. Compose siblings. Traverse links. Ingest through Memory. Mutate with revision. Scale through pipelines.**

| Situation | First choice |
|---|---|
| You know the exact content address | Exact get / exact write |
| You know properties, not the address | Bounded list / query |
| You need several content kinds for one subject | Composition |
| You need another subject | Relationship traversal / expansion |
| You need a semantic union (People) | Abstract list / get |
| Data arrives from an external system | Memory → raw → pipeline |
| Work is large, repeated, scheduled, or retryable | Pipeline with DataRef + ledger |
| One-off operator investigation | Bounded service API call — never Mongo |

**Runtime boundary (all ordinary clients):**

```text
Client
 → memorix-service (HTTP / OpenAPI)
 → canonical runtime packages
 → organization stores
```

Trusted in-process hosts may call `@x12i/memorix-data` (and siblings) **inside** the platform process with the same contracts. They still must not open Mongo as a product API.

**Clients must never use:** direct Mongo, collection names as business identity, Mongo `_id`, internal storage keys, internal package adapters as remote APIs, legacy Explorer/Hippox/associator routes, arbitrary relationship inserts, or arbitrary Mongo operators.

---

## 1. Client types and access style

Different callers should behave differently even when they share the same HTTP surface.

| Client type | Typical behavior | Prefer | Avoid |
|---|---|---|---|
| Interactive UI | Small pages, current reads, explicit user writes | Exact get, composition, metadata views, cursor pages | Export-sized scans in one HTTP request |
| Agent runtime | Bounded reads, composition/expansion, captured execution context | Compose + expand with caps; store run artifacts | Unbounded fan-out; treating expansion as history |
| Backend service | Point reads/writes and bounded queries | Service HTTP with revision preconditions | In-process Mongo “for speed” |
| Connector | Memory pull and raw landing | Memory + `credentialRef` | Writing `*-snapshots` from the connector |
| Pipeline step | DataRef inputs, canonical ports, idempotent writes | Pipeline ports + ledgers | Embedding batch engines in the step host |
| Batch job | Durable run, paginated input, partial-success ledger | Pipeline definition | List-and-mutate scripts without capture |
| Operator script | Bounded ad-hoc API calls with explicit scope | One revision-safe correction | Shell update-many / silent overwrite |
| Export / report client | Asynchronous export / artifact | Export job + artifact get | Unlimited `listAll` in the report process |

**Ops narrative:** Studio Explore and a host ops console both talk to the same `sandbox*` org with `agentIds: ["opx"]`. The console shows employee **Alex**, work item **WI-100**, and procedure **PROC-2**. Neither surface opens Mongo; both load metadata first, then choose exact, list, compose, abstract, or relationship access from this guide.

---

## 2. Master decision matrix

For every runtime goal: preferred method, acceptable ad-hoc, forbidden method, and the consistency / retry story.

| Goal | Best | Acceptable | Forbidden | Consistency / retry |
|---|---|---|---|---|
| Read one known content instance | Service `GET /api/data/content` (exact) | In-process `@x12i/memorix-data` `getContent` | Mongo `findOne` | Current revision; retry transient failures |
| Find records matching fields | Bounded list/query + opaque cursor | Temporary ad-hoc filter with hard limit | Unbounded `listAll` / decoded cursors | Cursor contract; retry **same** query+cursor |
| Read all content about one subject | Explicit composition | Two–three independent exact gets | Flatten siblings into the snapshot; invent `associated*` | Current siblings; per-sibling revisions |
| Read a union such as People | Abstract list / get | Concrete lists + UI label (diagnosis only) | Abstract collection / abstract `recordId` | Metadata-dependent projection |
| Follow relationships | Relationship query / `expandRelationships` | Bounded investigation queries | Client-side org-wide joins; copied payloads | Current targets; unresolved stays unresolved |
| Read current related properties | Related-property / expansion APIs | Two bounded queries | Persist expanded targets on the source | Current; may go stale if cached naïvely |
| Process a small known set | Batch get (if exposed) or bounded list of addresses | Parallel exact gets with concurrency cap | Cartesian fan-out | Per-item |
| Process a large set | Pipeline + DataRef | Async export for reporting | Interactive mega-list | Captured/ledgered; retry failed items |
| Import external data | Memory pull → raw | Controlled source test via Memory | Direct snapshot insert from connector | Source cursor; resume committed cursor |
| Write one content instance | `patch` / `upsert` / `replace` / `remove` + precondition | Operator script through the same API | Blind replace; Mongo update | Exact revision; never blind-retry after stale |
| Update an abstract property | `patchAbstract` reverse write | Temporary concrete patch when mapping is clear | Writing an “abstract record” | Maps to one concrete revision |
| Modify many records | Pipeline / batch with per-item revisions | Capture-all-then-write for tiny sets | Client update-many; list-mutate active sort | Partial success ledger |
| Create relationships | Relationship engine (discover / materialize / rebuild) | Dry-run / verify | Manual link inserts; `associated*` | Idempotent materialization |
| Inspect execution history | Operations / pipeline / Memory APIs | Studio Operate views | Treating ledgers as subject content | Operational identity |
| Inspect evidence / expected slots | `/api/data-journey/*` (matrix, coverage, evidence) | Studio Explore evidence UI | Inventing cells; treating journey rows as subject content | Read-only; views = slots only |
| Export large data | Asynchronous export artifact | Paginated offline consumer with checkpoint | Single-request dump | Export contract |
| Investigate once | Bounded ad-hoc API query | Studio Explore | Direct Mongo shell | Read-only unless labeled write |
| Save a repeated query | Metadata-defined view | Temporary client query | Hardcoded DB query in the product | Versioned with metadata |
| Repeated business process | Pipeline definition | Scheduled Memory + pipeline | UI-embedded “batch engine” | Durable run ledger |
| Propose new object type | Intelligence proposal → human install | Export draft for review | Intelligence auto-publish / AI writing records | Proposal ≠ live metadata |
| Secret to connector | `credentialRef` | — | Secret in body / Memory payload | Never log credentials |

---

## 3. Reading one known item (exact get)

Exact get is the fastest and safest point-read when the client already knows the address.

**Use when you know:**

- `recordId` (subject)
- `objectType` + `contentType` where required
- discriminator value when the content type is multi-instance

**Properties of an exact read:**

- Does **not** compose siblings
- Does **not** expand relationships unless you opt in
- Returned `revision` belongs to **that** content instance only
- Missing content ≠ missing subject (directory may still reserve the id)
- Internal Mongo `_id` is irrelevant

### Recipes (ops)

| Need | Address |
|---|---|
| WI-100 snapshot | `objectType=work-items`, `contentType=snapshots`, `id=wi-100` |
| WI-100 workflow sibling | same subject, `contentType=workflow` |
| Exact raw landing | `contentType=raw` (after Memory) |
| One analysis type | `contentType=analysis` + discriminator |
| One decision type | `contentType=decisions` + discriminator |
| One abstract projection | abstract get for People / KnowXEntity — still virtual |

```http
GET /api/data/content?objectType=work-items&contentType=snapshots&id=wi-100
x-memorix-org-id: sandbox<suffix>
x-memorix-agent-ids: opx
```

**Best:** service exact get. 
**Acceptable:** in-process `getContent` inside the trusted boundary. 
**Forbidden:** Mongo `findOne` by collection or `_id`.

---

## 4. Reading through lists

Use a list when the client must **discover** a set by object type, content type, filter, sort, time, lifecycle, discriminator, or abstract membership.

### Correct list loop

1. Construct one bounded query.
2. Preserve its exact shape for the whole walk.
3. Request one page (`limit` capped — content lists typically max 200).
4. Process the page.
5. Follow the opaque `nextCursor`.
6. Continue until the cursor is empty/missing.
7. Handle concurrent changes per the cursor consistency model — do not assume snapshot isolation unless documented.

### Do not

- Decode or forge cursors
- Change filters between pages of the same walk
- Request an unbounded page
- Create a new query for every item when one list would do
- Assume a total count exists
- Treat pages as a frozen snapshot of the org

```text
GET ...&limit=50&cursor=<opaque>
→ { records|items, pageInfo: { nextCursor } }
```

See [Lists, queries, pagination](../../03-lists-queries-pagination/developers/BOOK.md) for filter families and operational list params.

---

## 5. Exact read versus list

| Situation | Use |
|---|---|
| Known `recordId` / content address | Exact get |
| Unknown `recordId`, known properties | List / query |
| Several known addresses | Batch get or parallel exact gets (capped) |
| Same subject, several sibling types | Composition |
| Abstract union | Abstract list |
| Different related subjects | Relationship traversal |
| Large unknown set | Pipeline / export |

**Why not list a known id?** A list walk is more expensive, harder to reason about under concurrency, and invites “first hit wins” bugs. Exact get is O(1) identity.

**Why not guess `recordId` from business fields?** External ids live under `concept.identifiers`. Guessing subject ids from titles or provider keys is unsafe; list/filter by identifier fields or resolve through Memory/mapping metadata instead.

**OpX example:** Host knows `WI-100`’s Memorix `recordId` → exact snapshot get. Host only knows provider work-item key → bounded list/filter on identifier fields, then exact get thereafter.

---

## 6. Composition versus multiple gets

**Preferred:** request explicit composition when the client needs several sibling content types for one subject (snapshot + workflow + decisions for `wi-100`).

**Acceptable:** separate exact reads when only one or two independently refreshed siblings are needed (e.g. poll workflow alone).

**Avoid:**

- Fetching every registered sibling automatically
- Flattening siblings into the snapshot payload
- Treating analysis/workflow as relationships
- Caching composition as a new authoritative stored record

Composition is **virtual**. Sibling instances keep independent revisions. Payload size and freshness trade off: compose once for a dossier screen; refresh one sibling when only that pane changed.

```text
Subject wi-100
 ├─ snapshots (revision N)
 ├─ workflow (revision M)
 ├─ decisions[] (per-instance revisions)
 └─ analysis[] (per-instance revisions)
```

**Best:** compose API / `composeSubject`. 
**Acceptable:** multiple gets of **references / envelopes**, assembled in the client without copying neighbor `.data` into the source. 
**Forbidden:** embed copied neighbor payloads (`associated*` revival).

---

## 7. Relationships versus list joins

**Preferred:**

- Relationship traversal when the semantic link is materialized (four-field links)
- Related-property / expansion options when filtering or showing current target properties

**Acceptable ad-hoc:**

- Bounded concrete queries to investigate a *potential* relationship before metadata exists

**Forbidden:**

- Client-side full-organization joins
- Copying target payloads onto the source
- Creating root `associated*`
- Persisting expanded targets
- Manually inserting relationship documents

If a repeated join represents real domain meaning (people assigned to work items), author relationship metadata and run discovery/materialization — do not keep re-joining in every client.

**Ops path:**

```text
Metadata: people-assigned-to-work-items
 → relationship engine discover / materialize
 → GET content?expandRelationships=true
 → unresolved targets shown explicitly
 → fetch target snapshots only as needed
```

Clients may inspect definitions, trigger discovery/verify/rebuild where exposed, and query/traverse links. Clients may **not** invent unresolved recordIds or store relationship evidence on the link document.

---

## 8. Abstract versus concrete access

**Use abstract reads when:**

- The client wants a semantic union such as **People** (employees + contractors)
- It needs the shared abstract property contract
- Concrete member identity must still be retained in the projection

**Use concrete reads when:**

- Member-specific properties are needed
- The client already knows the object type
- A filter cannot be pushed through abstract bindings
- An abstract mapping is not reversible

`agentIds[]` determines **available abstract metadata** (and effective types/views). It does **not** hide organization data as a row-level security mask.

Abstracts are **virtual**. There is no abstract collection and no abstract `recordId`. Writes use `patchAbstract` → reverse mappings → one concrete writer patch.

**Best:** abstract list for unified People screens. 
**Acceptable:** concrete lists + label in UI for diagnosis. 
**Forbidden:** abstract collection writes; inventing abstract records.

---

## 9. Metadata-defined views versus ad-hoc queries

| | Ad-hoc query | Metadata-defined view |
|---|---|---|
| Purpose | Exploration | Product contract |
| Lifetime | Temporary | Versioned with installed metadata |
| Author | Client | Metadata author / Studio Design |
| Validation | Client diligence | Schema / install validation |
| Clients | One script | Many UIs and agents |

**Create a view when:**

- The same query/projection appears repeatedly
- Multiple clients need the same shape
- The result has product meaning
- The query needs governed bounds/projection

**Do not** create a metadata view for every temporary investigation. Bounded ad-hoc queries are fine for one-off ops work.

**Host app list screen recipe:**

1. Load list/view metadata from `/api/metadata`
2. `GET /api/data/content` or abstract list with cursor
3. Render columns from descriptors — do not hardcode field sets

---

## 10. Direct writes and the read-modify-write loop

Use direct writes for **one known content instance**.

**Supported patterns:** `add` (absent), `upsert` (absent or exact revision), `patch` (exact revision), `replace` (exact revision), `remove` (exact revision), `patchAbstract` (exact revision on the concrete target).

### Best-practice RMW

1. Read current content when necessary.
2. Retain its `revision`.
3. Compute the intended change.
4. Send the **smallest** valid mutation (`patch` over full `replace` when possible).
5. Supply exact revision / `absent` precondition.
6. Handle stale conflict explicitly.
7. Use the returned revision.

### Success path

```text
GET content → revision 4
compute change
PATCH with expectedRevision 4
success → revision 5
```

### Conflict path

```text
GET content → revision 4
another client writes → revision 5
PATCH expecting 4 → 409 stale
refetch revision 5
reconcile intent (do not blindly overwrite)
PATCH expecting 5
```

**Safe automated conflict strategies:** abort and report; refetch and recompute; merge only known independent fields; route to workflow/operator decision.

**Never:** omit preconditions; turn a stale patch into a blind upsert; overwrite the full record when a small patch is enough; modify immutable envelope fields; change discriminator in place; edit Mongo documents directly.

---

## 11. Ad-hoc writes versus list-driven writes versus pipelines

### Ad-hoc writes (acceptable when tiny and explicit)

Acceptable ad-hoc writing means: one-off mutation through `memorix-service`, explicit scope (`sandbox*` + `opx`), exact content address, exact revision, validated payload, correlation/request ID, and clear operator evidence.

**Acceptable examples:** correct one work-item title; archive one content instance; update one workflow sibling; patch one reversible People property.

**Not acceptable:** Mongo shell mutation; update-many over a collection; manually writing links; bypassing directory registration; rewriting without revision; copying related payload; changing metadata mid-job.

### List-driven writes (dangerous by default)

```text
list page → immediately update each result → request next page
```

If writes change active sort/filter fields, items move between pages; you skip or double-process.

**Preferred for anything beyond a handful of rows:**

1. Capture stable input addresses and revisions into a DataRef / durable input set.
2. Process that captured set through a pipeline.
3. Track per-item success/failure.
4. Retry failed items only.

**Small bounded ad-hoc exception:**

1. List and capture all addresses/revisions first.
2. Stop pagination.
3. Review the bounded set.
4. Apply exact-revision writes.
5. Report stale conflicts individually.

Do not mutate the active list while relying on its cursor unless the API explicitly guarantees safety for that operation.

### Pipelines (preferred large-operation mechanism)

Use pipelines when work is repeated, scheduled, long-running, multi-step, large batch, failure-prone, dependent on captured context, relationship-producing, service-calling, or requiring retry/resume/audit.

Pipelines give you: one durable run ledger, step attempts, at-least-once execution with idempotent mutation ports, DataRef/artifacts, partial-success ledger, retry-failed-step-only, captured input revisions/context, and the same runtime for scheduled and on-demand.

**Do not** build a second batch engine inside a client or UI.

| Scale | Mechanism |
|---|---|
| One instance | Direct revision-safe write |
| Small known set | Batch request or capture-then-write |
| Large / long / retryable | Pipeline |
| Full export | Asynchronous artifact |
| Cross-record business operation | Pipeline / workflow |
| Global atomicity across many records | Unsupported unless an explicit backend capability says otherwise |

---

## 12. Batch, export, and performance-aware choices

| Pattern | Latency | Scale | Consistency | Best use |
|---|---|---|---|---|
| Exact get | Lowest | One instance | Current | Known address |
| List page | Low | Bounded set | Cursor contract | Discovery |
| Composition | Moderate | One subject | Current siblings | Subject detail |
| Abstract union | Moderate | Bounded cross-type | Metadata-dependent | Semantic lists |
| Relationship expansion | Fan-out dependent | Bounded | Current targets | Related context |
| Batch | Moderate | Small known set | Per-item | Client batch |
| Pipeline | Async | Large/complex | Captured/ledgered | Production processing |
| Export artifact | Async | Very large | Export contract | Reporting/archive |

For each batch-like option, check OpenAPI for real limits, ordering, per-item preconditions, partial success, atomicity, retry, idempotency, and returned revisions. Prefer documented caps over inventing “page size = org size.”

**Interactive vs background:** interactive clients stay on small pages, low fan-out, fast bounded expansions, clear loading/errors, and current data. Background clients use durable runs, DataRef, checkpoint/resume, retries, captured context, and larger-but-bounded pages. Never run an export-sized scan inside an interactive HTTP request.

---

## 13. Memory as the preferred ingestion mechanism

When data comes from an external source, land it through Memory first.

```text
source (opx-source)
 → Memory pull (credentialRef, read-only connector)
 → exact raw V2 (*-raw)
 → downstream pipeline maps / validates / writes subject content
 → optional relationship materialization
```

**Not:** connector → snapshot collection directly.

Covered concerns: scheduled knowledge pulls, on-demand context pulls, source cursor, exact raw payload, content-hash dedup, TTL/discard for context mode, quarantine, failure followed by successful later pull, `credentialRef` only, read-only source interaction.

Memory stops at raw handoff. Promote, analysis, and relationship discovery belong to pipelines and the relationship engine — not the connector.

**Best:** Memory pull. 
**Forbidden:** writing `*-raw` or snapshots from a random script that skips Memory and directory rules; secrets in the request body.

---

## 14. Relationship creation

Runtime clients have a narrow, intentional set of options.

**Preferred:**

1. Define relationship metadata.
2. Run discovery / materialization.
3. Verify / rebuild explicitly when needed.

**Provider relationship-shaped input:**

1. Land it raw via Memory.
2. Metadata maps provider identifiers.
3. Relationship engine resolves `recordId`s.
4. Materialize four-field links (navigation only).

**Clients may:** inspect definitions; trigger discovery/verify/rebuild where exposed; query/traverse links; expand on read.

**Clients may not:** insert arbitrary links; copy associated target data; create unresolved/invented recordIds; store relationship evidence on links; pin current revisions on the link as history (capture that in operational artifacts instead).

---

## 15. Workflow, analysis, and decision content

### Workflow

Workflow is ordinary sibling content about the subject — not a separate persistence product.

**Best:** exact workflow read; patch/replace with revision; explicit composition for subject context; pipeline/workflow operation for repeated transitions. 
**Ad-hoc:** one revision-safe workflow correction. 
**Forbidden:** workflow-specific direct collection writes; flattening workflow into snapshot; a second “workflow service” with different semantics.

### Analysis, decision, discovery

- Discriminator required/default rules come from metadata profiles
- Multiple instances per subject are normal
- Exact addressing + independent revisions
- List by discriminator when discovering instances
- No in-place discriminator change (remove + recreate when the product allows)
- Outputs from AI/services are still ordinary V2 content once written

**Best:** pipeline produces a validated typed instance; clients read exact type or list instances. 
**Ad-hoc:** revision-safe correction of one existing instance if product rules allow.

---

## 16. Intelligence proposals

Clients may request profiling, inspect evidence, review proposals, export metadata drafts, and compare with installed metadata.

Clients may **not** treat proposals as live metadata, auto-install AI output, allow AI to write records/links, or bypass insufficient-evidence outcomes.

The explicit metadata install/publish operation is separate (`/api/metadata` or Studio Design). Intelligence proposes; humans install.

**Ops story:** profiling observes employee/contractor shapes in `sandbox*`, produces a People abstract proposal, an operator reviews evidence in Studio, then installs. Until install, product screens keep using the previously effective metadata.

---

## 17. Current versus captured reads

| Need | Use |
|---|---|
| Current application display | Current read / expansion |
| Reproducible job retry | Captured execution context |
| Audit of what a job saw | Run / operational artifact |
| Current related record | Relationship expansion |

Current runtime read returns current content/revision; relationship expansion loads **current** target content.

Captured execution context stores exact content/revisions used by a run; it remains unchanged when current content changes; it belongs in operational artifacts.

**Do not** pin revisions on relationship links. **Do not** store a current expansion as if it were historical job evidence.

---

## 18. Caching and retry choices

### Caching

- Cache effective metadata by checksum / install identity
- Cache immutable captured artifacts according to retention
- Carefully cache current records using revision / ETag where supported
- Treat expanded relationships as current views that may become stale
- Do not cache cursors as permanent identities
- Do not share scoped artifacts across orgs
- Invalidate or update local cache after successful writes

### Retry

| Operation | Retry rule |
|---|---|
| Exact read | Retry transient failures |
| List page | Retry same cursor/query |
| Add absent | Check whether creation succeeded before repeating |
| Patch revision | Never retry after stale without refetch |
| Pipeline start | Use idempotency / run identity |
| Failed step | Retry failed step only |
| Memory pull | Resume committed cursor |
| Relationship materialization | Idempotent rerun |
| Artifact get | Retry transient; not wrong-org / not-found |
| Export | Resume / inspect operation |

---

## 19. Observability

Every client pattern should carry enough identity to debug without opening Mongo:

- Correlation / request ID
- Run ID where async
- Step / attempt ID
- `recordId` / content address
- Expected and result revision
- `sourceId` / `pullId` where Memory
- `relationshipDefinitionId` where relationship operation
- Artifact reference where applicable

**Do not log:** credentials, unrestricted raw payloads, sensitive provider data unnecessarily, Mongo connection details.

---

## 20. Ad-hoc investigation cookbook

Every recipe is **bounded** and **read-only** unless labeled as a write. Scope: `sandbox*` + `opx`.

1. **Find a work item by external identifier** — bounded list/filter on `concept.identifiers` fields for `work-items` / `snapshots`; then exact get.
2. **List active work-item snapshots** — content list with lifecycle/state filter + cursor; stop when done.
3. **Inspect every analysis for WI-100** — list `analysis` for `recordId=wi-100` (or compose and read the analysis array); note discriminators.
4. **Compose WI-100 snapshot / workflow / decisions** — explicit composition; do not flatten.
5. **List People across employees and contractors** — abstract People list with cursor.
6. **Find People related to WI-100** — relationship query / expansion from the work item; show unresolved explicitly.
7. **Find relationships whose target has a current property** — related-property / expansion path; keep bounds.
8. **Inspect unresolved links** — relationship query filtered by unresolved target status.
9. **Inspect a failed pipeline run** — operations / pipeline run + attempt APIs; read ledger, not subject collections.
10. **Inspect Memory quarantine** — Memory quarantine list with limit/cursor.
11. **Compare current content with captured job input** — exact get vs run artifact / DataRef payload.
12. **Preview an intelligence proposal** — intelligence proposal get/compare; do not install.
13. **Inspect expected-vs-filled coverage for an object type** — `POST /api/data-journey/coverage` (views as slots only); drill with `/api/data-journey/evidence/:ref` when a cell is filled.

---

## 21. Ad-hoc correction cookbook

Safe one-off writes. Each requires pre-read, exact revision, request, response, conflict behavior, and correlation id.

1. **Patch one concrete property** — GET snapshot → `patch` with `expectedRevision` → handle 409.
2. **Patch one abstract reversible property** — `patchAbstract` People field → reverse map → concrete revision.
3. **Correct one workflow sibling** — exact workflow GET → patch/replace with revision.
4. **Archive one content instance** — lifecycle/archive write through the data plane with precondition.
5. **Change discriminator** — remove old instance with exact revision → add new instance with `absent` (never in-place discriminator edit).
6. **Retry one failed pipeline step** — operations API retry-failed-step; do not re-run the whole org scan casually.
7. **Rebuild one relationship definition** — relationship rebuild for that definition id only.

**Forbidden corrections:** Mongo update, blind upsert after stale, manual link insert, rewriting raw into snapshot without a pipeline when Memory ownership applies.

---

## 22. Large-data cookbook

| Need | Pattern |
|---|---|
| Paginated processing | Stable query + opaque cursor + checkpoint |
| Captured-address set | Materialize DataRef / address list first |
| Pipeline batch | Pipeline definition + per-item ledger |
| Partial-success retry | Retry failed items/steps only |
| Asynchronous export | Export job → artifact retrieval |
| Restart / resume | Run identity + committed cursors / checkpoints |

**When a simple script becomes unsafe:** the set no longer fits a reviewed capture; writes affect list membership; runtime exceeds interactive timeouts; you need audit, resume, or partial success. At that point, author a pipeline — do not keep growing the script.

---

## 23. Best / acceptable / forbidden catalog (summary)

| Goal | Best | Acceptable ad-hoc | Forbidden |
|---|---|---|---|
| Read one snapshot | Service exact GET | In-process `getContent` | Mongo `findOne` |
| Show subject dossier | Compose API | Multiple gets of references only | Embed copied neighbor payloads |
| Unified People list | Abstract list | Concrete lists + UI label | Abstract collection writes |
| Patch a field | `patch` + `expectedRevision` | Upsert with precondition | Blind replace without revision |
| Link two subjects | Relationship engine / pipeline step | Verify / dry-run | Manual link collection inserts |
| Land vendor JSON | Memory pull | Controlled Memory test pull | Writing `*-raw` / snapshots from a random script |
| Propose new OT | Intelligence proposal → human install | Draft export for review | Intelligence auto-publish |
| Secret to connector | `credentialRef` | — | Secret in body / Memory payload |
| Update 10,000 records | Pipeline | None | Client update-many / list-mutate |
| Find current related data | Expansion | Bounded two-query investigation | Copy payload onto source |
| Reusable filtered screen | Metadata view | Temporary bounded query | Hardcoded DB query |

---

## 24. Anti-pattern catalog

Explain why these fail in production:

1. **Mongo as API** — bypasses revision, directory, abstracts, relationship enforcement, and audit.
2. **`associated*` revival** — copied payloads go stale and break writes; navigation must be four-field links.
3. **Abstract `recordId` / abstract collections** — abstracts are virtual; reverse-write maps to concrete.
4. **Silent multi-instance overwrite** — ignoring discriminators merges distinct analyses/decisions.
5. **AI as source of truth** — intelligence proposes; humans install metadata; AI must not write records/links.
6. **Unbounded `listAll`** — interactive timeouts, memory blowups, and inconsistent partial views.
7. **Decoding cursors** — cursors are opaque server contracts; forging them skips or duplicates rows.
8. **List-and-mutate active sort fields** — items migrate across pages mid-walk.
9. **Blind upsert after stale revision** — overwrites a newer operator/agent edit.
10. **Client-side organization-wide joins** — Cartesian cost and duplicated join logic; author relationships instead.
11. **Writing raw source data directly as snapshots** — loses exact provider evidence and Memory cursor/quarantine semantics.
12. **Pipeline logic embedded in UI** — no durable ledger, no safe resume, no shared audit.
13. **`agentIds` as record visibility filters** — agent ids select effective metadata, not row ACLs.
14. **Treating operational artifacts as subject content** — runs/quarantines are not `memorix-record/2` product subjects.
15. **Storing current expansion as historical job evidence** — expansion is current; capture revisions in run artifacts.
16. **One client per old API** — obsolete Explorer/Hippox/associator routes are not the Memorix service boundary.
17. **Relationship sibling documents as authoritative storage** — authoritative links live in the relationship store, not copied into subject siblings.

---

## 25. Decision tree

```text
Do you know the exact content address?
├─ Yes → exact read / exact revision-safe write
└─ No → bounded list/query (then exact thereafter)

Do you need several content types for the same subject?
├─ Yes → composition
└─ No → keep exact content read

Do you need another subject?
├─ Existing semantic link → relationship traversal / expansion
├─ Investigating a possible link → bounded ad-hoc query
└─ Repeated meaningful join → relationship metadata + discovery

Is the write one small known instance?
├─ Yes → revision-safe direct write
└─ No → capture addresses → batch/pipeline

Does data come from an external source?
└─ Memory → raw first → pipeline promote

Is the work large, repeatable, scheduled, or retryable?
└─ Pipeline (DataRef + ledger)

Is this a one-off investigation?
└─ Bounded ad-hoc API query — never direct Mongo

Need evidence cone / expected slots for Explore?
└─ /api/data-journey/* (read-only) — see Runtime Data Client §17

Is this a reusable product screen/query?
└─ Metadata-defined view (not a hardcoded DB query)

Is this a metadata shape suggestion?
└─ Intelligence proposal → human install (never auto-publish)
```

Process debug (`/health`, `/_live`) is not an access pattern for domain data — see [Platform Overview](../../00-platform-overview/developers/BOOK.md). Remote clients do not install `@x12i/core-service` / ports kits.
---

## 26. Host recipes (ops)

### A. Host app list screen

1. Resolve scope: `sandbox*` + `opx`.
2. Load list/view metadata from `/api/metadata`.
3. Call content or abstract list with cursor and governed `limit`.
4. Render columns from descriptors — do not hardcode field sets.
5. On row open: exact get or compose for the dossier.

### B. Safe concurrent edit

1. GET content → note `revision`.
2. PATCH with `expectedRevision`.
3. On 409: GET again, merge only safe fields or abort to the operator, retry with the new revision.

### C. Graph neighborhood

1. GET content with `expandRelationships=true` **or** relationships query.
2. Show unresolved targets explicitly.
3. Fetch target snapshots only as needed — never copy `.data` onto the source.

### D. Ingest then relate

1. Memory pull from `opx-source` with `credentialRef`.
2. Pipeline maps raw → snapshots for employees / work-items / procedures.
3. Relationship engine materializes assignments for WI-100 ↔ Alex.
4. Explore composes WI-100 and expands links.

---

## 27. Conformance checklist

- [ ] Every call sends `x-memorix-org-id` and `x-memorix-agent-ids` (plural)
- [ ] Acceptance uses fresh `sandbox*` + `opx` — never live customer orgs
- [ ] Exact address → exact get; discovery → bounded list
- [ ] Composition for same-subject siblings; relationships for other subjects
- [ ] Abstracts stay virtual; writes reverse-map to concrete
- [ ] Writes carry `absent` or exact revision; 409 triggers refetch, not blind upsert
- [ ] External data lands through Memory; secrets only as `credentialRef`
- [ ] Links come from the relationship engine; no manual inserts / `associated*`
- [ ] Large/repeated work uses pipelines + DataRef + ledger
- [ ] Intelligence proposes; humans install
- [ ] No direct Mongo domain access from ordinary clients
- [ ] Correlation ids present; credentials and raw secrets never logged

---

## Next

- Wire these choices through the HTTP/SDK surface: [Runtime Data Client](../../02-runtime-data-client/developers/BOOK.md)
- Deepen list/cursor contracts: [Lists, queries, pagination](../../03-lists-queries-pagination/developers/BOOK.md)
- Envelope and revision law: [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md)
- Ingest and links: [Relationships & Memory](../../06-relationships-and-memory/developers/BOOK.md)
- Durable processing: [Pipelines](../../08-pipelines/developers/BOOK.md)
- Proposals, not auto-install: [Intelligence](../../10-intelligence/developers/BOOK.md)

*End of Memorix Runtime Data Access Patterns (Developers).*

---

# Metadata, Agents, Abstracts & Mappings — developers

# Metadata, Agents, Abstracts & Mappings — Developers

**Audience:** Integrators authoring or consuming declarative Memorix metadata (packs, effective resolution, abstracts, mappings, custom kinds). 
**Twin:** [Builders version](../builders/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Packages:** `@x12i/memorix-metadata`, `@x12i/memorix-metadata-runtime`, `@x12i/memorix-mapping`, `@x12i/memorix-opx-pack`, `@x12i/memorix-service`. 
**Prerequisites:** [Platform overview](../../00-platform-overview/developers/BOOK.md) · [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense product documentation. Deterministic opx fixtures only — **no live AI required**.

---

## How to read this guide

1. Start with the **central mental model** — metadata declares what the platform can run; data is separate.
2. Work chapters in order. Later chapters assume agents, object/content types, and abstracts.
3. Labels used throughout:
 - **Stored** — installed metadata items in the org metadata DB; subject records and links in data DBs
 - **Virtual / calculated** — effective metadata after inheritance; abstract projections; reverse-mapped patches
 - **Operational** — install/publish ledgers, intelligence proposals (not ordinary subject content)
4. Always drill in a fresh `sandbox*` org with `agentIds: ["opx"]`. Never use live/legacy customer orgs as acceptance.

---

## The central mental model

Metadata answers:

> **What kinds of subjects, content, views, transforms, and runs exist for this agent set — and how do they inherit?**

| Layer | Question it answers |
|---|---|
| Agent | Whose vocabulary and rules am I using? |
| Object type | What concrete kind of subject exists? |
| Content type | What kinds of content may that subject wear? |
| Abstract | Which virtual union / projection should callers read? |
| Mapping | How do fields transform deterministically? |
| Install | Which pack version is live for this org? |

Runtime data (`memorix-record/2`) is **not** metadata. Metadata **describes** the shapes and engines; the data plane **stores** subjects. Abstracts are **virtual** — there is no abstract collection and no abstract `recordId`.

```text
Pack (declarative)
 → explicit install / publish
 → effective metadata for agentIds[]
 → drives views, writes, abstracts, pipelines, relationships
 → data plane addresses concrete types (or reverse-writes through abstracts)
```

Ops narrative subjects: employee **Alex**, work item **WI-100**, procedure **PROC-2**. Fixture source: `memorix-opx-pack` + localhost `opx-source`.

---

## 1. Packages and HTTP surface

| Package | Role |
|---|---|
| `@x12i/memorix-metadata` | Declarative contracts, multi-agent resolution, metadata-only abstract unions, custom-kind declarations |
| `@x12i/memorix-metadata-runtime` | Install + publish into an org; unified effective-metadata client |
| `@x12i/memorix-mapping` | Deterministic mapping AST + execution (raw → snapshot, abstract property maps, …) |
| `@x12i/memorix-opx-pack` | Permanent opx acceptance pack (agents, types, abstracts, mappings, sources, pipelines) |
| `@x12i/memorix-service` | HTTP `/api/metadata/*` — the remote boundary |

### When to install which package

| You are… | Prefer |
|---|---|
| Remote host app | HTTP `/api/metadata` only (optional `memorix-format` for record validation) |
| Pack author / platform | `memorix-metadata` contracts + examples |
| Installing a pack into an org | `memorix-metadata-runtime` or `/api/metadata/install` · `/api/metadata/publish` |
| Writing deterministic transforms | `memorix-mapping` (usually inside pipeline steps, not as a host dependency) |

### In-process pack authors (platform only)

Remote services and host apps should stay on HTTP. If you author or validate packs **inside** a trusted platform process, the package APIs are:

| Package | Typical APIs |
|---|---|
| `@x12i/memorix-metadata` | `validateMetadataPack`, `resolveEffectiveMetadata`, `fingerprintDefinition` |
| `@x12i/memorix-metadata-runtime` | `createMetadataClient`, `installMetadataPack` / publish helpers (see package README) |

HTTP remains the product boundary for anything outside that process. Do not pull these packages into a remote UI just to “skip the service.”

### Core HTTP routes

```http
GET /api/metadata/effective
GET /api/metadata/raw
GET /api/metadata/kinds
GET /api/metadata/kinds/{kind}
GET /api/metadata/definition?kind=…&id=…
GET /api/metadata/abstracts
POST /api/metadata/validate
POST /api/metadata/install
POST /api/metadata/publish
GET /api/metadata/packs
```

Every request carries scope:

```http
x-memorix-org-id: sandbox<suffix>
x-memorix-agent-ids: opx
```

Singular `agentId` headers are rejected.

OpenAPI: `GET /api/openapi.json` on a running service.

---

## 2. Pack shape (`memorix-metadata/1`)

A pack is a declarative document (or folder of JSON fragments assembled into one). Conceptual top-level:

```text
agents[]
objectTypes[]
abstractTypes[]
abstractBindings[] # sometimes named bindings
relationshipDefinitions[]
mappings[]
views[]
writes[]
pipelines[]
sources[] / services[] / schedules[] / …
kindDeclarations[]
extensions{} # items keyed by custom kind id
```

Minimal sketch (opx-flavored):

```json
{
 "format": "memorix-metadata/1",
 "agentId": "opx",
 "version": "3.0.0-opx-example",
 "agents": [
 {
 "id": "opx",
 "name": "Operations",
 "parentId": null
 }
 ],
 "objectTypes": [
 {
 "id": "employees",
 "name": "Employees",
 "dataCategory": "entity",
 "contentTypes": [
 { "id": "raw" },
 { "id": "snapshots" }
 ]
 }
 ],
 "abstractTypes": [],
 "abstractBindings": [],
 "relationshipDefinitions": [],
 "mappings": [],
 "views": [],
 "writes": [],
 "kindDeclarations": [],
 "extensions": {}
}
```

**Why this shape:** one agent owns a vocabulary; object types declare concrete subjects; everything else (abstracts, relationships, pipelines) hangs off that vocabulary. Product words live in **tier 2/4** (instance metadata / packs), never hard-coded into tier-1 engines.

---

## 3. Agents and inheritance

### What an agent is

An agent is a **named metadata scope** — not a chatbot and not a user account. `opx` is the permanent parent agent for the opx journey. Child agents may inherit and override for project-specific packs.

Studio shows inheritance clearly:

![Agents inheritance](../screenshots/02-agents-inheritance.png)

### Ordered `agentIds[]` → effective metadata

The service resolves **effective metadata** from the ordered `agentIds[]` on the request:

```ts
// Conceptual
effective = resolveAgents(metadataStore, ["opx" /* , "project-opx", … */]);
```

Rules of thumb:

1. **Order matters.** Later agents override earlier ones for the same `(kind, id)`.
2. **Inheritance fills gaps.** A child that does not redefine `employees` still sees the parent’s `employees`.
3. **Effective ≠ raw.** `GET /api/metadata/raw` shows one agent’s stored items; `GET /api/metadata/effective` shows the resolved set for the whole `agentIds[]` list.
4. **Scope is orthogonal to data.** `orgId` selects the data universe; `agentIds[]` selects vocabulary. Same org can be queried with different agent sets.

### Ops parent agent

From `memorix-opx-pack`:

```json
{
 "id": "opx",
 "name": "Operations (generic parent)",
 "description": "Complete generic parent agent for ordinary operational work. Permanent Memorix acceptance pack.",
 "parentAgentId": null
}
```

Acceptance drills use:

| Field | Value |
|---|---|
| `orgId` | Fresh `sandbox*` (never a live customer org) |
| `agentIds` | `["opx"]` |

See [_shared/opx-scenario.md](../../_shared/opx-scenario.md).

---

## 4. Object types and content types

Object types and content types are **declared in agent metadata**, not in `@x12i/memorix-format`. Format only freezes the envelope (`memorix-record/2`); metadata says which `objectType` / `contentType` pairs are legal for this agent.

![Object & content types](../screenshots/04-object-content-types.png)

### Concrete types in the opx pack

| Object type | Data category | Typical content types |
|---|---|---|
| `employees` | `entity` | `raw`, `snapshots`, `discovery`, `analysis`, … |
| `contractors` | `entity` | `raw`, `snapshots`, … |
| `work-items` | `entity` | `raw`, `snapshots`, `workflow`, `context`, `analysis`, `decisions`, … |
| `procedures` | `knowledge` | `raw`, `snapshots`, … |
| `activity-events` | `event` | `raw`, `snapshots`, … |

### Discriminators

Some content types allow **many instances** on the same subject. Metadata declares a discriminator path (for example `data.analysisType`). Format and data plane treat that as a sibling instance key — not a separate subject.

```json
{
 "id": "analysis",
 "discriminator": {
 "path": "data.analysisType",
 "required": false,
 "defaultValue": "delivery-risk"
 }
}
```

### What belongs where

| Concern | Home |
|---|---|
| Envelope fields (`recordId`, `revision`, …) | `@x12i/memorix-format` |
| Which object/content types exist | Agent metadata (`objectTypes[]`) |
| How UI lists columns | `views[]` (or a custom kind your client owns) |
| How write forms validate | `writes[]` |
| How fields rename | `mappings[]` |

Never fork tier-1 engines to hard-code “employees exist.” Declare them in the pack.

---

## 5. Abstracts — virtual unions only

Hard rule **#4:** abstracts are virtual projections — **no** abstract collections, **no** abstract `recordId`.

![Abstracts & bindings](../screenshots/05-abstracts-bindings.png)

### Why abstracts exist

Callers often want “all people” without knowing whether a subject is an `employees` or `contractors` record. An abstract type (`People`) is a **union projection** over concrete bindings.

```text
People (abstract, virtual)
 ├─ employees (concrete) — snapshots via mapping
 └─ contractors (concrete) — snapshots via mapping
```

There is no `people` Mongo collection and no `recordId` namespace owned by the abstract.

### Abstract type (ops)

```json
{
 "id": "People",
 "name": "People",
 "meaning": "Employees and contractors who perform operational work",
 "category": "entity",
 "properties": [
 { "id": "title", "valueType": "string", "required": true },
 { "id": "role", "valueType": "string", "required": true },
 { "id": "lens", "valueType": "string", "required": false }
 ]
}
```

### Binding (union)

```json
{
 "id": "people-employees-contractors",
 "abstractType": "People",
 "concreteObjectTypes": ["employees", "contractors"],
 "union": true,
 "notes": "People = Employees ∪ Contractors. No abstract records exist.",
 "members": [
 {
 "objectType": "employees",
 "contentType": "snapshots",
 "properties": [
 {
 "abstractProperty": "title",
 "contentType": "snapshots",
 "mappingId": "opx-people-abstract-employees"
 }
 ]
 },
 {
 "objectType": "contractors",
 "contentType": "snapshots",
 "properties": [
 {
 "abstractProperty": "title",
 "contentType": "snapshots",
 "mappingId": "opx-people-abstract-contractors"
 }
 ]
 }
 ]
}
```

Studio Explore can list the union:

![People union](../screenshots/11-people-union.png)

### Reads vs writes

| Operation | API idea | Where it lands |
|---|---|---|
| List / get abstract | `listAbstract` / `getAbstract` (data plane) | Virtual projection over concrete stores |
| Patch abstract | `patchAbstract` | Reverse mapping → **one** concrete writer patch |
| Write concrete | ordinary data write | Concrete `objectType` + `contentType` |

```mermaid
flowchart LR
 Abs[Abstract People] --> Emp[employees]
 Abs --> Con[contractors]
 Patch[patchAbstract] --> Map[reverse mapping]
 Map --> Writer[concrete patch via memorix-data]
```

See also [diagrams/abstract-reverse-write.md](../diagrams/abstract-reverse-write.md).

### Reverse patch rules

1. The caller patches **abstract properties** (for example `title`).
2. Metadata bindings + mappings choose **which concrete object type** and **which content paths** receive the write.
3. Exactly **one** concrete writer target is selected for a given patch context (union members are not written in parallel as “abstract records”).
4. The data plane still enforces ordinary revision / concurrency rules on the concrete content.
5. If reverse mapping cannot resolve a unique writer, the patch fails — do not invent an abstract identity.

### What abstracts are not

- Not a second database
- Not a place to store relationship payloads
- Not a substitute for four-field relationship links
- Not auto-created when you invent a nice name in a host app — they must be declared and installed

---

## 6. Mappings — deterministic transforms

Package: `@x12i/memorix-mapping`.

Mappings are a **pure AST + executor** for field transforms. They are not pipelines, not services, and not relationship discovery.

### When to use a mapping

| Use mapping when… | Prefer something else when… |
|---|---|
| Raw → snapshot field renames | You need external I/O → **service** |
| Abstract property ↔ concrete path | You need ordered retries / ledgers → **pipeline** |
| Constant / path / reshape ops | You need to connect different `recordId`s → **relationship engine** |

### OpX example (source → raw)

Conceptual fragment from the opx pack style:

```json
{
 "id": "opx-source-employees-to-raw",
 "sourceId": "opx-source",
 "endpoint": "/employees",
 "objectType": "employees",
 "contentType": "raw",
 "mode": "knowledge",
 "map": {
 "recordId": { "op": "path", "path": "id", "required": true },
 "objectType": { "op": "constant", "value": "employees" },
 "data.displayName": { "op": "path", "path": "name", "required": true },
 "data.role": { "op": "path", "path": "role", "required": true }
 }
}
```

Pipeline **map** steps invoke the same deterministic core. Host apps rarely call `memorix-mapping` directly; they author mapping ids in metadata and let pipelines / abstract writers resolve them.

### Mapping vs intelligence

Intelligence may **propose** a mapping. Until a human (or explicit API client) **installs** that metadata, the mapping is not live. Deterministic execution never invents mappings at runtime from free-form prompts.

---

## 7. Views, writes, and the client effective-metadata pattern

Integrators should not invent list columns or write forms in the host. Load effective metadata, then drive UI from descriptors.

### Pattern

```text
1. Resolve effective metadata for (orgId, agentIds[])
2. Read views / writes / objectTypes / abstracts / custom kinds
3. Render UI columns and forms from those descriptors
4. Address data only with concrete types (or abstract reverse-write)
5. Never bypass the service with direct Mongo domain writes
```

Example client sketch:

```ts
const effective = await fetch("/api/metadata/effective", {
 headers: {
 "x-memorix-org-id": orgId, // sandbox…
 "x-memorix-agent-ids": "opx",
 },
}).then((r) => r.json());

const employeeSnapshotWrite = effective.writes.find(
 (w) => w.id === "opx-employees-snapshots",
);

// Drive form identity / required fields from employeeSnapshotWrite
// Then POST/PATCH via /api/data against objectType: "employees", contentType: "snapshots"
```

### Built-in views / writes vs custom kinds

Prefer built-in `views` and `writes` when the need is first-class Memorix shape or Studio-compatible UI composition. Use **custom kinds** (next chapter) when another client owns a descriptor vocabulary the platform should only store and resolve.

---

## 8. Built-in vs custom kinds

Platform engines understand a fixed set of **built-in** kinds, including:

`agents`, `object-types`, `abstract-types`, `abstract-bindings`, `relationships`, `mappings`, `views`, `writes`, `pipelines`, `sources`, `services`, `schedules`, … plus `kind-declarations`.

Clients that need product-specific descriptor families **declare** namespaced custom kinds instead of forking tier-1 code.

| Concern | How |
|---|---|
| Declare a kind | `kindDeclarations[]` — id like `acme.ui-tabs`, `name`, `schemaVersion`, optional `itemSchema` |
| Author items | `extensions["acme.ui-tabs"]` — each item needs `id` |
| Store / resolve | Same multi-agent inheritance as built-ins (`kind` + `id`) |
| Catalog | Built-ins keep fixed Catalox catalogs; custom items share `memorix-extensions` |
| Engines | **Do not** interpret custom kinds — data / relationships / pipelines ignore them |

**Kind id rules:**

- Dotted lowercase namespace (`acme.ui-tabs`)
- Not a built-in id
- Not a forbidden operational id (`pipeline-runs`, `leases`, …)

### Worked example

Pack fragment (see also [`memorix-metadata/examples/custom-kinds/acme-ui-tabs.pack.json`](../../../memorix-metadata/examples/custom-kinds/acme-ui-tabs.pack.json)):

```json
{
 "kindDeclarations": [
 {
 "id": "acme.ui-tabs",
 "name": "UI Tabs",
 "description": "Client-owned tab layouts for object detail screens (store/resolve only).",
 "schemaVersion": "acme.uiTabs.v1",
 "itemSchema": {
 "type": "object",
 "required": ["id", "title", "objectType"],
 "properties": {
 "id": { "type": "string" },
 "title": { "type": "string" },
 "objectType": { "type": "string" },
 "tabs": { "type": "array" }
 }
 }
 }
 ],
 "extensions": {
 "acme.ui-tabs": [
 {
 "id": "employee-detail",
 "title": "Employee detail",
 "objectType": "employees",
 "schemaVersion": "acme.uiTabs.v1",
 "tabs": [
 { "id": "overview", "label": "Overview" },
 { "id": "history", "label": "History" }
 ]
 }
 ]
 }
}
```

### HTTP for kinds

```http
GET /api/metadata/kinds
GET /api/metadata/kinds/acme.ui-tabs
GET /api/metadata/definition?kind=acme.ui-tabs&id=employee-detail
```

`GET /api/metadata/kinds` returns `builtIn` registry rows plus effective `custom` kind-declarations for the scope. Studio browsing for custom kinds may lag the API — treat HTTP as the source of truth today.

---

## 9. Install and publish — never auto

Hard product behavior: **Intelligence may propose metadata. Install/publish is always an explicit human or API step.**

![Explicit metadata installation](../screenshots/19-explicit-metadata-installation.png)

### Why never auto-install

- Metadata changes change what engines will run next.
- Accidental install on a shared org is a production incident.
- Proposals can be wrong, incomplete, or unsafe — review is the product.

### What install does / does not

| Install **does** | Install **does not** |
|---|---|
| Write agents, object/content types, abstracts, relationship definitions, sources, services, pipelines, views/writes into the **org metadata DB** | Create subject records (Alex, WI-100, PROC-2, …) |
| Change **effective metadata** for the scoped `agentIds[]` | Pull Memory or run pipelines |
| Make vocabulary legal for `/api/data` and Studio Design | Materialize relationship **instances** or seed subject DBs |
| Record install/publish ledgers (operational) | Replace the need for Operate / data-plane writes |

**One line:** install makes the vocabulary live; subject data still needs Memory/pipeline or `/api/data` writes.

### Routes

| Route | Meaning |
|---|---|
| `POST /api/metadata/validate` | Check a pack without making it live |
| `POST /api/metadata/install` | Install into the org metadata DB for the scoped agents |
| `POST /api/metadata/publish` | Publish / promote a validated set per product policy |
| `GET /api/metadata/packs` | List known packs |

`@x12i/memorix-metadata-runtime` is the in-process helper behind those routes. Host apps that only talk HTTP never need to import it.

### Ops acceptance flow

1. Start fresh `sandbox*` + `agentIds: ["opx"]`.
2. Validate the opx pack (or the subset you are changing).
3. Install explicitly.
4. Fetch `/api/metadata/effective` and confirm `People`, `work-items`, relationship definitions, sources.
5. Only then run Memory / pipelines / relationship verify.

If an intelligence proposal looks wrong, **reject it**. Do not install “to see what happens” on a shared org.

Full propose / profile / artifact flow (how proposals are produced and reviewed): [Structural Intelligence](../../10-intelligence/developers/BOOK.md). This book only owns the **install boundary**: proposals are never live until an explicit install/publish.

---

## 10. Ops narrative walkthrough

This chapter ties metadata to the permanent opx journey.

### Scope

```http
x-memorix-org-id: sandbox7
x-memorix-agent-ids: opx
```

### What the pack declares (subset)

| Kind | Examples |
|---|---|
| Agents | `opx` |
| Object types | `employees`, `contractors`, `work-items`, `procedures`, `activity-events` |
| Abstracts | `People`, `WorkItems`, `Procedures`, `ActivityEvents` |
| Relationships | `people-assigned-to-work-items`, `work-items-use-procedures`, … |
| Sources | `opx-source` (`credentialRef`, localhost fixture) |
| Mappings | source → raw; abstract property maps |
| Pipelines | knowledge pull, context pull, relationship build, analysis, workflow |

### Integrator checklist

1. **Effective metadata loads** for `opx` without errors.
2. **Object types** include employees and work-items with `raw` + `snapshots`.
3. **People** abstract lists both Alex (employee) and any contractor fixtures.
4. **`patchAbstract`** on a People title reverse-writes one concrete snapshot.
5. **Custom kinds** (if any) appear under `/api/metadata/kinds` and are ignored by engines.
6. **Install** was explicit — no intelligence auto-publish occurred.

### Failure modes

| Symptom | Likely cause | Fix |
|---|---|---|
| Empty effective metadata | Pack not installed / wrong agentIds | Install opx pack; check headers |
| Abstract list empty | Bindings missing or concrete content absent | Confirm bindings + seed/pull data |
| `patchAbstract` 4xx | Reverse mapping ambiguous or missing | Fix binding `mappingId` / writer target |
| Custom kind “does nothing” | Expected — engines ignore it | Consume in your client only |
| Proposal “already live” | Mistaken assumption | Proposals are not live until install |

---

## 11. Hard rules for metadata authors

1. Acceptance orgs are fresh `sandbox*` with `agentIds: ["opx"]` — never live customer orgs.
2. Permanent narrative is non-cyber **ops** (employees, work-items, procedures).
3. Abstracts are virtual only — no abstract collections / abstract `recordId`.
4. Object types live in agent metadata, not in format.
5. Custom kinds store/resolve only — engines do not execute them.
6. Install/publish is explicit — intelligence never auto-installs.
7. Mutations of subject data go through the data plane — never direct Mongo domain writes.
8. Deterministic fixtures only for acceptance — no live AI required.

Full list: [HARD-RULES.md](../../HARD-RULES.md).

---

## 12. What not to do

- Do not invent abstract `recordId`s or abstract Mongo collections.
- Do not put copied neighbor payloads into metadata “for convenience.”
- Do not fork `@x12i/memorix-format` to add product object types.
- Do not treat custom kinds as pipeline steps or relationship definitions.
- Do not auto-install intelligence proposals.
- Do not use live/legacy customer orgs as acceptance playgrounds.
- Do not present cyber/vuln narratives as the permanent product story.

---

## 13. Recommended next reading

| Next | Why |
|---|---|
| [Relationships & Memory (developers)](../../06-relationships-and-memory/developers/BOOK.md) | Four-field links + Memory raw landing consume this metadata |
| [Pipelines (developers)](../../08-pipelines/developers/BOOK.md) | Map/validate/write steps execute mappings and writers |
| [Services (developers)](../../07-services/developers/BOOK.md) | When a transform needs I/O instead of a pure mapping |
| [Structural Intelligence (developers)](../../10-intelligence/developers/BOOK.md) | Proposals, profiles, review — never auto-install |
| [Agent packs & ops (developers)](../../11-agent-packs-and-opx/developers/BOOK.md) | Pack authoring and opx-pack deep dive |
| [Builders twin](../builders/BOOK.md) | Studio Design screens and builder workflow |

Tutorial companion: [documentation/tutorials/03-metadata-and-abstracts.md](../../../documentation/tutorials/03-metadata-and-abstracts.md).

---

## Appendix A — Quick reference tables

### Packages

| Package | Install in host? |
|---|---|
| `memorix-metadata` | Only if authoring/resolving packs in-process |
| `memorix-metadata-runtime` | Trusted platform / install tooling |
| `memorix-mapping` | Usually inside platform pipelines, not remote hosts |
| HTTP `/api/metadata` | Preferred remote boundary |

### Abstract vs concrete

| | Abstract | Concrete |
|---|---|---|
| Identity | None (virtual) | `recordId` + objectType |
| Collection | No | Yes (via data plane / DB router) |
| Read | `getAbstract` / `listAbstract` | `getContent` / `listContent` |
| Write | `patchAbstract` → reverse map | Direct content write |

### Built-in vs custom kind

| | Built-in | Custom |
|---|---|---|
| Declared how | Platform registry | `kindDeclarations[]` |
| Items live where | Kind-specific catalogs | `extensions[kindId]` |
| Engines | Interpret | Ignore |
| Inheritance | Yes | Yes |

---

## Appendix B — Glossary (metadata-focused)

| Term | Meaning |
|---|---|
| Agent | Named metadata scope; inherits/overrides |
| Effective metadata | Resolved set for `agentIds[]` |
| Object type | Concrete subject kind (`employees`) |
| Content type | Kind of content about a subject (`snapshots`) |
| Abstract | Virtual union / projection over concrete types |
| Binding | Maps abstract properties to concrete members |
| Mapping | Deterministic field transform AST |
| Kind | Metadata family (`object-types`, `acme.ui-tabs`, …) |
| Install | Explicit step that makes a pack live for an org |
| Proposal | Intelligence suggestion — not live until installed |

Full glossary: [_shared/glossary.md](../../_shared/glossary.md).

---

# Metadata, Agents, Abstracts & Mappings — builders

# Metadata, Agents, Abstracts & Mappings — Builders

**Audience:** Studio authors managing agents, types, abstracts, mappings, and installs. 
**Twin:** [Developers version](../developers/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Prerequisites:** [Platform overview (builders)](../../00-platform-overview/builders/BOOK.md) · [Format for Dummies (builders)](../../01-format-for-dummies/builders/BOOK.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense Studio guide. Deterministic opx fixtures only — **no live AI required**.

---

## How to read this guide

1. Start with the **central idea** — metadata is the vocabulary Studio and engines share.
2. Walk the **Design screens** with the screenshots — that is your daily surface.
3. Use the **builder workflow** and **ops checklist** when installing or changing packs.
4. Always work in a fresh `sandbox*` organization with agent **`opx`**. Never use live customer orgs as playgrounds.

---

## The central idea

Metadata answers:

> **What kinds of things exist here, how do they look when unified, and what is allowed to run?**

| You see in Studio | What it means |
|---|---|
| Agent | Whose vocabulary you are using (`opx`) |
| Object / content types | Concrete subjects and the kinds of information they can wear |
| Abstract | A unified *view* (People) over concrete types — not a separate database |
| Mapping | A deterministic field rename / reshape |
| Install / publish | The deliberate step that makes a pack live |

Runtime records are separate. You design metadata in **Design**, run jobs in **Operate**, and inspect results in **Explore**.

Ops subjects you will meet: employee **Alex**, work item **WI-100**, procedure **PROC-2**.

---

## 1. Scope before everything

In the Studio scope bar:

1. Choose a **fresh** `sandbox*` organization.
2. Set agent to **`opx`** (or the agent set your pack documents).
3. Confirm Design / Operate / Explore are looking at that scope.

Wrong scope is the most common “metadata disappeared” false alarm.

---

## 2. Design screens you will use

### Agents and inheritance

![Agents inheritance](../screenshots/02-agents-inheritance.png)

Agents are named vocabularies, not chatbots. Child agents inherit from parents and can override. What your scope actually uses is the **effective** set after inheritance — not a single file you edit blindly.

For acceptance drills, stay on parent **`opx`**.

### Object and content types

![Object & content types](../screenshots/04-object-content-types.png)

Object types are concrete subjects (`employees`, `work-items`, `procedures`, …). Content types are kinds of information about them (`raw`, `snapshots`, `workflow`, …).

Same subject keeps the same id. Different content kinds become siblings (composition). You do not flatten everything into one blob.

### Abstracts and bindings

![Abstracts & bindings](../screenshots/05-abstracts-bindings.png)

An abstract such as **People** is a virtual union of employees and contractors. There is no separate “people database” and no abstract subject id you can invent.

Bindings say which concrete types participate and how abstract fields map onto concrete snapshot fields.

### People union in Explore

![People union](../screenshots/11-people-union.png)

After metadata is installed and subject content exists, Explore can list **People** as one unified view. Writers still always target a concrete type (or an abstract patch that reverse-maps to one concrete writer).

### Explicit install

![Explicit metadata installation](../screenshots/19-explicit-metadata-installation.png)

Intelligence may **propose** metadata. Proposals are not live until you **install / publish** deliberately. Never expect auto-install.

### What install does / does not

| Install **does** | Install **does not** |
|---|---|
| Make agents, types, abstracts, relationships, sources, services, and pipelines **live** for this org | Create Alex, WI-100, or PROC-2 subject rows |
| Refresh Design so vocabulary appears | Pull Memory or start a pipeline by itself |
| Change what Operate / Explore are allowed to use | Seed fixture data into subject databases |

**One line:** install = vocabulary only. Subject data still comes from Operate (Memory / pipelines) or Explore writes.

---

## 3. Builder workflow (ops)

Use this sequence every time you change vocabulary:

1. **Choose agent scope** — `opx` on a fresh `sandbox*` org.
2. **Define or confirm object/content types** — employees, contractors, work-items, procedures, activity-events.
3. **Bind abstracts** — People = employees ∪ contractors; remember: virtual only.
4. **Author mappings** where transforms are deterministic (source fields → raw/snapshot paths, abstract property maps).
5. **Validate**, then **install / publish** deliberately.
6. **Prove in Explore** — People list, Alex, WI-100 composition — before you run Memory or pipelines for real work.

```text
Design vocabulary
 → validate
 → explicit install
 → Explore proves types/abstracts
 → Operate runs Memory / pipelines / relationships
```

### Tips while authoring

- Prefer clear titles and identifiers over clever nested payloads.
- Keep cyber/vendor vocabulary out of the permanent opx narrative.
- If a proposal looks wrong, **reject it** — do not install “to see what happens” on a shared org.
- Treat abstract names as views, not storage.

---

## 4. Custom kinds (when built-ins are not enough)

Some product clients need their own descriptor vocabulary (for example UI tab packs) that Memorix should **store and inherit**, but **not execute**.

1. **Declare** a namespaced kind (`acme.ui-tabs`) under kind declarations.
2. **Author** items for that kind.
3. **Install** the pack — same explicit install step as other metadata.
4. Read them back via effective metadata / metadata kinds APIs (your host app consumes them).

Use **object types**, **views**, and **writes** for Memorix-native shape and screens. Use **custom kinds** only for client-owned descriptor families. Platform pipelines and the data plane will not run custom kinds for you.

| Need | Use |
|---|---|
| Subject kinds and content kinds | Object / content types |
| Studio-friendly lists and forms | Views / writes |
| Client-only UI descriptors | Custom kinds |
| Field renames | Mappings |
| Connect different subjects | Relationship definitions (next book) |

---

## 5. Abstracts in plain language

Think of abstracts as **saved searches that look like types**.

- **People** means “show me employees and contractors together.”
- Editing a People field in a reverse-write flow updates **one** real employee or contractor record.
- There is no orphan “abstract person” sitting in a hidden table.

If Explore’s People list is empty:

1. Confirm the abstract and bindings are installed for `opx`.
2. Confirm concrete snapshots exist (Alex as an employee, …).
3. Confirm you are in the right `sandbox*` org.

---

## 6. Mappings in plain language

A mapping is a **recipe for renaming and reshaping fields** without calling an external system.

Examples:

- Source `name` → raw `data.displayName`
- Abstract `title` → employee snapshot title path

If the work needs an HTTP call, a model, or retries, that is a **service** or **pipeline** — not a mapping alone.

---

## 7. Ops checklist (builders)

Work subjects: **Alex**, **WI-100**, **PROC-2**.

- [ ] Scope bar shows fresh `sandbox*` + agent `opx`
- [ ] Design → Agents shows `opx` inheritance as expected
- [ ] Design → Types lists employees, contractors, work-items, procedures, activity-events
- [ ] Design → Abstracts shows People bound to employees + contractors
- [ ] Mappings for source/abstract paths are present (or intentionally unchanged)
- [ ] Metadata validated without blocking errors
- [ ] **Install / publish completed deliberately** (screenshot `19` style confirmation)
- [ ] Explore → People lists expected subjects once data exists
- [ ] No intelligence proposal was left “assumed installed”

When something fails, capture: org, agents, abstract or type id, install action time, and whether you were looking at raw vs effective metadata — then escalate with that packet.

---

## 8. Rules builders enforce

- Fresh `sandbox*` + agent `opx` for acceptance and drills.
- No live customer orgs as playgrounds.
- Abstracts are views — you always write to a **concrete** type (directly or via reverse-write).
- Intelligence proposes; humans install.
- Custom kinds do not become secret engines.
- Deterministic fixtures for acceptance — do not require live AI to prove metadata.
- Permanent narrative stays non-cyber ops.

Hard rules (full): [HARD-RULES.md](../../HARD-RULES.md).

---

## 9. What not to expect

- An abstract subject id you can create in Explore
- A separate “people” database
- Auto-install of intelligence proposals
- Custom kinds that pipelines magically execute
- Metadata changes that appear in another org without installing there
- Nested “associated employee” blobs as a substitute for relationships

---

## 10. Recommended next reading

1. [Relationships & Memory (builders)](../../06-relationships-and-memory/builders/BOOK.md) — sources, pulls, link definitions
2. [Workflow & Studio (builders)](../../09-workflow-and-studio/builders/BOOK.md) — Operate / Explore depth
3. [Pipelines (builders)](../../08-pipelines/builders/BOOK.md) — how mappings run inside steps
4. [Agent packs & ops (builders)](../../11-agent-packs-and-opx/builders/BOOK.md) — pack authoring
5. API-level detail → [developers twin](../developers/BOOK.md)

Tutorial companion: [documentation/tutorials/03-metadata-and-abstracts.md](../../../documentation/tutorials/03-metadata-and-abstracts.md).

---

# Relationships & Memory — developers

# Relationships & Memory — Developers

**Audience:** Engineers working with four-field relationship links and source landing via Memory. 
**Twin:** [Builders version](../builders/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Packages:** `@x12i/memorix-relationships`, `@x12i/memorix-relationship-store`, `@x12i/memorix-memory`, `@x12i/memorix-data`, `@x12i/memorix-opx-pack`, `@x12i/memorix-service`. 
**Prerequisites:** [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md) · [Metadata, Agents, Abstracts](../../05-metadata-agents-abstracts/developers/BOOK.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense product documentation. Deterministic opx fixtures only — **no live AI required**.

---

## How to read this guide

1. Start with the **central mental model** — Memory lands exact bytes; relationships navigate subjects; neither copies neighbor payloads.
2. Read **Memory** before **Relationships** if you are wiring ingest; reverse the order if you already have subject content.
3. Labels used throughout:
 - **Stored** — `*-raw` content; four-field relationship links; operational ledgers
 - **Virtual / calculated** — relationship expansion on read; composed subject views
 - **Operational** — memory-pull runs/cursors/quarantine; relationship job evidence (not ordinary subject content)
4. Always drill in a fresh `sandbox*` org with `agentIds: ["opx"]`. Never use live/legacy customer orgs as acceptance.

---

## The central mental model

Two complementary engines:

> **Memory remembers what the source said. Relationships remember who links to whom.**

| Concern | Package | Stored shape |
|---|---|---|
| Exact source landing | `@x12i/memorix-memory` | `<objectType>-raw` via data plane |
| Navigation between subjects | `@x12i/memorix-relationships` | Four-field link only |
| Link persistence | `@x12i/memorix-relationship-store` | Candidate C, org-scoped per data category |

Hard rules **#3** and **#8**:

- Relationships are **four-field links only** — no root `associated*` storage, no copied payloads.
- Mutations go through the data plane / trusted writers — never direct Mongo domain writes.

```text
opx-source (credentialRef)
 → Memory pull lands exact raw (*-raw)
 → Pipeline promotes to snapshots / siblings
 → Relationship engine discover → materialize → verify
 → Data reads may expand links virtually
```

See [diagrams/memory-to-links.md](../diagrams/memory-to-links.md).

Ops subjects: employee **Alex**, work item **WI-100**, procedure **PROC-2**.

---

## 1. Packages and HTTP surface

| Package | Role |
|---|---|
| `@x12i/memorix-memory` | Scheduled / on-demand source pulls; lands exact provider payloads as raw |
| `@x12i/memorix-relationships` | **Only** trusted production writer of relationship links |
| `@x12i/memorix-relationship-store` | Logical link contract, deterministic ids, Candidate C Mongo adapter |
| `@x12i/memorix-data` | Writes raw/subject content; virtual relationship expansion on read |
| `@x12i/memorix-service` | HTTP `/api/memory/*`, `/api/relationships/*`, `/api/data/*` |

### Install guidance

| You are… | Memory | Relationships |
|---|---|---|
| Remote host app | HTTP `/api/memory` | HTTP `/api/relationships` (verify / rebuild / query) |
| Trusted platform | May install `memorix-memory` | May install `memorix-relationships` (sole writer) |
| Tests seeding links | — | Store `upsertBatch` only inside trusted tests — not product APIs |

### Scope headers

```http
x-memorix-org-id: sandbox<suffix>
x-memorix-agent-ids: opx
```

---

## 2. Memory — exact source landing

**Package:** `@x12i/memorix-memory` 
**HTTP:** `/api/memory`

Memory is a **collector**, not a promoter. It pulls from declared sources and lands **exact** provider payloads as `<objectType>-raw` through `@x12i/memorix-data`.

### Knowledge vs context

| Mode | Retention | Landing | Typical use |
|---|---|---|---|
| `knowledge` | Persistent; content-hash dedup | `<objectType>-raw` | Employees, work-items, procedures from opx-source |
| `context` | Work-scoped TTL / discard | `<objectType>-raw` | Short-lived work-item context pulls |

Both modes land **raw**. Promotion to snapshots (or other subject content) is a **pipeline** concern, not Memory.

### Boundaries (non-goals)

Memory:

- Is **read-only** toward sources (connectors do not mutate the provider).
- Accepts secrets via **`credentialRef` only** — never raw secrets in request bodies.
- **Stops at raw handoff** — no promote, no analysis, no relationship discovery inside Memory.
- Writes operational ledgers in `{orgId}_operational`:
 - `memory-pull-runs`
 - `memory-pull-cursors` (source checkpoints)
 - `memory-quarantine`

### Source checkpoints

For every **organization × source × object type × route**, Memorix stores the position reached in the external source during the last successful pull.

| Method | When |
|--------|------|
| `cursor` | Provider-issued continue token |
| `watermark` | Last timestamp / version / sequence / ID |
| `fullScanHash` | No incremental API — re-read all; content-hash dedup |

Rules:

- Declared on the route (`checkpoint.method`); runtime state is per-org.
- Advance only after corresponding raw is stored successfully.
- After failure, resume from the last committed checkpoint.
- Manual and scheduled pulls share the same checkpoint.
- Reset clears the position — the next pull starts from the beginning.
- Studio Operate → Memory shows a **Source checkpoints** table (`Saved` / `Fingerprints saved` / `Not established`).

HTTP (narrow API for Studio + called services):

| Method | Path |
|--------|------|
| GET | `/api/memory/sources/{sourceId}/checkpoints` |
| PUT | `/api/memory/sources/{sourceId}/checkpoints` |
| POST | `/api/memory/sources/{sourceId}/checkpoints/reset` |
| POST | `/api/memory/sources/{sourceId}/commit-page` |

SDK: `@x12i/memorix-sdk/client` → `client.checkpoints.*` and `client.memory.commitPage` (atomic land + checkpoint). `@x12i/memorix-client` remains the HTTP foundation underneath. Standalone `checkpoints.put` is administrative — not proof that raw landed.

```mermaid
flowchart LR
 Src[opx-source] -->|credentialRef| Pull[Memory pull]
 Pull --> Raw["*-raw via memorix-data"]
 Pull --> Ledger[operational ledgers]
 Raw --> Pipeline[Pipelines consume]
```

### Source declaration (opx pack)

Conceptual source shape:

```json
{
 "id": "opx-source",
 "name": "Ops fixture source",
 "kind": "rest-fixture",
 "defaultBaseUrl": "http://127.0.0.1:5102",
 "credentialRef": "opx-source-local",
 "endpoints": [
 "/employees",
 "/contractors",
 "/work-items",
 "/activity-events",
 "/procedures",
 "/work-items/:id/context"
 ]
}
```

Studio Design / Operate surfaces sources and pulls:

![Sources / Memory](../screenshots/03-sources-memory.png)

### credentialRef

| Do | Do not |
|---|---|
| Reference a configured credential id (`opx-source-local`) | Paste API keys into pull bodies |
| Rotate credentials in the secret store | Embed secrets in metadata packs checked into git |
| Fail closed when the ref is missing | Fall back to “temporary plaintext for debugging” in shared orgs |

### Raw landing example

After a knowledge pull for employees, expect content addressed roughly as:

```text
objectType: employees
contentType: raw
recordId: <stable subject id from source mapping, e.g. alex’s id>
data: { …exact provider payload fields per mapping… }
```

The **exact** bytes (or canonical structured raw) matter later for provenance (`docHash` in KnowX) and for proving what arrived.

### Failure modes

| Symptom | Likely cause | Fix |
|---|---|---|
| Pull rejected for secrets | Raw secret in body | Use `credentialRef` only |
| Quarantine entries | Mapping / identity failure on land | Fix source mapping; retry pull |
| No `*-raw` after “success” | Wrong org/agent scope | Confirm `sandbox*` + `opx` |
| Expecting snapshots from Memory | Misunderstood boundary | Run the promote pipeline |

### Tutorial-shaped flow

See [documentation/tutorials/04-memory-and-pipeline.md](../../../documentation/tutorials/04-memory-and-pipeline.md):

```text
Source → /api/memory → *-raw → /api/pipelines → snapshots → /api/data
```

---

## 3. Relationships — four-field links only

**Trusted writer:** `@x12i/memorix-relationships` only. 
**Store:** `@x12i/memorix-relationship-store` (Candidate C, org-scoped per data category).

Hard rule **#3**: links remember endpoints and meaning — they never copy target payloads.

### The only stored link shape

```ts
{
 relationshipId: string; // deterministic
 relationshipDefinitionId: string; // which definition produced this link
 fromRecordId: string;
 toRecordId: string;
}
```

Nothing else belongs on the persisted link:

- No nested employee snapshots
- No job context blobs
- No revision copies of either side
- No `associated*` root buckets on subject records

Job context and revisions live in **operational evidence**, captured via data-plane expansion when the engine needs them — never denormalized onto the link.

### Why four fields

Older shapes nested full related records under `associated*`. That duplicated data, mixed navigation with evidence, and made writes unsafe. The new path:

```text
Same recordId → composition (siblings)
Different recordIds → relationship (four-field link)
```

### Definition (metadata) vs link (data)

| Layer | Example |
|---|---|
| Relationship **definition** (metadata) | `people-assigned-to-work-items` |
| Materialized **link** (data) | `{ relationshipId, relationshipDefinitionId, fromRecordId: "…Alex…", toRecordId: "wi-100" }` |

Definitions are authored in packs and installed with other metadata. Links are produced by the engine.

Studio Design shows definitions:

![Relationship definition](../screenshots/06-relationship-definition.png)

### Ops definitions (subset)

From `memorix-opx-pack`:

| Id | Meaning |
|---|---|
| `people-assigned-to-work-items` | Person assigned to perform a work item |
| `work-items-use-procedures` | Work item follows reusable procedures |
| `activity-events-about-work-items` | Activity event reports progress on a work item |
| `work-items-depend-on-work-items` | Work item depends on another work item |
| `people-collaborate-with-people` | Two people collaborate (undirected) |

Example definition sketch:

```json
{
 "id": "people-assigned-to-work-items",
 "name": "People assigned to work items",
 "meaning": "A person (employee or contractor) is assigned to perform a work item",
 "directionality": "directed",
 "from": { "abstractType": "People", "property": "assignedWorkItems" },
 "to": { "abstractType": "WorkItems", "property": "assignedPeople" },
 "discoveryRules": [
 {
 "id": "from-source-assignee",
 "kind": "source-field",
 "sourcePath": "assigneeId",
 "notes": "opx-source work-items.assigneeId resolves across employees and contractors"
 }
 ]
}
```

Abstract endpoints (`People`, `WorkItems`) compile to concrete binding plans — the engine does not invent Cartesian products.

---

## 4. Engine lifecycle — discover / verify / rebuild / cleanup

The relationship engine owns a deliberate lifecycle. Product callers do **not** PUT/DELETE individual links.

### Responsibilities

1. **Compile** abstract relationship definitions → concrete binding plans.
2. **Discover** candidate links (bounded, non-Cartesian).
3. **Materialize** idempotent four-field links via the store.
4. **Verify** existing links against current definitions and subject content.
5. **Rebuild** a definition’s links when definitions or source evidence change.
6. **Cleanup** stale links for a definition according to policy.

Conceptual trusted call:

```ts
import { createRelationshipEngine } from "@x12i/memorix-relationships";

await engine.discoverAndMaterialize(scope, "people-assigned-to-work-items");
```

### HTTP surface

| Capability | Via |
|---|---|
| List definitions | `GET /api/relationships/definitions` (and related) |
| Verify / rebuild / query | `/api/relationships` operator routes |
| Ad-hoc link PUT/DELETE | **405** — forbidden for product callers |

There is:

- **No** public manual put/delete relationship API for product callers
- **No** auto-run on metadata change (installing a definition does not silently rewrite all links)
- **No** AI on the deterministic path

### Discover (bounded)

Discovery rules read evidence (often source fields on raw/snapshots) and resolve endpoints across concrete types that bind to the abstract sides.

Guarantees you should expect:

- Candidates are **bounded** — not “every person × every work item.”
- Unresolved targets are **first-class** (quarantined / reported), not silently invented.
- Repeated discover+materialize is **idempotent** for the same logical link (`relationshipId` is deterministic).

### Verify / rebuild / cleanup

| Operation | Intent |
|---|---|
| Verify | Compare materialized links to what discovery would produce now; report drift |
| Rebuild | Re-run discover+materialize for one definition; reconcile store |
| Cleanup | Remove stale links for one definition per policy |

Always operate **one definition at a time** unless a higher-level pipeline explicitly sequences multiple definitions.

### Pipeline pairing (ops)

The opx pack includes an explicit relationship pipeline pattern:

```text
relationship-discover → relationship-write → relationship-verify
```

Pipelines **invoke** the engine; they do not become a second link writer.

---

## 5. Reads — virtual expansion

`@x12i/memorix-data` can expand relationships **virtually** at read time:

```ts
// Conceptual read options
{
 relationships: {
 include: true,
 depth: 1,
 expandContentTypes: ["snapshots"], // optional projection of targets
 },
}
```

Rules:

- Expansion is **calculated** for the response — it does not paste payloads into stored links.
- Unresolved targets remain visible as unresolved (Studio shows this clearly).
- Expanding content types is a read convenience; writers still address concrete content separately.

![Link traversal / expansion](../screenshots/07-link-traversal-expansion.png)

![Unresolved relationship](../screenshots/08-unresolved-relationship.png)

### Composition vs relationship on read

| Same `recordId` | Different `recordId`s |
|---|---|
| Composition / siblings | Relationship expansion |
| `getContent` / compose APIs | `relationships.include` |
| Snapshot + workflow + analysis on WI-100 | Alex → WI-100 assignment link |

---

## 6. Opx narrative walkthrough

### Scope

```http
x-memorix-org-id: sandbox7
x-memorix-agent-ids: opx
```

### Happy path

1. **Metadata installed** for `opx` (sources, object types, relationship definitions).
2. **Memory knowledge pull** from `opx-source` using `credentialRef` → `employees-raw`, `work-items-raw`, …
3. **Pipeline promote** → snapshots for Alex, WI-100, PROC-2, …
4. **Relationship discover/materialize** for `people-assigned-to-work-items` (and others as needed).
5. **Verify** the definition — no unexpected drift.
6. **Read** WI-100 or Alex with `relationships.include: true` — navigation only, no copied payloads.
7. If a target is missing, Explore shows **unresolved** — fix data or rebuild; do not hand-edit links.

### Integrator checklist

- [ ] Source uses `credentialRef`, not pasted secrets
- [ ] `*-raw` exists before you expect snapshots
- [ ] Snapshots exist on both sides before you expect links
- [ ] Relationship definition installed for the agent
- [ ] Links created via discover/materialize or Operate verify/rebuild — never ad-hoc PUT
- [ ] Expansion shows four-field navigation; unresolved list explained or empty
- [ ] No `associated*` blobs on subject records

### Failure modes

| Symptom | Likely cause | Fix |
|---|---|---|
| 405 on link PUT/DELETE | Calling forbidden ad-hoc API | Use verify/rebuild/query only |
| Empty links after promote | Discovery not run / wrong definition id | Run discover+materialize for that definition |
| Unresolved targets | Missing subject content or bad source ids | Seed/fix data; rebuild |
| Cartesian explosion fear | Misread discovery | Confirm bounded rules; report if unbounded |
| Secrets in quarantine | Credential misuse | Switch to `credentialRef` |

Tutorial companion: [documentation/tutorials/05-relationships.md](../../../documentation/tutorials/05-relationships.md).

---

## 7. Store details (Candidate C)

Product callers should not depend on Mongo collection names. For platform engineers:

- Links are org-scoped and separated by data category as implemented by the Candidate C adapter.
- `relationshipId` is **deterministic** from definition + endpoints (and engine identity rules).
- The in-memory oracle exists for tests; production path is Mongo via the store adapter.
- Seeding via `upsertBatch` is for **trusted tests**, not for host apps.

If you find yourself designing a “manual link editor” for operators, stop — expose verify/rebuild/cleanup and unresolved diagnostics instead.

---

## 8. How Memory and Relationships relate to other planes

```mermaid
flowchart TB
 Meta[Metadata: sources + relationship definitions]
 Mem[Memory: raw landing]
 Pipe[Pipelines: promote / map / validate]
 Rel[Relationship engine]
 Data[Data plane reads/writes]
 Meta --> Mem
 Meta --> Rel
 Mem --> Data
 Pipe --> Data
 Pipe --> Rel
 Rel --> Store[relationship-store]
 Data --> Expand[virtual expansion]
 Store --> Expand
```

| Plane | Owns |
|---|---|
| Metadata | Source defs, relationship defs, mappings |
| Memory | Exact raw landing + pull ledgers |
| Pipeline | Promote, analysis, ordered relationship jobs |
| Relationships | Discover/materialize/verify/rebuild/cleanup |
| Data | Subject content + virtual expansion |
| Intelligence | May propose defs — never auto-install; never write links |

---

## 9. Hard rules (relationships & memory)

1. Acceptance orgs are fresh `sandbox*` with `agentIds: ["opx"]`.
2. Permanent narrative is non-cyber **ops**.
3. Relationships are **four-field links only** — no `associated*` storage, no copied payloads.
4. Abstracts stay virtual — relationship definitions may *reference* abstracts; links still store concrete `recordId`s.
5. Mutations go through the data plane / trusted writers — never direct Mongo domain writes.
6. Memory secrets via **`credentialRef` only**.
7. Memory stops at raw — no promote / analysis / relationship discovery inside Memory.
8. No ad-hoc product link PUT/DELETE (**405**).
9. Deterministic fixtures only for acceptance — no live AI required.

Full list: [HARD-RULES.md](../../HARD-RULES.md).

---

## 10. What not to do

- Do not PUT/DELETE product links by hand.
- Do not copy related payloads onto source records “for the UI.”
- Do not auto-run relationship rebuild on every metadata save without an explicit Operate/API action.
- Do not put raw secrets in Memory pull bodies.
- Do not expect Memory to emit snapshots.
- Do not treat unresolved links as silent successes — surface them.
- Do not use live/legacy customer orgs as acceptance playgrounds.
- Do not invent a second graph database beside Memorix links + KnowX content.

---

## 11. Recommended next reading

| Next | Why |
|---|---|
| [Pipelines (developers)](../../08-pipelines/developers/BOOK.md) | Promote raw → snapshots; sequence relationship jobs |
| [Services (developers)](../../07-services/developers/BOOK.md) | When connectors need custom I/O beyond Memory fixtures |
| [KnowX (developers)](../../12-knowx/developers/BOOK.md) | Claims + provenance; materialize navigation from accepted edges |
| [Metadata (developers)](../../05-metadata-agents-abstracts/developers/BOOK.md) | Definitions, abstracts, install |
| [Builders twin](../builders/BOOK.md) | Studio Operate / Explore checklist |

---

## Appendix A — Quick reference

### Memory modes

| Mode | Retention | Stops at |
|---|---|---|
| `knowledge` | Persistent + content-hash dedup | `*-raw` |
| `context` | TTL / discard on work completion | `*-raw` |

### Relationship lifecycle verbs

| Verb | Product meaning |
|---|---|
| Discover | Find candidates from rules + evidence |
| Materialize | Upsert four-field links idempotently |
| Verify | Detect drift vs current discovery |
| Rebuild | Reconcile one definition’s links |
| Cleanup | Remove stale links for one definition |
| Expand | Virtual read-time navigation |

### Forbidden

| Action | Result |
|---|---|
| Ad-hoc link PUT/DELETE via service | **405** |
| Raw secrets in Memory bodies | Rejected |
| Direct Mongo domain writes | Forbidden (hard rule #8) |
| Copied `associated*` payloads | Forbidden (hard rule #3) |

---

## Appendix B — Glossary (this book)

| Term | Meaning |
|---|---|
| Memory | Collector that lands exact source payloads as raw |
| `credentialRef` | Indirection to a secret — never the secret itself |
| Knowledge mode | Persistent raw landing with content-hash dedup |
| Context mode | Work-scoped TTL raw landing |
| Relationship definition | Metadata describing how two abstract sides connect |
| Four-field link | `relationshipId`, `relationshipDefinitionId`, `fromRecordId`, `toRecordId` |
| Discover | Bounded candidate search |
| Expand | Virtual inclusion of links (and optional target content) on read |
| Unresolved | Link or candidate whose target subject is missing |

Full glossary: [_shared/glossary.md](../../_shared/glossary.md).

---

# Relationships & Memory — builders

# Relationships & Memory — Builders

**Audience:** Studio users configuring sources, pulls, and relationship definitions. 
**Twin:** [Developers version](../developers/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Prerequisites:** [Format for Dummies (builders)](../../01-format-for-dummies/builders/BOOK.md) · [Metadata (builders)](../../05-metadata-agents-abstracts/builders/BOOK.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense Studio guide. Deterministic opx fixtures only — **no live AI required**.

---

## How to read this guide

1. Start with the **plain-language** sections — Memory and Relationships in everyday words.
2. Match the **screenshots** to Design / Operate / Explore as you click through.
3. Use the **checklist** at the end every time you accept a pack or debug missing links.
4. Always work in a fresh `sandbox*` organization with agent **`opx`**.

---

## The central idea

> **Memory keeps the exact source material. Relationships only remember who connects to whom.**

| Piece | Plain meaning |
|---|---|
| Memory pull | “Fetch what the source said and store it as raw proof.” |
| Pipeline | “Turn raw proof into business snapshots and other content.” |
| Relationship definition | “How two kinds of subjects are allowed to connect.” |
| Link | “This subject points to that subject for this reason” — four fields, no pasted neighbor data |
| Unresolved | “We expected a target subject, but it is not there yet.” |

Ops subjects: employee **Alex**, work item **WI-100**, procedure **PROC-2**.

---

## 1. Scope bar first

Before pulls or verify/rebuild:

1. Fresh **`sandbox*`** organization.
2. Agent **`opx`**.
3. Confirm Design / Operate / Explore share that scope.

Wrong scope looks like “Memory ran but Explore is empty.”

---

## 2. Memory in plain language

Memory stores the **exact** bytes (or structured raw) from a source so you can always prove what arrived. Pipelines turn those bytes into subject content. Memory itself does **not** “finish” the business record.

![Sources / Memory](../screenshots/03-sources-memory.png)

### Knowledge vs context (builder wording)

| Mode | How long it keeps raw | When you use it |
|---|---|---|
| **Knowledge** | Keeps it (deduped by content hash) | Ordinary opx-source subjects — people, work items, procedures |
| **Context** | Temporary (TTL / discard when work completes) | Short-lived work-item context pulls |

Both land as **raw**. Snapshots appear only after a pipeline promotes them.

### What success looks like

After a pull, Studio Operate → Memory shows **Source checkpoints** for each object type / route: **Saved**, **Fingerprints saved**, or **Not established — the next pull starts from the beginning.** You can reset a checkpoint so the next pull starts over.

### Credential references

Sources use a **credential reference** (`credentialRef`) — a pointer to a secret — never a pasted API key in the pull form or in a shared pack.

If a pull fails because of secrets, fix the credential configuration. Do not paste the secret “just this once” into a shared org.

### What success looks like

1. Operate shows a Memory pull run for `opx-source`.
2. Explore (or technical details) shows `*-raw` for the object type.
3. A pipeline run then creates snapshots for Alex / WI-100 / PROC-2.
4. You can still open raw later to prove what the source said.

### What Memory will not do

- Will not invent snapshots by itself
- Will not discover relationship links
- Will not call analysis engines
- Will not accept raw secrets in the request body

---

## 3. Relationships in plain language

A relationship **definition** says *how* two kinds of subjects connect (for example People assigned to Work items). After runs, Explore lets you walk **links**. Links remember endpoints and meaning — they do **not** paste the other record’s fields into this one.

![Relationship definition](../screenshots/06-relationship-definition.png)

### The only link shape (why builders care)

Stored links have four fields:

```text
relationship id
definition id
from subject id
to subject id
```

If a screen shows a full nested “associated employee” blob on a work item, that is the old wrong model — escalate; do not teach it as normal.

### Traverse and expand

![Link traversal / expansion](../screenshots/07-link-traversal-expansion.png)

In Explore you walk from Alex to WI-100 (or the reverse) using links. Expansion is a **read-time** convenience. It does not rewrite stored records to embed neighbors.

### Unresolved

![Unresolved relationship](../screenshots/08-unresolved-relationship.png)

If a target is missing, Studio shows **unresolved**. That is a first-class outcome:

- Fix missing subject content (pull / pipeline), or
- Rerun **verify / rebuild** from Operate for that definition,

…rather than hand-editing a link.

---

## 4. Builder workflow (ops)

```text
Design: confirm source + relationship definitions installed
 → Operate: Memory pull (credentialRef)
 → Operate: promote pipeline
 → Operate: relationship verify / rebuild for a definition
 → Explore: walk links; check unresolved list
```

### Design

1. Confirm **opx-source** exists and uses a credential reference.
2. Confirm relationship definitions you care about (assignment, procedures, activity, dependencies, collaboration) are installed for `opx`.
3. Do not assume installing metadata auto-rebuilds all links — rebuild is an explicit Operate action.

### Operate

1. Run a **Memory** knowledge pull; confirm the run and any quarantine.
2. Run the **promote / knowledge** pipeline until snapshots exist for both sides of the link.
3. Run **relationship verify** (or rebuild) for `people-assigned-to-work-items` (then others as needed).
4. Capture run / attempt ids when something fails.

### Explore

1. Open **Alex** and **WI-100**.
2. Confirm composition siblings (snapshot, workflow, …) as expected.
3. Traverse assignment links — navigation only.
4. Open unresolved list — empty or explained.

---

## 5. Ops checklist (builders)

Work subjects: **Alex**, **WI-100**, **PROC-2**.

### Memory

- [ ] Scope bar: fresh `sandbox*` + agent `opx`
- [ ] Source uses **credential reference**, not pasted secrets
- [ ] Memory pull run visible in Operate
- [ ] `*-raw` landed for expected object types
- [ ] Promote pipeline produced snapshots (Memory alone is not enough)

### Relationships

- [ ] Relationship definition installed for the agent
- [ ] Both sides have subject content before you expect links
- [ ] Verify / rebuild run from Operate (no hand-made links)
- [ ] Explore shows link traversal for assignment (and others you care about)
- [ ] Unresolved list is empty **or** each item is understood
- [ ] No nested “associated payload” teaching moment left unexplained

When something fails, capture: org, agents, source id, pull run id, relationship definition id, subject ids, and whether the failure was Memory, pipeline, or relationship verify — then escalate with that packet.

---

## 6. Rules builders enforce

- Fresh `sandbox*` + agent `opx` for acceptance and drills.
- No live customer orgs as playgrounds.
- Relationships never copy the other record’s payload into this one.
- Memory lands raw proof; pipelines finish business content.
- Secrets only via credential references.
- Unresolved links are investigated, not ignored.
- Intelligence may propose definitions; humans install; Operate rebuilds deliberately.
- Deterministic fixtures for acceptance — do not require live AI to prove links.

Hard rules (full): [HARD-RULES.md](../../HARD-RULES.md) — especially **#3** (four-field links) and **#8** (data-plane mutations).

---

## 7. What not to expect

- Memory that “also creates snapshots”
- A button that pastes full related records into a subject
- Ad-hoc create/delete of individual product links (the platform rejects that path)
- Auto-rebuild of all links the moment metadata is installed
- Empty Explore while looking at a different org than the one you pulled into
- Cyber/vuln stories as the permanent product narrative

---

## 8. Recommended next reading

1. [Pipelines (builders)](../../08-pipelines/builders/BOOK.md) — promote and relationship job steps
2. [Workflow & Studio (builders)](../../09-workflow-and-studio/builders/BOOK.md) — Operate depth
3. [KnowX (builders)](../../12-knowx/builders/BOOK.md) — when links carry knowledge-claim semantics
4. [Metadata (builders)](../../05-metadata-agents-abstracts/builders/BOOK.md) — definitions and install
5. API-level detail → [developers twin](../developers/BOOK.md)

Tutorials: [04 — Memory → pipeline](../../../documentation/tutorials/04-memory-and-pipeline.md) · [05 — Relationships](../../../documentation/tutorials/05-relationships.md).

---

# Building Services — developers

# Building Memorix Services — Developers

**Audience:** Service, connector, pipeline, agent-package, and AI-job developers. 
**Twin:** [Builders version](../builders/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Packages:** `@x12i/memorix-metadata`, `@x12i/memorix-pipeline`, `@x12i/memorix-pipeline-runtime`, `@x12i/memorix-mapping`, `@x12i/memorix-memory`, `@x12i/memorix-data`, `@x12i/memorix-service`, `@x12i/memorix-opx-pack`, `@x12i/core-service`, `@x12i/ports-manager`, `@x12i/api-live-view`. 
**Prerequisites:** [Platform overview](../../00-platform-overview/developers/BOOK.md) · [Format](../../01-format-for-dummies/developers/BOOK.md) · [Metadata & mappings](../../05-metadata-agents-abstracts/developers/BOOK.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense product documentation for the Memorix service model. Deterministic opx fixtures only — **no live AI required**.

---

## How to read this guide

1. Start with the **central mental model** — one service model, roles as capabilities.
2. Use the **decision guide** before writing code; most “service” requests are mappings, Memory, relationships, or pipelines.
3. Work **define → implement → bind → invoke → operate** in order.
4. Remember services are **side B** in the authoring order (vocabulary → services → pipeline) — see [Getting started — three sides](../../14-getting-started-and-tutorials/developers/BOOK.md).
5. Labels used throughout:
 - **Stored** — subject content or relationship links
 - **Virtual / calculated** — composition, abstracts, expansions
 - **Operational** — runs, attempts, artifacts, receipts (not ordinary subject content)
6. Every example uses a fresh `sandbox*` org, agent `opx`, and subjects **Alex**, **WI-100**, **PROC-2**.

---

## The central mental model

A Memorix service is a **named capability** that performs one logical operation for a pipeline or platform workflow.

> **Services perform capabilities. Pipelines orchestrate them. Canonical ports persist their results. Metadata describes them. Deployment binds them. Operational ledgers explain what happened.**

A service may:

- pull external data (read-only toward the source),
- call an external system,
- validate or calculate,
- enrich existing subjects,
- produce relationship candidates,
- analyze, recommend, narrate, or complete fields,
- perform an external action,
- execute an AI job through the single model adapter.

A service does **not** own orchestration, scheduling, retries, persistence, metadata installation, relationship storage, or Mongo access. Those belong to pipeline runtime, Memory, the data plane, the relationship engine, and metadata runtime.

### One service model

There is one declarative `services` metadata kind. Roles such as collect, enrich, resolve, action, and AI are **classifications** — not separate product frameworks.

Every service uses:

| Shared surface | Role |
|---|---|
| One logical definition | Agent metadata (`services[]` with pack field `id`) |
| One registry | `ServiceRef` resolve (`serviceId`) — binding only |
| One invocation contract | `PortHandler(StepExecutionRequest)` on a `StepKind` |
| One pipeline runtime | `@x12i/memorix-pipeline` + runtime ledger |
| One artifact / DataRef model | Large payloads leave the ledger |
| One scope model | `{ orgId, agentIds[] }` unchanged |

### Logical definition versus deployment

**Service metadata** (stable, installable):

- `id`, `name`, `description`
- `capabilities[]` (roles)
- logical `inputs` / `outputs` (`schemaRef`)
- `invocation` (`kind`: `http` | `worker` | `internal`, optional `operation`)

**Runtime / deployment** (environment-specific, never in Catalox secrets):

- endpoint or in-process handler
- `credentialRef` resolution
- timeout, concurrency, build identity
- health / readiness

Secrets, URLs, and deployment state do **not** belong in agent metadata. Metadata names a logical service; the registry binds it.

### Scope

Every invocation receives:

```ts
{
 orgId: string; // complete organization data universe — e.g. sandbox7a3c
 agentIds: string[]; // effective metadata — e.g. ["opx"]
}
```

- `orgId` selects the data universe.
- `agentIds[]` selects effective metadata; it does **not** filter organization data.
- A service must not invent its own scope interpretation.
- Pass scope unchanged to canonical ports (`@x12i/memorix-data`, relationships, Memory).

HTTP equivalent:

```http
x-memorix-org-id: sandbox7a3c
x-memorix-agent-ids: opx
```

Studio mirrors the same catalog and binding story:

![Services in Design](../screenshots/placeholder-services-design.png)

![Service step in a pipeline](../screenshots/placeholder-service-in-pipeline.png)

Decision tree (compact): [diagrams/decision-tree.md](../diagrams/decision-tree.md).

---

## 1. Do I need a service?

### Decision guide

| Prefer | When |
|---|---|
| **Mapping** (`@x12i/memorix-mapping`) | Pure deterministic field transform; no I/O; expressible in the shared AST |
| **Memory** (`@x12i/memorix-memory`) | External source material must land exact raw first (`*-raw`) |
| **Relationship metadata** | Connecting different `recordId`s with deterministic rules/operators |
| **Composition** | Same `recordId`, different content types (siblings) |
| **Pipeline** | Ordered, scheduled, retryable, multi-step work with run evidence |
| **Service** | Custom code, remote I/O, specialized compute, reusable external action, or model execution |

### Decision tree

```text
Is it a pure deterministic field transformation?
├─ Yes → mapping (@x12i/memorix-mapping)
└─ No

Is it external source material that must land raw?
├─ Yes → Memory source + connector (collect role)
└─ No

Is it connecting different recordIds?
├─ Expressible by metadata rules → relationship engine
├─ Needs custom resolution → resolver service + relationship engine
└─ Same recordId → composition (siblings), not a relationship service

Is it several ordered / retryable operations?
├─ Yes → pipeline (may invoke services as steps)
└─ No

Does it require custom computation, I/O, or AI?
└─ Service
```

See also [diagrams/decision-tree.md](../diagrams/decision-tree.md).

### Things that are not services

Object types, content types, mappings, abstract bindings, relationship definitions, schedules, pipelines, workflow content, the record writer/reader, the relationship store, the artifact store, and the metadata catalog are **platform surfaces**. A service may *use* them through canonical contracts; it must not reimplement them.

---

## 2. One model — roles catalog

Service “types” are **roles** (capabilities) on one model. A single service declares one or more capabilities; pipelines and operators still see one `serviceId`.

| Role | Capability examples | Purpose |
|---|---|---|
| **Collect** | `collect`, `connector`, `source-pull` | Read material from an external source for Memory |
| **Enrich** | `enrich`, `normalize`, `completion`, `lookup` | Compute or fetch additional information for an existing subject |
| **Resolve** | `resolve`, `identity`, `dedupe`, `relationship-candidates` | Produce identity or relationship *candidates* — engine writes links |
| **Action** | `action`, `notify`, `external-command`, `workflow-action` | Deliberate external side effect with idempotency |
| **AI job** | `ai`, `extract`, `classify`, `summarize`, `narrative`, `analyze` | Model call behind the single AI adapter |

Ops-pack examples use capabilities such as `normalize` and `completion` on services `opx-normalize` and `opx-completion` — same model, different roles.

### Side-effect classifications

| Classification | Retry safe? | Idempotency key | Typical evidence |
|---|---|---|---|
| Pure / read-only | Yes | Optional | Input/output hashes |
| External read | Usually yes | Recommended | Source receipt, cursor |
| Deterministic computation | Yes | Optional | Output artifact |
| Platform draft producer | Yes (draft only) | Recommended | Draft + validation result |
| External action | Only with key | **Mandatory** | External receipt |
| AI / model call | Re-callable; output may differ | Capture input; one accepted write | Request/response artifacts |

---

## 3. Define the service

### Metadata shape

Exact logical contract (`ServiceDefinition` in `@x12i/memorix-metadata`):

```ts
type ServiceIoPort = {
 id: string;
 schemaRef?: string; // logical schema reference — not a runtime payload
 description?: string;
};

type ServiceInvocation = {
 kind: "http" | "worker" | "internal";
 operation?: string;
};

type ServiceDefinition = {
 id: string;
 name: string;
 description?: string;
 capabilities: string[];
 inputs?: ServiceIoPort[];
 outputs?: ServiceIoPort[];
 invocation?: ServiceInvocation;
};
```

Example — enrich WI-100 with an external priority scorer:

```json
{
 "id": "opx-wi-priority-enrich",
 "name": "Work-item priority enricher",
 "description": "Returns a patch plan for work-item snapshot priority from an external scorer",
 "capabilities": ["enrich", "lookup"],
 "inputs": [
 { "id": "snapshot", "schemaRef": "memorix-record/snapshots" }
 ],
 "outputs": [
 { "id": "patch-plan", "schemaRef": "memorix-patch-plan/1" }
 ],
 "invocation": { "kind": "worker", "operation": "enrichPriority" }
}
```

### Identity rules

- Stable `id` is the logical name pipelines reference (`serviceId`).
- Agent ownership and inheritance follow the same multi-agent resolution as other kinds.
- Child agents may replace a service definition; unrelated agents must not silently collide.
- Service ID is **not** an endpoint URL.
- Metadata has no mutable runtime state; build/version belongs in run evidence.

### Input contract

Prefer references over copying large records into pipeline definitions or ledgers:

| Form | Use when |
|---|---|
| Small inline JSON | Tiny control values |
| DataRef / artifact | Large payloads, batches, model responses |
| Concrete record reference | Address by `(objectType, contentType, recordId)` + revision |
| Abstract projection reference | Read through virtual union (People), write concrete later |
| Captured context reference | Exact revision frozen for the run |
| Source / pull reference | Memory work |
| Relationship definition reference | Resolver / discovery helpers |
| Batch reference | Per-item manifests |

Every input is schema-validated. Prefer captured revisions for retry-sensitive work.

### Output contract

| Form | Meaning |
|---|---|
| Inline result | Small structured result |
| Artifact / DataRef | Large or binary output |
| Record draft / patch plan | Candidate content — **not** yet stored |
| Validation / mapping result | Structured issues |
| Relationship candidate result | Candidates for the relationship engine |
| Analysis / decision / narrative draft | Sibling content drafts with discriminators |
| External-action receipt | Proof of side effect |
| Diagnostics / metrics | Operational only |
| Retryable / non-retryable error | Machine-readable codes |

A service output is **not** automatically persisted as subject content. The pipeline validates and persists through canonical ports.

Forbidden in metadata: secret values, deployment URLs as identity, fake “always succeed” stubs presented as production contracts.

---

## 4. Implement the handler

### Lifecycle

1. Identify the capability.
2. Confirm it should be a service (decision guide).
3. Select roles / capabilities.
4. Define input/output schemas in pack metadata (`ServiceDefinition`).
5. Declare side effects and errors.
6. Implement a **`PortHandler`** for the canonical pipeline **`StepKind`** that will run the work (usually `enrich` / `analyze` / `decide` / `narrative`).
7. Register a runtime **`ServiceRef`** (`serviceId` + `kind`) — from pack bridge, pipeline `services[]`, or env defaults.
8. Add service metadata to the agent pack (`opx`) as `services[]` with pack field `id`.
9. Reference it from a pipeline step: `stepKind` selects the handler; optional `serviceId` must resolve in the registry.
10. Validate / install metadata.
11. Run conformance tests.
12. Deploy; execute through the pipeline; observe runs / attempts / artifacts.

### Real invocation contract (pipeline v2)

Handlers are **not** free-standing “service handlers.” They are **step-kind port handlers**:

```ts
type StepKind =
 | "memory-pull" | "persist-raw" | "map-normalize" | "validate"
 | "identity-dedupe" | "write-record" | "relationship-discover"
 | "relationship-write" | "compose" | "verify" | "workflow"
 | "enrich" | "analyze" | "decide" | "narrative";

type StepExecutionRequest = {
 scope: { orgId: string; agentIds: string[] };
 runId: string;
 stepId: string;
 attempt: number;
 stepType: StepKind;
 idempotencyKey?: string;
 deadline?: string;
 inputs: Record<string, unknown>;
 capturedContextRef?: /* DataRef */;
 config?: unknown; // from step.portConfig
};

type StepExecutionResult = {
 status: "succeeded" | "failed" | "partial";
 output?: unknown;
 itemOutcomes?: /* BatchItemOutcome[] */;
 diagnostics?: Record<string, unknown>;
 mutationRefs?: unknown[];
 retryable?: boolean; // failed + retryable ≈ safe to attempt again
 error?: { code: string; message: string };
};

type PortHandler = (req: StepExecutionRequest) => Promise<StepExecutionResult>;
```

Package: `@x12i/memorix-pipeline` (`createLivePortRegistry`, `createPortInvoker`, `createServiceRegistry`).

**Critical:** `serviceId` on a step does **not** select which function runs. Execution is always:

```text
step.stepKind → PortRegistry[stepKind] → PortHandler(req)
```

If `step.serviceId` is set, the runtime **resolves** that id in the service registry (fail closed if missing), then still invokes the **stepKind** port.

Do not invent private Mongo handles. Use the adapters the live port registry already wires (`@x12i/memorix-data`, relationships, Memory).

### Result contract

Every result is machine-readable — callers must not parse free-text messages:

| `status` | Meaning |
|---|---|
| `succeeded` | Full success |
| `partial` | Batch: some items ok (`itemOutcomes`) |
| `failed` | Failure — set `retryable: true` when safe to attempt again; omit or `false` when terminal |

Timeouts, cancellations, and skips are expressed as `failed` (+ diagnostics / error codes) or by the pipeline runtime — there is no separate port-status enum for `timed_out` / `cancelled` / `skipped`.

### Pack definition versus runtime binding

| Layer | Shape | Role |
|---|---|---|
| Pack metadata | `ServiceDefinition`: `{ id, name, capabilities, inputs?, outputs?, invocation? }` | Declarative catalog (`services[]` in agent pack) |
| Runtime registry | `ServiceRef`: `{ serviceId, kind, endpoint?, credentialRef?, … }` | Deploy binding |
| Bridge | `servicesFromMetadata` | Maps pack `id` → `serviceId`, `invocation.kind` → `kind` (default `internal`) |

```ts
// Pack (opx-pack metadata/services.json)
{ id: "opx-enrich-priority", capabilities: ["enrich"], invocation: { kind: "worker" } }

// Runtime after bridge
{ serviceId: "opx-enrich-priority", kind: "worker", capabilities: ["enrich"] }
```

### In-process versus remote

| | In-process | Remote |
|---|---|---|
| Use when | Trusted, low-latency, deterministic platform code | Separate process, vendor API, isolation |
| Binding | Live `PortHandler` on the step kind + `ServiceRef` | `ServiceRef.endpoint` + HTTP worker process |
| Credentials | `credentialRef` → runtime vault | Same — never in metadata |
| Large I/O | DataRef / artifacts | Same |
| Failure | Process-local errors → attempt ledger | Unavailable → `failed` + `retryable`; missing binding → fail closed in prod |

Helper `createServiceInvokePorts().invokeRemote` POSTs a versioned **`CapabilityInvocation`** envelope (`memorix-capability/1`) to `endpoint` — carrying scope, service/operation identity, inputs, attempt / idempotency / deadline, and optional `memorix` callback context — and expects HTTP 200 + JSON. It is **not** a first-class `StepKind` — wire it from a custom `enrich`/`analyze`/… port when you need remote dispatch.

**Product path:** implement remote workers with `@x12i/memorix-sdk/worker` (`defineCapability` + `createMemorixWorker`), which sits on `@x12i/core-service` for health/live compliance.

### Bi-directional called services

A called service **may** run an **internal multi-step** flow (fetch → decide → fetch more → return). It still must not own platform scheduling, retries, or subject persistence — return drafts/pages; Memory/pipeline ports land/write.

When the service declares `memorixAccess` (`explore`, `checkpoints`, and/or `raw`), Memorix injects a callback context **plus a server-issued HMAC grant**. Access is enforced server-side; mutating the `access` array cannot escalate privileges.

```ts
memorix: {
  baseUrl: string;
  orgId: string;
  agentIds: string[];
  runId?: string;
  stepId?: string;
  access: Array<"explore" | "checkpoints" | "raw">;
  grant?: string;
}
```

Prefer `@x12i/memorix-sdk/client` (re-exports `@x12i/memorix-client`):

```ts
import { fromInvokeContext } from "@x12i/memorix-sdk/client";

const mx = fromInvokeContext(request.memorix);
const existing = await mx.explore.listContent({
  objectType: "knowx",
  contentType: "knowx",
  limit: 200,
});
await mx.checkpoints.list("knowx-source");
```

Allowed Memorix surfaces for called services: **Explore** (`/api/data/*`), **Source checkpoints**, **raw read** (`/api/memory/raw`), **`commitPage`** (atomic land + checkpoint). Forbidden: Mongo, inventing scope, writing snapshots outside pipeline/Memory ports.
Production rejects missing bindings and silent stub handlers. Development stubs require **explicit** opt-in (`mode: "dev"` + `allowStubPorts: true`) and clear labeling.

---

## 5. Roles, one by one

Each role below uses the same structure: purpose, when, inputs/outputs, rules, anti-patterns, opx example.

### 5.1 Collect (source / connector)

**Purpose:** Read material from an external source for Memory.

**Rules:**

- External direction is read-only.
- Scheduled or on-demand pull; cursor / pagination supported.
- Credentials only via `credentialRef`.
- Exact provider payload preserved; every accepted item lands `<objectType>-raw`.
- Malformed items quarantine; failures do not block later scheduled pulls.

**Must not:** write back to the source, write snapshots directly, enrich/analyze, create relationships, bypass Memory, decorate raw `.data`.

**OpX example:** Pull procedures from localhost `opx-source` into `sandbox7a3c`; land `procedures-raw` for **PROC-2**.

### 5.2 Enrich (lookup, normalize, completion)

**Purpose:** Generate additional information for an existing subject, or convert complex input into a draft when mapping alone is insufficient.

**Preferred flow:**

```text
capture input revision
 → invoke enrich service
 → validate output (patch plan / draft)
 → write-record through canonical writer
 → handle stale conflicts
```

**Two lookup cases:**

| Case | Behavior |
|---|---|
| External source material | Preserve and land raw first; pipeline relates/enriches explicitly |
| Transient computational lookup | Return operational artifact/result; pipeline decides persistence |

**Must not:** silently mutate the subject, open Mongo, bypass revision preconditions.

**OpX example:** Enrich **WI-100** snapshot with a scored priority; apply exact-revision patch.

Use **mapping** for deterministic completion. Use a service when completion needs external lookup, custom computation, or AI.

### 5.3 Resolve (identity, dedupe, relationship candidates)

**Purpose:** Resolve external identities or produce relationship candidates when metadata operators are insufficient.

**Output:** `resolved` | `ambiguous` | `unresolved` + candidate `recordId`s + diagnostics.

**Must not:** invent `recordId`s silently, merge subjects, mutate the record directory, write links, write root `associated*`, copy target payloads.

Only the relationship engine creates deterministic relationship IDs, routes storage, and writes four-field links.

**OpX example:** People union (employees + contractors including **Alex**) produces assignment candidates for **WI-100**; engine materializes links.

Distinguish:

- Exact content dedup → Memory content hashing
- Exact identity enforcement → record directory
- Semantic duplicate suggestion → service candidates until an explicit merge workflow

### 5.4 Action (external command, notification, workflow action)

**Purpose:** Deliberate external side effect (notify, open ticket, approved remote command).

**Requirements:** explicit side-effect declaration, **idempotency key**, timeout, retry policy, duplicate-delivery handling, external receipt, sanitized evidence, optional compensation.

At-least-once pipeline execution must not cause duplicate external actions.

Notification is an action subtype: recipients, template input, delivery result, sensitive-data handling. Do not store delivery state on subject records unless modeled as domain content.

Workflow content remains ordinary V2 sibling content. The action service performs the effect; it does not invent a separate workflow persistence model.

**OpX example:** Approved action on **WI-100** posts to an ops ticket system once; retries reuse the same idempotency key and return the same receipt.

### 5.5 AI job

An AI job is a service invocation whose implementation calls a model through the **single** AI/model adapter. It still uses service metadata, pipeline steps, captured inputs, durable ledgers, DataRefs, structured outputs, validation, and retries.

AI is never inserted into the deterministic data/relationship runtime.

Categories: extraction, classification, completion, enrichment, summarization, narrative, analysis, recommendation, identity/relationship *candidates*, structural-intelligence proposals.

**Pipeline flow for untrusted model output:**

1. Receive raw model response.
2. Store response artifact / hash.
3. Parse structured result.
4. Validate schema + business/metadata constraints.
5. Reject or repair per explicit policy.
6. Create draft / patch plan.
7. Persist through canonical writer only if allowed.

Never allow a model to emit a Mongo update or arbitrary internal API call.

AI may suggest relationship definitions or candidate pairs as operational evidence. AI may **not** write arbitrary links, bypass the relationship engine, or install metadata. Only human/operator metadata workflows approve, install, and publish.

**Reproducibility captures:** exact input revisions, relationship-expanded context, prompt/template fingerprint, model/provider/build id, service build id, request/response hashes, attempt identity. Do not store this provenance on relationship links.

**Air-gapped:** runtime without AI credentials; optional skip/fail policies; recorded-response adapters for tests; production fails clearly when a required AI binding is absent — no fake production AI fallback.

**OpX example:** Narrative AI for **WI-100** using captured snapshot + PROC-2 link context; acceptance uses recorded adapter from `memorix-opx-pack` (no live AI required).

---

## 6. Bind and deploy

### Registry

`createServiceRegistry` merges (later wins):

1. environment defaults (`ServiceRef[]`),
2. effective metadata services (via `servicesFromMetadata` — pack `id` bridged to `serviceId`),
3. pipeline-local `services[]` on the v2 `PipelineDefinition`.

Pipeline steps may set optional `serviceId`. The registry **resolves identity / endpoint / credentialRef** — it does **not** dispatch to a per-serviceId function. Dispatch remains `stepKind` → `PortHandler`.

`credentialRef` (or `auth.secretRef`) resolves at runtime. Resolved views expose `hasCredential` — never the secret.

### Production versus development

| Production | Development / test |
|---|---|
| Required real `PortHandler` for every used `StepKind` | Explicit stub / recorded adapter |
| Missing `serviceId` binding fails when the step references it | Opt-in only, clearly labeled |
| Durable ledger + artifact store | May use isolated fixtures |
| No fake success | Cannot silently become production default |

Startup fails closed when: required binding missing, handler is a silent stub, durable stores missing, `credentialRef` unresolvable for a required operation, or contracts incompatible.

---

## 7. Invoke from a pipeline

Preferred split:

```text
read / capture
 → StepKind port (enrich / analyze / …) [optional serviceId resolve]
 → validate result
 → write-record or relationship-write
```

Avoid a service that hides computation and persistence in one opaque call. External-action services are the exception — the side effect is their declared purpose.

### Adding a service to a pipeline (opx recipe)

1. Add `services[]` entry to the agent pack (`id`, `capabilities`, `invocation`) — see `memorix-opx-pack/metadata/services.json` (`opx-enrich-priority`).
2. Install / resolve effective metadata for `sandbox*` + `agentIds: ["opx"]`.
3. Bridge pack services → `ServiceRef`s (`servicesFromMetadata`) when constructing the pipeline runtime.
4. Implement or override the **`enrich`** (or other) live port so the handler returns a **draft / patch plan**, not a silent Mongo write.
5. Author a v2 pipeline step:
 ```ts
 {
 stepId: "enrich",
 stepKind: "enrich", // selects PortHandler
 serviceId: "opx-enrich-priority", // must resolve in registry
 portConfig: { /* optional */ },
 }
 ```
 Pack authoring may use the same `serviceId` with kind `enrich` (see `opx-enrich-priority` pipeline in opx-pack). Pack kinds are not always identical to v2 `StepKind` — the platform compile/bind path must produce a valid v2 definition.
6. Add a separate mutation step (`write-record` / `relationship-write`) that applies the draft with revision preconditions.
7. Start the run; inspect `/api/pipelines` attempts and artifacts (or Studio Operate).

There is **no** v2 `StepKind` named `invoke-service`. Older pack sketches that used `kind: "invoke-service"` are not executable until rewritten to a canonical step kind.

### Partial batches

- Per-item input and per-item success/failure.
- Retry only failed items.
- No false global atomicity unless a port explicitly declares atomic batch support.
- Output artifact holds the partial manifest.

### Retry matrix (summary)

| Kind | Repeat same invocation? |
|---|---|
| Pure calculation | Yes |
| External read | Yes (watch rate limits / cursors) |
| Enrichment draft | Yes; one accepted write |
| AI call | Yes; outputs may differ; capture input; one accepted mutation |
| External action | Only with idempotency key + receipt reuse |
| Relationship candidates | Yes; engine write remains idempotent |
| Artifact processing | Yes if content-addressed |

---

## 8. Data access — may and must-not

### May

Read through canonical data ports:

- exact content,
- bounded list / query,
- composition,
- abstract view,
- relationship expansion,
- captured context.

Write **drafts / plans** as service results. Canonical pipeline mutation steps use:

- writer `add` / `upsert` / `patch` / `replace` / `remove`,
- abstract reverse patch,
- relationship engine materialization.

Every write retains revision / precondition behavior.

### Must not

- Direct Mongo domain writes
- Ad-hoc relationship link mutation (product `PUT`/`DELETE` of links → rejected)
- Auto-install intelligence / metadata proposals
- Embed raw secrets in Memory, records, run evidence, artifacts, errors, logs, or client responses
- Create duplicate subjects merely to attach generated content
- Bypass Memory for provider source material

### External payload rule

| Material | Destination |
|---|---|
| Provider / source material | Memory raw first; exact `.data` |
| Generated computational output | Artifact or derived-content draft |
| External enrichment that is source material | Do not disguise as an internally generated snapshot |

Same `recordId` → sibling content / composition. Different `recordId`s → relationship.

---

## 9. Secrets, validation, reliability

### Credential references

Service metadata and Memory configs carry `credentialRef` only. Runtime resolves credentials outside Catalox and never places them in metadata, records, run evidence, artifacts, errors, logs, or client responses.

```http
POST /api/memory/pull
x-memorix-org-id: sandbox7a3c
x-memorix-agent-ids: opx
Content-Type: application/json

{
 "sourceId": "opx-source",
 "credentialRef": "secret://opx-source"
}
```

### Validation

Schema validation, maximum sizes, path safety, content types, artifact type/hash, scope validation, unknown fields, structured error codes.

### Timeouts and cancellation

Deadline propagation, cancellation signal, cleanup, remote cancellation limits, cancelled attempt status, and honest uncertainty when an external side effect may already have occurred.

### Idempotency declaration

For every service declare: supported?, key shape, duplicate behavior, result reuse, side-effect receipt handling.

Recommended key material: `(orgId, pipelineId, stepId, subjectRecordId, inputRevision)` or an explicit business key for external actions.

### Error catalog (codes to handle)

| Code class | Retryable? | Operator action |
|---|---|---|
| Configuration / missing binding | No | Fix pack / registry; redeploy |
| Invalid input / validation | No | Fix caller or mapping |
| Dependency unavailable | Yes | Wait / retry; check health |
| Timeout | Often yes | Inspect attempt; raise timeout or fix hang |
| Retryable external failure | Yes | Automatic / manual retry |
| Terminal external failure | No | Fix external system or disable step |
| Partial batch | Mixed | Retry failed items only |
| Invalid output | No | Fix service contract / parser |
| Artifact error | Depends | Restore artifact store / wrong-org check |
| Stale write after service | No auto | Re-read; new revision; retry write |
| Ambiguous identity / insufficient evidence | No | Human resolution / richer input |
| AI parse / validation failure | Policy | Repair policy or fail terminal |

Sanitize all errors — no secrets, no unrestricted payloads.

---

## 10. Testing

### Pyramid

1. Metadata / schema validation
2. Pure unit tests for handler logic
3. Contract tests (input/output schemas)
4. Registry binding tests
5. Pipeline invocation tests
6. Retry / idempotency tests
7. Artifact / DataRef tests
8. Production missing-binding fail-closed tests
9. Ops integration with `memorix-opx-pack` fixtures in a fresh `sandbox*` org
10. Credential / evidence scan (no secrets in ledgers)

### Recorded dependencies

Use recorded adapters for remote APIs and AI responses. They are explicit test/dev tools — never silent production defaults. Platform acceptance requires **no live AI**.

### Conformance (every service)

Definition validation · input/output validation · scope propagation · no-secret evidence · timeout · cancellation · retry classification · idempotency declaration · inline/DataRef limits · production registry resolution · no direct Mongo · no hidden writer/store access · structured errors.

Add specialized suites for connectors, external actions, enrichment, relationships, AI, artifacts, and batches.

---

## 11. Operate and troubleshoot

### Observability per invocation

`serviceId`, pipeline/run/step/attempt IDs, scope, duration, result status, retry classification, input/output artifact refs, external receipt, deployment/build identity, model evidence for AI.

HTTP surfaces: `/api/pipelines` (runs/attempts/artifacts), OpenAPI at `GET /api/openapi.json`. Studio Operate mirrors the same ledger.

### Operator playbook

1. Find the run for `sandbox*` + pipeline that invoked the service.
2. Open the failed step attempt; read structured error code.
3. Inspect input DataRef / captured revision (compare to current Explore data).
4. Inspect output artifact or receipt.
5. Fix metadata, binding, or handler; reinstall if definition changed.
6. Retry the step / resume the run — do not invent a second schedule.
7. Confirm Explore shows expected siblings/links after success.

### Update procedure

- Change logical contract only when behavior/schema changes.
- Update implementation/binding; run conformance.
- Update affected pipelines; validate agent packs; install deliberately.
- Running jobs use captured definitions/context per runtime contract — do not invent unsupported metadata-version machinery.

### Retire procedure

1. Inventory pipeline / source references.
2. Replace consumers.
3. Disable new use.
4. Keep required historical run artifacts.
5. Remove metadata deliberately; remove binding; delete implementation.
6. Verify no production references.

---

## 12. Complete opx tutorials

Use a fresh org each time (`sandbox*` + `agentIds: ["opx"]`). Fixtures from `memorix-opx-pack` / localhost `opx-source`.

### 12.1 Procedure source connector

Pull paginated procedures → land exact raw → deduplicate by content hash → resume cursor → quarantine malformed items. Confirm **PROC-2** raw exists before any snapshot promotion.

### 12.2 Work-item enrichment (runnable shape)

Scope: fresh `sandbox*` + `agentIds: ["opx"]`. Pack fixture: `opx-enrich-priority` in `@x12i/memorix-opx-pack`.

1. Install opx-pack (includes `metadata/services.json` + pipeline `opx-enrich-priority`).
2. Confirm effective metadata lists service id `opx-enrich-priority`.
3. Construct pipeline runtime with `metadataServices: { services: effective.services }` so `servicesFromMetadata` bridges `id` → `serviceId`.
4. Ensure a live **`enrich`** `PortHandler` returns a patch-plan draft (no Mongo).
5. Run v2 definition equivalent to:
 ```ts
 {
 pipelineId: "opx-enrich-priority",
 steps: [
 { stepId: "enrich", stepKind: "enrich", serviceId: "opx-enrich-priority" },
 { stepId: "write", stepKind: "write-record" },
 ],
 services: [{ serviceId: "opx-enrich-priority", kind: "worker" }],
 }
 ```
6. Read **WI-100** snapshot at exact revision before write; after success, exact-get shows scored fields; if Explore edited meanwhile, writer returns stale conflict — retry with fresh revision.

Unit bridge coverage: `memorix-pipeline` `service-registry.unit.test.ts`.

### 12.3 People relationship resolver

Employees and contractors (including **Alex**) as People abstract → resolver returns candidates → relationship engine writes four-field links → verify/rebuild. Service never writes links itself.

### 12.4 Work-item analysis

Capture snapshot / workflow / relationships for **WI-100** → analysis service returns draft with `analysisType` → pipeline persists sibling analysis content.

### 12.5 Decision service

Consume captured analysis → produce `decisionType` draft → persist decision sibling. Decision content does not itself perform an external action.

### 12.6 Narrative AI (recorded)

Capture exact context → invoke recorded AI adapter → validate structured output → store artifact or narrative content → replay with same captured input.

### 12.7 External action

Receive approved **WI-100** action → idempotency key → external receipt → safe retry returns same receipt.

### 12.8 Structural intelligence

Profile fixture data → produce People proposal artifact → review evidence → **explicitly** install metadata separately (never auto-install).

---

## 13. Antipatterns and troubleshooting

### Antipatterns

| Antipattern | Prefer |
|---|---|
| New service for a rename/shape | Mapping AST |
| Service writes Mongo / bypasses writer | Draft out → `write-record` |
| Service `PUT`s relationship links | Candidates → relationship engine |
| Secrets in metadata, records, or artifacts | `credentialRef` only |
| AI installs metadata or writes links | Proposal artifact + human install |
| Collect that also promotes snapshots | Memory raw → separate pipeline write |
| Action without idempotency key | Mandatory key + receipt reuse |
| Live AI required for opx acceptance | Recorded adapter / fixtures |
| Silent stub as production default | Fail closed; opt-in stubs only |
| Duplicate subjects to attach analysis | Sibling content on same `recordId` |

### Troubleshooting decision tree

```text
Missing binding / fail before run?
├─ Yes → register production binding; redeploy
└─ No

Invalid input / validation failure?
├─ Yes → fix caller, mapping, or schemaRef
└─ No

External / AI dependency flaky?
├─ Action → check idempotency key + receipt
├─ AI → use recorded adapter for acceptance; inspect response artifact
└─ Read/lookup → retry; inspect cursor / rate limits

Green service step, empty Explore?
├─ Missing write-record / relationship-write after draft?
└─ Wrong org / agent scope?

Links missing after resolve?
└─ Engine materialize / verify not run — service only proposed candidates
```

---

## 14. Reference

### Role comparison

| Role | Reads external? | Writes external? | Returns draft? | Uses AI? | Direct Memorix mutation? | Idempotency | Persistence owner |
|---|---|---|---|---|---|---|---|
| Collect | Yes | No | Raw handoff | No | Via Memory only | Cursor / content hash | Memory + data plane raw |
| Enrich | Optional | No | Yes | Optional | No — writer step | Key + revision write | Pipeline writer |
| Resolve | Optional | No | Candidates | Optional | No — relationship engine | Engine keys | Relationship engine |
| Action | Optional | Yes | Receipt | Optional | Domain write separate | **Mandatory** | External system + ledger |
| AI job | Via adapter | No | Yes | Yes | No — writer step | Captured input; one write | Pipeline writer |

### Packages

| Package | Use |
|---|---|
| `@x12i/memorix-metadata` | `services[]` definitions |
| `@x12i/memorix-pipeline` | Steps that invoke services |
| `@x12i/memorix-pipeline-runtime` | Run/attempt/artifact ledger |
| `@x12i/memorix-mapping` | Prefer before inventing a service |
| `@x12i/memorix-memory` | Collect / raw landing |
| `@x12i/memorix-data` | Canonical read/write ports |
| `@x12i/memorix-service` | HTTP `/api/*` |
| `@x12i/memorix-opx-pack` | Acceptance fixtures |
| `@x12i/core-service` | HTTP process compliance (ports, `/health`, `/_live`) |
| `@x12i/ports-manager` | Org port zones (Memorix **5100–5119**) |
| `@x12i/api-live-view` | Embeddable live request UI |

### HTTP process compliance (local services)

**Terminology:** TCP/HTTP **ports** (this section) are not Memorix **canonical data ports** (`PortHandler` / `write-record`). Different layers, same English word.

| Audience | Action |
|---|---|
| Remote Memorix **client** (calls `/api/*`) | Do **not** install these kits |
| New HTTP **worker** Memorix (or Studio) will call | **Must** use `@x12i/core-service` |
| Studio / opx-source / memorix-service | Already compliant |

Any x12i/exellix process that **binds an HTTP port** should use `@x12i/core-service` (pulls in `@x12i/ports-manager` + `@x12i/api-live-view`).

#### Compliance contract

1. Port from the org zone map (`port-map.json`) — env override OK; avoid/quarantine enforced 
2. `GET /health` → `{ ok, service, … }` 
3. `/_live` (+ inbound/worker rings) via api-live-view 
4. `x-correlation-id` on request/response 
5. Startup banner: origin · health · live · docs 

#### Fastify

```ts
import Fastify from "fastify";
import { attachCoreFastify } from "@x12i/core-service/fastify";

const app = Fastify();
const core = await attachCoreFastify(app, {
 identity: {
 id: "@x12i/my-enrich-worker",
 title: "Ops enrich worker",
 zoneId: "memorix",
 portKey: "api", // even API default in zone
 portEnv: "MY_WORKER_PORT",
 docsPath: "/openapi.json",
 },
 getHealth: async () => ({ ok: true, service: "@x12i/my-enrich-worker" }),
});

await app.listen({ port: core.port, host: "127.0.0.1" });
```

#### Node `http`

```ts
import http from "node:http";
import { createCoreNodeHttp, listenAndAnnounce } from "@x12i/core-service";

const core = createCoreNodeHttp({
 identity: {
 id: "opx-source",
 title: "Ops Source",
 zoneId: "memorix",
 portKey: "opxSource",
 portEnv: "OPX_SOURCE_PORT",
 docsPath: "/openapi.json",
 },
});
// tryHandleCore → /health + /_live; then your routes
```

#### Ports manager (Memorix zone)

| Default | Port | Env |
|---|---|---|
| `service` / `api` | **5100** | `MEMORIX_SERVICE_PORT` |
| `studio` / `ui` | **5101** | `MEMORIX_STUDIO_PORT` / `VITE_DEV_PORT` |
| `opxSource` | **5102** | `OPX_SOURCE_PORT` |
| `statusHub` | **5103** | `STACK_STATUS_PORT` |

Rules: zone **5100–5119** (20 contiguous); **even = API**, **odd = UI**; never bind avoid list (AirPlay **5000/7000**, …); quarantine **5000–5099**. CLI: `npx x12i-ports zone memorix`. Deep reference: package READMEs for `ports-manager`, `core-service`, `api-live-view`.

Prefer `core-service` over wiring `api-live-view` alone. Register the worker’s `ServiceRef.endpoint` to the even API origin the registry will POST to.

### Authoring checklist

- [ ] Decision guide says “service,” not mapping/Memory/relationship/pipeline alone
- [ ] Metadata has logical I/O + capabilities; no secrets/URLs as identity
- [ ] `ServiceRef` registered (`serviceId`); prod fails closed without it
- [ ] Handler is a `PortHandler` on a real `StepKind` (not invented `invoke-service`)
- [ ] Side effects and idempotency declared
- [ ] Drafts vs persistence split clear
- [ ] HTTP workers use `@x12i/core-service` (`/health`, `/_live`, zone port)
- [ ] Conformance + opx fixture tests green on `sandbox*`
- [ ] Evidence scan: no secrets in ledgers/artifacts

### Glossary (service terms)

| Term | Meaning |
|---|---|
| Service | Named capability with one logical pack definition (`id`) |
| ServiceRef | Runtime binding (`serviceId` + `kind` + optional endpoint) |
| Role / capability | Classification on that one model |
| StepKind / PortHandler | Executable pipeline step surface — dispatch key |
| Binding | Registry resolve for a `serviceId` (not handler dispatch) |
| Draft | Proposed content — not yet stored |
| DataRef | Reference to inline or artifact payload |
| Credential ref | Pointer to a secret; never the secret |
| Attempt | One try of a pipeline step that may invoke a service |
| TCP port / zone | Local bind address from ports-manager — not a data port |

### Next

- [Building Pipelines — Developers](../../08-pipelines/developers/BOOK.md)
- [Relationships & Memory](../../06-relationships-and-memory/developers/BOOK.md)
- [Intelligence](../../10-intelligence/developers/BOOK.md)
- Studio-oriented twin → [Builders](../builders/BOOK.md)

---

# Building Services — builders

# Building Memorix Services — Builders

**Audience:** Studio authors deciding when a service is needed and how to operate it. 
**Twin:** [Developers version](../developers/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Prerequisites:** [Platform overview (builders)](../../00-platform-overview/builders/BOOK.md) · [Metadata (builders)](../../05-metadata-agents-abstracts/builders/BOOK.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense Studio guide. Deterministic opx fixtures only — **no live AI required**.

---

## How to read this guide

1. Learn **what a service is** in product language.
2. Use the **decision guide** before asking engineering for a new service.
3. Work in Studio: **Design → Pipelines/Services → Operate → Explore**.
4. Always use a fresh `sandbox*` org and agent **`opx`** for drills (Alex, WI-100, PROC-2).
5. Deep implementation detail lives in the [developers twin](../developers/BOOK.md).

---

## The central idea

> **Services do the specialized work. Pipelines decide *when*. Memorix writers and the relationship engine decide *what gets stored*.**

A **service** is a named capability the platform can invoke — usually from a **pipeline** step. It may pull from outside, enrich a subject, propose relationship candidates, call a model, or perform an approved external action.

A service does **not** own scheduling, retries, Mongo access, relationship storage, or metadata install. Those belong to pipelines, Memory, the data plane, the relationship engine, and Studio metadata workflows.

Ops subjects: employee **Alex**, work item **WI-100**, procedure **PROC-2**.

---

## 1. What a service is (builder view)

| Need | Service role | Not a service |
|---|---|---|
| Pull procedures from opx-source | Collect (via Memory connector) | Writing snapshots yourself in Explore |
| Score priority for WI-100 | Enrich | A simple field rename (use a mapping) |
| Suggest People assignment links | Resolve (candidates only) | Hand-editing relationship links |
| Notify an external ops channel | Action | Pasting an API key into a record |
| Draft a narrative for WI-100 | AI job | Expecting live AI for acceptance |

Services do **not** replace mappings, Memory, relationship definitions, or pipelines. They plug into them.

### Bi-directional called services

A remote service may be granted limited Memorix callback access (`memorixAccess`: Explore, Source checkpoints, and/or raw). Memorix issues a server-enforced grant — the worker can fetch during its own multi-step work, then return drafts. It still must not open Mongo or write snapshots itself. Engineering implements this with `@x12i/memorix-sdk/worker` + `/client` (see the [developers twin](../developers/BOOK.md)).

![Services in Design](../screenshots/placeholder-services-design.png)

![Service step in a pipeline](../screenshots/placeholder-service-in-pipeline.png)

---

## 2. Do I need a service?

Ask for a service only when Studio metadata alone cannot:

- call an external system,
- run specialized code,
- execute an AI job with recorded prompts/responses.

Otherwise prefer:

| Prefer | When |
|---|---|
| **Mapping** | Pure field transform (rename, shape, fill from known fields) |
| **Memory source** | Land exact external material as raw |
| **Relationship definition** | Connect different subjects with rules |
| **Pipeline** | Order several steps with retries and run history |
| **Composition** | Same subject, different content types (siblings) |

```text
Pure field transform? → mapping
External source material? → Memory
Connect recordIds by rules? → relationship definition
Ordered retryable work? → pipeline
Custom / remote / AI? → service
```

Full tree: [diagrams/decision-tree.md](../diagrams/decision-tree.md).

### Things that are not services

Object types, content types, mappings, abstract bindings, relationship definitions, schedules, pipelines, workflow content, and the metadata catalog are **platform surfaces**. A service may *use* them; it must not reimplement them.

---

## 3. One model, five roles

Studio shows **one** Services catalog. Roles are labels on that catalog — not five different products.

| Role | Plain meaning | OpX example |
|---|---|---|
| **Collect** | Fetch from outside (read-only toward the source) | Procedure pull for PROC-2 |
| **Enrich** | Add or compute fields for an existing subject | Priority for WI-100 |
| **Resolve** | Propose matches / candidates | People → work-item assignment candidates |
| **Action** | Cause an external effect (once, safely) | Approved notification for WI-100 |
| **AI job** | Call a model; store proof; validate output | Narrative draft for WI-100 |

### How roles show up in a healthy run

```text
Memory collect lands PROC-2 raw
 → pipeline maps / writes snapshots
 → enrich service proposes WI-100 priority patch
 → write step persists the patch
 → resolve service proposes People ↔ WI-100 candidates
 → relationship engine writes four-field links
 → AI job (recorded) drafts a narrative sibling
 → action service notifies once with an idempotency key
```

---

## 4. In Studio

### Scope bar (always)

1. Fresh `sandbox*` organization — never a live customer org.
2. Agent **`opx`**.
3. Confirm Design / Operate / Explore show that same scope.

Wrong scope looks like “the service ran but Explore is empty.”

### Design

1. Open **Design → Services** — register or review logical name, role/capabilities, and I/O description.
2. Confirm the service is part of the **opx** agent pack (or inherited correctly).
3. Open **Design → Pipelines** — attach a step that points at the service id.
4. Keep mutation separate: service produces a draft/result; a later write or relationship step persists.
5. Never paste API keys into record fields, pipeline JSON, or service descriptions. Use **credential references** only.

### Operate

1. Start or schedule the pipeline that uses the service.
2. Open the run → step → attempt.
3. Read the failure code and artifacts if something breaks.
4. Retry from Operate after the fix — do not create duplicate schedules.

### Explore

After a green run, confirm the subject (for example **WI-100**) shows the expected sibling content or relationship links — not copied neighbor payloads.

---

## 5. Secrets and safety

| Do | Don't |
|---|---|
| Use `credentialRef` / Studio secret references | Paste tokens into fixtures or records |
| Expect Memory to land exact raw first | Ask a service to “just write Mongo” |
| Treat AI output as untrusted until validated | Auto-install intelligence proposals |
| Keep acceptance on recorded/fixture AI | Require live AI for opx acceptance |
| Expect relationship engine to write links | Hand-create product links |

Hard rules always apply: fresh `sandbox*`, four-field relationships only, no direct Mongo domain writes, intelligence never auto-installs.

---

## 6. Retries and idempotency (builder view)

Pipelines may retry a step. That is normal.

| Kind of service | What “safe retry” means for you |
|---|---|
| Enrich / resolve / AI draft | Retry is fine; Explore should still show **one** accepted write |
| Collect / Memory | Cursor and content-hash prevent duplicate raw |
| Action (notify / ticket) | Same idempotency key → same external receipt — not two tickets |

If Operate shows repeated action attempts with **different** receipts for one approved WI-100 action, escalate — that is a platform defect, not something to “fix” by scheduling again.

---

## 7. Operate checklist

- [ ] Scope bar: fresh `sandbox*` + agent `opx`
- [ ] Service appears in Design for this agent
- [ ] Pipeline step points at the correct service id
- [ ] Binding exists for this environment (missing binding fails in production — that is correct)
- [ ] Failures show in Operate with artifacts
- [ ] Retries are intentional (button / schedule), not silent loops
- [ ] Explore confirms Alex / WI-100 / PROC-2 outcomes after success
- [ ] No secrets visible in artifacts or records

### Diagnose quickly

| Symptom | Likely cause |
|---|---|
| Run fails before start | Missing service binding or invalid pipeline |
| Step retry loops | Non-idempotent action or bad error classification |
| Explore empty after “green” | Service returned a draft but write step missing/failed |
| Links missing after resolve | Engine write/verify not run — service only proposed candidates |
| Secrets in artifact | Misconfigured service — stop and escalate |
| AI step flaky in acceptance | Expect recorded adapter; do not require live model |

---

## 8. Short opx tour

1. Install opx metadata for your `sandbox*` org.
2. Confirm Memory can pull from opx-source (**PROC-2** raw lands) — collect path.
3. Run the pipeline that invokes an enrich or normalize service for **WI-100**.
4. In Operate, open attempts and any artifacts.
5. In Explore, open **WI-100** and **Alex**; confirm composition and links.
6. If an AI narrative step exists, confirm acceptance path uses recorded responses — no live model required.
7. If intelligence proposed metadata, review and **explicitly** install or reject.

Companion: [documentation/tutorials/04-memory-and-pipeline.md](../../../documentation/tutorials/04-memory-and-pipeline.md).

---

## 9. What not to expect

- A service that “also schedules itself”
- A service that writes Mongo directly “for speed”
- Ad-hoc product link create/delete from a service form
- Auto-install of intelligence proposals
- Live AI as a requirement for opx acceptance
- Five separate “service products” instead of one catalog with roles
- Cyber/vuln stories as the permanent product narrative

---

## Next

- Full implementation detail → [developers twin](../developers/BOOK.md)
- [Building Pipelines — Builders](../../08-pipelines/builders/BOOK.md)
- [Relationships & Memory](../../06-relationships-and-memory/builders/BOOK.md)
- [Platform overview](../../00-platform-overview/builders/BOOK.md)
- [Hard rules](../../HARD-RULES.md)

---

# Building Pipelines — developers

# Building Memorix Pipelines — Developers

**Audience:** Pipeline authors, service developers, connector and AI-job developers, operators, and platform integrators. 
**Twin:** [Builders version](../builders/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Packages:** `@x12i/memorix-pipeline`, `@x12i/memorix-pipeline-runtime`, `@x12i/memorix-mapping`, `@x12i/memorix-memory`, `@x12i/memorix-data`, `@x12i/memorix-relationships`, `@x12i/memorix-metadata`, `@x12i/memorix-service`, `@x12i/memorix-opx-pack`. 
**Prerequisites:** [Platform overview](../../00-platform-overview/developers/BOOK.md) · [Services](../../07-services/developers/BOOK.md) · [Relationships & Memory](../../06-relationships-and-memory/developers/BOOK.md) · [Metadata & mappings](../../05-metadata-agents-abstracts/developers/BOOK.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense product documentation for the Memorix pipeline runtime. Deterministic opx fixtures only — **no live AI required**.

---

## How to read this guide

1. Start with the **central mental model** — one pipeline runtime, many patterns.
2. Learn **definition → DAG → run/step/attempt → artifacts**.
3. Walk **every step kind**, then the reusable **patterns**.
4. Finish with **AI**, **operations**, and the **complete opx tutorial**.
5. Pipelines are **side C** — author vocabulary (A) and services (B) first; see [Getting started — three sides](../../14-getting-started-and-tutorials/developers/BOOK.md).
6. Labels: **Stored** subject content · **Virtual** composition/abstracts · **Operational** runs, attempts, artifacts.

Every example uses a fresh `sandbox*` organization, agent `opx`, and subjects **Alex**, **WI-100**, **PROC-2**. Never reuse live customer orgs for acceptance.

---

## The central mental model

A Memorix pipeline is a **declarative graph of steps** executed by one durable runtime.

> **Pipelines orchestrate the work. Steps declare behavior. Mappings transform. Services compute or act. Canonical ports read and write. Artifacts carry large data. The durable ledger makes execution retryable and understandable.**

A pipeline coordinates:

- pulling external data and consuming Memory raw,
- mapping and normalization,
- validation, identity, and record writes,
- relationship discovery and materialization,
- composition and enrichment,
- analysis, decisions, narratives, workflows,
- external actions (via services),
- verification and maintenance.

### One pipeline model

Ingestion, enrichment, relationships, AI jobs, workflows, maintenance, scheduled jobs, on-demand jobs, and batches are **patterns** — not separate engines.

All pipelines share:

| Surface | Package / contract |
|---|---|
| Definition schema | Agent metadata + pipeline definition types |
| Step taxonomy | `StepKind` in `@x12i/memorix-pipeline` |
| Service registry | Logical `serviceId` → binding |
| Runtime | `@x12i/memorix-pipeline` |
| Operational ledger | `@x12i/memorix-pipeline-runtime` |
| Schedule model | Schedule definitions referencing pipelines |
| DataRef / artifacts | Inline under threshold; artifacts above |
| Scope | `{ orgId, agentIds[] }` captured on the run |

HTTP: `memorix-service` `/api/pipelines`. OpenAPI: `GET /api/openapi.json`.

### Pipeline versus mapping versus service

| Construct | Owns |
|---|---|
| **Mapping** | Pure deterministic field transform AST (`@x12i/memorix-mapping`) |
| **Service** | Custom / external / AI capability |
| **Pipeline** | Ordering, retries, resume, run evidence, step graph |
| **Memory** | Exact source landing (`*-raw`) |
| **Relationship engine** | Link compile / discover / materialize |
| **Data plane** | Revision-safe reads/writes |

Use a **mapping** when the transform is pure, deterministic, and expressible in the AST. 
Use a **service** when you need custom code, remote I/O, AI, or a reusable external action. 
Use a **pipeline** when operations must be ordered, several components participate, work needs retry/resume, work is scheduled, partial success matters, exact inputs must be captured, or operational evidence is required.

### Pipeline versus workflow

A pipeline describes **technical execution**. Workflow content describes the subject’s process state as ordinary V2 sibling content. A pipeline may read/update workflow through the writer and invoke workflow-related services — it does not invent a separate workflow database.

### Pipeline versus Memory

Memory owns connector/cursor/quarantine semantics and raw landing. A pipeline may trigger Memory and consume raw results; it must not recreate those semantics.

### Pipeline versus relationship engine

The pipeline orders discovery, write, verify, and rebuild. The engine owns deterministic relationship IDs, candidate/link behavior, storage routing, and link writes.

```mermaid
flowchart LR
 Mem[Memory raw] --> Pipe[Pipeline steps]
 Pipe --> Map[Mapping]
 Pipe --> Svc[Service / AI job]
 Pipe --> Write[Data plane write]
 Write --> Rel[Relationship materialize]
 Pipe --> Art[Artifacts / ledgers]
```

See also [diagrams/pipeline-flow.md](../diagrams/pipeline-flow.md).

Studio Operate evidence:

![Failed step](../screenshots/13-failed-pipeline-step.png)

![Successful retry](../screenshots/14-successful-retry.png)

![Captured context artifact](../screenshots/15-artifact-captured-context.png)

![Pipeline designer](../screenshots/placeholder-pipeline-designer.png)

---

## 1. Pipeline definition

### Identity

- Stable pipeline metadata ID (`pipelineId`) belongs to an agent (e.g. `opx`).
- Inheritance and child replacement follow multi-agent metadata rules.
- Pipeline ID is **not** a run ID.
- Metadata does not hold runtime state; fingerprints and build evidence live on the run.
- Do not invent unsupported metadata-version machinery beyond what the runtime captures.

### Scope

Every run uses:

```ts
{
 orgId: string; // e.g. sandbox7a3c — complete data universe
 agentIds: string[]; // e.g. ["opx"] — effective metadata only
}
```

Scope is captured on the run. Every step receives the same canonical scope. Steps must not reinterpret scope. `agentIds` does not filter organization data.

```http
x-memorix-org-id: sandbox7a3c
x-memorix-agent-ids: opx
```

### Definition fields (runtime)

Core runtime shape (`PipelineDefinition` in `@x12i/memorix-pipeline`):

```ts
type PipelineDefinition = {
 pipelineId: string;
 version: string;
 description?: string;
 steps: StepDefinition[];
 services?: ServiceRef[];
 schedules?: ScheduleDefinition[];
 retryPolicy?: RetryPolicy;
 dataRefThresholdBytes?: number; // default 1_048_576
 metadata?: Record<string, unknown>;
};

type StepDefinition = {
 stepId: string;
 stepKind: StepKind;
 label?: string;
 dependsOn?: string[];
 inputs?: DataRefExpr[];
 outputKey?: string;
 retryPolicy?: RetryPolicy;
 idempotencyKeyExpr?: string; // required for mutating steps
 port?: string;
 portConfig?: unknown;
 serviceId?: string;
 batchSize?: number;
 atomicBatch?: boolean; // opt-in; rejected unless adapter supports
 timeoutMs?: number;
 disabled?: boolean;
 inputSchema?: unknown;
 outputSchema?: unknown;
};
```

Agent-pack metadata may use a lighter authoring shape (`id`, `name`, `trigger`, `mode`, `steps[]` with `kind` / `serviceId` / `mappingId`). The runtime compiles or binds that into the executable definition. Always validate packs with `@x12i/memorix-metadata` before install.

### Inputs and outputs

**Inputs:** small inline JSON, DataRef/artifact, record/content address, list/query, abstract reference, relationship reference, Memory source/pull reference, captured context, batch manifest, workflow reference, external trigger payload.

**Outputs:** inline result, DataRef, record address/revision, relationship run result, partial batch manifest, external action receipt, workflow/analysis/decision/narrative references, diagnostics.

Prefer references to stored results over copying large payloads into the run ledger. Default DataRef threshold: **1 MiB**.

### DAG and dependencies

Rules:

- Unique `stepId`s
- `dependsOn` edges; topological execution
- Fan-out / fan-in and parallelism for independent branches
- Cycle rejection; unreachable steps rejected at validate
- Output-to-input via `inputs` DataRef expressions (`fromStep`, `key`, `path`)
- Disabled steps skipped

Loops are not a graph feature. Repeated work uses batches, retries, schedules, or an explicit redesign — not hidden cycles.

### Control flow (supported)

Sequential execution · parallel independent branches · per-item batch work · short-circuit on terminal failure · continue-on-item-failure for batches · optional/disabled steps · cancellation.

**Unsupported as global features:** cross-service distributed transactions, hidden graph cycles, silent compensation without an explicit step. Atomic batch is **opt-in** and only when the backing port declares support.

---

## 2. Execution model

### Run, step, attempt

```text
Pipeline definition
 → Run (one execution)
 → Step (one node in the definition)
 → Attempt (one try)
```

Retries create **attempts**; they do not duplicate logical steps.

### Run statuses

| Status | Meaning | Terminal? |
|---|---|---|
| `pending` | Queued | No |
| `running` | Active | No |
| `succeeded` | All required work ok | Yes |
| `failed` | Terminal failure | Yes |
| `partial` | Explicit partial success (e.g. batch) | Yes |
| `cancelled` | Operator/runtime cancel | Yes |

### Step statuses

`pending` · `running` · `succeeded` · `failed` · `skipped`

### Durable production runtime

Production requires durable run ledger, durable artifact store, real canonical ports, real service bindings, and explicit production mode.

Production rejects: missing ledger, missing artifact store, required missing ports, stub services without opt-in, in-memory operational fallback presented as prod, unknown step kinds, invalid definitions.

Development stubs require explicit opt-in.

### At-least-once execution

A mutating step may be attempted again. Services and mutation ports must be idempotent. A successful external effect may occur before the runtime sees the response — retries must not create duplicate records, links, or external actions.

### Idempotency

| Concern | Mechanism |
|---|---|
| Memory pulls | Cursor + content-hash dedup |
| Raw persistence | Content hash / absent-or-revision |
| Record writes | Precondition + idempotency key |
| Relationship writes | Deterministic `relationshipId` |
| Mapping | Pure replay |
| Analysis / decision / narrative writes | Discriminator + revision / key |
| External actions | Mandatory key + receipt reuse |
| AI jobs | Captured input; one accepted mutation |
| Batch items | Per-item status (`already_applied`) |
| Artifacts | Content hash |

Mutating step kinds include: `persist-raw`, `write-record`, `relationship-write`, `workflow`, `narrative`. These require `idempotencyKeyExpr`.

### Retry

Default policy: `maxAttempts: 3`, `backoffMs: 1000`, `backoffMultiplier: 2`, `maxBackoffMs: 60000`.

Classify errors as retryable vs terminal. Support automatic retry, manual retry of one failed step, and retry of failed batch items only. Each attempt is recorded.

### Resume

If the runtime process stops, the durable ledger determines completed steps. Succeeded upstream work is not rerun unnecessarily. Failed/incomplete work resumes with captured context and stable artifact references.

### Partial success

Examples: record write succeeded + relationship write failed; valid batch items + invalid items. Per-item ledger; retry failed portion only; final run status may be `partial`; artifact holds the partial manifest.

### Atomicity

Default: **no** global pipeline transaction. Successful prior steps remain successful. Atomic batch is opt-in only when supported — otherwise rejected before execution. Do not claim cross-service global atomicity.

### Cancellation

Cancel queued work or active attempts. Respect remote-service limits. If an external side effect may already have occurred, record uncertainty honestly. Retain artifacts per retention policy.

---

## 3. Data movement

### Inline vs DataRef

Use inline for small JSON only. Use artifacts for large payloads, batches, profiles, model responses, documents, captured context, and exports.

```ts
type DataRefV2 = {
 kind: "inline" | "artifact";
 data?: unknown;
 artifactId?: string;
 contentHash: string;
 sizeBytes: number;
 // optional addressing / association
 collection?: string;
 objectType?: string;
 contentType?: string;
 recordId?: string;
 runId?: string;
 stepId?: string;
 attemptId?: string;
 mediaType?: string;
};
```

Wrong-org or missing/expired artifacts fail closed with structured errors.

### Captured inputs

Before retry-sensitive work, capture exact record revisions, selected sibling content, abstract projections, relationship-expanded content, mapping/service fingerprints where contracted, prompt/template fingerprint for AI, and source/pull references.

**Record reference** asks for *current* Memorix content. **Captured DataRef** asks for the exact artifact frozen for the run. Retries use captured context even if Explore data later changes.

### Batch manifests

Per item: `itemId`, content address/reference, expected revision, status (`succeeded` | `failed_retryable` | `failed_terminal` | `skipped_dependency` | `already_applied`), attempt, output reference, error, retry eligibility.

---

## 4. Step types, one by one

Registered `StepKind` values:

`memory-pull` · `persist-raw` · `map-normalize` · `validate` · `identity-dedupe` · `write-record` · `relationship-discover` · `relationship-write` · `compose` · `verify` · `workflow` · `enrich` · `analyze` · `decide` · `narrative`

Behavioral steps may reference `serviceId` for custom/AI work. Generic invoke is expressed through these kinds (and pack-level `invoke-service` where metadata uses that authoring alias) — not a second runtime.

### 4.1 `memory-pull`

**Purpose:** Invoke one configured Memory source.

Cover: scheduled/on-demand, knowledge vs context mode, `sourceId`, workRef, cursor, pull/run reference, quarantine, context expiry, dedup, exact raw payload. Credentials via `credentialRef` only.

Memory owns raw landing end-to-end for the pull; `persist-raw` is used when the graph needs an explicit handoff step in front of later ports — do not document two competing “ways to land raw.”

**Ops:** Pull procedures from `opx-source` into `sandbox*`; land raw for **PROC-2**.

### 4.2 `persist-raw`

**Purpose:** Persist normalized Memory handoff as `<objectType>-raw` with exact provider `.data`, provenance, absent/revision behavior, dedup hash, directory registration — when the plan separates pull from persist.

**Anti-pattern:** Decorating raw `.data` or promoting to snapshot inside this step.

### 4.3 `map-normalize`

**Purpose:** Execute the shared mapping AST (`@x12i/memorix-mapping`).

Source mapping, registered transforms, no arbitrary JavaScript, no I/O, deterministic replay. Mapping errors are structured. Prefer this over inventing a service for renames/shapes.

### 4.4 `validate`

**Purpose:** Validate data/drafts/contracts.

Distinguish V2 format validation, metadata validation, mapping output validation, service output validation, and domain validation services. Output structured errors — not message-only failures.

### 4.5 `identity-dedupe`

**Purpose:** Establish or resolve identity safely.

`recordId`, external identifiers, record directory, Memory content hash, duplicate identity conflict, semantic duplicate *candidates*. No silent merge. No invented record IDs.

### 4.6 `write-record`

**Purpose:** Canonical writer — `add` | `upsert` | `patch` | `replace` | `remove`.

Absent/revision preconditions, discriminator addressing, lifecycle, stale conflicts, directory behavior, idempotency, partial batch writes. This is how drafts become stored subject content.

### 4.7 `relationship-discover`

**Purpose:** Compile/evaluate relationship metadata and calculate expected candidates.

Abstract endpoint unions, deterministic operators, registered resolvers, provider raw references, bounded discovery plans, dry-run, diagnostics. Discovery-only does not mutate links.

### 4.8 `relationship-write`

**Purpose:** Materialize candidates through the relationship engine.

Deterministic `relationshipId`, four-field link, idempotent writes, category routing, stale-link removal where required. No copied payload. No manual arbitrary links. Partial failure/retry supported.

### 4.9 `compose`

**Purpose:** Retrieve selected sibling content for one `recordId`.

Selected content types, arrays per type, discriminated instances. No flattening. No relationship definition. Prefer captured composition for retry-sensitive downstream AI/analysis.

### 4.10 `enrich`

**Purpose:** Deterministic or service-based enrichment.

Preferred flow: capture → service/mapping → validate → `write-record`. Do not hide uncontrolled persistence inside enrich.

**Ops:** Enrich **WI-100** priority via `opx-completion` / enrich service.

### 4.11 `analyze`

**Purpose:** Produce analysis content/artifacts.

Captured context, deterministic or AI service, required `analysisType` discriminator, output validation, DataRef, sibling V2 persistence through writer, repeated-run idempotency.

### 4.12 `decide`

**Purpose:** Produce decision/recommendation content with `decisionType`.

Consume analysis inputs; keep rationale as domain content where appropriate; operational provenance separate. A decision does **not** perform an external action — that is a later action service step.

### 4.13 `narrative`

**Purpose:** Produce narrative content or artifact.

Current/captured context, relationship expansion, deterministic/AI service, validation, content versus artifact. No legacy narrative mutation path. Mutating when it persists — requires idempotency key.

### 4.14 `workflow`

**Purpose:** Read/update workflow sibling content or invoke workflow actions.

Exact revision, composition, workflow action service, state transition, retries. No separate workflow persistence model.

### 4.15 `verify`

**Purpose:** Explicit verification — relationship verify, data/directory checks, output postconditions — selected by configuration. Do not use one vague verify to hide unrelated behaviors.

### External actions and intelligence

External actions are **service invocations** (action role) from enrich/workflow/generic service steps with declared side effects — not a separate hidden step kind unless your pack aliases one. Structural intelligence is a service/pipeline pattern: deterministic profiling first, proposal artifacts, optional AI, **no** metadata auto-install, **no** data/link mutation inside the proposal step.

---

## 5. Pipeline patterns

Patterns reuse the same runtime. Each is a graph shape, not a product line.

| Pattern | Typical graph |
|---|---|
| Scheduled knowledge ingestion | `memory-pull` → raw → `map-normalize` → `validate` → `identity-dedupe` → `write-record` → relationship discover/write |
| On-demand context | `memory-pull`(context) → capture → work steps → TTL/discard per Memory policy |
| Raw-only enrichment | External object stays raw; enriches another subject — no auto snapshot |
| Raw-to-snapshot promotion | Explicit map/validate/write — never automatic merely because raw exists |
| Record enrichment | Capture → enrich → validate → exact-revision write |
| Completion | Mapping or service/AI proposes fields → validate → one write |
| Relationship | Discover → write → verify; optional rebuild / stale removal |
| Analysis / decision / narrative | Capture → analyze/decide/narrative → validate → writer |
| Workflow | Read workflow sibling → actions → exact-revision update |
| External-action | Validate prerequisites → idempotent action service → receipt → separate domain/workflow update |
| AI | Capture → AI service → store raw response artifact → validate → persist via ports |
| Structural intelligence | Profile → proposal artifacts → human install outside the step |
| Verification / maintenance | Relationship verify/rebuild, directory reconciliation, metadata validation, artifact cleanup via canonical ports |
| Small / large batch | Manifest + per-item results; DataRef + bounded concurrency + retry failed items |
| Export / report | Paginate internally; durable artifact; avoid unbounded HTTP lists |
| Multi-service | Several services connected through artifacts; clear persistence boundaries |
| Scheduled vs on-demand | Same definition/runtime; different trigger |

---

## 6. Authoring process

1. Define the business/technical outcome (e.g. “PROC-2 raw → procedure snapshot + WI-100 links”).
2. Confirm a pipeline is necessary (not a lone mapping/service).
3. Define inputs/outputs and artifact boundaries.
4. Identify mappings and services.
5. Choose step kinds; draw dependencies.
6. Mark mutation/side-effect steps; define idempotency keys.
7. Define retries, timeouts, captured context, batch/atomicity.
8. Validate definition; simulate pure steps.
9. Run in an isolated `sandbox*` scope; inspect graph/ledger/artifacts.
10. Install metadata deliberately; schedule or invoke.

### Step design checklist

Every step needs: one clear responsibility, typed I/O, bounded work, declared side effects, idempotency behavior, timeout, retry classification, useful error codes, artifact policy, observable result.

### Graph design best practices

- Separate computation from persistence.
- Separate relationship discovery from link writes.
- Capture inputs before retry-sensitive work.
- Parallelize only independent branches.
- Avoid hidden service-side orchestration.
- Use DataRef between large steps.
- Keep mutation boundaries obvious.

### Scheduling

Schedule metadata references a pipeline (`scheduleId`, cron, timezone, enabled, input, concurrency). Overlapping-run and missed-run policies follow the final schedule contract. Scheduling is not a second pipeline definition.

### On-demand invocation

Provide input + scope (+ optional run key). Receive `runId`. Wait / poll / cancel via the Capability SDK (or Operate / `/api/pipelines`). Read outputs via artifacts and record references.

**Preferred — `@x12i/memorix-sdk/client`:**

```ts
import { createMemorixClient } from "@x12i/memorix-sdk/client";

const mx = createMemorixClient({
  baseUrl: "http://127.0.0.1:5100",
  orgId: "sandbox7a3c",
  agentIds: ["opx"],
});

const run = await mx.pipelines.start("opx-knowledge-pull", {
  sourceId: "opx-source",
});
const completed = await mx.pipelines.wait(run.runId);
const arts = await mx.pipelines.listArtifacts(run.runId);
```

Raw HTTP (same wire contract):

```http
POST /api/pipelines/start
x-memorix-org-id: sandbox7a3c
x-memorix-agent-ids: opx
Content-Type: application/json

{
 "pipelineId": "opx-knowledge-pull",
 "input": { "sourceId": "opx-source" }
}
```

Also: `GET …/runs/{runId}`, `POST …/cancel`, `POST …/retry-step`, `POST …/resume`, `GET …/artifacts`.
---

## 7. Services inside pipelines

- Pipeline references logical `serviceId`.
- Metadata defines the logical contract.
- Runtime registry resolves handler/endpoint.
- Missing production binding fails closed.
- Stubs require explicit dev mode.
- Large I/O through DataRef.
- Timeouts/retries follow step + service policy.
- External actions require idempotency keys and receipts.

See [Building Services](../../07-services/developers/BOOK.md).

---

## 8. AI pipelines

AI is **optional service execution**, not core pipeline intelligence.

1. Capture exact record revisions, composition, relationship-expanded context, prompt/template fingerprint, model/service binding evidence.
2. Invoke AI service step.
3. Store raw response artifact/hash.
4. Parse and validate structured output.
5. Persist only through canonical ports.
6. Several model attempts may differ; **one** accepted mutation applies.
7. Recorded-response testing for acceptance; air-gapped execution supported.
8. Failure/skip/fallback is explicit policy — never silent fake success.
9. Metadata proposal pipelines never install metadata directly.

Deterministic fixtures only for platform acceptance — **no live AI required**.

---

## 9. Batches, concurrency, and writes

- Per-item batch ledger and partial success.
- Bounded concurrency and backpressure.
- Rate-limited services respect retry backoff.
- Revision conflicts → re-read and retry write with new revision.
- Duplicate delivery → `already_applied` / receipt reuse.
- Relationship and external-action idempotency as above.
- Atomic batch opt-in only; retry failed items only by default.

---

## 10. Operations

| Need | Where |
|---|---|
| Start / retry runs | `/api/pipelines` or Studio Operate |
| Run graph / ledger | Pipeline runtime collections (org-scoped operational DB) |
| Step / attempt inspection | Attempt documents with status, error, item outcomes |
| Artifacts | Artifact store + DataRef |
| Schedules | Design schedules + Operate history |
| Memory diagnostics | Memory pull/quarantine ledgers |
| Relationship diagnostics | `/api/relationships` verify/rebuild |
| AI evidence | Request/response artifacts |

### Operator diagnosis

1. Open failed attempt — read structured `error.code`.
2. Compare captured context artifact vs current Explore data.
3. Fix mapping/service/metadata; reinstall if needed.
4. Retry failed step or failed batch items only.
5. After process restart, confirm resume without redoing succeeded steps.
6. Confirm Explore for **WI-100** / **Alex** / **PROC-2**.

### Retention

Keep run evidence needed for audit; clean up expired context artifacts per Memory/policy. Do not delete ledgers that explain production incidents until retention allows.

---

## 11. Errors (catalog)

| Code class | Layer | Retryable? | Safe action |
|---|---|---|---|
| Invalid definition / unknown step / cycle | Validate | No | Fix pack |
| Missing dependency / invalid reference | Validate | No | Fix graph |
| Missing service / port | Startup/run | No | Bind registry / ports |
| Missing prod ledger/artifact store | Startup | No | Configure durable stores |
| Invalid scope / input | Run | No | Fix caller |
| Mapping / validation failure | Step | No | Fix mapping/data |
| Identity conflict | Step | No | Resolve identity |
| Stale revision | Write | No auto | Re-read; new revision |
| Relationship discover/write failure | Step | Often partial | Retry relationship portion |
| Memory / cursor failure | Step | Often yes | Fix source; resume cursor |
| Artifact missing / wrong org | DataRef | No | Restore / fix scope |
| Timeout / cancelled | Attempt | Policy | Retry or accept cancel |
| Retry exhausted | Step | No | Manual intervene |
| Partial batch | Run | Mixed | Retry failed items |
| Invalid AI output | Step | Policy | Repair or fail |
| External action uncertain/failed | Step | Keyed only | Inspect receipt |
| Unsupported atomicity | Validate | No | Disable atomicBatch |
| Schedule error | Scheduler | No | Fix cron/config |

Sanitize all evidence — no secrets, no unrestricted payloads.

---

## 12. Testing

Require:

- Definition/schema and graph/compiler tests
- Pure-step unit tests; service contract tests; port wiring tests
- Production fail-closed tests
- Real Mongo ledger/artifact tests where contracted
- Retry/resume and process-restart tests
- Duplicate-delivery and partial-batch tests
- Atomicity rejection/support tests
- Schedule tests
- AI recorded-response tests
- Complete opx E2E on fresh `sandbox*` with `agentIds: ["opx"]`

Templates and examples must validate against the accepted tree in CI.

---

## 13. Complete opx tutorial

End-to-end non-cyber drill. Use a **fresh** `sandbox*` org every acceptance run.

1. **Define procedure source** in opx metadata (`opx-source`).
2. **Define connector** (Memory collect; `credentialRef` only).
3. **Define knowledge-ingestion pipeline** (`memory-pull` → raw → map → validate → write → relationships).
4. **Pull procedures**; confirm raw for **PROC-2**.
5. **Inspect raw** in Explore / data API (`procedures-raw`).
6. **Promote** selected procedure snapshot via map/validate/`write-record` — never automatic.
7. **Pull work-item context** for **WI-100** (context mode as configured).
8. **Normalize** employee/contractor data (mapping and/or normalize service).
9. **List People** abstract (employees + contractors including **Alex**).
10. **Materialize assignments** (relationship discover + write).
11. **Create work-item ↔ procedure relationship** (WI-100 ↔ PROC-2).
12. **Run enrichment** (priority completion / enrich service).
13. **Produce analysis** (`analyze` + `analysisType`).
14. **Produce decision** (`decide` + `decisionType`).
15. **Generate narrative** (recorded AI adapter acceptable).
16. **Update workflow** sibling with exact revision.
17. **Trigger controlled relationship failure** (e.g. break a definition temporarily).
18. **Retry relationship step only**; confirm records not duplicated.
19. **Restart runtime** during another run.
20. **Resume** without repeating completed work.
21. **Inspect captured context** artifacts vs current data.
22. **Inspect artifacts** (hashes, sizes, step association).
23. **Inspect final records/links** in Explore — four-field links, no copied payloads.
24. **Run scheduled** invocation; confirm schedule history.
25. **Run on-demand** invocation with explicit input; confirm distinct `runId`.

Tutorial companion narrative: [documentation/tutorials/04-memory-and-pipeline.md](../../../documentation/tutorials/04-memory-and-pipeline.md).

---

## 14. Antipatterns and troubleshooting

### Antipatterns

| Antipattern | Prefer |
|---|---|
| Pipeline for a pure rename | Mapping |
| Service that hides persistence | Capture → service → validate → write |
| Auto-promote raw to snapshot | Explicit map/validate/`write-record` |
| Ad-hoc relationship `PUT` | Discover → relationship-write via engine |
| Secrets in pipeline JSON | `credentialRef` on sources/services |
| AI auto-installs metadata | Proposal artifact + human install |
| Global transaction assumption | Explicit partial success; retry failed portion |
| Live AI for opx acceptance | Recorded adapter / fixtures |
| Re-schedule to “fix” resume | Resume from durable ledger |
| Direct Mongo domain writes | Data plane + relationship engine |

### Troubleshooting decision tree

```text
Fails before any step?
├─ Invalid definition / cycle / unknown step → fix pack
├─ Missing service/port/ledger → bind durable stores
└─ Wrong scope → fresh sandbox* + ops

Map/validate fails?
└─ Fix mapping against raw for PROC-2 / WI-100

Write stale?
└─ Re-read or new run; do not force overwrite blindly

Green run, empty Explore?
├─ Missing write-record?
└─ Wrong objectType/contentType?

Links missing?
├─ Discover without write?
└─ Verify/rebuild not run?

Partial batch?
└─ Retry failed items only — do not redo succeeded writes

AI flaky in acceptance?
└─ Recorded adapter; inspect request/response artifacts
```

---

## 15. Reference

### Status transitions (summary)

```text
Run: pending → running → succeeded | failed | partial | cancelled
Step: pending → running → succeeded | failed | skipped
Attempt: recorded per try with startedAt / finishedAt
```

### Idempotency matrix (short)

| Step family | Key / precondition |
|---|---|
| Memory / raw | Cursor + content hash |
| write-record / workflow / narrative | `idempotencyKeyExpr` + revision/absent |
| relationship-write | Deterministic relationshipId |
| External action service | Business idempotency key + receipt |
| AI | Captured input; single accepted write |

### Packages

| Package | Role |
|---|---|
| `@x12i/memorix-sdk` | **Developer edge** — typed pipeline client, worker kit, authoring |
| `@x12i/memorix-client` | Canonical HTTP foundation under the SDK |
| `@x12i/memorix-pipeline` | Engine, step kinds, validation |
| `@x12i/memorix-pipeline-runtime` | Org-scoped runs/attempts/artifacts |
| `@x12i/memorix-mapping` | Deterministic map steps |
| `@x12i/memorix-memory` | Pull / raw landing / commitPage |
| `@x12i/memorix-data` | Writer/reader ports |
| `@x12i/memorix-relationships` | Discover/materialize |
| `@x12i/memorix-service` | HTTP API |
| `@x12i/memorix-opx-pack` | Fixtures + acceptance |
### Production-readiness checklist

- [ ] One clear outcome; pipeline justified vs mapping/service alone
- [ ] Valid DAG; mutating steps have idempotency keys
- [ ] Captured context for retry-sensitive / AI steps
- [ ] Prod bindings + durable ledger/artifact store
- [ ] Partial-batch and resume tested on `sandbox*`
- [ ] No secrets in definitions or evidence
- [ ] Explore confirms Alex / WI-100 / PROC-2 outcomes

### Glossary

| Term | Meaning |
|---|---|
| Pipeline | Declarative step graph |
| Run | One execution of a pipeline |
| Step | One node in the definition |
| Attempt | One try of a step |
| DataRef | Inline or artifact reference |
| Partial | Explicit incomplete success |
| Pattern | Reusable graph shape on one runtime |

### Next

- [Building Services](../../07-services/developers/BOOK.md)
- [Intelligence](../../10-intelligence/developers/BOOK.md)
- [KnowX](../../12-knowx/developers/BOOK.md)
- Studio twin → [Builders](../builders/BOOK.md)

---

# Building Pipelines — builders

# Building Memorix Pipelines — Builders

**Audience:** Studio operators designing, running, and troubleshooting pipelines. 
**Twin:** [Developers version](../developers/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Prerequisites:** [Services (builders)](../../07-services/builders/BOOK.md) · [Relationships & Memory (builders)](../../06-relationships-and-memory/builders/BOOK.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense Studio guide. Deterministic opx fixtures only — **no live AI required**.

---

## How to read this guide

1. Learn the **builder mental model** — pipeline vs mapping vs service vs Memory.
2. Use Studio **Design → Operate → Explore** in that order.
3. Practice retries and artifact inspection on a fresh `sandbox*` org (Alex, WI-100, PROC-2).
4. Deep implementation detail lives in the [developers twin](../developers/BOOK.md).

---

## The central idea

> **Pipelines orchestrate. Mappings transform fields. Services do specialized or remote work. Memory lands exact source material. Relationships become navigation links. Explore shows the result.**

A **pipeline** is an ordered, retryable set of steps that moves work from sources to stored subjects — with a full run history.

Host products start and wait on runs via **`@x12i/memorix-sdk/client`** (`pipelines.start` / `wait`) or Studio Operate — see the [developers twin](../developers/BOOK.md).

```text
Source → Memory (exact bytes)
 → Pipeline transforms & writes subjects
 → Relationships appear as links
 → Explore shows composition + links
```

```mermaid
flowchart LR
 Mem[Memory raw] --> Pipe[Pipeline]
 Pipe --> Map[Mapping]
 Pipe --> Svc[Service / AI]
 Pipe --> Write[Subject content]
 Write --> Rel[Relationship links]
 Pipe --> Art[Run artifacts]
```

See also [diagrams/pipeline-flow.md](../diagrams/pipeline-flow.md).

Ops subjects: employee **Alex**, work item **WI-100**, procedure **PROC-2**.

---

## 1. Pipeline vs the neighbors

| Construct | You use it when… |
|---|---|
| **Mapping** | Pure field transform (no outside calls) |
| **Service** | Custom code, external call, or AI job |
| **Memory** | Land exact provider payload as raw |
| **Relationship definition** | Connect different subjects by rules |
| **Pipeline** | Several steps must run in order, retry, schedule, and leave evidence |

A **workflow** on a work item is ordinary subject content (process state). A pipeline may update it — it is not a second product.

Ingestion, enrichment, relationships, AI jobs, and maintenance are **patterns** on one pipeline product — not separate engines.

---

## 2. Where in Studio

| Need | Go to |
|---|---|
| Define pipeline | Design → Pipelines |
| Attach services / mappings | Design → Services / Mappings, then pipeline steps |
| Schedule | Design → Schedules |
| Watch a run | Operate → Runs / Attempts |
| Inspect failure | Operate → Failures + Artifacts |
| Confirm outcome | Explore → subject composition + relationships |

![Pipeline designer](../screenshots/placeholder-pipeline-designer.png)

![Failed step](../screenshots/13-failed-pipeline-step.png)

![Successful retry](../screenshots/14-successful-retry.png)

![Captured artifact](../screenshots/15-artifact-captured-context.png)

### Scope bar (always)

1. Fresh `sandbox*` organization — never a live customer org.
2. Agent **`opx`**.
3. Confirm Design / Operate / Explore show that same scope.

---

## 3. What you author (without drowning in fields)

In Design you declare:

- Pipeline name and id (stable logical name)
- Trigger: **scheduled** or **on-demand**
- Steps in order (pull → map → validate → write → relate → enrich/analyze/…)
- Which service or mapping each step uses
- That mutating steps are safe to retry (engineering sets idempotency; you verify in Operate)

You do **not** paste secrets into the pipeline. Sources and services use **credential references**.

### Step kinds you will see

| Kind (plain language) | Job |
|---|---|
| Memory pull | Fetch from a source into raw |
| Map / normalize | Deterministic field shaping |
| Validate | Structured checks before write |
| Write record | Persist subject content |
| Relationship discover / write | Propose then materialize links |
| Compose | Gather siblings for one subject |
| Enrich / analyze / decide / narrative | Add derived content (may call a service or AI) |
| Workflow | Update process state on the subject |
| Verify | Check links or postconditions |

### Healthy promotion shape

```text
memory-pull (opx-source)
 → map-normalize / validate
 → write-record (snapshots)
 → relationship discover → write
 → enrich / analyze (optional)
```

Promotion is never automatic merely because raw exists. Raw for **PROC-2** can sit until a pipeline explicitly maps and writes a snapshot.

---

## 4. Runs, attempts, and artifacts

| Word | Meaning |
|---|---|
| **Run** | One execution of a pipeline |
| **Step** | One box in the design |
| **Attempt** | One try of a step (retries add attempts) |
| **Artifact** | Stored proof of large input/output or captured context |

Operate shows status: pending, running, succeeded, failed, partial, cancelled.

**Partial** means some items or branches succeeded — retry only what failed. Do not assume the whole world rolled back.

**Captured context** freezes the exact revisions a step saw. If someone edits WI-100 in Explore later, a retry still uses the captured artifact unless you intentionally start a new run.

---

## 5. Builder rules that prevent pain

| Do | Don't |
|---|---|
| Keep steps small and restartable | Hide all business logic only in unreproducible live AI |
| Use credential refs | Paste secrets into fixtures or records |
| Separate “compute” from “write” | Expect a green enrich step to mean data was stored if no write step exists |
| Verify Explore after a green run | Assume links exist without opening relationships |
| Retry from Operate after a fix | Duplicate schedules blindly |
| Prefer mappings for simple transforms | Ask for a service for every rename |
| Treat intelligence proposals as reviewable | Expect auto-install of metadata |

Hard rules: four-field relationships only (no copied neighbor payloads), abstracts are virtual views, intelligence proposals never auto-install, acceptance uses deterministic fixtures (**no live AI required**).

---

## 6. Retry etiquette

1. Read the failure code and artifact context in Operate.
2. Fix metadata, mapping, service binding, or source issue.
3. Retry from Operate (failed step or failed batch items).
4. Confirm input revisions in artifacts after success.
5. Open Explore — **WI-100**, **Alex**, **PROC-2** — and confirm composition + links.

If the runtime restarted mid-run, resume should **not** redo completed steps. If it does, escalate — that is a platform defect, not something to “fix” by re-scheduling.

### Partial success in plain words

Example: snapshot write for WI-100 succeeded, relationship write failed. Retry the relationship step only. Do not re-pull the whole world unless Memory itself failed.

---

## 7. AI pipelines (builder expectations)

AI steps are ordinary pipeline steps that call an AI **service**:

1. Pipeline captures exact context.
2. Model returns a draft.
3. Platform stores the raw response as an artifact.
4. Output is validated.
5. A write step persists sibling content if allowed.

You review proposals and installs explicitly. Air-gapped / recorded responses are valid for opx acceptance.

AI may **suggest** relationship rules or metadata. AI may **not** write arbitrary links or install packs by itself.

---

## 8. Short opx tour (success path)

Memory → pipeline → relationships — the acceptance path.

1. **Design** — confirm opx metadata and the knowledge-ingestion pipeline are installed for your `sandbox*` org.
2. **Operate** — run Memory pull from opx-source; confirm **PROC-2** raw landed (`credentialRef` only).
3. **Operate** — start the promotion / enrichment pipeline; watch attempts.
4. **Operate** — if a step fails, open the artifact, fix, retry once intentionally.
5. **Explore** — open **WI-100**; see snapshot / workflow / analysis siblings as designed.
6. **Explore** — traverse links to **Alex** and **PROC-2**; confirm navigation, not pasted fields.
7. **Design / Operate** — if intelligence proposed metadata, review and **explicitly** install or reject.
8. **Operate** — optional: run scheduled invocation once; confirm a distinct `runId` from on-demand.

Companion tutorial: [documentation/tutorials/04-memory-and-pipeline.md](../../../documentation/tutorials/04-memory-and-pipeline.md).

---

## 9. Operate checklist

- [ ] Scope: fresh `sandbox*` + `opx`
- [ ] Pipeline definition valid in Design
- [ ] Services/mappings referenced by steps exist for this agent
- [ ] Schedule enabled only when intended; overlapping runs understood
- [ ] Failures show attempts + artifacts
- [ ] Retries intentional
- [ ] Explore confirms subjects and links after success
- [ ] No secrets visible in artifacts or records
- [ ] Intelligence proposals reviewed — never assumed installed

### Quick diagnose

| Symptom | Likely cause |
|---|---|
| Fails before any step | Invalid definition, missing binding, or missing durable store in prod |
| Map step fails | Mapping/schema mismatch on raw |
| Write fails “stale” | Subject edited since capture — re-read or new run |
| Green run, empty Explore | Missing write step or wrong object/content type |
| Links missing | Discover without write, or verify/rebuild not run |
| AI step flaky | Expect recorded adapter for acceptance; check validation artifacts |
| Duplicate external ticket | Action step missing/broken idempotency — escalate |

---

## 10. What not to expect

- Automatic snapshot creation merely because raw exists
- A second “workflow product” separate from subject content
- Ad-hoc product link create/delete outside the relationship engine
- Global undo of every successful step after a later failure
- Live AI as a requirement for opx acceptance
- Direct Mongo “fixes” from Studio
- Cyber/vuln stories as the permanent product narrative

---

## Next

- Full step/runtime reference → [developers twin](../developers/BOOK.md)
- [Building Services — Builders](../../07-services/builders/BOOK.md)
- [Relationships & Memory](../../06-relationships-and-memory/builders/BOOK.md)
- [Workflow & Studio](../../09-workflow-and-studio/builders/BOOK.md)
- [Hard rules](../../HARD-RULES.md)

---

# Workflow & Studio — developers

# Workflow & Studio — Developers

**Audience:** Integrators calling workflow APIs or embedding Studio-adjacent host clients. 
**Twin:** [Builders version](../builders/BOOK.md) — **primary** Studio operator guide. 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Packages:** `@x12i/memorix-studio`, `@x12i/memorix-service`, `@x12i/memorix-format`, `@x12i/memorix-data`. 
**Prerequisites:** [Platform overview](../../00-platform-overview/developers/BOOK.md) · [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md) · [Data access](../../04-data-access-patterns/developers/BOOK.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense product documentation. Deterministic opx fixtures only — **no live AI required**.

---

## How to read this guide

1. Start with the **central mental model** — Studio is a service-only UI; workflow is ordinary sibling content.
2. Work chapters in order. HTTP embedding assumes you already know V2 envelopes and scope headers.
3. Labels used throughout:
 - **Stored** — workflow (and other) subject content behind `/api/workflow` and `/api/data`
 - **Virtual / calculated** — composition, abstract lists, relationship expansion in Explore
 - **Operational** — pipeline runs, memory pulls, ledgers (Operate)
4. Always drill in a fresh `sandbox*` org with `agentIds: ["opx"]`. Never use live/legacy customer orgs as acceptance.
5. For screen-by-screen Studio IA, prefer the **builders twin**. This book owns HTTP contracts and host embedding.

---

## The central mental model

Studio answers:

> **How do humans design metadata, operate runs, and explore subjects — without talking to Mongo or inventing a second UI?**

Workflow answers:

> **What process state sits on this subject as a first-class V2 sibling (`contentType: "workflow"`)?**

| Layer | Question it answers |
|---|---|
| Studio | Which operator surface am I on (Design / Operate / Explore)? |
| Scope bar | Which `orgId` + `agentIds[]` am I acting under? |
| Service | What is the only HTTP boundary Studio (and hosts) may call? |
| Workflow sibling | What assignment / review state lives on this `recordId`? |
| Revision | Which exact edit am I reading or safely replacing? |

```text
Host / Studio
 → only memorix-service (/api + /health)
 → scope headers (org + agentIds)
 → data plane / workflow / metadata / pipelines / …

Subject (e.g. wi-100 or ASSERTION-99)
 ├─ snapshots / raw / knowx / …
 └─ workflow ← ordinary sibling content, not a product database
```

Ops narrative: employee **Alex**, work item **WI-100**, procedure **PROC-2**. Fixture source: `memorix-opx-pack` + localhost `opx-source`.

---

## 1. Studio constraint (service-only UI)

`@x12i/memorix-studio` is the **sole Memorix product UI**. It talks **only** to `@x12i/memorix-service` via proxied `/api` and `/health`.

Rules:

1. Do **not** point custom UIs at package internals (`memorix-data`, Mongo drivers, metadata JSON on disk).
2. Do **not** embed Studio as a shell around direct database access.
3. Do **not** invent a parallel “admin console” that mutates domain collections.
4. Host apps that need Memorix capabilities call the **same OpenAPI surface** Studio uses.

### Local run

Prefer the monorepo stack:

```bash
node scripts/dev-stack.mjs
# starts opx-source + service + studio + stack-status; opens /_live tabs
```

Or local pair:

```bash
# terminal 1 — service (default :5100)
cd memorix-service && MEMORIX_SERVICE_MODE=dev npm run dev

# terminal 2 — studio (default :5101, proxies to service)
cd memorix-studio && npm run dev
```

| Surface | Default |
|---|---|
| Studio | http://127.0.0.1:5101 (`MEMORIX_STUDIO_PORT` / `VITE_DEV_PORT`, odd = UI) |
| Service | http://127.0.0.1:5100 (`MEMORIX_SERVICE_PORT`, even = API) |
| Service live view | http://127.0.0.1:5100/_live |
| Stack status | http://127.0.0.1:5103/ |
| Ops-source | http://127.0.0.1:5102/ (`OPX_SOURCE_PORT`) |
| Proxy target | `MEMORIX_SERVICE_PROXY_TARGET` or derived from service port |

Zone map: `@x12i/ports-manager` (Memorix **5100–5119**). HTTP compliance: `@x12i/core-service`. Studio binds an **odd** UI port and proxies API via `MEMORIX_SERVICE_PROXY_TARGET` (or derived service port) — it must not bind an even API port. Open for debug: Studio Explore/Operate, service `/_live` (`:5100/_live`), stack hub (`:5103`). Authoring a worker process: [Building Services — HTTP compliance](../../07-services/developers/BOOK.md#http-process-compliance-local-services).

Acceptance workspace: isolated `sandbox*` + `agentIds: ["opx"]` in the scope bar (builders twin shows the screenshot).

### Why this boundary exists

- One auth / scope / revision model for humans and machines.
- One OpenAPI contract (`GET /api/openapi.json`) for generated clients.
- No accidental Mongo-shaped writes that bypass directory, G4 relationships, or abstract rules.

---

## 2. Design · Operate · Explore (developer map)

Studio IA is three areas. Host UIs that “feel like Memorix” should mirror this split even when they only implement a subset.

```mermaid
flowchart TB
 subgraph Design
 A[Agents] --> T[Types]
 T --> Abs[Abstracts]
 Abs --> Rel[Relationships]
 Rel --> Pipe[Pipelines]
 end
 subgraph Operate
 R[Runs] --> F[Failures]
 F --> Art[Artifacts]
 end
 subgraph Explore
 C[Compose] --> L[Links]
 L --> W[Workflow]
 end
 Design --> Operate --> Explore
```

| Area | Purpose | Typical HTTP families |
|---|---|---|
| **Design** | Declare metadata; review intelligence proposals; install/publish | `/api/metadata/*`, `/api/intelligence/*` |
| **Operate** | Runs, attempts, schedules, failures, artifacts, memory, relationship verify/rebuild | `/api/pipelines/*`, `/api/memory/*`, `/api/relationships/*` |
| **Explore** | Concrete/abstract records, composition, relationships, workflow, analysis, decisions | `/api/data/*`, `/api/workflow` |

Operator UX rule (also for hosts): prefer **lists, forms, status, confirmations**. Raw JSON belongs under expandable technical details — never as the primary operator path.

---

## 3. Scope bar ≡ HTTP headers

Studio’s scope bar is not decoration. It is the same tenant boundary every API call must send.

```http
x-memorix-org-id: sandbox<suffix>
x-memorix-agent-ids: opx
```

| Rule | Detail |
|---|---|
| Org | Fresh `sandbox*` for acceptance — never live customer orgs |
| Agents | Plural header only; comma-separated when multiple |
| Singular `agentId` | Rejected with **400** |
| Effective metadata | Resolved for the supplied `agentIds[]` after install |

```bash
ORG=sandboxdemo01
H=(-H "x-memorix-org-id: $ORG" -H "x-memorix-agent-ids: opx")

curl -s "${H[@]}" "$SERVICE/api/metadata/effective" | head
curl -s "${H[@]}" "$SERVICE/health"
```

If Studio “looks empty,” check scope first — then whether the opx pack is installed for that agent set.

---

## 4. Workflow is sibling content

Workflow is **not** a separate product database and not a Mongo collection you invent. It is ordinary `memorix-record/2` content:

```json
{
 "format": "memorix-record/2",
 "recordId": "wi-100",
 "revision": 2,
 "objectType": "work-items",
 "contentType": "workflow",
 "dataCategory": "entity",
 "concept": {
 "title": "Workflow for WI-100",
 "identifiers": {
 "primary": {
 "kind": "workItemId",
 "value": "WI-100",
 "issuer": "opx-source"
 }
 }
 },
 "data": {
 "state": "in-progress",
 "assignedTo": ["opx-reviewers"],
 "steps": [
 { "id": "triage", "status": "done" },
 { "id": "opx-review", "status": "open" }
 ]
 },
 "_system": {
 "state": "active",
 "createdAt": "2026-07-22T09:00:00.000Z",
 "modifiedAt": "2026-07-22T11:00:00.000Z"
 }
}
```

Composition with the same `recordId` yields siblings — for example `work-items-snapshots` + `work-items-workflow` — without flattening.

KnowX review pattern (book `12`): assertions often carry:

- `contentType: "knowx"` — the claim
- `contentType: "workflow"` — assignment / pending-review
- `contentType: "confirmations"` — accept/reject decision

Do **not** stuff review UI state into provenance. Use workflow / confirmations siblings.

---

## 5. Workflow HTTP (`/api/workflow`)

OpenAPI family: **Workflow**. Two operations mirror Studio Explore workflow edit.

### 5.1 Read — `GET /api/workflow`

```http
GET /api/workflow?id=wi-100
x-memorix-org-id: sandbox…
x-memorix-agent-ids: opx
```

Response shape (conceptual):

```ts
{
 scope: MemorixScope;
 id: string | null;
 contentType: "workflow";
 record: MemorixRecordV2 | null;
 currentRevision: number | null;
 note?: string;
}
```

Notes:

- `contentType` is fixed to `"workflow"`.
- Missing workflow returns a null `record` (and null revision) rather than inventing empty steps.
- You can also read via the data plane (`/api/data/content?objectType=…&contentType=workflow&id=…`). The workflow family exists so operator clients have a stable, narrow contract.

### 5.2 Write — `PUT /api/workflow`

Exact-revision edit (optimistic concurrency):

```http
PUT /api/workflow
x-memorix-org-id: sandbox…
x-memorix-agent-ids: opx
content-type: application/json
```

```json
{
 "id": "wi-100",
 "expectedRevision": "2",
 "record": {
 "format": "memorix-record/2",
 "recordId": "wi-100",
 "revision": 2,
 "objectType": "work-items",
 "contentType": "workflow",
 "dataCategory": "entity",
 "concept": {
 "title": "Workflow for WI-100",
 "identifiers": {
 "primary": {
 "kind": "workItemId",
 "value": "WI-100",
 "issuer": "opx-source"
 }
 }
 },
 "data": {
 "state": "pending-review",
 "assignedTo": ["opx-reviewers"]
 },
 "_system": {
 "state": "active",
 "createdAt": "2026-07-22T09:00:00.000Z",
 "modifiedAt": "2026-07-22T11:00:00.000Z"
 }
 }
}
```

| Field | Role |
|---|---|
| `id` | Subject `recordId` |
| `record` | Full V2 workflow envelope (`contentType` must be `workflow`) |
| `expectedRevision` | Exact revision the client believes is current |
| `currentRevision` (optional) | When set and mismatched, service returns **409** |

Success response (conceptual): `scope`, `id`, `expectedRevision`, `currentRevision`, `status`, `conflict: false`.

Conflict: **409** — re-GET, merge operator intent, retry with the new revision. Studio shows this as a stale-edit conflict (see migration book screenshot `17-stale-conflict`).

### 5.3 Error matrix (workflow-focused)

| Status | Likely meaning | Fix |
|---|---|---|
| 400 | Bad body / validation / singular agent header | Fix envelope + plural `agent-ids` |
| 403 | Forbidden for this scope | Check org / agents / auth |
| 409 | Stale revision / conflict | Re-read; retry exact revision |
| 503 | Service unavailable (often durable mode without Mongo) | Set prod mode + `MONGO_URI`, or use documented dev mode |

Always validate locally when iterating:

```bash
npx memorix-format validate-v2 ./work-items-workflow.json
```

---

## 6. Ops narrative: WI-100 workflow edit

End-to-end acceptance sketch (deterministic fixtures):

```bash
ORG=$(node -e "console.log('sandbox'+Math.random().toString(36).slice(2,10))")
H=(-H "x-memorix-org-id: $ORG" -H "x-memorix-agent-ids: opx")

# 1) install opx pack (explicit confirm)
curl -s "${H[@]}" -X POST "$SERVICE/api/metadata/install" \
 -H "content-type: application/json" \
 -d '{"packId":"opx","confirm":true}'

# 2) ensure work-item snapshot + workflow exist (pipeline / fixtures)

# 3) read workflow sibling
curl -s "${H[@]}" "$SERVICE/api/workflow?id=wi-100"

# 4) put with exact expectedRevision from step 3
curl -s "${H[@]}" -X PUT "$SERVICE/api/workflow" \
 -H "content-type: application/json" \
 -d @wi-100-workflow-put.json

# 5) compose subject — workflow appears beside snapshots
curl -s "${H[@]}" \
 "$SERVICE/api/data/compose?objectType=work-items&id=wi-100"
```

Assertions:

- [ ] `format === "memorix-record/2"` and `contentType === "workflow"`
- [ ] Stale `expectedRevision` yields 409 (no silent overwrite)
- [ ] Composition lists workflow as a sibling, not nested inside snapshots
- [ ] Org matches `sandbox*`; agents `["opx"]`
- [ ] No direct Mongo mutation

---

## 7. Embedding guidance for host apps

When you build a host that embeds Memorix operator flows (or a thin custom panel beside Studio):

### Do

1. **Generate types** from `GET /api/openapi.json` — do not hand-roll divergent envelopes.
2. **Mirror Studio scope** — same org + `agentIds` headers on every call; surface them in your chrome the way Studio’s scope bar does.
3. **Mirror Studio IA** — Design / Operate / Explore (or a documented subset). Do not mix “edit metadata” and “edit subject workflow” into one undifferentiated screen without labels.
4. **Prefer operator widgets** — tables, forms, status chips, confirm dialogs. Hide raw JSON behind “Technical details.”
5. **Reuse workflow + data + relationships routes** — composition for subject views; `/api/workflow` for review assignment; relationship query for navigation.
6. **Respect revisions** — never “last write wins” without showing conflict.

### Don’t

| Anti-pattern | Why it fails |
|---|---|
| Point UI at Mongo / package internals | Bypasses directory, G4, abstracts, ledgers |
| Primary UI = raw JSON editor | Operators cannot review safely |
| Invent a second workflow store | Diverges from composition / KnowX review |
| Hard-code `fieldstone` / `neo` orgs | Violates isolation hard rules |
| Singular `agentId` header | Rejected |
| Auto-install intelligence proposals from a host button labeled “Apply” without confirm | Violates explicit metadata install |

### Minimal host sketch

```ts
const headers = {
 "x-memorix-org-id": orgId, // sandbox…
 "x-memorix-agent-ids": agentIds.join(","), // ["opx"]
 "content-type": "application/json",
};

async function getWorkflow(id: string) {
 const res = await fetch(`${serviceBase}/api/workflow?id=${encodeURIComponent(id)}`, {
 headers,
 });
 if (!res.ok) throw new Error(`workflowGet ${res.status}`);
 return res.json();
}

async function putWorkflow(body: {
 id: string;
 expectedRevision: string;
 record: MemorixRecordV2;
}) {
 const res = await fetch(`${serviceBase}/api/workflow`, {
 method: "PUT",
 headers,
 body: JSON.stringify(body),
 });
 if (res.status === 409) {
 // surface conflict UI — re-fetch and retry
 }
 if (!res.ok) throw new Error(`workflowPut ${res.status}`);
 return res.json();
}
```

For KnowX human review, sequence:

1. `GET` assertion composition (knowx + workflow + confirmations).
2. Operator decides accept/reject in UI (not by editing provenance JSON).
3. Write confirmations sibling + revision-safe patch on knowx epistemic state (book `12`).
4. Optionally update workflow `state` via `PUT /api/workflow`.

---

## 8. Relation to data plane and OpenAPI

| Need | Prefer |
|---|---|
| Generic content CRUD / list / compose | `/api/data/*` |
| Operator workflow edit | `/api/workflow` |
| Metadata design | `/api/metadata/*` |
| Runs / attempts | `/api/pipelines/*` |
| Contract discovery | `GET /api/openapi.json` |

Workflow writes still obey V2 validation, directory identity, and revision preconditions. The dedicated route does not create a side channel around the data plane — it is a **narrow façade** for a common operator content type.

---

## 9. Decision tree

```text
Am I building operator UX?
├─ Yes → Studio first; or host that calls only memorix-service
└─ No, headless integration
 └─ OpenAPI clients + same scope headers

Do I need process / assignment state on a subject?
├─ Yes → contentType workflow sibling (same recordId)
└─ No → do not invent a parallel “tasks” collection

Is this a KnowX claim review?
├─ Assignment → workflow sibling
└─ Decision → confirmations sibling (+ epistemic patch)

Am I about to show raw JSON as the main screen?
└─ Stop. Forms / status / confirmations first.

Did PUT /api/workflow return 409?
└─ Re-GET currentRevision; merge; retry expectedRevision

Am I tempted to write Mongo directly “just for Studio”?
└─ Stop. Service-only.
```

---

## 10. Anti-patterns

| Anti-pattern | Why it fails | Do this instead |
|---|---|---|
| Second admin UI on Mongo | Divergent truth / skipped contracts | Studio + service OpenAPI |
| Workflow as nested snapshot field | Breaks composition / revisions | Sibling `contentType: "workflow"` |
| Review state inside KnowX provenance | Mixes evidence with process | workflow + confirmations |
| Ignore `expectedRevision` | Silent overwrites | Exact revision + 409 handling |
| Singular agent header | 400 | `x-memorix-agent-ids` |
| Acceptance on live orgs | Contaminates customer data | Fresh `sandbox*` |
| Host bypasses service for “speed” | Breaks hard rule #8 (mutations) | Always `/api/*` |
| Treat Studio screenshots as API | Drift | OpenAPI + this book |

---

## 11. Tiny journey sketch

1. Fresh `sandbox*` + install opx pack with confirm.
2. Open Studio Explore (or host) on `wi-100` — composition shows snapshots + workflow.
3. `GET /api/workflow?id=wi-100` — note `currentRevision`.
4. Operator assigns `opx-reviewers` — `PUT /api/workflow` with exact revision.
5. Stale concurrent edit → 409 → refresh → retry.
6. For KnowX assertions, same pattern on `ASSERTION-99` with confirmations sibling (book `12`).

---

## 12. OpenAPI generation workflow

1. Run service locally (`dev-stack` or `memorix-service` alone).
2. Fetch `GET /api/openapi.json`.
3. Generate TypeScript (or other) clients with your org’s generator.
4. Prefer generated `workflowGet` / `workflowPut` operationIds over hand-rolled paths.
5. Regenerate when service OpenAPI changes — do not forever-pin a stale client against Studio.

Smoke after regenerate:

```bash
curl -s "$SERVICE/api/openapi.json" | jq '.paths["/api/workflow"] | keys'
```

Expect `get` and `put`. Treat other workflow-adjacent needs (list by assignee, bulk assign) as **product features on top of** sibling content — not as a reason to invent a second store.

---

## 13. Testing checklist (integrator)

- [ ] Scope headers on every call
- [ ] `GET /api/workflow` null-safe when sibling absent
- [ ] `PUT /api/workflow` 409 path covered in UI/tests
- [ ] Composition still lists workflow separately after edit
- [ ] KnowX review uses workflow + confirmations without provenance mutation
- [ ] Host never opens Mongo connection for domain writes
- [ ] Acceptance org is fresh `sandbox*`
- [ ] Operator primary UI is not a raw JSON blob

---

## 14. Next

| Track | Link |
|---|---|
| Studio screens (primary) | [Builders twin](../builders/BOOK.md) |
| KnowX review siblings | [KnowX developers](../../12-knowx/developers/BOOK.md) |
| Data plane compose / revisions | [Data access](../../04-data-access-patterns/developers/BOOK.md) |
| Metadata install | [Metadata & agents](../../05-metadata-agents-abstracts/developers/BOOK.md) |
| Stale conflicts / migrate | [Migration & troubleshooting](../../13-migration-troubleshooting/developers/BOOK.md) |
| OpX pack harness | [Agent packs & ops](../../11-agent-packs-and-opx/developers/BOOK.md) |

---

*End of Workflow & Studio — Developers.*

---

# Workflow & Studio — builders

# Workflow & Studio — Builders

**Audience:** Studio operators and product builders — **primary** guide for Design · Operate · Explore. 
**Twin:** [Developers version](../developers/BOOK.md) (HTTP / embedding). 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense operator documentation. Deterministic fixtures only — **no live AI required**.

---

## How to read this guide

1. Start with the **central idea** — Studio is the only product UI; it talks only to Memorix Service.
2. Set the **scope bar** before anything else.
3. Learn the **three areas** (Design / Operate / Explore), then the **workflow sibling** pattern.
4. Use the **opx tour** with Alex / WI-100 as your acceptance drill.
5. API detail and host embedding → [developers twin](../developers/BOOK.md).

---

## The central idea

> **Studio is how humans design, operate, and explore Memorix — without raw JSON as the primary surface and without touching Mongo.**

| You see | Meaning |
|---|---|
| Scope bar | Which org + agent set you are acting under |
| Design | Metadata and install/publish |
| Operate | Runs, failures, memory, relationship verify |
| Explore | Subjects, composition, links, workflow, analysis |
| Workflow editor | Process state on a subject — ordinary sibling content |

```mermaid
flowchart LR
 Design[Design] --> Operate[Operate]
 Operate --> Explore[Explore]
 Explore -.->|find gaps| Design
```

---

## 1. Scope bar first (always)

![Isolated workspace scope](../screenshots/01-isolated-workspace-scope.png)

Before any meaningful work:

1. Choose a **fresh** `sandbox*` organization (never a live customer org).
2. Set agent to **`opx`** (or the agent set your pack documents).
3. Confirm Design / Operate / Explore reflect that scope.
4. If screens look empty or “wrong types,” fix scope and metadata install before filing a bug.

Forbidden for acceptance drills: legacy/live customer org names. Always `sandbox*` + `opx`.

---

## 2. Design

Author and govern what the platform *can* run:

| Screen family | You do this |
|---|---|
| Agents | Inheritance and which vocabulary is in scope |
| Sources | Where Memory pulls from (e.g. opx-source) |
| Object / content types | What subjects exist and what content they may wear |
| Abstracts | Virtual unions (e.g. People) — not separate databases |
| Relationships | Definitions for navigation links |
| Mappings / views | Deterministic transforms and list shapes |
| Services / pipelines / schedules | Capabilities and ordered runs |
| Intelligence proposals | Review suggested metadata — **never** auto-live |
| Install / publish | Explicitly make a pack version effective |

Operator rule: proposals are suggestions. You **install** deliberately (see Intelligence builders book).

---

## 3. Operate

Watch and recover work in flight:

| Need | Go to |
|---|---|
| Pipeline / schedule activity | Runs / Attempts / Schedules |
| Why something failed | Failures + Artifacts |
| Source landing | Memory |
| Link health | Relationship verify / rebuild |
| Intelligence profiling runs | Intelligence runs (still does **not** install) |

Operate is where you prove the path:

```text
opx-source → Memory (exact raw) → Pipeline → subjects + links → Explore
```

Do not “fix” a failed step by hand-editing storage. Fix metadata/mapping/service, then retry from Operate.

---

## 4. Explore

![Composed subject](../screenshots/10-composed-subject.png)

Explore is where you inspect **truth about subjects**:

- Concrete records (employees, work-items, procedures, …)
- Abstract unions (People, …)
- **Composition** — siblings on the same id (snapshot + workflow + analysis, …)
- **Relationships** — four-field links between different ids (expansion shows current targets)
- Workflow / analysis / decisions editors

![Workflow edit](../screenshots/16-workflow-edit.png)

### Composition vs relationship (builder wording)

```text
Same subject id → composition (siblings on one page)
Different subject ids → relationship (link + expansion)
```

You should never see a pasted “associated employee payload” block inside a work-item snapshot. If you do, that content is wrong — escalate with a screenshot.

---

## 5. Workflow content (what you are editing)

Workflow is an ordinary **sibling content type** on a subject (`contentType: "workflow"`), not a separate product database.

Examples in the opx narrative:

- **WI-100** — work-item workflow (state, assignees, steps)
- **KnowX assertion** — review assignment while the claim stays in `knowx` and the decision lives in `confirmations` (see KnowX builders)

### Editing etiquette

1. Open the subject in Explore → Workflow.
2. Change state / assignment in the form (not by inventing nested snapshot fields).
3. Save — Studio sends an exact **revision**.
4. If you get a **stale conflict**, refresh, re-apply your intent, save again (screenshot in migration book).

### KnowX review pattern (Studio)

When reviewing a knowledge claim:

1. Open the assertion subject.
2. Check provenance / epistemic state on KnowX (evidence).
3. Use **workflow** for “who must review.”
4. Record accept/reject on **confirmations**.
5. After accept, relationship links appear for navigation in Explore — links are not a copy of the claim JSON.

---

## 6. Operator UX rules

| Do | Don’t |
|---|---|
| Prefer tables, forms, status, confirmations | Make raw JSON the primary review surface |
| Open technical JSON only when needed | Ask engineers to “just patch Mongo” |
| Keep Design / Operate / Explore jobs separate | Mix install, run, and subject edit without labels |
| Use fresh `sandbox*` for drills | Practice on live customer orgs |
| Explicit metadata install | Expect intelligence to “just apply” |
| Capture org, agents, subject, revision, run id on failure | Escalate with only “it broke” |

Studio talks **only** to Memorix Service. If someone claims Studio “writes Mongo,” they are wrong about the architecture.

---

## 7. Short opx tour (acceptance)

Subjects: employee **Alex**, work item **WI-100**, procedure **PROC-2**.

1. **Scope** — fresh `sandbox*` + agent `opx`.
2. **Design** — confirm opx metadata installed (explicit install if needed).
3. **Operate** — Memory pull from opx-source; confirm raw landed.
4. **Operate** — run the documented pipeline; open artifacts if anything fails.
5. **Explore** — open `WI-100`; confirm composition siblings (snapshot + workflow, …).
6. **Explore** — edit workflow assignment; save; confirm revision bump.
7. **Explore** — traverse relationships; confirm expansion, not copied payloads.
8. Optional: open a KnowX assertion and walk the review siblings (book `12`).

---

## 8. Decision tree

```text
Is the screen empty / wrong types?
└─ Check scope bar → then metadata install → then Memory/pipeline

Am I changing what kinds of things exist?
└─ Design (then explicit install/publish)

Am I watching a run or fixing a failure?
└─ Operate (artifacts first)

Am I inspecting or editing a subject?
└─ Explore (composition / workflow / links)

Is this a review assignment?
└─ Workflow sibling

Is this an accept/reject decision on a claim?
└─ Confirmations (+ KnowX epistemic) — not provenance edits

Did save conflict?
└─ Refresh → re-apply → save (exact revision)
```

---

## 9. Anti-patterns

| Anti-pattern | Why it hurts | Do this instead |
|---|---|---|
| Raw JSON as the main UI | Operators miss conflicts and evidence | Forms + status + confirmations |
| Nested “associated*” blobs | Stale copies / wrong model | Relationships + expansion |
| Review state inside provenance | Mixes process with evidence | Workflow / confirmations |
| Install on a shared live org “to try” | Contaminates customers | Fresh `sandbox*` |
| Retry blindly without reading artifacts | Hides root cause | Operate → Failure → Artifact |
| Second “admin tool” outside Studio | Divergent truth | Studio + service only |

---

## 10. Next

| Track | Link |
|---|---|
| HTTP / embedding | [Developers twin](../developers/BOOK.md) |
| Platform map | [Platform overview (builders)](../../00-platform-overview/builders/BOOK.md) |
| Metadata & install | [Metadata builders](../../05-metadata-agents-abstracts/builders/BOOK.md) |
| Pipelines | [Pipelines builders](../../08-pipelines/builders/BOOK.md) |
| KnowX review | [KnowX builders](../../12-knowx/builders/BOOK.md) |
| Intelligence review | [Intelligence builders](../../10-intelligence/builders/BOOK.md) |
| Install opx pack | [Agent packs builders](../../11-agent-packs-and-opx/builders/BOOK.md) |
| Conflicts & escalate | [Migration & troubleshooting builders](../../13-migration-troubleshooting/builders/BOOK.md) |

---

*End of Workflow & Studio — Builders.*

---

# Structural Intelligence — developers

# Structural Intelligence — Developers

**Audience:** Engineers integrating structural profiling and evidence-backed metadata proposals. 
**Twin:** [Builders version](../builders/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Packages:** `@x12i/memorix-intelligence`, `@x12i/memorix-service`, `@x12i/memorix-metadata-runtime`, `@x12i/memorix-opx-pack`. 
**Prerequisites:** [Metadata, agents & abstracts](../../05-metadata-agents-abstracts/developers/BOOK.md) · [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense product documentation. Deterministic fixtures first — **no live AI required** for acceptance.

---

## How to read this guide

1. Start with the **central mental model** — intelligence suggests; humans (or explicit APIs) install.
2. Internalize **owns vs does-not-own** before calling any API.
3. Walk **profile → propose → human install** in order; do not skip the human gate.
4. Labels used throughout:
 - **Stored** — installed metadata after explicit install/publish; ordinary subject records elsewhere
 - **Virtual / calculated** — structural profiles, proposal diffs, compare/export drafts
 - **Operational** — intelligence run ledgers / proposal artifacts (not subject content)
5. Always drill in a fresh `sandbox*` org with `agentIds: ["opx"]`.

---

## The central mental model

Structural intelligence answers:

> **Given sample records (and optional AI hints), what metadata changes are *evidenced* — and which must a human still approve?**

| Stage | Question it answers | Mutates live metadata? |
|---|---|---|
| Profile | What structure / cardinality / overlap do we see? | No |
| Propose | What typed suggestions follow from that evidence? | No |
| Compare / export | How does a draft differ from an approved pack? | No |
| Human review | Is the evidence strong enough? | No |
| Install / publish | Make definitions effective for this org | **Yes** (metadata-runtime / `/api/metadata`) |

```mermaid
flowchart LR
 Profile[profile] --> Propose[proposal]
 Propose --> Human[human review]
 Human --> Install[metadata install]
```

Hard product rule: **intelligence never auto-installs / auto-publishes metadata.** Deterministic fixtures are enough for acceptance; live AI is optional and untrusted until validation + human install.

---

## 1. Package boundary

**Package:** `@x12i/memorix-intelligence`

### Owns

- Structural profiles (paths, types, null vs missing, uniqueness, cardinality, overlap)
- Evidence-backed suggestions (object/content types, abstracts, mappings, relationships, views, pipeline stubs)
- Insufficient-evidence and rejected-by-validation outcomes
- Proposal compare / export against approved packs
- Optional AI adapter behind **one** boundary (candidates remain untrusted)

### Does not own

| Concern | Owner |
|---|---|
| Metadata install / publish | `@x12i/memorix-metadata-runtime` / `/api/metadata/*` |
| Record or relationship writes | Data plane / relationship engine |
| Pipeline execution | Pipeline runtime / `/api/pipelines` |
| Connectors / Memory pulls | Memory |
| Studio / HTTP façade | `@x12i/memorix-service` routes that *call* this package |

### When to install which package

| You are… | Prefer |
|---|---|
| Remote host app | HTTP `/api/intelligence/*` only |
| Trusted platform embedding profiling in-process | `@x12i/memorix-intelligence` |
| Applying a proposal | `/api/metadata/install` or `/api/metadata/publish` — **not** an intelligence route |

```ts
import { profileRecords, createProposal } from "@x12i/memorix-intelligence";

const profile = profileRecords(records, {
 scope: { orgId: "sandbox…", agentIds: ["opx"] },
 label: "employees",
});
```

---

## 2. Deterministic profile first

Profiling is **structural and deterministic**. Feed it sample records (fixtures or bounded reads) and get evidence — not vibes.

What a profile typically reports:

| Signal | Why it matters |
|---|---|
| Paths & value types | Candidate field maps / content shapes |
| Null vs missing | Optional vs required / mapping quirks |
| Uniqueness | Identifier / key candidates |
| Cardinality | One-vs-many content / relationship hints |
| Overlap across samples | Abstract union / shared shape candidates |

Air-gapped rule: heuristics must work with **no model credentials**. If AI is unavailable, profiling and proposal scaffolding still run; AI-only enrichment is skipped or marked insufficient.

### HTTP — profile

```http
POST /api/intelligence/profile
x-memorix-org-id: sandbox…
x-memorix-agent-ids: opx
content-type: application/json
```

Body (conceptual): scoped sample set + label (e.g. `employees`, `work-items`). Response: structural profile artifact — **not** an install.

Acceptance orgs: fresh `sandbox*` only — never live customer orgs as profiling playgrounds that might leak into install habits.

---

## 3. Propose from evidence

A proposal is a **draft metadata pack fragment** (or diff) tied to profile evidence. Outcomes you must handle:

| Outcome | Meaning | Next step |
|---|---|---|
| Evidence-backed suggestion | Enough structure to propose types/maps/… | Human review |
| Insufficient evidence | Samples too thin / conflicting | Gather more fixtures; do not install |
| Rejected by validation | Draft fails metadata/format rules | Fix samples or proposal rules; do not install |

```ts
const proposal = createProposal(profile, {
 scope: { orgId: "sandbox…", agentIds: ["opx"] },
 basePackId: "opx", // compare against approved opx when relevant
});
```

### HTTP — proposals & helpers

| Route | Role |
|---|---|
| `POST /api/intelligence/suggest/people` | Focused people/abstract-oriented suggestions |
| `POST /api/intelligence/proposals` | Create / list proposal artifacts |
| `POST /api/intelligence/compare` | Diff draft vs approved pack |
| `POST /api/intelligence/export-draft` | Export a draft for review / offline inspect |

None of these routes install metadata. If a host “Apply” button skips `/api/metadata/install` with `confirm: true` (or Studio’s explicit install), it is a product bug.

---

## 4. Human install (the only write path)

After review:

```bash
ORG=sandboxintel01
H=(-H "x-memorix-org-id: $ORG" -H "x-memorix-agent-ids: opx")

# Review proposal in Studio Design or via compare/export
curl -s "${H[@]}" -X POST "$SERVICE/api/intelligence/compare" \
 -H "content-type: application/json" \
 -d '{"proposalId":"…","againstPackId":"opx"}'

# Explicit install — separate family
curl -s "${H[@]}" -X POST "$SERVICE/api/metadata/install" \
 -H "content-type: application/json" \
 -d '{"packId":"opx","confirm":true}'
```

Rules:

1. **Never** auto-install on proposal create.
2. **Never** treat AI confidence as permission to publish.
3. Prefer installing into a fresh `sandbox*` first; promote only after acceptance.
4. Reject thin proposals — insufficient evidence is a successful, honest outcome.

Studio path: Design → intelligence proposal → review evidence → **Install / publish** deliberately (builders twin).

---

## 5. Optional AI adapter (untrusted)

AI may propose additional candidates behind one adapter:

```text
deterministic profile
 → (optional) AI candidates
 → validation / schema checks
 → proposal artifact
 → human install gate
```

| Rule | Detail |
|---|---|
| Air-gapped acceptance | Must pass without model credentials |
| Trust | AI output is untrusted until validation + human install |
| Secrets | No provider keys in fixtures; use credential refs if a service calls out |
| Failure mode | Missing AI → heuristics-only proposal or insufficient-evidence |

Do not require live AI in CI gates or opx-pack acceptance.

---

## 6. Ops narrative drill (employees / WI-100)

Deterministic path using opx fixtures:

1. Fresh `sandbox*` + `agentIds: ["opx"]`.
2. Install baseline opx pack (explicit).
3. Pull or load fixture employee / work-item samples (Memory or static JSON).
4. `POST /api/intelligence/profile` labeled `employees` (and/or `work-items`).
5. Create proposal; `compare` against approved `opx`.
6. In Studio (or HTTP), **reject** anything thin; **install** only after evidence review.
7. Re-resolve `/api/metadata/effective` — only then should new types/views appear.

Assertions:

- [ ] Profile works with AI disabled / no credentials
- [ ] Proposal create does not change effective metadata
- [ ] Install requires explicit confirm
- [ ] Rejected / insufficient-evidence paths are first-class
- [ ] Org is `sandbox*`; no live customer org

---

## 7. Embedding intelligence in a host

```ts
async function profile(scope: { orgId: string; agentIds: string[] }, samples: unknown[]) {
 const res = await fetch(`${serviceBase}/api/intelligence/profile`, {
 method: "POST",
 headers: {
 "x-memorix-org-id": scope.orgId,
 "x-memorix-agent-ids": scope.agentIds.join(","),
 "content-type": "application/json",
 },
 body: JSON.stringify({ label: "employees", records: samples }),
 });
 if (!res.ok) throw new Error(`profile ${res.status}`);
 return res.json();
}

// Separate function — never call from profile() automatically
async function installPack(scope: { orgId: string; agentIds: string[] }, packId: string) {
 const res = await fetch(`${serviceBase}/api/metadata/install`, {
 method: "POST",
 headers: {
 "x-memorix-org-id": scope.orgId,
 "x-memorix-agent-ids": scope.agentIds.join(","),
 "content-type": "application/json",
 },
 body: JSON.stringify({ packId, confirm: true }),
 });
 if (!res.ok) throw new Error(`install ${res.status}`);
 return res.json();
}
```

UI requirement: the control that installs must be a distinct, confirmed action — not a side effect of “Generate proposal.”

---

## 8. Evidence model (what proposals must carry)

A proposal without evidence is marketing copy. Platform proposals should be inspectable:

| Evidence class | Example | Install implication |
|---|---|---|
| Path frequency | `$.email` present in 98% of employee samples | Strong field candidate |
| Type stability | `priority` always string enum-like | Mapping / content schema hint |
| Null vs missing | `managerId` sometimes absent key vs null | Optional vs nullable semantics |
| Uniqueness | `employeeId` unique across samples | Identifier / primary key hint |
| Cardinality | many analyses per work-item | Discriminator / multi-instance content |
| Overlap | employees ∩ contractors share name/email paths | Abstract `People` binding candidate |
| Insufficient | &lt; N samples or conflicting shapes | **Do not install** — gather fixtures |
| Validation reject | Draft violates metadata contracts | **Do not install** — fix generator |

AI narrative text (“this looks like a person”) is **not** evidence. Structural counters are.

When comparing against the approved opx pack, treat the pack as the **baseline truth** for acceptance. Proposals that invent cyber/vuln vocabulary into the permanent opx narrative should be rejected even if structurally clever.

---

## 9. Compare and export drafts

### Compare

`POST /api/intelligence/compare` answers: *what would change if we installed this draft relative to pack X?*

Use it to:

- Bound review to a diff (types added/changed, bindings, relationship defs, views)
- Spot accidental deletions or renames
- Produce an audit artifact for acceptance ledgers

### Export

`POST /api/intelligence/export-draft` produces a reviewable pack fragment (JSON) for:

- Offline review / PR attachment
- Re-validation with metadata tooling
- Hand-editing **before** install (still requires explicit install afterward)

Exported drafts are not live. Installing a modified export is still a metadata-runtime / `/api/metadata` concern.

---

## 10. People-oriented suggestions

`POST /api/intelligence/suggest/people` is a focused helper for abstract-union style suggestions (employees + contractors → People-like projections). It still:

- returns suggestions / drafts only,
- does not write abstracts into the org,
- must be followed by human review + explicit install of whatever pack fragment you accept.

Do not special-case “people” as an abstract collection. Abstracts remain **virtual** after install (book `05` / Format hard rules).

---

## 11. Failure modes developers must surface

| Symptom | Likely cause | Correct handling |
|---|---|---|
| “Intelligence did nothing” | Expecting auto-install | Document explicit install step |
| Proposal empty | Thin samples / wrong label | Return insufficient-evidence |
| Proposal fails validation | Generator bug or bad samples | rejected-by-validation outcome |
| Effective metadata unchanged after propose | **Correct** | Only install changes it |
| Effective metadata changed after propose | Auto-install bug | Fail acceptance; fix service |
| AI timeout / no credentials | Expected in air-gap | Heuristics-only path |
| 400 singular `agentId` | Wrong header | Plural `agent-ids` |
| Profiling wrong tenant | Bad org header | Fresh `sandbox*` only |

---

## 12. In-process vs HTTP

| Mode | Use when | Caution |
|---|---|---|
| HTTP `/api/intelligence/*` | Remote hosts, Studio, demos | Prefer for product boundaries |
| In-process `@x12i/memorix-intelligence` | Trusted platform workers | Still must not call install implicitly |

Even in-process, keep proposal creation and metadata install in **separate functions / jobs**. A single “profileAndApply” helper is an anti-pattern.

```ts
// Good: explicit stages
const profile = profileRecords(samples, opts);
const proposal = createProposal(profile, opts);
// ... persist proposal artifact, notify reviewers ...
// later, after approval:
await metadataRuntime.install({ pack: approvedPack, confirm: true, scope });
```

---

## 13. Acceptance script (outline)

```bash
ORG=$(node -e "console.log('sandbox'+Math.random().toString(36).slice(2,10))")
H=(-H "x-memorix-org-id: $ORG" -H "x-memorix-agent-ids: opx")

# baseline
curl -s "${H[@]}" -X POST "$SERVICE/api/metadata/install" \
 -H "content-type: application/json" \
 -d '{"packId":"opx","confirm":true}'

BEFORE=$(curl -s "${H[@]}" "$SERVICE/api/metadata/effective" | shasum)

# profile + propose (AI disabled / no keys in env)
curl -s "${H[@]}" -X POST "$SERVICE/api/intelligence/profile" \
 -H "content-type: application/json" \
 -d @./fixtures/employees-sample.json

curl -s "${H[@]}" -X POST "$SERVICE/api/intelligence/proposals" \
 -H "content-type: application/json" \
 -d '{"fromProfile":true,"label":"employees"}'

AFTER_PROPOSE=$(curl -s "${H[@]}" "$SERVICE/api/metadata/effective" | shasum)
test "$BEFORE" = "$AFTER_PROPOSE" # must hold — no auto-install

# human gate simulated: only then install a reviewed pack/draft
# curl ... /api/metadata/install ...
```

Gate assertions:

- [ ] Profile succeeds without AI credentials
- [ ] Propose does not change effective metadata hash
- [ ] Insufficient-evidence / rejected-by-validation are representable
- [ ] Install requires `confirm: true` (or Studio equivalent)
- [ ] Org matches `sandbox*`; agents `["opx"]`

---

## 14. Decision tree

```text
Do I need live metadata changed?
├─ No → profile / propose / compare / export only
└─ Yes → explicit /api/metadata/install|publish after human review

Are samples deterministic fixtures?
├─ Yes → good for acceptance
└─ No, production scrape only
 └─ Still never auto-install; prefer shadowed sandbox* rehearsal

Is AI unavailable?
└─ Continue with heuristics; mark insufficient if needed

Did validation reject the draft?
└─ Do not install; fix evidence or rules

Am I about to write records from intelligence?
└─ Stop. Intelligence does not own data-plane writes.

Did effective metadata change on propose alone?
└─ Fail the gate — auto-install regression
```

---

## 15. Anti-patterns

| Anti-pattern | Why it fails | Do this instead |
|---|---|---|
| Auto-install on propose | Silent production metadata drift | Explicit install with confirm |
| Treat AI confidence as epistemic truth | Wrong gate | Evidence + human review |
| Intelligence writes snapshots / links | Wrong package boundary | Pipelines / relationship engine |
| Require live AI for CI | Blocks air-gapped gates | Deterministic fixtures |
| Profile on live customer org “for fun” | Risk + habit | Fresh `sandbox*` |
| Hide insufficient-evidence | Forces bad installs | Surface as first-class outcome |
| Merge propose + install in one host API | Skips review | Two calls, two UX steps |
| Invent abstract collections from suggestions | Violates virtual abstracts | Bindings only after install |
| Use proposal text as record payload | Wrong layer | Metadata vs data plane |

---

## 16. Relation to other books

| Topic | Book |
|---|---|
| Agents, packs, install semantics | `05` Metadata |
| Studio proposal review UX | Builders twin + `09` Studio |
| OpX pack as approved baseline | `11` Agent packs |
| “Intelligence did nothing” symptom | `13` Troubleshooting |

Intelligence does **not** change `memorix-record/2`. It only drafts **metadata** that later constrains writers/readers after install.

Hard-rule cross-check: explicit metadata install; intelligence never auto-publishes; deterministic fixtures for acceptance (see [HARD-RULES.md](../../HARD-RULES.md) and documentation hard rules).

---

## 17. Worked proposal shapes (conceptual)

### Insufficient evidence

```json
{
 "outcome": "insufficient-evidence",
 "label": "employees",
 "scope": { "orgId": "sandbox…", "agentIds": ["opx"] },
 "reasons": [
 "sampleCount < minimum",
 "conflicting types at $.role"
 ],
 "profileFingerprint": "…",
 "suggestions": []
}
```

Host UI must render this as a terminal-but-honest state — not as an empty spinner that invites “Apply anyway.”

### Rejected by validation

```json
{
 "outcome": "rejected-by-validation",
 "errors": [
 { "code": "METADATA_SCHEMA", "path": "objectTypes[0].contentTypes", "message": "…" }
 ],
 "draft": null
}
```

Do not install. Fix the generator or samples; re-profile.

### Evidence-backed suggestion (installable only after review)

```json
{
 "outcome": "suggestion",
 "label": "work-items",
 "evidence": {
 "paths": ["$.status", "$.priority", "$.assigneeId"],
 "cardinalityHints": [{ "path": "$.priority", "approxDistinct": 4 }]
 },
 "draft": {
 "objectTypes": ["…additive fragment…"],
 "relationshipDefinitions": ["…optional…"]
 },
 "aiAssisted": false
}
```

When `aiAssisted: true`, Studio/host must visually separate AI extras from deterministic evidence.

---

## 18. Security and tenancy notes

- Profiles and proposals are **org-scoped**. Never reuse a proposal artifact across orgs.
- Do not log raw customer PII from production scrapes into shared proposal stores for demos — use opx fixtures.
- AI adapters must not receive secrets embedded in samples; redact or use synthetic fixtures.
- Compare/export outputs may contain structural paths — treat them as sensitive metadata about the tenant’s shape.

---

## 19. Next

| Track | Link |
|---|---|
| Review / reject / install UX | [Builders twin](../builders/BOOK.md) |
| Metadata install contracts | [Metadata developers](../../05-metadata-agents-abstracts/developers/BOOK.md) |
| Studio Design area | [Workflow & Studio builders](../../09-workflow-and-studio/builders/BOOK.md) |
| OpX pack baseline | [Agent packs developers](../../11-agent-packs-and-opx/developers/BOOK.md) |
| Failure matrix | [Migration & troubleshooting](../../13-migration-troubleshooting/developers/BOOK.md) |

---

*End of Structural Intelligence — Developers.*

---

# Structural Intelligence — builders

# Structural Intelligence — Builders

**Audience:** Studio builders reviewing structural profiles and metadata proposals. 
**Twin:** [Developers version](../developers/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense operator documentation. Deterministic fixtures first — live AI optional.

---

## How to read this guide

1. Memorize the **one sentence rule**: intelligence suggests; you install.
2. Learn the **profile → propose → review → install/reject** loop.
3. Use the **ops drill** on a fresh `sandbox*` workspace.
4. Escalate thin or confusing proposals — do not install “to see what happens” on a shared org.

---

## The central idea

> **Intelligence profiles structure and proposes metadata. It never silently changes what your org runs.**

![Intelligence proposal](../screenshots/18-intelligence-proposal.png)

```mermaid
flowchart LR
 Profile[Profile samples] --> Propose[Proposal]
 Propose --> Review[You review evidence]
 Review -->|strong| Install[Explicit install]
 Review -->|thin / wrong| Reject[Reject / revise samples]
```

| Stage | Your job |
|---|---|
| Profile | Confirm samples are the right opx fixtures (Alex, WI-100, …) |
| Proposal | Read evidence — paths, cardinality, overlap — not just titles |
| Compare | Diff against the approved opx pack when offered |
| Install | Explicit Design → install/publish with confirmation |
| Reject | Leave effective metadata unchanged |

---

## 1. Where in Studio

| Need | Go to |
|---|---|
| Run / inspect profiling | Operate → intelligence runs (and related artifacts) |
| Read a proposal | Design → intelligence proposals |
| Install after approval | Design → metadata install / publish |
| Confirm it took effect | Design → effective types / agents · Explore lists |

Scope bar first: fresh `sandbox*` + agent `opx`. Never rehearse installs on live customer orgs.

---

## 2. What “good evidence” looks like

Ask before installing:

1. Were samples **deterministic** (opx fixtures) or a one-off scrape?
2. Does the proposal explain **paths / types / cardinality**, or only vague labels?
3. Is anything marked **insufficient evidence** or **rejected by validation**?
4. Does compare-against-opx show a **small, understandable** diff?
5. Will this change writers/readers for Alex / WI-100 in ways you can explain?

If you cannot answer those, **reject** (or request more fixtures). Rejection is a successful outcome.

---

## 3. Review → reject → install checklist

### Review

- [ ] Scope is `sandbox*` + `opx`
- [ ] Proposal id / timestamp recorded
- [ ] Evidence section opened (not only summary chips)
- [ ] Compare against approved `opx` (when available)
- [ ] AI-only suggestions flagged as untrusted extras

### Reject when

- Evidence is thin or contradictory
- Diff invents cyber/vendor vocabulary into the permanent opx narrative
- Validation already rejected the draft
- You are on the wrong org / agent set
- Someone asks you to “just apply” without reading

### Install only when

- [ ] Human review completed
- [ ] Explicit install/publish confirmation clicked
- [ ] Effective metadata refreshed afterward
- [ ] Explore still shows expected subjects (smoke check WI-100 / Alex)

Install is a **different** action from “generate proposal.” If Studio ever conflates them, treat that as a defect.

---

## 4. Air-gapped / no-AI demos

Acceptance demos must work **without** model credentials:

1. Use opx-pack fixtures / Memory pull from localhost opx-source.
2. Run profile + proposal with heuristics only.
3. Show insufficient-evidence honestly if samples are too small.
4. Install only a reviewed, validated draft (often: stick with the shipped opx pack).

Live AI is optional sugar — never a gate.

---

## 5. Short opx tour

1. Fresh `sandbox*` + agent `opx`.
2. Ensure baseline opx metadata is installed.
3. Operate: pull or load employee / work-item samples.
4. Run structural profile for `employees` (and/or `work-items`).
5. Open the proposal in Design — read evidence.
6. Compare to ops; reject anything thin.
7. If approving a deliberate draft: **explicit install**.
8. Explore: confirm lists/types match expectation; WI-100 composition still healthy.

---

## 6. Decision tree

```text
Did types change after “Generate proposal”?
├─ Yes → unexpected — stop; file bug (auto-install)
└─ No → correct; continue review

Is evidence insufficient?
└─ Reject / gather fixtures — do not install

Is this a live customer org?
└─ Stop. Move to fresh sandbox*

Do I only need the shipped opx pack?
└─ Install ops explicitly — skip speculative proposals
```

---

## 7. Anti-patterns

| Anti-pattern | Why it hurts | Do this instead |
|---|---|---|
| Install to “see what happens” | Pollutes shared metadata | Fresh `sandbox*` + reject thin drafts |
| Trust AI wording over evidence | Wrong gate | Read structural evidence |
| Expect intelligence to write records | Wrong boundary | Pipelines / Explore edits |
| Require live AI for a demo | Fragile / air-gap fail | Deterministic fixtures |
| Skip compare against ops | Surprise diffs | Always compare when offered |

---

## 8. Escalate packet

If a proposal looks wrong or install fails, send developers:

- Org id (`sandbox*`) and agent ids
- Proposal id / compare output
- Whether AI was enabled
- Screenshot of proposal evidence
- Whether baseline opx pack was already installed

---

## 9. Next

| Track | Link |
|---|---|
| APIs / package boundaries | [Developers twin](../developers/BOOK.md) |
| Studio IA | [Workflow & Studio builders](../../09-workflow-and-studio/builders/BOOK.md) |
| Metadata install | [Metadata builders](../../05-metadata-agents-abstracts/builders/BOOK.md) |
| OpX pack install checklist | [Agent packs builders](../../11-agent-packs-and-opx/builders/BOOK.md) |
| “Did nothing” / conflicts | [Troubleshooting builders](../../13-migration-troubleshooting/builders/BOOK.md) |

---

*End of Structural Intelligence — Builders.*

---

# Agent Packs & OpX — developers

# Agent Packs & OpX — Developers

**Audience:** Authors packaging metadata + fixtures for an agent, and engineers running opx acceptance. 
**Twin:** [Builders version](../builders/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Packages:** `@x12i/memorix-opx-pack`, `@x12i/memorix-metadata`, `@x12i/memorix-metadata-runtime`, `@x12i/memorix-format`, `@x12i/memorix-service`. 
**Prerequisites:** [Metadata, agents & abstracts](../../05-metadata-agents-abstracts/developers/BOOK.md) · [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense product documentation. Deterministic fixtures only — **no live AI required**.

---

## How to read this guide

1. Start with the **central mental model** — a pack is versioned metadata + fixtures + a localhost source, not a runtime dependency for customer apps.
2. Learn the **opx-pack layout**, then the **harness** (`up` / `down` / `reset` / `accept`).
3. Add **custom kinds** only when built-in descriptor families are not enough.
4. Validate **V2 goldens** before claiming acceptance.
5. Labels used throughout:
 - **Stored** — installed metadata items; subject records written by pipelines
 - **Virtual / calculated** — effective metadata after inheritance; abstracts
 - **Operational** — harness runs, memory/pipeline ledgers during acceptance
6. Platform gate acceptance: fresh `sandbox*` + `agentIds: ["opx"]`. Never live/legacy customer orgs.

---

## The central mental model

An agent pack answers:

> **What vocabulary, sources, pipelines, and deterministic samples prove this agent works on Memorix?**

| Piece | Question it answers |
|---|---|
| `metadata/` | What agents, types, abstracts, relationships, views, mappings, pipelines exist? |
| `fixtures/` | What exact source bytes/JSON can we replay offline? |
| `opx-source/` | How do we serve those fixtures as a local REST source? |
| `scripts/` | How do we bring the source up, reset, accept, and tear down? |
| `examples/v2/` | What golden `memorix-record/2` shapes must stay valid? |

```text
Pack (declarative + fixtures)
 → explicit metadata install into sandbox*
 → Memory pull from localhost opx-source
 → Pipelines write subjects / materialize links
 → Explore / HTTP asserts composition + relationships
```

`@x12i/memorix-opx-pack` is the permanent **opx** acceptance pack for the platform narrative (employees, work-items, procedures, activity-events) — not a production dependency for remote customer apps.

---

## 1. When to use the opx pack

| Audience | Decision |
|---|---|
| Local demo / platform acceptance | **Yes** — dedicated opx org + `agentIds: ["opx"]` |
| Production customer org | **No** as the acceptance target |
| Remote product runtime dependency | **No** — customers consume service APIs / their own packs |
| Authoring a new agent pack | **Yes** as a structural pattern to copy (not customer data to copy) |

Never point acceptance tooling at live customer orgs.

---

## 2. Pack layout

```text
memorix-opx-pack/
 metadata/
 manifest.json
 agents/opx.json
 agents/project-opx.json
 abstract-types.json
 abstract-bindings.json
 object-types.json
 relationship-definitions.json
 view-definitions.json
 write-and-source-mappings.json
 pipelines-and-sources.json
 fixtures/ # deterministic source data
 opx-source/server.mjs # local REST fixture (no internet)
 scripts/harness.mjs # up / down / reset / accept
 examples/v2/ # memorix-record/2 goldens
```

Diagram (same idea):

```text
pack/
 metadata/
 fixtures/
 opx-source/
 scripts/
 examples/v2/
```

### Manifest

`metadata/manifest.json` lists includes for agent `opx` (versioned). Install tooling resolves those files into installable metadata items.

Typical includes: parent agent `opx`, child `project-opx`, abstracts + bindings, object types, relationship definitions, views, write/source mappings, pipelines and sources.

### Subjects in the narrative

| Subject | Role in fixtures |
|---|---|
| Employee **Alex** | Stable `recordId` across snapshot / workflow / optional knowx |
| Work item **WI-100** | Prepare quarterly operations review |
| Procedure **PROC-2** | Knowledge-category procedure content |
| Activity events | Event-category samples |
| Malformed / deleted items | Negative paths for quarantine / 404 checks |

---

## 3. Harness: up / down / reset / accept

```bash
cd memorix-opx-pack
npm install && npm test
node scripts/harness.mjs up # OPX_SOURCE_PORT default 5102
node scripts/harness.mjs reset # reset fixture control state
node scripts/harness.mjs accept # smoke: metadata files + source routes
node scripts/harness.mjs down
```

Prefer the monorepo stack when exercising Studio + service together:

```bash
node scripts/dev-stack.mjs
# includes opx-source on :5102 (even ports = APIs), stack-status :5103, /_live on APIs
```

| Command | Purpose |
|---|---|
| `up` | Start localhost `opx-source` (PID under `.run/`) |
| `down` | Stop opx-source via PID file |
| `reset` | POST fixture control reset; re-establish known source state |
| `accept` | Verify manifest files exist; hit key source routes; assert negative paths |

### What `accept` is (and is not)

**Is:** a deterministic smoke that the pack’s metadata files and fixture server behave.

**Is not:** a license to use live customer orgs; not a substitute for service-level gates on a fresh `sandbox*` with explicit metadata install + pipeline green.

Platform product acceptance always pairs pack smoke with:

```http
x-memorix-org-id: sandbox<suffix>
x-memorix-agent-ids: opx
```

and an explicit `/api/metadata/install` (or Studio install) into that org.

### Harness sketch in CI

```bash
node scripts/harness.mjs up
node scripts/harness.mjs accept
npx memorix-format validate-v2 examples/v2
node scripts/harness.mjs down
```

---

## 4. Author a new agent pack (phased)

Copy **structural patterns** from `memorix-opx-pack` — not customer payloads. Work in phases so every later id already exists. Three sides: **A vocabulary → B services (only if needed) → C Memory/pipeline**.

```text
your-pack/
  metadata/     # agents, types, abstracts, relationships, views, services, pipelines
  fixtures/     # deterministic JSON / files
  <source>/     # localhost server or recorded pages
  scripts/      # up/down/reset/accept
  examples/v2/  # goldens
```

| Phase | What | Why | Exit criteria |
|---|---|---|---|
| **1. Agent** | Declare agent id (+ optional `parentAgentId` / child of `opx`) | Named vocabulary scope; inheritance fills gaps | Agent appears in pack / Design after install |
| **2. Object / content types** | Concrete subjects + content they may wear (`raw`, `snapshots`, …) | Format freezes the envelope; metadata declares legal OT/CT | Effective metadata lists your types |
| **3. Abstracts + relationship definitions** | Virtual unions (optional); allowed link shapes | UI unions without fake DBs; definition before instances | Abstracts bind; relationship defs resolve |
| **4. Sources / mappings / views / writes** | Memory inputs; deterministic transforms; list/form shapes | Exact raw first; renames are mappings, not services | Source + mapping validate; views/writes present |
| **5. Services (only if needed)** | Custom code, remote I/O, or AI capability in `services[]` | Prefer mapping/Memory/relationships when enough | Service id in effective metadata; runtime `ServiceRef` bound |
| **6. Pipeline (+ optional schedule)** | Ordered steps that wire A/B; schedule is “when” | Orchestration last so every step references existing ids | Pipeline validates; steps reference known service/source ids |
| **7. Validate → install → prove** | `validate` → install on fresh `sandbox*` → Memory → Explore | Pack files are not live until install; data is separate | Effective metadata green; one subject composition + links |

Hard rules while authoring:

1. Keep connectors / sources **read-only** toward upstream systems.
2. Ship **V2 golden examples**; validate with `memorix-format validate-v2`.
3. Provide a **localhost fixture server** for air-gapped acceptance.
4. Secrets: never embed raw credentials; Memory uses `credentialRef` only.
5. Keep the permanent opx narrative non-cyber unless you are deliberately authoring a separate domain pack.

Deep dive: [Getting started — three sides](../../14-getting-started-and-tutorials/developers/BOOK.md) · [Building Services](../../07-services/developers/BOOK.md) · [Building Pipelines](../../08-pipelines/developers/BOOK.md).

---


## 5. Custom kinds in a pack

When a client needs a descriptor family that is **not** a built-in kind, declare it on the pack and ship items:

```json
{
 "format": "memorix-metadata/1",
 "agentId": "opx",
 "kindDeclarations": [
 {
 "id": "acme.ui-tabs",
 "name": "UI Tabs",
 "schemaVersion": "acme.uiTabs.v1"
 }
 ],
 "extensions": {
 "acme.ui-tabs": [
 {
 "id": "employee-detail",
 "title": "Employee detail",
 "objectType": "employees"
 }
 ]
 }
}
```

Full worked pack: [`memorix-metadata/examples/custom-kinds/acme-ui-tabs.pack.json`](../../../memorix-metadata/examples/custom-kinds/acme-ui-tabs.pack.json).

### Rules of thumb

| Rule | Detail |
|---|---|
| Namespaced ids | `vendor.feature` (e.g. `acme.ui-tabs`) |
| Inheritance | Child agents may override items by same `kind` + `id` |
| Parent kinds | A child pack may reference a kind declared on an already-installed parent |
| Execution | Custom kinds are **store/resolve only** — pipelines and the data plane do **not** execute them |
| Prefer built-ins | Use object types, views, and writes for Memorix-native shape; custom kinds for client-owned descriptors |

Read back via effective metadata / `GET /api/metadata/kinds` after install.

---

## 6. Golden V2 validation

OpX pack ships goldens under `examples/v2/` (snapshots, raw, workflow, analysis, decisions, lightweight-refs, …).

```bash
npx memorix-format validate-v2 examples/v2
```

Rules:

- Every golden must be `format: "memorix-record/2"`.
- No copied associated payloads (`V2-ASSOC-COPIED`).
- Discriminators present when required (`V2-DISCRIMINATOR`).
- Workflow / analysis / decisions siblings stay separate content types.
- Relationship **navigation** is not proven by fat goldens — use four-field links + engine verify in service gates.

Migration note: present → v2 belongs in book `13` / Format §§18–19. Pack authors ship **V2**, not present envelopes, as target truth.

---

## 7. Install into a fresh acceptance org

Prefer isolation helpers over inventing org ids by hand:

```bash
# generateSandboxOrgId / assertSafeSandboxOrgId from @x12i/memorix-service
ORG=$(node --input-type=module -e "import { generateSandboxOrgId } from '@x12i/memorix-service'; console.log(generateSandboxOrgId())")

H=(-H "x-memorix-org-id: $ORG" -H "x-memorix-agent-ids: opx")

curl -s "${H[@]}" -X POST "$SERVICE/api/metadata/install" \
  -H "content-type: application/json" \
  -d '{"packId":"opx","confirm":true}'

curl -s "${H[@]}" "$SERVICE/api/metadata/effective" | head

# Memory pull from opx-source (credentialRef per Memory contracts)
# Pipeline start → assert WI-100 composition + links
```

### Acceptance tooling rename

| Old | New |
|---|---|
| `sandbox*` org prefix | `sandbox*` |
| `G12_ORG_ID` | `SANDBOX_ORG_ID` |
| `generateG12OrgId` / `assertSafeG12OrgId` | `generateSandboxOrgId` / `assertSafeSandboxOrgId` |
| `test:live-g12` / `G12_LIVE_PIPELINE` | `test:live-sandbox` / `SANDBOX_LIVE_PIPELINE` |
| `scripts/phase*` gate runners | removed |
| Stack profile | `local-stack` (`node scripts/dev-stack.mjs`) |

Isolation sessions still pin `agentIds: ["opx"]`; only the org prefix and helper names changed.

Assertions for a full gate:

- [ ] Manifest files present; `harness accept` green
- [ ] `validate-v2` green on `examples/v2`
- [ ] Install with `confirm: true` into `sandbox*`
- [ ] Memory pull lands raw for employees / work-items / …
- [ ] Pipeline green; Explore/HTTP composition for `wi-100`
- [ ] Relationship verify for documented definitions
- [ ] Custom kinds (if any) visible via `/api/metadata/kinds` and **not** executed by pipelines
- [ ] No live customer org; agents `["opx"]`

---

## 8. Child agents and inheritance

OpX includes a child agent pattern (`project-opx`) so authors see inheritance:

```text
ops (parent vocabulary)
 └─ project-opx (overrides / extensions by kind+id)
```

Effective metadata for `agentIds: ["opx"]` vs `["opx","project-opx"]` differs. Acceptance docs must state which agent set is required. Singular `agentId` headers are rejected.

---

## 9. Decision tree

```text
Am I accepting platform behavior?
└─ Fresh sandbox* + agentIds ["opx"] + opx-pack fixtures

Am I shipping a customer runtime dependency on opx-pack?
└─ No — packs are for demo/acceptance/patterning

Do I need a new descriptor family?
├─ Memorix-native shape → object types / views / writes
└─ Client-owned UI descriptors → namespaced custom kind (store/resolve only)

Are goldens present-format?
└─ Migrate to V2 (book 13); validate-v2 before merge

Is opx-source required for air-gap?
└─ Yes — harness up + localhost routes; no internet scrape
```

---

## 10. Anti-patterns

| Anti-pattern | Why it fails | Do this instead |
|---|---|---|
| Accept on live customer orgs | Contaminates production | Fresh `sandbox*` |
| Copy customer PII into pack fixtures | Compliance + drift | Synthetic opx fixtures |
| Expect custom kinds to run pipelines | By design they don’t | Object types + pipeline steps |
| Ship present envelopes as goldens | Wrong target truth | `memorix-record/2` + validate-v2 |
| Embed secrets in fixtures | Leak risk | `credentialRef` only |
| Skip `confirm` on install | Accidental publish | Explicit confirm |
| Treat harness accept as full platform gate | Incomplete | + service install + pipeline + Explore asserts |
| Cyber narrative as permanent opx story | Violates product rules | Non-cyber opx subjects |

---

## 11. Tiny journey sketch

1. `harness up` → opx-source healthy on `:5102`.
2. `harness accept` → metadata files + WI-100 routes OK.
3. `validate-v2 examples/v2`.
4. Start service/Studio (`dev-stack`) with fresh `sandbox*`.
5. Explicit metadata install for `opx`.
6. Memory pull → pipeline → compose `wi-100` (Alex linked as defined).
7. `harness down` when finished.

---

## 12. Fixture server expectations (`opx-source`)

The localhost source is part of the pack’s acceptance surface. Typical routes exercised by `harness accept`:

| Route | Expectation |
|---|---|
| `GET /health` | Healthy before accept |
| `GET /employees` | List includes Alex-class fixtures |
| `GET /contractors` | Contractor samples for People union |
| `GET /work-items` | Includes WI-100; may include malformed items for quarantine paths |
| `GET /procedures` | Procedure samples (e.g. PROC-2) |
| `GET /activity-events` | Event-category samples |
| `GET /work-items/WI-100/context` | Context sibling source material |
| `GET /work-items/WI-DELETED` | **404** — deleted/missing negative path |
| `POST /_control/reset` | Deterministic reset used by `harness reset` |

Default listen: `127.0.0.1:5102` (`OPX_SOURCE_PORT` / `OPX_SOURCE_HOST`). Even ports are API-side in the monorepo port convention; Studio remains on an odd UI port. The fixture server uses `@x12i/core-service` for `/health`, `/_live`, correlation id, and zone port resolution.

Air-gapped rule: acceptance must not require downloading fixtures from the public internet at gate time.

---

## 13. Golden inventory (ops `examples/v2`)

Validate the directory, not a single file:

```bash
npx memorix-format validate-v2 examples/v2
```

Representative goldens (names may evolve; treat the directory as SoT):

| Golden | Teaches |
|---|---|
| `employees-snapshots.json` | Entity snapshot for Alex-class subjects |
| `employees-discovery.json` | Discovery discriminator pattern |
| `work-items-snapshots.json` | WI-100 snapshot |
| `work-items-raw.json` | Exact raw landing shape |
| `work-items-workflow.json` | Workflow sibling |
| `work-items-context.json` | Context sibling |
| `work-items-analysis.json` | Analysis discriminator |
| `work-items-decisions.json` | Decision discriminator |
| `work-items-snapshots-lifecycle.json` | Lifecycle on snapshots |
| `work-items-snapshots-lightweight-refs.json` | Compat lightweight refs (not authoritative store) |
| `work-items-relationships-candidate.json` | Non-authoritative format example only |
| `procedures-knowledge.json` | Knowledge-category procedure |
| `activity-events-snapshots.json` | Event-category snapshot |

CI should fail the pack if any golden regresses `validate-v2`.

---

## 14. Mapping packs to Studio / HTTP

| Pack piece | Studio | HTTP |
|---|---|---|
| `metadata/*` | Design install | `/api/metadata/install` |
| Effective agents/types | Design browsers | `/api/metadata/effective` |
| Custom kinds | Design kinds | `/api/metadata/kinds` |
| `opx-source` | Operate → Memory source | Memory pull APIs |
| Pipelines in metadata | Design → Pipelines · Operate → Runs | `/api/pipelines/*` |
| `examples/v2` | (authoring) | `memorix-format validate-v2` |

Builders twin is the install checklist; this chapter is the authoring/acceptance contract.

---

## 15. Versioning and publish etiquette

1. Bump pack/`manifest` version when metadata semantics change.
2. Keep fixtures compatible with the declared pipeline/source ids or update both together.
3. Do not silently change primary identifiers for Alex / WI-100 in ways that break goldens and KnowX tutorials.
4. Publish/install into acceptance orgs first; only then promote to shared demo orgs your team owns.
5. Intelligence drafts that “improve” opx should be reviewed as **diffs against this pack**, not as silent replacements.

---

## 16. Next

| Track | Link |
|---|---|
| Studio install checklist | [Builders twin](../builders/BOOK.md) |
| Metadata / custom kinds depth | [Metadata developers](../../05-metadata-agents-abstracts/developers/BOOK.md) |
| Pipelines consuming sources | [Pipelines developers](../../08-pipelines/developers/BOOK.md) |
| Memory pull contracts | [Relationships & Memory](../../06-relationships-and-memory/developers/BOOK.md) |
| Migrate present → v2 | [Migration & troubleshooting](../../13-migration-troubleshooting/developers/BOOK.md) |
| Intelligence vs pack baseline | [Intelligence developers](../../10-intelligence/developers/BOOK.md) |

---

## Appendix A — Layout cheat sheet

```text
pack/
 metadata/ # agents, OTs, abstracts, relationships, views, pipelines, mappings
 fixtures/ # deterministic source JSON
 opx-source/ # localhost REST server (default :5102)
 scripts/ # harness up/down/reset/accept
 examples/v2/ # memorix-record/2 goldens
```

See [diagrams/pack-layout.md](../diagrams/pack-layout.md).

Typical `metadata/manifest.json` includes:

- `agents/opx.json`, `agents/project-opx.json` (child example)
- `object-types.json`, `abstract-types.json`, `abstract-bindings.json`
- `relationship-definitions.json`, `view-definitions.json`
- `write-and-source-mappings.json`, `pipelines-and-sources.json`

## Appendix B — opx-source routes

```text
GET /health
GET /employees
GET /contractors
GET /work-items
GET /activity-events
GET /procedures
GET /work-items/:id/context
```

Memory pulls use **`credentialRef` only** — never paste secrets into fixtures or pull bodies.

## Appendix C — Install / kinds HTTP

```http
POST /api/metadata/install
{"packId":"opx","confirm":true}

GET /api/metadata/effective
GET /api/metadata/kinds
GET /api/metadata/kinds/acme.ui-tabs
```

Custom kinds are **store/resolve only** — pipelines and the data plane do not execute them.

## Appendix D — Acceptance assertions

- [ ] Fresh `sandbox*` + `agentIds: ["opx"]`
- [ ] Explicit install with `confirm: true`
- [ ] opx-source healthy; Memory pull lands `*-raw`
- [ ] Pipelines write V2 subjects (`format === "memorix-record/2"`)
- [ ] No copied `associated*` payloads on authoritative content
- [ ] Composition shows siblings for WI-100 / Alex
- [ ] Relationship verify/rebuild — no ad-hoc link PUT (**405**)
- [ ] `npx memorix-format validate-v2 examples/v2` green
- [ ] No live AI required

## Appendix E — Glossary (this book)

| Term | Meaning |
|---|---|
| Pack | Versioned metadata + fixtures + acceptance helpers |
| Harness | `up` / `down` / `reset` / `accept` lifecycle scripts |
| Golden V2 | Checked-in `memorix-record/2` example that must validate |
| Custom kind | Namespaced metadata descriptor family — store/resolve only |
| Effective metadata | Resolved union for supplied `agentIds` after install |
| opx-source | Localhost fixture REST server for air-gapped pulls |

Full glossary: [_shared/glossary.md](../../_shared/glossary.md).

---

*End of Agent Packs & OpX — Developers.*

---

# Agent Packs & OpX — builders

# Agent Packs & OpX — Builders

**Audience:** Studio builders installing and verifying the opx pack on a fresh acceptance workspace. 
**Twin:** [Developers version](../developers/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense operator checklist. Deterministic fixtures only — **no live AI required**.

---

## How to read this guide

1. Understand **what a pack is** in Studio terms.
2. Run the **fresh sandbox\* install checklist** in order.
3. Confirm **Explore** results for Alex / WI-100 before calling the day done.
4. Use the escalate packet if Memory or pipelines fail after a correct install.

---

## The central idea

> **A pack is a versioned bundle of metadata + sample source data. You install it explicitly into an isolated opx workspace, then operate and explore.**

![Explicit metadata installation](../screenshots/19-explicit-metadata-installation.png)

```text
Fresh sandbox* + agent opx
 → Design: install opx pack (confirm)
 → Operate: Memory pull from opx-source
 → Operate: pipeline green
 → Explore: composition + links for WI-100 / Alex
```

The opx pack is for **acceptance and demos**, not for experimenting on live customer orgs.

---

## 1. What you should see after a good install

| Area | Healthy signal |
|---|---|
| Design → Agents | `opx` (and documented children) present |
| Design → Types | employees, work-items, procedures, … |
| Design → Abstracts | People-style unions bound to concrete types |
| Design → Relationships | Documented definitions available |
| Operate → Memory | Pull from opx-source succeeds |
| Operate → Runs | Pipeline attempts succeed; artifacts readable |
| Explore | WI-100 composition (snapshot, workflow, …); Alex visible; links expand |

Custom kinds (if your pack declares them): appear under metadata kinds / effective metadata after install. They do **not** by themselves run pipelines.

---

## 2. Install checklist (fresh `sandbox*`)

Work top to bottom. Do not skip scope.

### A. Workspace

- [ ] Create / select a **fresh** `sandbox*` organization
- [ ] Scope bar agent = **`opx`** (or the documented agent set)
- [ ] You are **not** on a live customer org
- [ ] Service + Studio + opx-source are up (prefer `dev-stack`)

### B. Metadata install

- [ ] Design → metadata install / publish for the **opx** pack
- [ ] Confirmation step completed (explicit — not a silent side effect)
- [ ] Effective agents/types refresh in Design
- [ ] (Optional) If the pack declares custom kinds, confirm they appear under kinds / effective metadata

### C. Memory

- [ ] Operate → Memory pull from **opx-source** (localhost fixture)
- [ ] Raw material landed for employees / work-items / procedures / activity-events as documented
- [ ] No raw secrets pasted into forms — credential refs only if prompted

### D. Pipeline

- [ ] Operate → start the documented opx pipeline / schedule
- [ ] Attempts green (or failures opened with readable artifacts)
- [ ] Do not “fix” by inventing subjects outside the data plane

### E. Explore verification

- [ ] Open **WI-100** — composition siblings present (snapshot, workflow, …)
- [ ] Open **Alex** (employee) — snapshot healthy
- [ ] Traverse a relationship link — expansion shows current targets, not pasted neighbor blobs
- [ ] Unresolved targets (if any in the scenario) are visible as unresolved — not silently invented

### F. Optional intelligence

- [ ] If you run intelligence proposals, **review then install/reject** — proposals alone must not change types
- [ ] Prefer shipped opx pack for acceptance demos

---

## 3. Subjects to keep in mind

| Subject | Why it matters |
|---|---|
| **Alex** | Stable employee across siblings |
| **WI-100** | Flagship work item for composition + workflow |
| **PROC-2** | Procedure / knowledge path |
| Activity events | Event-category path |

If these are missing after a “green” run, capture run/attempt ids and escalate — do not hand-edit storage.

---

## 4. Author a new agent pack (phased)

When `opx` is not enough, author a child or domain pack in Design — copy **structure** from opx-pack, not customer data. Order matters:

| Phase | What you do in Design | Why | Done when |
|---|---|---|---|
| **1. Agent** | Create agent (or child of `opx`) | Vocabulary scope | Agent visible under Design → Agents |
| **2. Types** | Object + content types | Subjects before jobs | Types list shows your OT/CT |
| **3. Abstracts + relationships** | People-style unions; link definitions | Views and edges before runs | Bindings + defs present |
| **4. Sources / mappings / views** | Memory inputs; transforms; list shapes | Exact raw + deterministic renames | Sources and mappings validate |
| **5. Services** | Only if external/custom/AI needed | A rename is a mapping, not a service | Service listed; not required for every pack |
| **6. Pipeline + schedule** | Ordered steps last; schedule optional | Every step must reference existing ids | Pipeline saves without dangling refs |
| **7. Install → Operate → Explore** | Explicit install on fresh `sandbox*` | Pack files are not live until install | Explore shows composition + links |

Install still **does not** create subjects — Operate Memory/pipelines (or Explore writes) do.

Deep dive: [Getting started — three sides](../../14-getting-started-and-tutorials/builders/BOOK.md).

---

## 5. Custom kinds (builder note)

Some packs declare extra descriptor families (for example UI tab packs):

1. They show up after **install** like other metadata.
2. You can browse them in Design / metadata kinds.
3. Pipelines will **not** execute them for you — use object types, views, and writes for Memorix-native behavior.

If a custom kind “does nothing” in Operate, that is often **by design**.

---

## 6. Decision tree

```text
Empty Design types?
└─ Scope bar → then explicit install → refresh

Memory pull failed?
└─ Is opx-source up? credentialRef? correct agent?

Pipeline green but Explore empty?
└─ Wrong org/agent · or pipeline wrote different object types · check artifacts

Saw nested associated payloads?
└─ Defect — screenshot + escalate (relationships must be links)

Proposal changed types with no install click?
└─ Defect — intelligence must not auto-install
```

---

## 7. Anti-patterns

| Anti-pattern | Why it hurts | Do this instead |
|---|---|---|
| Install on live customer org | Contaminates production | Fresh `sandbox*` |
| Skip confirm on install | Accidental publish | Explicit confirmation |
| Assume custom kinds run jobs | Wrong expectation | Check object types / pipelines |
| Ignore artifacts on failure | Hidden root cause | Operate → Failures → Artifacts |
| Require live AI for opx demo | Fragile | Deterministic fixtures |

---

## 8. Escalate packet

Send developers:

- Org id (`sandbox*`) and agent ids
- Pack id / version and install timestamp
- Memory pull run id (if any)
- Pipeline run / attempt id + failure artifact reference
- Screenshot of scope bar + Explore subject
- Whether fixtures were deterministic (opx-source) or not

---

## 9. Next

| Track | Link |
|---|---|
| Pack layout / harness / goldens | [Developers twin](../developers/BOOK.md) |
| Studio areas | [Workflow & Studio builders](../../09-workflow-and-studio/builders/BOOK.md) |
| Metadata & kinds | [Metadata builders](../../05-metadata-agents-abstracts/builders/BOOK.md) |
| Pipelines | [Pipelines builders](../../08-pipelines/builders/BOOK.md) |
| Intelligence review | [Intelligence builders](../../10-intelligence/builders/BOOK.md) |
| Quick checks when stuck | [Troubleshooting builders](../../13-migration-troubleshooting/builders/BOOK.md) |

---

*End of Agent Packs & OpX — Builders.*

---

# Building KnowX — developers

# Building KnowX as a Memorix Agent Pack

**Audience:** Developers implementing the first-party KnowX agent — vocabulary, services, and pipeline.  
**Twin:** [Builders version](../builders/BOOK.md)  
**Scenario:** Fresh `sandbox*` · install `opx` then `knowx` · `agentIds: ["knowx"]` (inherits opx).  
**Packages:** `@x12i/memorix-knowx-pack`, `@x12i/memorix-opx-pack`, `@x12i/memorix-service`, `@x12i/memorix-pipeline`, `@x12i/memorix-format`, `@x12i/memorix-relationships`.  
**Prerequisites:** [Agent packs](../../11-agent-packs-and-opx/developers/BOOK.md) · [Services](../../07-services/developers/BOOK.md) · [Pipelines](../../08-pipelines/developers/BOOK.md) · [Format](../../01-format-for-dummies/developers/BOOK.md).  
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md)  
**Status:** Present-tense tutorial for the shipped `@x12i/memorix-knowx-pack`. Deterministic fixture stubs — **no live AI required**. Replace stubs with real service bodies using the Memorix contracts in §B.

---

## How to read this tutorial

Work in authoring order: **A vocabulary → B services → C pipeline → D install & run**.

1. KnowX is a **content type** (`knowx`) plus a first-party **agent pack** — not a siloed graph engine.
2. Agent `knowx` is a **child of `opx`**. Scope with `agentIds: ["knowx"]` so inheritance collapses ops + knowx.
3. Three ordinary Memorix **services** extract and associate; they **do not persist**.
4. Pipeline `knowx-ingest-associate` persists via `write-record` and `relationship-write`.
5. When your service implementations return contract-shaped outputs, KnowX works end-to-end.
6. Provenance is mandatory; `epistemic.state` ≠ extraction confidence.
7. Four-field RelationshipLinks only — no `associated*`, no abstract collections.

Labels: **Stored** subject content · **Virtual** abstract projection · **Operational** run ledgers.

---

## The central mental model

```text
text + knowledgeSource
  → knowx-extract          (drafts only)
  → write-record knowx     (knowledge added)
  → knowx-associate-knowledge
  → relationship-write     (known / supports / contradicts / derived-from)
  → knowx-associate-entities
  → relationship-write     (edge-from / edge-to)
```

```mermaid
flowchart LR
  Text[text plus knowledge source] --> Extract[knowx-extract]
  Extract --> Write[write-record knowx]
  Write --> AssocK[knowx-associate-knowledge]
  AssocK --> RelK[relationship-write knowledge links]
  RelK --> AssocE[knowx-associate-entities]
  AssocE --> RelE[relationship-write entity links]
```

---

## A — Vocabulary

### A.1 Content type (platform-native)

`memorix-format` registers `knowx` and `confirmations`. Payload rules:

| Layer | Name | Stored? |
|---|---|---|
| Content type | `knowx` | Yes |
| Discriminator | `data.kind`: `NODE` \| `EDGE` | Inside `.data` |
| Abstract | `KnowXEntity` | Virtual only |
| Forbidden | `KnowXNode` / `KnowXEdge` types or collections | — |

### A.2 Agent pack layout

Package: `@x12i/memorix-knowx-pack`

```text
metadata/
  agents/knowx.json          # parentAgentId: ops
  abstract-types.json        # KnowXEntity (+ People/WorkItems restated for standalone validate)
  abstract-bindings.json     # knowx-entity-union
  object-types.json          # employees/contractors/work-items overlays + assertions
  relationship-definitions.json
  services.json
  pipelines-and-sources.json
fixtures/knowx-activity-log.jsonl
src/knowx-services.ts        # deterministic stubs
```

Known pack id: `knowx` → `memorix-metadata/examples/knowx/normalized/pack.json` (explicit install only).

### A.3 Object types

| OT | Role |
|---|---|
| `employees` / `contractors` / `work-items` | Host OTs — knowx pack **replaces** ops defs to add CT `knowx` + `KnowXEntity` |
| `assertions` | Claim subjects — CTs `knowx` (required), `workflow`, `confirmations` |

NODE facets share the host `recordId`. EDGE claims get their own `recordId` on `assertions`.

### A.4 Relationships

| Id | Classification / use |
|---|---|
| `knowx-same-as` | **known** — matches existing claim |
| `knowx-supports` | Reinforces existing |
| `knowx-contradicts` | **contradicts** existing |
| `knowx-derived-from` | Derived from another claim |
| `knowx-edge-from` | EDGE → domain from-endpoint |
| `knowx-edge-to` | EDGE → domain to-endpoint |

Domain assignment links may still use ops `people-assigned-to-work-items` after human accept (operator flow).

### A.5 Provenance (mandatory)

```ts
type SourceSpan =
  | { jsonPath: string }
  | { lineStart: number; lineEnd: number }
  | { byteStart: number; byteEnd: number };

type KnowxProvenance = {
  sourceClass: "measured" | "inferred" | "asserted" | "imported";
  sources: Array<{
    docId: string;
    docType: string;
    docHash: string; // sha-256 of exact source bytes
    span: SourceSpan;
    retrievedAt: string;
  }>;
  extraction: {
    method: string;
    extractorId: string;
    confidence: number; // ≠ epistemic.state
  };
};
```

Missing provenance ⇒ reject writes / fail the extract step.

---

## B — Service specs (Memorix contracts)

These are the Memorix-facing contracts. Implement real bodies against them; fixture stubs in `@x12i/memorix-knowx-pack` satisfy acceptance without live AI.

**Shared rules for all three**

- Capability class: `analyze` (pipeline `stepKind: "analyze"`).
- Invocation: `kind: "worker"` + `operation` matching the service id.
- **Must not** write Mongo, install metadata, or invent `associated*`.
- Scope headers: `orgId` + `agentIds` (use `["knowx"]`).
- Fail closed: return structured errors the pipeline can surface.

### B.1 `knowx-extract`

| | |
|---|---|
| **Role** | Turn text + knowledge source into KnowX drafts (NODE/EDGE), not yet classified. |
| **serviceId** | `knowx-extract` |
| **operation** | `knowx-extract` |
| **Inputs** | `text` — string to analyze; `knowledgeSource` — exact bytes or Memory DataRef + envelope (`docId`, `docType`, `retrievedAt`) |
| **Output** | `knowxDrafts: KnowxDraft[]` plus `docHash` of exact source bytes |
| **Draft shape** | `{ objectType, contentType: "knowx", recordId, data }` with `data.kind`, `provenance`, `epistemic`; EDGE also has `data.refs.{from,to,predicate}` |
| **Errors** | `MISSING_SOURCE`, `INVALID_SOURCE_BYTES`, `MISSING_PROVENANCE` |
| **Handoff** | Pipeline step `write-knowx` (`write-record`, `contentType: "knowx"`) consumes `knowxDrafts` |
| **Fixture stub** | `extractKnowxFromActivityLog` — parses `fixtures/knowx-activity-log.jsonl` → employee NODE, work-item NODE, `ASSERTION-99` EDGE |

### B.2 `knowx-associate-knowledge`

| | |
|---|---|
| **Role** | Classify drafts against existing KnowX scope: **known**, **novel**, **contradicts**; emit knowledge relationship plans. |
| **serviceId** | `knowx-associate-knowledge` |
| **operation** | `knowx-associate-knowledge` |
| **Inputs** | `drafts` — from extract (or already written knowx); `existingKnowx` — current `contentType=knowx` in scope |
| **Output** | `classification: { known[], novel[], contradicts[] }`; `relationshipPlans[]` |
| **Plan item** | `{ definitionId, from: { objectType, recordId }, to: { objectType, recordId }, reason? }` where `definitionId` ∈ `knowx-same-as` \| `knowx-supports` \| `knowx-contradicts` \| `knowx-derived-from` |
| **Errors** | `MISSING_DRAFTS`, `INVALID_CLASSIFICATION` |
| **Handoff** | Step `write-knowledge-links` (`relationship-write`) consumes `relationshipPlans` |
| **Fixture stub** | Same `objectType+recordId` → known + `knowx-same-as`; EDGE same endpoints different predicate → contradicts + `knowx-contradicts`; else novel |

### B.3 `knowx-associate-entities`

| | |
|---|---|
| **Role** | Associate **novel** KnowX (especially EDGE) to existing domain entities. |
| **serviceId** | `knowx-associate-entities` |
| **operation** | `knowx-associate-entities` |
| **Inputs** | `novelDrafts` — from classification.novel (resolve full drafts); `entityCatalog` — available entity endpoints / abstracts |
| **Output** | `entityRelationshipPlans[]` with `definitionId` ∈ `knowx-edge-from` \| `knowx-edge-to` (optional domain defs) |
| **Errors** | `MISSING_NOVEL`, `UNRESOLVED_ENDPOINT` |
| **Handoff** | Step `write-entity-links` (`relationship-write`) consumes `entityRelationshipPlans` |
| **Fixture stub** | For each novel EDGE, emit plans from `data.refs.from` / `data.refs.to` |

### B.4 Swap stub → real implementation

1. Keep the same `serviceId` / `operation` / output field names.
2. Bind your worker in the pipeline PortHandler registry (same pattern as other `analyze` services).
3. Return contract-shaped JSON; pipeline persist steps stay unchanged.
4. Keep a deterministic path for acceptance (fixture or golden inputs).

---

## C — Pipeline `knowx-ingest-associate`

| Step id | kind | Binding |
|---|---|---|
| `extract` | `analyze` | `serviceId: knowx-extract` |
| `write-knowx` | `write-record` | `contentType: knowx` |
| `associate-knowledge` | `analyze` | `serviceId: knowx-associate-knowledge` |
| `write-knowledge-links` | `relationship-write` | plans → knowledge defs |
| `associate-entities` | `analyze` | `serviceId: knowx-associate-entities` |
| `write-entity-links` | `relationship-write` | plans → edge-from/to |

Optional later (operator-driven): `workflow` + `confirmations` before treating epistemic as `fact`; materialize domain links such as `people-assigned-to-work-items`.

Run evidence lives in the pipeline operational ledger — not in subject content.

---

## D — Install and sandbox run

```bash
# 1) Install parent ops, then knowx (never auto-install)
POST /api/metadata/install  { "packId": "opx", "confirm": true }
POST /api/metadata/install  { "packId": "knowx", "confirm": true }

# 2) Scope — child agent collapses ops inheritance
X-Memorix-Org-Id: sandbox…
X-Memorix-Agent-Ids: knowx

# 3) Start pipeline with text + knowledge source inputs
POST /api/pipelines/start
{ "pipelineId": "knowx-ingest-associate", … }
```

Isolation: fresh `sandbox*` org; `beginIsolationSession({ agentIds: ["knowx"] })` when running KnowX acceptance. Default isolation remains `["opx"]`.

Package tests: `cd memorix-knowx-pack && npm test`.

---

## Human review (siblings)

On assertion `recordId`:

- `knowx` — claim + provenance + epistemic  
- `workflow` — review assignment (`PUT /api/workflow`)  
- `confirmations` — accept/reject  

Accepted EDGE → optional domain relationship materialize (Explore / relationship APIs). Heavy JSON stays on the claim; links are navigation only.

---

## Checklist

- [ ] Agent `knowx` with `parentAgentId: "opx"`
- [ ] `KnowXEntity` union + `assertions` OT + knowx CT on host OTs
- [ ] Six KnowX relationship definitions
- [ ] Three services with Memorix contracts above
- [ ] Pipeline `knowx-ingest-associate` wired end-to-end
- [ ] Known pack `packId: "knowx"` installable after ops
- [ ] No live AI required for fixture acceptance
- [ ] No `associated*`, no abstract collections, provenance on every draft

---

## Related

- [Builders twin](../builders/BOOK.md) · [Diagram](../diagrams/knowx-flow.md)  
- [Agent packs & ops](../../11-agent-packs-and-opx/developers/BOOK.md)  
- [Services](../../07-services/developers/BOOK.md) · [Pipelines](../../08-pipelines/developers/BOOK.md)  
- Use case: `knowx-claims-review`

---

# Building KnowX — builders

# Building KnowX — Builders

**Audience:** Studio builders reviewing knowledge claims and running the KnowX ingest pipeline.  
**Twin:** [Developers version](../developers/BOOK.md)  
**Scenario:** Fresh `sandbox*` · install opx then knowx · `agentIds: ["knowx"]`.  
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md)  
**Status:** Operator guide for the first-party `@x12i/memorix-knowx-pack`. Live AI not required. Studio NODE/EDGE screenshots may still be placeholders.

---

## How to read this guide

1. KnowX is ordinary Memorix content plus a **knowx** agent pack — not a separate graph product.
2. Pipeline adds knowledge, associates it to other knowledge, then links novel claims to entities.
3. You review claims via workflow/confirmations; Explore shows four-field links.
4. Implementation contracts → [developers twin](../developers/BOOK.md).

---

## The central idea

> **KnowX is `contentType: "knowx"` with mandatory provenance. Services draft; the pipeline persists. Navigation is four-field relationships — never copied claim JSON.**

| You see | Meaning |
|---|---|
| NODE knowx on an employee / work item | Subject is graph-visible with provenance |
| EDGE knowx on an assertion | A claim linking subjects |
| known / novel / contradicts | Associate-knowledge classification |
| Knowledge relationship links | same-as / supports / contradicts / derived-from |
| Entity edge-from / edge-to links | Assertion attached to domain endpoints |
| Workflow + confirmations | Human review siblings |

One abstract name: **KnowXEntity**. Scope with agent **`knowx`** (inherits **ops**). Always use a fresh **`sandbox*`** org.

---

## 1. Five questions every claim should answer

1. What is asserted? (label / refs)  
2. Where did it come from? (`docHash` + span)  
3. How sure is the extractor? (extraction confidence)  
4. What is the epistemic state? (hypothesis / fact / disputed / …)  
5. Who must confirm? (workflow sibling)

Extraction confidence and epistemic state are **different** fields.

---

## 2. Operator journey

1. **Install** known packs `opx` then `knowx` (explicit confirm).  
2. **Run** pipeline `knowx-ingest-associate` with text + knowledge source.  
3. **Inspect** written knowx content (NODE/EDGE).  
4. **Inspect** knowledge association links and entity edge links.  
5. **Review** assertions via workflow / confirmations when required.  
6. **Explore** four-field links — do not expect claim JSON on the link.

---

## 3. Review rules before accept

- Provenance present (`docHash` + span).  
- `data.kind` is NODE or EDGE.  
- Contradictions are visible as `knowx-contradicts` (do not silently overwrite).  
- Accept writes `confirmations`; update epistemic deliberately.  
- Never store `associated*` on content roots.

---

## 4. What builders do not do

- Do not call Mongo.  
- Do not invent a separate KnowX database.  
- Do not expect auto-install of the knowx pack.  
- Live AI extractors are optional behind your service implementation — fixtures prove the Memorix path.

---

## Related

- [Developers twin](../developers/BOOK.md) — full service contracts and pipeline map  
- [Agent packs](../../11-agent-packs-and-opx/builders/BOOK.md) · [Studio / workflow](../../09-workflow-and-studio/builders/BOOK.md)

---

# Migration & Troubleshooting — developers

# Migration & Troubleshooting — Developers

**Audience:** Engineers migrating present → `memorix-record/2` and debugging runtime failures. 
**Twin:** [Builders checklist](../builders/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Packages:** `@x12i/memorix-format`, `@x12i/memorix-service`, `@x12i/memorix-data`, `@x12i/memorix-relationships`, `@x12i/memorix-opx-pack`. 
**Prerequisites:** [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md) §§18–19 · [Data access](../../04-data-access-patterns/developers/BOOK.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense product documentation. Deterministic fixtures only — **no live AI required**.

---

## How to read this guide

1. Start with the **central mental model** — present is migration input; V2 is target truth.
2. Run **migrate → validate** before writing migrated records.
3. Use the **failure matrix** as a lookup when HTTP/runtime misbehaves.
4. Labels used throughout:
 - **Stored** — V2 subject content and four-field relationship links
 - **Virtual / calculated** — composition, expansion, abstracts, unresolved target status
 - **Operational** — run ledgers, artifacts, quarantine (not subject snapshots)
5. Rehearse only on fresh `sandbox*` orgs. Never migrate experiments on live/legacy customer orgs.

---

## The central mental model

Migration answers:

> **How do I turn a present/V1-shaped document into a clean `memorix-record/2` envelope without bringing copied association payloads with me?**

Troubleshooting answers:

> **Which layer failed — scope, revision, relationships, durability, abstracts, intelligence, or discriminators — and what is the corrective write path?**

```text
present.json
 → migrate-to-v2
 → { v2, extractedRelationships, warnings }
 → validate-v2
 → (separate step) persist v2 via data plane
 → (separate step) materialize links via relationship engine
```

```mermaid
flowchart LR
 Present[present.json] --> Migrate[migrate-to-v2]
 Migrate --> V2[v2 record]
 Migrate --> Rel[extractedRelationships]
 Migrate --> Warn[warnings]
 V2 --> Validate[validate-v2]
```

Authoritative navigation after migration is the **central relationships collection** (four-field links) — not nested `associated*` payloads on snapshots.

---

## 1. Present → memorix-record/2

### CLI

```bash
npx memorix-format migrate-to-v2 ./present.json \
 --object-type=work-items \
 --content-type=snapshots \
 --data-category=entity \
 --issuer=opx-source
```

### Library

`migratePresentToV2(present, options) → { ok, v2, extractedRelationships, warnings }`

| Output | Meaning |
|---|---|
| `v2` | Clean V2 envelope — **no** copied associated payloads |
| `extractedRelationships` | Out-of-band candidates — **not** auto-persisted |
| `warnings` | Timestamp defaults, dropped roots, quarantine notes, … |
| `ok` | A V2 envelope was produced (may still need validate + link materialization) |

Always:

```bash
npx memorix-format validate-v2 ./out-v2.json
# or validateMemorixRecordV2(result.v2)
```

### Behavior highlights (Format §§18–19)

- Copied related payload removed (`droppedPayloadKeys` on extracted candidates).
- Resolvable `recordId` links preserved out-of-band (`mapped` / pending metadata).
- Unresolvable associations quarantined or dropped with status — not silently invented as subjects.
- `_extIdentifier` → primary identifier (`kind` + `value`).
- Missing timestamps default with warnings.
- Subject `_graphRuns` / similar dropped (ops ledgers own history).
- Present format remains **migration input**, not target truth.

Optional map when you know definition ids:

```ts
associatedPropertyMap: {
 associatedPeople: "people-assigned-to-work-items",
}
```

---

## 2. `associated*` rejection (what V2 forbids)

### Present world (legacy)

Root buckets like `associatedPeople` often held **copied** related records (names, titles, nested `.data`). That caused staleness and unsafe writes.

### V2 / production writer world

| Shape | Format validate | Production writer |
|---|---|---|
| Copied payloads under `associated*` | **Reject** (`V2-ASSOC-COPIED`) | Reject (`WRITE_G4_*`) |
| Lightweight refs only (`relationshipDefinitionId` + `recordId`) | May be format-valid (Option A / migration compat) | **Not** authoritative storage — prefer central links |
| Four-field relationship link in category store | N/A (separate collection) | **Authoritative** navigation |

```json
// BAD — copied payload (rejected)
{
 "associatedPeople": [
 {
 "recordId": "employee-10",
 "title": "Alex",
 "data": { "email": "alex@example.com" }
 }
 ]
}
```

```json
// GOOD — central link (conceptual)
{
 "relationshipId": "…deterministic…",
 "relationshipDefinitionId": "people-assigned-to-work-items",
 "fromRecordId": "wi-100",
 "toRecordId": "employee-10"
}
```

Migration extracts candidates **out-of-band**. You must still run relationship discover/verify/rebuild (or pipeline relationship-write) to persist links. Do not paste candidates back onto the V2 snapshot as fat `associated*`.

---

## 3. Validation error categories (quick)

From Format §18 — plain-language map:

| Code / situation | Fix direction |
|---|---|
| `V2-FORMAT` | `"format": "memorix-record/2"` |
| `V2-ROOT` | Move unknown roots into `.data` or drop |
| `V2-RECORD-ID` / `V2-CONCEPT-TITLE` / `V2-STATE` | Complete envelope |
| Directory conflicts | New subject → new `recordId` |
| `V2-DISCRIMINATOR` | Supply discriminator or metadata default |
| `WRITE_DISCRIMINATOR_IMMUTABLE` | remove + add new instance |
| `V2-ASSOC-COPIED` | Strip copies; use relationship engine |
| `WRITE_STALE_REVISION` | Re-read; retry exact revision |
| `WRITE_G4_ASSOCIATED_ROOT` / `WRITE_G4_COPIED_PAYLOAD` | Central relationships collection |
| Abstract write errors | `patchAbstract` / reverse maps — never invent abstract collections |

---

## 4. Failure matrix (runtime HTTP)

Use this as the first lookup when demos break.

### 409 — stale revision / conflict

| Likely cause | Fix |
|---|---|
| Concurrent edit; wrong `expectedRevision` | Re-GET content; merge; retry with current revision |
| Workflow PUT with stale revision | Same — Studio shows stale conflict UI |

```bash
# pattern
curl -s "${H[@]}" "$SERVICE/api/data/content?objectType=work-items&contentType=workflow&id=wi-100"
# note revision → PUT /api/workflow with expectedRevision
```

### 405 — relationship link mutation

| Likely cause | Fix |
|---|---|
| Ad-hoc PUT/DELETE on links | Use `/api/relationships` verify / rebuild / discover |
| Host treating links like ordinary documents | Relationship engine only |

### 503 — durable routes unavailable

| Likely cause | Fix |
|---|---|
| Prod/durable mode without Mongo | Set `MEMORIX_SERVICE_MODE=prod` + `MONGO_URI` |
| Service down | Check `/health`; restart `dev-stack` |
| Port already in use / AirPlay conflict | Avoid **5000/7000**; stay in zone **5100–5119**; see `@x12i/ports-manager` `port-map.json` avoid list (`npx x12i-ports avoid`) |

### Unresolved relationships

| Likely cause | Fix |
|---|---|
| Target subject/content missing | Write targets first; or rebuild definition after targets exist |
| Wrong definition / candidate set | Discover/rebuild for the definition id |
| Expansion shows `targetStatus: "unresolved"` | **Expected** calculated signal — do not invent fake targets |

### Abstract write error

| Likely cause | Fix |
|---|---|
| Treating abstract type as a write collection | Write concrete object types |
| Non-reversible abstract property | Adjust bindings / use concrete path |
| Multi-target ambiguity | Disambiguate concrete type / id |

### Intelligence “did nothing”

| Likely cause | Fix |
|---|---|
| Expecting auto-install | Explicit `/api/metadata/install` or Studio install |
| Insufficient evidence | Gather fixtures; do not force install |
| Wrong org/agent | Fresh `sandbox*` + `opx` |

### Discriminator failures

| Likely cause | Fix |
|---|---|
| Missing `data.analysisType` (etc.) when required | Supply value or profile default |
| Changing immutable discriminator in place | remove old instance + add new |

### Scope / headers

| Symptom | Fix |
|---|---|
| 400 on `agentId` | Use plural `x-memorix-agent-ids` |
| Empty / wrong types | Install pack; fix agent set |
| Wrong tenant data | Fix `x-memorix-org-id` |

### Memory / pipelines

| Symptom | Fix |
|---|---|
| Secret rejected | Pass `credentialRef` only |
| Raw present, no snapshots | Check pipeline runs & artifacts |
| Quarantine growing | Inspect mapping / provider shape |

### Stack / networking

| Symptom | Fix |
|---|---|
| Studio loads, API fails | Service `:5100/health`; proxy target |
| Port in use | Change `MEMORIX_*_PORT` (even=API, odd=UI) |

---

## 5. Isolation & acceptance orgs

| Rule | Detail |
|---|---|
| Acceptance org | Fresh `sandbox*` only |
| Agents | `["opx"]` via plural header |
| Databases | Touch only ledgered acceptance DBs for gates |
| Forbidden | Live/legacy customer org names as acceptance targets |
| Rollback mindset | Prefer re-run from fixtures over hand-editing Mongo |
| Hash rehearsals | Canonical hash compares — re-migrate + re-validate |

### Acceptance tooling rename

| Old | New |
|---|---|
| `g12ops*` | `sandbox*` |
| `G12_ORG_ID` | `SANDBOX_ORG_ID` |
| `generateG12OrgId` / `assertSafeG12OrgId` | `generateSandboxOrgId` / `assertSafeSandboxOrgId` |
| `test:live-g12` / `G12_LIVE_PIPELINE` | `test:live-sandbox` / `SANDBOX_LIVE_PIPELINE` |
| `scripts/phase*` gate runners | removed |
| Stack profile | `local-stack` |

```bash
# Prefer isolation helpers from @x12i/memorix-service
ORG=$(node --input-type=module -e "import { generateSandboxOrgId } from '@x12i/memorix-service'; console.log(generateSandboxOrgId())")
H=(-H "x-memorix-org-id: $ORG" -H "x-memorix-agent-ids: opx")
```

If a script hard-codes a legacy customer org for “convenience,” change it before the gate. Product ebooks and acceptance must not instruct engineers to use those orgs.

Studio e2e and live pipeline gates expect `SANDBOX_ORG_ID` (must match `sandbox*`).

---

## 6. Ops narrative migration drill (WI-100)

1. Take a present-shaped work-item snapshot fixture (local file).
2. `migrate-to-v2` with `--object-type=work-items --content-type=snapshots --data-category=entity`.
3. Confirm `v2` has no copied `associated*` payloads.
4. Inspect `extractedRelationships` — map definition ids where known.
5. `validate-v2` on `v2`.
6. Install opx metadata on fresh `sandbox*`.
7. Write `v2` through `/api/data` (absent / exact revision preconditions).
8. Materialize links via relationship discover/verify (or pipeline).
9. Compose `wi-100` — siblings only; expansion for Alex without nested copies.
10. Force a stale workflow PUT → expect **409**.

Assertions:

- [ ] `format === "memorix-record/2"`
- [ ] No `V2-ASSOC-COPIED`
- [ ] Links are four fields; expansion virtual
- [ ] Stale revision → 409
- [ ] Org `sandbox*`; agents `["opx"]`
- [ ] No direct Mongo domain writes

---

## 7. Rollback and repair strategies

| Problem | Prefer |
|---|---|
| Bad migrated batch | Re-run migrate + validate from present inputs; rewrite via data plane |
| Bad links | Relationship rebuild/cleanup for definition — not manual link JSON surgery |
| Bad metadata install | Install known-good opx pack version into a **new** `sandbox*`; compare |
| Quarantine pile-up | Fix mapping; replay Memory/pipeline from fixtures |
| Temptation to “fix Mongo” | Stop — violates mutation hard rule |

Operational history (pipeline attempts, memory pulls) lives in operational ledgers — do not stuff run history back into subject snapshots during repair.

---

## 8. Embedding migration in host tooling

```ts
import { migratePresentToV2, validateMemorixRecordV2 } from "@x12i/memorix-format";

function migrateOne(present: unknown, opts: {
 objectType: string;
 contentType: string;
 dataCategory: "entity" | "event" | "knowledge";
 issuer?: string;
 associatedPropertyMap?: Record<string, string>;
}) {
 const result = migratePresentToV2(present, opts);
 if (!result.ok) throw new Error("migrate failed");
 const v = validateMemorixRecordV2(result.v2);
 if (!v.ok) throw new Error(JSON.stringify(v.errors));
 return result; // caller persists v2 + handles extractedRelationships separately
}
```

Never auto-persist `extractedRelationships` onto the V2 document as `associated*`. Hand them to the relationship engine with explicit definition ids.

---

## 9. Decision tree

```text
Do I still have present envelopes in the wild?
└─ migrate-to-v2 → validate-v2 → write V2 via data plane

Do I see associated* with titles/data blobs?
└─ Reject / strip — extract links out-of-band

HTTP 409?
└─ Re-read revision → retry

HTTP 405 on links?
└─ Relationship verify/rebuild — not ad-hoc CRUD

HTTP 503 on durable routes?
└─ Mongo/prod mode configuration

Unresolved expansion?
└─ Missing target content or rebuild — do not invent subjects

Abstract write failed?
└─ Concrete objectType + reverse maps

Intelligence did nothing?
└─ Explicit metadata install

Wrong types / empty Studio?
└─ Scope headers + pack install
```

---

## 10. Anti-patterns

| Anti-pattern | Why it fails | Do this instead |
|---|---|---|
| Treat present as target truth | Divergent platform | V2 only after migrate |
| Re-attach fat `associated*` after migrate | Fails validate / G4 writers | Central four-field links |
| Auto-persist extractedRelationships onto snapshots | Wrong store | Relationship engine |
| Hand-edit Mongo to “unblock” | Breaks hard rules | Data plane + rebuild |
| Accept on live customer orgs | Contamination | Fresh `sandbox*` |
| Ignore warnings from migrate | Silent identifier/timestamp debt | Review warnings before write |
| Collapse unresolved into fake targets | Lies in Explore | Keep unresolved status |
| One-shot “migrate and install intelligence” | Mixes layers | Separate migrate, write, propose, install |

---

## 11. Escalate packet (for humans / builders)

When filing or receiving a failure, require:

| Field | Example |
|---|---|
| Org id | `sandboxab12cd` |
| Agent ids | `["opx"]` |
| Subject / content | `wi-100` / `workflow` |
| Revision observed | `3` |
| Route + status | `PUT /api/workflow` → `409` |
| Run / attempt id | pipeline attempt uuid |
| Artifact ref | failure artifact id |
| Fixture mode | deterministic opx-source: yes/no |
| Screenshot | scope bar + error |

Builders twin turns this into a Studio quick-check list.

---

## 12. Associated* migration table (detail)

| Input | V2 record | `extractedRelationships` |
|---|---|---|
| Item with `recordId` (+ copied fields) | `associated*` removed | status `mapped` or `pending-metadata-resolution`; `droppedPayloadKeys` lists copies |
| Item without target `recordId` | removed | `quarantined-missing-recordId` |
| Non-array `associated*` | removed | warning; skipped |

Copied payload is **never** written onto V2. Resolvable links are **never** silently discarded — they leave as out-of-band candidates.

Success vs failure:

- Migration can succeed **with warnings** when a V2 envelope is produced (`ok: true`) even if relationships are pending/quarantined.
- Callers must still `validateMemorixRecordV2(result.v2)`.
- Migration does not throw merely because an association target is missing; it quarantines.

Gate G4 reminder: authoritative physical relationship storage is the centralized org-prefixed per-category `relationships` collection — not Option A roots on snapshots.

---

## 13. End-to-end repair playbooks

### Playbook A — Stale workflow edit (409)

1. `GET /api/workflow?id=wi-100` (or data content read).
2. Diff operator intent against `record.data`.
3. `PUT /api/workflow` with `expectedRevision` = current.
4. If 409 again, another writer won — loop once; then escalate with both revisions.

### Playbook B — Missing links after migrate

1. Confirm V2 subjects for endpoints exist (`employee-10`, `wi-100`).
2. Map extracted candidates → `relationshipDefinitionId` values installed in opx metadata.
3. `POST /api/relationships/discover` (bounded) and/or pipeline `relationship-write`.
4. `POST /api/relationships/verify` for the definition.
5. Expand in compose/query — unresolved targets stay visible if still missing.

### Playbook C — Abstract write confusion

1. Confirm the abstract type is virtual (People).
2. Choose concrete `objectType` (`employees` / `contractors`) or use documented reverse-map patch APIs.
3. Never create an abstract collection or abstract `recordId`.

### Playbook D — Intelligence silence

1. Confirm proposal artifact exists (Design / `/api/intelligence/proposals`).
2. Confirm effective metadata hash unchanged after propose (expected).
3. Explicit install with confirm — or reject.
4. Re-check `/api/metadata/effective`.

### Playbook E — 503 on durable routes

1. `GET /health`.
2. Check `MEMORIX_SERVICE_MODE` and `MONGO_URI` for durable/prod.
3. Restart `dev-stack` / service; re-try the same scoped call.

---

## 14. Sample migrate CLI session

```bash
# 1) migrate
npx memorix-format migrate-to-v2 ./fixtures/present-wi-100.json \
 --object-type=work-items \
 --content-type=snapshots \
 --data-category=entity \
 --issuer=opx-source \
 > ./out/migrate-wi-100.json

# 2) extract v2 document from tool output (shape depends on CLI pretty-print)
# validate the v2 object:
npx memorix-format validate-v2 ./out/wi-100-v2.json

# 3) write via service into fresh org
ORG=sandboxmig01
H=(-H "x-memorix-org-id: $ORG" -H "x-memorix-agent-ids: opx")
curl -s "${H[@]}" -X POST "$SERVICE/api/metadata/install" \
 -H "content-type: application/json" \
 -d '{"packId":"opx","confirm":true}'
curl -s "${H[@]}" -X PUT "$SERVICE/api/data/content" \
 -H "content-type: application/json" \
 -d @./out/wi-100-write-body.json
```

Do not skip validate. Do not write present envelopes to the data plane as if they were V2.

---

## 15. Next

| Track | Link |
|---|---|
| Studio quick checks | [Builders twin](../builders/BOOK.md) |
| Format validation depth | [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md) §§18–19 |
| Relationship engine | [Relationships & Memory](../../06-relationships-and-memory/developers/BOOK.md) |
| Workflow 409 UX | [Workflow & Studio](../../09-workflow-and-studio/developers/BOOK.md) |
| OpX pack goldens | [Agent packs](../../11-agent-packs-and-opx/developers/BOOK.md) |
| Intelligence install gate | [Intelligence](../../10-intelligence/developers/BOOK.md) |

---

*End of Migration & Troubleshooting — Developers.*

---

# Migration & Troubleshooting — builders

# Migration & Troubleshooting — Builders

**Audience:** Studio operators diagnosing issues and preparing a clean escalate packet. 
**Twin:** [Developers version](../developers/BOOK.md) 
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md). 
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md) 
**Status:** Present-tense operator checklist. Deterministic fixtures preferred.

---

## How to read this guide

1. Run **quick checks** before escalating.
2. Learn the **stale conflict** and **unresolved link** patterns you will see in Explore.
3. When you escalate, send the **packet** — not just “it broke.”
4. Migration mechanics (CLI present → v2) live in the [developers twin](../developers/BOOK.md).

---

## The central idea

> **Most Studio failures are scope, install, revision, or missing targets — not “Mongo is wrong.” Fix the layer you can see; escalate with evidence.**

![Stale conflict](../screenshots/17-stale-conflict.png)

```text
Scope bar OK?
 → Metadata installed?
 → Memory pull OK?
 → Pipeline artifact readable?
 → Explore subject + revision?
 → Escalate with packet
```

---

## 1. Quick checks (in order)

1. **Scope bar** — fresh `sandbox*` + agent `opx`? (Wrong org looks like “data vanished.”)
2. **Metadata installed** for this agent? (Empty types → Design install.)
3. **Memory pull** succeeded? (No raw → pipelines cannot promote.)
4. **Pipeline failure** — open Failures + Artifacts (readable context?).
5. **Stale edit** — refresh subject, re-apply change, save again (exact revision).
6. **Relationship unresolved** — is the target subject present? Rebuild/verify from Operate if definitions exist.
7. **Intelligence** — did you expect a proposal to change types without install? It will not.
8. **Discriminator** — analysis/decision missing a type label? Supply it or use the defaulted form.

If all eight look healthy and the product still misbehaves, escalate with the packet below.

---

## 2. Stale conflicts (what you should do)

When Explore/Workflow save fails because someone else (or you in another tab) saved first:

1. Read the conflict message — do not force-overwrite.
2. Refresh the subject.
3. Re-apply **your** intent on the latest revision.
4. Save again.

This is normal optimistic concurrency — not a broken database.

---

## 3. Composition and relationships (visual tells)

| You see | Meaning | Action |
|---|---|---|
| Several siblings on WI-100 | Composition — healthy | Edit the right sibling (workflow vs snapshot) |
| Link with missing target label / unresolved | Target content absent or stale definition | Create/fix target or Operate → relationship verify/rebuild |
| Nested “associated employee” blob inside a snapshot | **Wrong model** | Screenshot + escalate — should be a link + expansion |
| Abstract People list | Virtual union | Writes still go to concrete employee/contractor |

---

## 4. Symptom → where to click

| Symptom | Studio area | Likely next step |
|---|---|---|
| Empty types / agents | Design | Explicit metadata install |
| Proposal didn’t change anything | Design | Review → **install** or reject (by design) |
| Pull failed / quarantine | Operate → Memory | Fix source/credentialRef; re-pull |
| Step failed | Operate → Failures / Artifacts | Read artifact; fix mapping/service; retry |
| Save conflict | Explore | Refresh + retry |
| Link missing | Operate → Relationships | Verify/rebuild definition |
| Wrong tenant’s subjects | Scope bar | Fix org / agents |

---

## 5. Isolation rules builders enforce

- Acceptance and drills: **fresh `sandbox*`** only.
- Agent: **`opx`** unless your pack documents otherwise.
- Never practice destructive installs on live customer orgs.
- Prefer re-running fixtures over asking engineering to “just patch storage.”

---

## 6. Escalate packet (copy/paste)

Send this block to developers:

```text
Org id: sandbox________
Agent ids: ops
Subject id: (e.g. wi-100 / employee-10)
Content type: (e.g. workflow / snapshots)
Revision seen: ________
Screen: Design / Operate / Explore → ________
Error text: ________
HTTP status if known: ________
Memory pull run: ________
Pipeline run/attempt: ________
Artifact id: ________
Deterministic fixtures (opx-source)? yes / no
Screenshot: attached (scope bar + error)
What I already tried:
 - [ ] scope bar
 - [ ] metadata install
 - [ ] refresh + retry save
 - [ ] opened failure artifact
```

Without org, agents, and a screenshot of the scope bar, debugging starts blind.

---

## 7. Decision tree

```text
Data missing?
└─ Scope → install → Memory → pipeline → Explore

Save failed?
└─ Stale? refresh+retry · else capture status + packet

Links wrong?
└─ Target exist? · Operate verify/rebuild · no nested copies

Types didn’t change after intelligence?
└─ Expected — install explicitly or reject

Still stuck?
└─ Escalate packet (section 6)
```

---

## 8. Anti-patterns

| Anti-pattern | Why it hurts | Do this instead |
|---|---|---|
| Escalate with only “broken” | No reproduction | Packet above |
| Retry without reading artifacts | Hides cause | Operate → Artifacts first |
| Install speculative proposals on shared orgs | Drift | Fresh `sandbox*` |
| Ask for Mongo edits | Violates product rules | Data plane / rebuild |
| Ignore unresolved and invent targets | False graph | Fix targets or leave unresolved visible |

---

## 9. Next

| Track | Link |
|---|---|
| Migrate CLI / failure matrix | [Developers twin](../developers/BOOK.md) |
| Studio IA | [Workflow & Studio builders](../../09-workflow-and-studio/builders/BOOK.md) |
| OpX pack checklist | [Agent packs builders](../../11-agent-packs-and-opx/builders/BOOK.md) |
| Intelligence review | [Intelligence builders](../../10-intelligence/builders/BOOK.md) |
| Format mental model | [Format builders](../../01-format-for-dummies/builders/BOOK.md) |

---

*End of Migration & Troubleshooting — Builders.*

---

# Getting Started & Tutorials — developers

# Getting Started & Tutorials — Developers

**Audience:** Engineers taking Memorix from zero to the core loop: metadata → data → Memory/pipeline → relationships.
**Twin:** [Builders version](../builders/BOOK.md)
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md).
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md)

---

## 1. Installation for agents

Give your AI tooling the Memorix docs as a package before anything else:

```bash
npm i -D @x12i/memorix-docs
```

### Why install it

- **Versioned product truth for agents.** Cursor / Codex / CI agents get a pinned, upgradable knowledge surface — no stale prompts, no scraped HTML.
- **Structured Markdown.** Per-book MD, chapter indexes, use-case packs, and `agent-manifest.json` ship inside the package.
- **Typed discovery.** `listUseCases`, `getBookMarkdown`, `getUseCaseMarkdown` answer “which book / chapter?” without hallucinated paths.
- **Knowledge out of prompts.** Product rules (four-field relationships, explicit installs, V2 envelope) live in a dependency you update deliberately.

### The devDependency rule

`@x12i/memorix-docs` documents the product for tools. It is **never** a production runtime dependency:

- Do: `devDependencies`, pin or caret the version, point agent rules/skills at the package.
- Do not: add to `dependencies`, import from shipped product code, or fork the content.

Humans do not need the package — this website is the human surface.

---

## 2. Getting started

**Mental model:** `sandbox*` is a disposable **tenant** (DB isolation). `opx` is the **agent pack** you install inside that tenant — not part of the org name.

Scope headers (every call):

```http
x-memorix-org-id: sandbox<suffix>
x-memorix-agent-ids: opx
```

Fresh `sandbox*` orgs only — never live customer orgs. Singular `agentId` headers are rejected. Prefer `generateSandboxOrgId` from isolation helpers over inventing ids by hand.

### Working loop (do this in order)

| Step | What | Why |
|---|---|---|
| 1 | Install docs for agents: `npm i -D @x12i/memorix-docs` (chapter 1) | Agents need versioned product truth, not scraped HTML |
| 2 | Run the local stack: `node scripts/dev-stack.mjs` (profile `local-stack`, chapter 4) | Turns on service + Studio + opx-source so you have a real boundary |
| 3 | Set scope: fresh `sandbox*` + `agentIds: ["opx"]` | Tenant and vocabulary are separate; wrong scope looks like “missing types” |
| 4 | **Explicitly install** `@x12i/memorix-opx-pack` into that org | Vocabulary must be live before data work; nothing auto-installs |
| 5 | Read/write through `/api/data` only (chapter 5) | Only mutation path — never direct Mongo |
| 6 | Memory → pipeline → relationships (chapters 7–8) | Land exact raw, promote deliberately, then materialize four-field links |

**Install populates metadata only** — agents, object/content types, abstracts, relationships, sources, services, pipelines in the org metadata DB. It does **not** create Alex / WI-100 / PROC-2, pull Memory, run pipelines, or seed subject data. Those come from steps 5–6.

Subjects you will meet everywhere: employee **Alex**, work item **WI-100**, procedure **PROC-2**.

---

## 3. Best practice — three sides and authoring order

Build Memorix work in **three sides**. Order matters: every later piece must reference something that already exists.

| Side | What you author | Why it comes here |
|---|---|---|
| **A — Vocabulary** | Object/content types, abstracts + bindings, relationship **definitions** | Nouns first — engines and UIs need legal addresses |
| **B — Capabilities** | Services (only if mappings / Memory / relationships are not enough) | Verbs that need custom code, remote I/O, or AI |
| **C — Runtime** | Memorix data plane (`/api/data`), Memory, pipelines (+ optional schedule) | Ordered, retryable runs that wire A and B and persist through canonical ports |

### Authoring checklist (inside a pack)

1. **Scope** — org + agent in every call. *Why:* isolation + effective metadata resolution.
2. **Object / content types** — what subjects exist and what content they may wear. *Why:* format freezes the envelope; metadata declares legal OT/CT pairs.
3. **Sources** — where Memory pulls from (`credentialRef` only, never raw secrets). *Why:* exact provider bytes land before promotion.
4. **Abstracts + bindings** — optional unified views (People); virtual only. *Why:* UI unions without inventing abstract databases.
5. **Relationship definitions** — how subjects may link (four fields, never copied payloads). *Why:* definition first; instances later.
6. **Mappings / views / writes** — deterministic transforms and list shapes. *Why:* renames are mappings, not services.
7. **Services** — only for custom code, external calls, or AI jobs. *Why:* prefer a mapping for renames; services do not auto-persist.
8. **Pipeline** — ordered run: `memory-pull → map-normalize → validate → write-record → relationship discover/write → enrich/analyze`. *Why:* orchestration last so every step id already exists.
9. **Schedule** — optional trigger. *Why:* separate “when” from “what.”
10. **Validate → explicit install → Operate → Explore.** *Why:* proposals and pack files are not live until install; Explore proves the result.

Rule of thumb: **nouns (A) → verbs that need code (B) → ordered run (C).** Promotion is never automatic merely because raw exists.

Deep dive: [Building Pipelines](../../08-pipelines/developers/BOOK.md) · [Building Services](../../07-services/developers/BOOK.md) · [Author a new agent pack](../../11-agent-packs-and-opx/developers/BOOK.md).

---

## 4. Tutorial — run the local stack

Goal: opx-source + memorix-service + Studio on default ports.

**Why:** you need a live HTTP boundary and fixture source before install or data calls mean anything.

```bash
# from monorepo root
node scripts/dev-stack.mjs
```

Open:

- Studio: `http://127.0.0.1:5101/`
- Stack status: `http://127.0.0.1:5103/`
- OpenAPI: `http://127.0.0.1:5100/api/openapi.json`
- Health: `http://127.0.0.1:5100/health` · `http://127.0.0.1:5102/health`

Variants: `--list` prints URLs without starting; `--dev` runs the service in dev mode. Prod mode needs `MONGO_URI`. Ctrl+C stops the stack. Stack profile name: `local-stack`.

Check:

```bash
curl -sS http://127.0.0.1:5100/health
curl -sS http://127.0.0.1:5102/health
curl -sS http://127.0.0.1:5103/api/status
```

---

## 5. Tutorial — first read and write

Goal: call `/api/data` with correct scope; understand revision preconditions.

**Why:** revision-safe writes through the data plane are the only legal mutation path; stale conflicts are expected, not bugs.

Prerequisite: stack running, opx metadata installed for your `sandbox*` org.

Scope every call:

```bash
ORG=sandbox<suffix>
AGENTS=ops

curl -sS \
  -H "x-memorix-org-id: $ORG" \
  -H "x-memorix-agent-ids: $AGENTS" \
  http://127.0.0.1:5100/api/openapi.json | head
```

Read metadata first — resolve effective object/content types and views so you know legal addresses (paths under `/api/metadata`, see OpenAPI).

Write then read:

1. Create with revision precondition `absent`; update with the exact `revision`.
2. Get / list / compose the same `recordId`.
3. On a stale conflict: re-read, re-apply intent, retry.

Do not: omit scope headers, write with the Mongo driver, or invent list columns in your host — drive UI from metadata views.

---

## 6. Tutorial — metadata and abstracts

Goal: see how agents, object types, and abstracts drive reads/writes.

**Why:** vocabulary (side A) drives every address and screen; empty Design/Explore is almost always wrong scope or missing install.

1. Fetch effective metadata for `x-memorix-agent-ids: opx`.
2. Find an **abstract** (People) and its concrete bindings (employees, contractors).
3. Call abstract list/get routes under `/api/data`.
4. Attempt an abstract patch — observe reverse mapping to **one** concrete writer.

Rules to notice: no abstract collections, no abstract `recordId`, custom kinds store/resolve only — engines ignore them.

Depth: [Metadata, Agents, Abstracts & Mappings](../../05-metadata-agents-abstracts/developers/BOOK.md).

---

## 7. Tutorial — memory to pipeline

Goal: land exact provider payloads, then promote to subject content.

**Why:** raw must land exactly first; promotion is an explicit pipeline so failures stay quarantineable and retryable.

```text
opx-source → /api/memory → objectType-raw → /api/pipelines → snapshots
```

1. Ensure a **source** is declared in metadata (opx-pack provides `opx-source`).
2. Trigger a Memory pull via `/api/memory` — **`credentialRef`** only; raw secrets in the body are rejected.
3. Confirm `*-raw` content exists for the object type (**PROC-2** lands here).
4. Run the promotion **pipeline**; watch runs / attempts / artifacts under `/api/pipelines`.
5. Read the resulting `snapshots` via `/api/data`.

Failure modes: mapping errors quarantine the attempt — fix the mapping and retry; promotion never happens just because raw exists.

Depth: [Building Pipelines](../../08-pipelines/developers/BOOK.md) — including the 25-step complete opx tutorial.

---

## 8. Tutorial — relationships

Goal: materialize and verify four-field links between subjects.

**Why:** navigation links are first-class; copied neighbor payloads on source records are a defect.

1. Confirm relationship **definitions** in effective metadata.
2. Ensure both sides have subject content (`recordId`s exist).
3. Call `/api/relationships` **verify** / **rebuild** for a definition.
4. Query links; expand related subjects on read via data plane options.
5. Confirm the stored shape is only:

```text
relationshipId
relationshipDefinitionId
fromRecordId
toRecordId
```

Do not: PUT/DELETE ad-hoc product links (the service returns 405) or copy related payloads onto the source record.

You now have the core loop: **docs → stack → scope → install → data → memory/pipeline → relationships**.

---

## 9. Where to go next

| Need | Book |
|---|---|
| Whole platform map | [Platform Overview](../../00-platform-overview/developers/BOOK.md) |
| Record envelope | [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md) |
| HTTP client patterns | [Runtime Data Client](../../02-runtime-data-client/developers/BOOK.md) |
| Pipelines in depth | [Building Pipelines](../../08-pipelines/developers/BOOK.md) |
| New agent / pack phases | [Agent Packs & OpX](../../11-agent-packs-and-opx/developers/BOOK.md) |

Or take the interactive path: open **/journey** on the docs site.

---

# Getting Started & Tutorials — builders

# Getting Started & Tutorials — Builders

**Audience:** Studio operators taking Memorix from zero to the core loop — Design → Operate → Explore, without living in the API.
**Twin:** [Developers version](../developers/BOOK.md)
**Scenario:** Non-cyber `opx` · fresh `sandbox*` · `agentIds: ["opx"]` — see [_shared/opx-scenario.md](../../_shared/opx-scenario.md).
**Site:** [https://ebooks.memorix.x12i.com](https://ebooks.memorix.x12i.com) · [Hard rules](../../HARD-RULES.md)

---

## 1. Installation for agents

Two audiences consume Memorix knowledge:

| Audience | Form | Their “install” |
|---|---|---|
| **Humans (you)** | This website + Studio | None — browse and operate |
| **AI agents & tooling** | `@x12i/memorix-docs` knowledge package | `npm i -D @x12i/memorix-docs` |

Why this matters to you even if you never run npm:

- Everything in these books ships **verbatim** to agents as Markdown — the same rules you enforce in Studio (four-field links, explicit installs) are what agents read.
- If your team wires an AI assistant, ask engineering to install the docs package as a **devDependency** and point the assistant at it — that keeps its answers on the accepted architecture.
- The package is documentation for tools, **never** part of the shipped product.

---

## 2. Getting started

**Mental model:** `sandbox*` is a disposable **tenant**. `opx` is the **agent** (vocabulary pack) — not part of the org name.

Scope bar first, always:

1. Choose a **fresh** `sandbox*` organization — never a live customer org.
2. Set agent to **`opx`**.
3. Confirm Design / Operate / Explore show that same scope.

### Working loop (do this in order)

| Step | What | Why |
|---|---|---|
| 1 | Ask engineering to install `@x12i/memorix-docs` for AI tooling (chapter 1) | Agents read the same rules you enforce in Studio |
| 2 | Run the local stack (chapter 4) — or ask engineering for a running Studio (`local-stack`) | You need Design / Operate / Explore against a live service |
| 3 | Set scope: fresh `sandbox*` + agent `opx` | Tenant vs vocabulary; wrong scope looks like empty types |
| 4 | **Design** — explicitly install the opx pack (nothing auto-installs) | Vocabulary must be live before Operate or Explore mean anything |
| 5 | **Explore** — open a subject; edit with revision awareness (chapter 5) | Prove the data plane; never “patch Mongo” |
| 6 | **Operate** — Memory pull, then promotion pipeline, then relationships (chapters 7–8) | Land raw → promote deliberately → materialize links |

**Install populates metadata only.** It does not create Alex / WI-100 / PROC-2, pull Memory, or run pipelines. Those happen in Operate / Explore after install.

If screens look empty or show wrong types: fix scope and metadata install before filing a bug.

---

## 3. Best practice — three sides and authoring order

In Design, build in **three sides**:

| Side | What you author | Why |
|---|---|---|
| **A — Vocabulary** | Object/content types, abstracts, relationship definitions | Nouns first — Studio and engines need legal addresses |
| **B — Capabilities** | Services only when metadata alone cannot do it | External call, custom code, or AI — a rename is a mapping |
| **C — Runtime** | Memory, pipelines, schedules; Operate runs them | Ordered work that wires A and B and writes through the data plane |

### Authoring checklist (Design)

1. **Scope** — fresh `sandbox*` + agent `opx` in the scope bar. *Why:* isolation + effective metadata.
2. **Object / content types** — what subjects exist (employees, work-items, procedures, …). *Why:* subjects before jobs.
3. **Sources** — where Memory pulls from; credential references only. *Why:* exact source bytes before promotion.
4. **Abstracts** — People-style unions; virtual views, not databases. *Why:* unified lists without fake collections.
5. **Relationships** — definitions for navigation links. *Why:* definition first; instances later.
6. **Mappings / views** — deterministic transforms and list shapes. *Why:* renames are mappings, not services.
7. **Services** — only when Studio metadata alone cannot do it. *Why:* keep side B thin.
8. **Pipeline** — steps in order: pull → map → validate → write → relate → enrich. Separate compute from write. *Why:* every step must reference existing ids.
9. **Schedule** — only when intended. *Why:* “when” is separate from “what.”
10. **Validate → install / publish deliberately → prove in Operate and Explore.** *Why:* pack files are not live until install.

A green enrich step does not mean data was stored — there must be a write step. Promotion is never automatic merely because raw exists.

Deep dive: [Building Pipelines (builders)](../../08-pipelines/builders/BOOK.md) · [Agent Packs & OpX (builders)](../../11-agent-packs-and-opx/builders/BOOK.md).

---

## 4. Tutorial — run the local stack

Goal: a running Studio against the local opx fixture stack.

**Why:** Design / Operate / Explore need a live service and opx-source; empty screens usually mean the stack is down or scope is wrong.

1. From the monorepo root: `node scripts/dev-stack.mjs` (or ask engineering). Profile: `local-stack`.
2. Open Studio at `http://127.0.0.1:5101/`.
3. Check stack status at `http://127.0.0.1:5103/` — all services healthy.
4. Set the scope bar: fresh `sandbox*` + agent `opx`.

Ctrl+C stops the stack. Never point drills at live customer orgs.

---

## 5. Tutorial — first read and write

Goal: create and edit subject content from Studio, understanding revisions.

**Why:** revision conflicts are the product’s concurrency model — not a failure of Studio.

1. **Explore** — open an existing subject (or create one where the pack allows).
2. Edit a field in the form; save. Studio sends an exact **revision**.
3. Have a colleague (or second tab) edit the same subject; save again — observe the **stale conflict**.
4. Refresh, re-apply your intent, save. That is the intended conflict loop.

Do not ask anyone to “just patch Mongo” — Studio talks only to Memorix Service.

---

## 6. Tutorial — metadata and abstracts

Goal: see how the vocabulary drives every screen.

**Why:** side A (types / abstracts) is what Design installs and what Explore lists; empty People is almost always scope or missing install.

1. **Design → Agents** — inheritance for `opx`.
2. **Design → Types** — employees, contractors, work-items, procedures, activity-events.
3. **Design → Abstracts** — People bound to employees + contractors.
4. **Explore → People** — one unified list; open **Alex**.
5. Edit an abstract field — observe the write landing on **one** concrete record.

If the People list is empty: check scope, then metadata install, then whether concrete snapshots exist.

---

## 7. Tutorial — memory to pipeline

Goal: land exact source material, then promote it — the acceptance path.

**Why:** Memory lands exact raw; pipelines promote deliberately so failures stay visible and retryable.

1. **Operate → Memory** — pull from opx-source; confirm **PROC-2** raw landed (`credentialRef` only).
2. **Operate → Runs** — start the promotion pipeline; watch runs / attempts.
3. If a step fails: open the **artifact**, fix metadata/mapping/service, retry from Operate — once, intentionally.
4. **Explore** — open **WI-100**; confirm snapshot / workflow siblings appeared as designed.

Raw sitting in Memory is not a bug — promotion is always an explicit pipeline path.

---

## 8. Tutorial — relationships

Goal: links as navigation, never copied payloads.

**Why:** four-field links are the product; pasted “associated*” blobs on a record are a defect.

1. **Design** — confirm relationship definitions exist for the pack.
2. **Operate** — run relationship verify / rebuild.
3. **Explore** — open **WI-100**; traverse links to **Alex** and **PROC-2**.
4. Confirm expansion shows current targets — no pasted “associated employee” blocks. If you see one, escalate with a screenshot.

You now have the core loop: **docs → stack → scope → install → data → memory/pipeline → relationships**.

---

## 9. Where to go next

| Need | Book |
|---|---|
| Studio areas in depth | [Workflow & Studio](../../09-workflow-and-studio/builders/BOOK.md) |
| Platform mental model | [Platform Overview](../../00-platform-overview/builders/BOOK.md) |
| Pipelines in depth | [Building Pipelines](../../08-pipelines/builders/BOOK.md) |
| Install / author packs | [Agent Packs & OpX](../../11-agent-packs-and-opx/builders/BOOK.md) |
| Reviewing intelligence | [Structural Intelligence](../../10-intelligence/builders/BOOK.md) |

Or take the interactive path: open **/journey** on the docs site.