Architecture

Canon Ops: architecture

Canon on live data at the platform's published scale (350,000+ orders a day, 14,000+ active locations).

This repository is the reference implementation and runs only on a synthetic world (known gaps). Section 8 maps each component to its code or its gap. Platform facts come from the platform's public documentation and site; every other statement about the platform is marked "(assumption)". The repo says "branch" where this document says "store": one brand at one location.

1. On one page

What runs. One region, next to the order feed (5.5), runs a data plane and a control plane. The data plane moves order, POS, menu, store and dispatch events onto a bus, keeps hot state per store, evaluates real-time definitions on the stream and business definitions in batch, and keeps history. The control plane turns detections into incidents, proposes one action for each, checks it against a written policy, calls the platform (4.3), and records every step in an append-only ledger. People approve what the policy reserves for them.

What decides.

1. Definitions say what is true. A stuck order is what stuck_order version 2 says it is, and every detection carries that version. Each definition names its owner.
2. The policy says what may be done: tier, conditions, budgets, cooldowns and a never-list. A controller enforces it on every proposal. The COO owns it, and nothing automated can change it.
3. Rules pick the action where the runbook leaves one option. A model is asked only where there is a real choice among the actions and arguments the policy lists, or where tier 2 is possible and the approver needs a reason. Its answer is a proposal like any other.

What is measured. Precision and recall per detector per day; the share of incidents closed without a person and of actions reverted; time to detect and time to fix; model cost per resolved incident; the age of every input; drift on all of these.

flowchart LR
    subgraph SRC["Sources"]
        ORD["Order feed"]
        POS["POS integration logs"]
        MENU["Menu service"]
        STORE["Store state"]
        DISP["Dispatch"]
        SLOW["Support desk, CRM, billing"]
    end

    subgraph DP["Data plane"]
        BUS[("Event bus: Redpanda")]
        FLINK["Stream jobs: real-time terms"]
        HOT[("Hot state: Redis")]
        COLD[("Cold store: Parquet")]
        BATCH["Batch jobs: business terms"]
    end

    subgraph CP["Control plane"]
        DEFS["Definitions: term, variant, version"]
        INC["Incidents"]
        PROP["Proposer: rules, model only for a choice"]
        CTRL["Controller: policy, budgets"]
        ADPT["Adapters: platform calls"]
        GW["Key gateway: policy verbs only"]
        LEDG[("Ledger: append-only")]
        QUEUE["Approval queue and on-call"]
        MEAS["Measurement"]
    end

    ORD --> BUS
    POS --> BUS
    MENU --> BUS
    STORE --> BUS
    DISP --> BUS
    SLOW --> COLD
    BUS --> FLINK
    BUS --> COLD
    FLINK --> HOT
    COLD --> BATCH
    DEFS -.-> FLINK
    DEFS -.-> BATCH
    FLINK --> INC
    BATCH --> INC
    INC --> PROP
    PROP --> CTRL
    HOT -.->|evidence| CTRL
    CTRL -->|tier 0-1| ADPT
    CTRL -->|tier 2| QUEUE
    QUEUE -->|approved| ADPT
    ADPT --> GW
    GW --> API["Platform APIs"]
    CTRL --> LEDG
    ADPT --> LEDG
    QUEUE --> LEDG
    LEDG --> MEAS
    SLOW -.->|tickets as labels| MEAS
    LEDG --> COLD

The control plane changes the world only through the platform's APIs, behind a gateway that holds the platform key (4.7), and through drafts a person sends.

2. Data plane

2.1 Sources

The order-feed and dispatch rows, gData's menu tables and item-availability log, and store status, working mode (busyUntil) and working hours come from the platform's docs; every other cell, including what the POS logs contain and a channel listing status, is an assumption.

gData is a push: the platform writes, from four IP addresses of its own, into a Postgres 10+ database the receiver owns, in AWS eu-west-2 (London). CDC (Debezium with pgoutput) runs on Canon's own primary. The feed starts the day it is configured; earlier weeks need a history sync request. No lag is published, only "near-real-time", so measuring it is week one's first job. That one destination can take every customer's feed is an assumption. Without CDC, Canon polls gdata_order_status_history every one to two seconds, re-reading an overlap at least as long as the measured commit lag (60 seconds until known) to catch rows committed out of timestamp order.

2.2 Event bus

Redpanda, three brokers. It speaks the Kafka protocol, so Debezium and Kafka clients work unchanged, and it is one binary per node with no JVM, which suits a three-person team. The community edition is under the Business Source License; if that fails the platform's policy, Apache Kafka in KRaft mode is the drop-in.

Topics are keyed by store id (the docs suggest a v4 UUID), so each store's events stay in order in one partition: 48 partitions per topic, replication factor 3, 7 days' retention, matching the definitions' data window. Market-wide checks, such as no orders on one app in one market, run as a second stage keyed by market and channel.

2.3 Stream jobs and hot state

A keyed consumer service first; Flink only if a number demands it. The hard cases are absences: an order not injected 90 seconds after acceptance, a courier who has not arrived, a channel with no orders. Their timers run on processing time: an event-time timer fires only when later events move the watermark, so it stalls exactly when events stop (an aggregator outage, idle partitions at 03:00). A timer fires at the source timestamp plus the threshold plus a lateness allowance (2.6). Freshness comes from each source's position and heartbeat, not from arrivals (2.5). At 225 to 290 events a second (5.1), a consumer group with per-key state in RocksDB and a timer wheel is enough; gate 1 runs it. Flink replaces it if state outgrows one node or sustained load passes 10,000 events a second (assumption: about three times the burst). The definition is the contract, not the engine.

Detector state is snapshotted with its bus offsets every 30 seconds. A projection goes to Redis, which the controller, the interface and the model's evidence read:

The menu hash covers item ids, availability and modifier mapping but not prices, because a channel price above the till price is the merchant's markup, not drift.

2.4 Cold store

Parquet on object storage, by market, day and topic. Raw events stay 13 months (assumption), and the ledger is copied in hourly. DuckDB runs the batch definitions and replays over it. That is the engine the reference implementation uses, so the definitions' SQL runs over views with the same table names.

2.5 Freshness per fact

Every hot-state key carries as_of, and in production every definition lists the maximum age of each input. A detector with an input past its bound reports "blind", not "nothing found", and the controller does not act on that evidence.

2.6 Order, lateness, duplicates

Exactly-once. Snapshots and bus offsets keep state consistent inside the data plane; nothing is exactly-once against an external API. Canon assumes at-least-once delivery and makes every effect idempotent: deterministic incident ids, a unique action key in the ledger, idempotency keys and read-backs in the adapters. The controller's dedupe window and cooldowns (4.2) stop repeats.

3. Taxonomy: the definitions

3.1 A definition is code

Each term is one YAML file: term, title, unit, description, synonyms, params, a canonical variant or null, variants, and where it applies a default period (on the term or the variant). A variant has a key, owner, department, version, dated change note, history, SQL and tests (row-count bands, uniqueness, not-null, evidence shape). Detector terms also name a subject, and most a data window. Thresholds are parameters outside the SQL, so changing one is a versioned edit with an author and a reason. That is the repo's format. Production adds stream, inputs (each fact read, with its maximum age) and approvers.

3.2 The terms Canon acts on

Every detector in canon/dictionary/definitions/, the grouping the agent adds, and the batch back-office checks. Variant, version, owner and thresholds come from the YAML (owners are the repo's role names, so assumed roles on the platform); action and tier from policy version 3, the action being the rule's usual first pick; age bounds from 2.5 (production; the repo enforces none).

Second variants no detector runs, still answered by Ask: acceptance_risk cs v1 (Head of CS), churn_risk finance v1 (CFO), overdue_invoice cs v1 (Head of CS), overloaded_branch cs v1 (Head of CS), menu_drift integrations v3 (Head of Integrations). The other terms (active client, GMV, late order, onboarded merchant, stage gap) answer questions and never trigger an action.

3.3 Compiled twice

• Business terms and batch checks (active client, GMV, late order, onboarded merchant, stage gap, overdue invoice, churn risk, acceptance risk, release regression) run as batch SQL on the cold store, on the policy's cadence: acceptance and release checks hourly, invoices and churn daily at 08:00.
• Real-time terms compile to stream jobs from three templates: absence (an opening event with no closing event in time), windowed rate (failures over orders in a sliding window) and comparison (two states that should agree, such as the channel menu and the POS menu).
• The SQL stays the reference. A parity test replays a recorded day through both forms and requires the same subjects, with start times within 30 seconds (the SQL side's replay step). A stream form that disagrees with its SQL does not ship.

3.4 Ambiguity

Canon never picks a meaning silently. A term with several variants and no canonical one is answered with every variant, owner and number, plus the gaps between them, as the repo's Ask does. Detectors never run an ambiguous term: each names its variant or uses the canonical one. The model is told the definition and version behind the incident and cannot swap it.

3.5 Lineage

Every answer carries the term, variant, version, owner, parameters, period and tables read. Every ledger row carries the definition term and version, policy version, evidence, who decided, the call and the undo. Production adds the source position of each evidence fact (topic, partition and offset, or the feed's row key), with those events kept in the cold store as long as the row.

3.6 Changing a definition

Definitions and policy live in git: a change is a reviewed commit, and CI runs the third column. The repo has the layer underneath: each parameter edit bumps the version of the variants that use it, and each canonical choice is recorded; both go to a history file with author and reason, and what_changes() lists the rows a parameter change adds and drops before it is committed. It has no approval step. In production, rollback switches the acting version back, and the previous version runs in shadow for 24 hours after each change.

3.7 Business term: active client

Finance, Growth and Ops count clients differently. The repo keeps all three at version 1, with canonical: null:

-- finance (active_client.yaml)
WHERE i.issued_at >= {period_start} AND i.issued_at < {period_end} AND i.amount > 0
-- growth
WHERE m.contract_signed_at <= {now} AND (m.churned_at IS NULL OR m.churned_at > {now})
-- ops: orders from the daily aggregates plus the order-level rows
WHERE m.first_order_at IS NOT NULL AND m.first_order_at < {period_end}
  AND (daily.merchant_id IS NOT NULL OR live.merchant_id IS NOT NULL)

The three are stages of one pipeline, from contract to pilot store to handover. Growth counts from the signature, Ops from the first order, Finance from the first positive invoice; the inactivity fee means a store that stopped trading can still be invoiced. So "how many active clients" gets three numbers and four gaps: signed but not live, billed but not ordering, live but quiet (no order in 30 days, not billed), ordering but not billed. The repo's stage_gap term lists the gaps by merchant (130, 14, 6 and 9 in the seed-23 synthetic world, all generator settings). Making one variant canonical is recorded with author and reason; it bumps no version, and every answer names the variant it used (3.5).

3.8 Real-time term: stuck order

One variant, owned by the Head of Integrations, at version 2 (the change in version 2: the token state is read from the auth event stream as of the time asked). Parameters: stuck_seconds: 90, high_after_seconds: 600. An order is stuck when it was accepted at least 90 seconds ago and is not in the POS, not cancelled or rejected, and not dine-in. Severity is high when its POS token has expired or the refresh failed, or after 600 seconds.

WHERE o.accepted_at >= {period_start} AND o.accepted_at < {period_end}
  AND o.accepted_at <= {now} - INTERVAL ({stuck_seconds}) SECOND
  AND (o.injected_at IS NULL OR o.injected_at > {now})
  AND c.kind <> 'dine_in'
  AND NOT (o.status IN ('cancelled', 'rejected') AND coalesce(x.closed_at, o.placed_at) <= {now})

The evidence row names the order, store, integration, channel, wait, POS vendor and version, token state, failed attempts and last error; the controller checks targets against it. The stream form uses the absence template:

stream:                                   # production addition, not in the repo
  template: absence
  key: pos_integration                    # orders re-keyed from store to its POS integration
  lookup: store_to_integration            # broadcast table: store id -> POS integration id
  opens_on: order.accepted                # processing-time timer at accepted_at + stuck_seconds + allowance
  closes_on: [pos.injected, order.cancelled, order.rejected]
  skip_when: {channel_kind: dine_in}
  state: integration_auth_state           # latest auth event, held under the same key
  raise_severity: [auth_expired, refresh_failed, {age_seconds: high_after_seconds}]
  inputs: {order.accepted: 5s, pos.injected: 5s, integration_auth_state: 5m}

The key is the POS integration, not the order, because keyed state cannot read another key's state. Ninety seconds is a setting, not a platform fact: it equals the manual accept window but measures accepted to injected. The production version must also read deliveryplan and measure a SCHEDULED order from its requested time (assumption: such orders reach the POS nearer that time).

4. Decision model

4.1 From incident to proposal

Alarms with one cause are merged first, so one cause gets one action. In the repo, stuck orders behind one POS integration with a recent auth event, or three stuck within 10 minutes, become one pos_link_down incident re-sent as a batch; alarms on an aggregator that is down in a market fold into the outage; a bad POS release is one incident for all its stores.

Each incident kind has a runbook in the policy: the allowed actions, in order. For a stuck order: refresh the POS token, re-send the order, re-send a batch, escalate to the POS partner, draft a store message. Where the kind's runbook leaves one action at tier 0 or 1, a rule fills it in and no model is called. Stuck orders, a POS link down and an expiring token are always rules.

The model is asked when two or more actions are possible, or when a person may review the decision. It gets the incident, its evidence, the definition and version, and for each allowed action its tier, description, argument names and limits (maximum minutes, minimum error rate, allowed values, when a person must decide). It returns JSON: action, arguments, a short reason, a confidence. If the answer is missing, late or malformed, uses an action or argument not offered, or names an id not in the evidence, the rule's proposal is used instead, and the ledger records which path decided.

4.2 The controller

The controller trusts no proposal, from a rule or a model. It checks in this order and stops at the first failure:

When many stores need the same action at once, the cause is usually upstream (an aggregator, a POS release, the platform itself), so the blast-radius rule hands the batch to a person as one decision.

4.3 Adapters on the platform's calls

Where an assumed call does not exist, the action becomes a queue item with steps for a person. The platform documents only generic HTTP codes, so adapters retry only 429, 5xx and timeouts, with bounded backoff, and never retry a POST without a read-back. A circuit breaker per verb raises one platform incident instead of one per store. Deprecated endpoints live three months, so adapters pin an API version and run contract tests nightly and on every change against the developer portal's sandbox, UAT, starter account (a sample brand, menu and location) and Virtual KDS (assumption: they expose these verbs).

4.4 Ledger

Every proposal, decision, call, approval and rollback is a row, and nothing is updated or deleted: a status change is a new row pointing at its parent, approvals have their own table, and repeated declines inside a cooldown go to a side table. A row holds the evidence, definition and policy versions, who decided and at what token cost, the call, the result and the undo. In production it is Postgres, with no UPDATE or DELETE grant for the application, a unique key on incident, action, target and attempt number over the rows that start a call (a partial unique index; status rows point at that row and are not constrained), a hash of the previous row in each row, and an hourly copy to the cold store.

4.5 People

Tier 2 items go to a queue, and during the evening shift each also notifies the on-call in one line (the repo drafts it; production pages). A person can approve, approve a narrower version (offline for 45 minutes instead of 240), reject with a note, or roll back from the recorded before-state; a rejection or rollback is final for that incident. "Approve and remember" also files a policy suggestion (this kind, this action, cases like this, tier 2 to 1) for the policy owner and never changes the policy. A kind with no runbook goes on a process backlog with a runbook drafted from past human decisions; the agent does nothing until a person writes it.

Production adds single sign-on for approvers, a deadline per queue item (an offline proposal for an overloaded kitchen is useless after 20 minutes; expired means "not done"), and drafted tickets tagged with the platform's published support priorities (assumption: P1 for a market-wide outage, P2 for one store's integration, P5 for menu and configuration).

4.6 State machines

An incident. In the repo these states are implicit in ledger rows; production makes them explicit and closes an incident when the detector stops returning it.

stateDiagram-v2
    [*] --> Detected
    Detected --> Dropped: duplicate, or inside the dedupe window
    Detected --> Folded: same cause already open
    Detected --> Proposed: new
    Proposed --> Declined: controller refuses
    Proposed --> Queued: needs a person
    Proposed --> Acting: allowed
    Queued --> Acting: approved
    Queued --> Closed: rejected or expired
    Acting --> Watching: call succeeded
    Acting --> Queued: call failed, follow-up needs a person
    Watching --> Resolved: detector no longer returns it
    Watching --> Proposed: still true after the cooldown
    Watching --> Closed: a person rolled it back
    Declined --> Proposed: still true after the dedupe window
    Declined --> Resolved: detector no longer returns it
    Folded --> Closed: parent incident closes
    Resolved --> Closed
    Dropped --> [*]
    Closed --> [*]

An action. A successful action may start one follow-up (then: re-send the orders behind a refreshed token) and a failed one its fallback (if_it_fails: escalate to the POS partner); either enters as a new proposal. "Expired" is a production addition.

stateDiagram-v2
    [*] --> Proposed
    Proposed --> Declined: unknown, never, off runbook, bad arguments, condition, stale evidence or maintenance or kill switch (production), cooldown
    Proposed --> Waiting: platform rate limit
    Waiting --> Proposed: next minute
    Proposed --> Queued: tier 2, ask-first rule, budget used up, blast radius (production)
    Proposed --> Allowed: tier 0 or 1 within budget
    Queued --> Allowed: approved
    Queued --> Rejected: rejected
    Queued --> Expired: deadline passed
    Allowed --> Shadow: shadow mode
    Allowed --> Calling: live
    Calling --> Done: success, or read-back confirms
    Calling --> Failed: final error
    Done --> RolledBack: a person undoes it
    Done --> [*]
    Failed --> [*]
    Declined --> [*]
    Rejected --> [*]
    Expired --> [*]
    Shadow --> [*]
    RolledBack --> [*]

4.7 Who holds which keys

The documented X-API-KEY has no scopes, and the OAuth 2.0 pages read describe an app-install flow (assumption: no client-credentials grant or scopes; the install and refresh pages were not read in full). So a gateway holds the key and passes only the policy's verbs for one market: ideally the platform's (assumption: none exists), until then Canon's, as the only process holding the key.

One controller is active, with a standby holding a lease in the ledger database. Decisions are serial, so no race overspends a budget (the repo also decides on one thread). The model holds nothing because it can be wrong, can be misled by free text in its evidence (ticket bodies, error payloads, dish names), and can be replaced next month; none of that widens authority. The controller ignores the reasoning and checks only whether the action is permitted for this incident, on targets the detector named. In the repo the controller is a separate class in the same process and the model has no tools; separate processes and credentials are production work.

5. Scale and latency

5.1 Volume

None of this is large; components are sized for availability, retention and bursts.

5.2 Latency budget

For rule-decided stuck orders, the only measured number is the repo's: a 63 s mean from the tick an order behind a dead POS link is first seen stuck to its re-send (README, seed 23; the wait before detection is not counted). It reflects the replay, not processing: the clock moves in ticks (30 s on the Friday evening, 5 minutes on the days before) and nothing happens between them; a store gets one batch per 5 minutes at most (BATCH_EVERY in canon/agent/linkdown.py), so orders stuck in between wait; and when the token is the cause, the first batch waits for the refresh step. Production replaces the first two with timers and a per-store lock; the third stays, hence the refresh row.

The production column is a target for gate 1 to test, not a measurement. Model-decided kinds stay off this path: the median call took 5.6 seconds (p95 13.0 s) in the seed-23 Friday replay, and the client gives up at 20.

5.3 Hot state size

By Little's law, about 50 minutes from acceptance to delivery (assumption) at 4 orders a second is 12,000 open orders on average, about 57,000 at 19 a second: under 60 MB at 1 KB each. Per-store and per-integration aggregates (assumption: 20,000 stores on 14,000+ locations, about 1.5 integrations each, 60 one-minute buckets) are under 150 MB. Item availability (assumption: 150 items on 4 channels) is about 12 million rows, roughly 500 MB, in the stream job's disk-backed state; Redis holds only the hash per store and channel. It all fits in one machine's memory; the second Redis node is for availability.

5.4 Model calls

Each call has a 20-second timeout with one retry, fallback models in order, and a 60-second pause after three calls in which every model failed. In the repo a call over the cap waits for a free slot, and a call that times out or fails is recorded as unavailable before the rule decides. In production, stuck orders, a POS link down and an expiring token are rule-decided and never wait for a model (in the repo a tick's decisions wait until that tick's model calls return, which the replay's logical clock hides); for other kinds, a late or down model costs the timeout and one retry per configured model (40 s with a single model) before the rule decides, until the breaker opens.

5.5 One region

The gData docs name only AWS eu-west-2 (London), so the feed lands there for every market (assumption until the platform confirms), and in-country partitions would not keep order data in-country: it leaves before Canon sees it. Canon runs one region there; markets keep their own gateway scopes, budgets, kill switches and queue. A second region pays only once a market's rules require in-country processing and the platform delivers that market's feed in-country (assumption). A failed bus or stream job stops automatic actions with one notice ("no fresh data since 19:42; no automatic actions"); section 7 has the rest.

5.6 Cost per month

Assumptions: on-demand eu-west-2 prices for self-run components, no reserved capacity, 2,500 input and 200 output tokens per model call. Managed Kafka and managed Flink cost roughly 1.5 to 2 times the self-run lines. The team in section 9 costs more than all of it.

6. Measurement and rollout

6.1 Shadow mode

Every tier 0 and tier 1 decision is written as "would do", with its evidence and call, and nothing runs; approvals are recorded, not executed. The repo has the switch (CANON_SHADOW=1). A row ops would have reverted means the tier or the definition is wrong.

6.2 Labels in production

Production has no answer key. Labels come from people, arrive late, and each source has a known bias.

6.3 Metrics

6.4 Gates

Token refresh goes first: rule-decided, invisible to guests, harmless when wrong. The re-send behind it is not (a double order reaches the kitchen), so it ships with the synchronous read-back, only for POS vendors confirmed to drop a repeated order id.

6.5 What the first two weeks need from the platform

1. gData pointed at Canon's Postgres in eu-west-2, which whitelists the platform's four sending addresses, and a history sync. No access to the platform's own databases.
2. The facts the feed does not carry: POS injection results, token expiry and auth events, store status and working mode, item availability per channel. An export or read-only API is enough for shadow. Without them a stuck order can only be approximated as "accepted, with no KOT RECEIVED after 90 seconds", which holds only where the platform's own kitchen system receives the ticket (assumption).
3. No write credentials during shadow. Before gate 2, a write credential (an unscoped X-API-KEY), which the gateway (4.7) limits to one market, the token refresh and the POS relay re-send.
4. A named on-call per dinner shift who answers queue items, and an ops reviewer for 30 to 60 minutes a day.
5. A ticket export for labels, the maintenance calendar, and what each aggregator supports for busy mode.

7. Failure modes and safety

Automation may reduce its own authority without asking, by dropping a detector, a kind or a market to shadow. It may never increase it; only the policy owner can.

Never automated: money, contracts and messages to clients. These are tier 3 (4.2), and the never-list wins even if one is added to the actions by mistake. Detectors may find such cases (churn risk, an overdue invoice); Canon hands them to the named team with the evidence. A wrong busy mode costs minutes of orders and is undone; a wrong message to a client's finance director cannot be unsent, and a refund moves money. A store offline sits between: guests see it, so it is tier 2.

8. Reference implementation and production

What exists: a Python reference implementation on a synthetic world of 4,700 stores in 7 markets with a week of full orders (about 650,000), 17 definitions, 13 detector specs, policy version 3, the controller, the ledger, the approval queue, shadow mode, accuracy reports and an operator interface. It reads one DuckDB file on a clock tick. Nothing streams and nothing calls a real API. Its accuracy figures are scored against the generator's answer key, with thresholds tuned on seeds 7 and 11 and results reported on seed 23 (canon/dictionary/TUNING.md): over seven replayed days, 249 injected incidents, 774 detections, precision 98%, recall 99%.

9. The first 90 days

One metric ties the plan together: for stuck orders in the UAE, the minutes from the 90-second mark until the order is in the POS or a person has taken it over. Its baseline is measured in the shadow weeks, when Canon watches and does not act.

The dates assume the feed flows by week two and write credentials arrive by day 46 (assumption); a slip in either moves what follows.