Demo Runbook

A hands-on, end-to-end guide to running the DSLCore Orchestrator as a live control plane against its three domain peers (eis / epa / elwpm): how to start it, the internal API (the GETs and PUTs), how information flows, and how to fire each of the four acceptance flows and watch them work.

For what the app is and the data story, read 00_OVERVIEW.md and 01_QUICK_REFERENCE.md first. For the design rationale, see docs/implementation/orchestrator_control_plane.md. This document is the operational companion to those.

Scope. The demo runs locally from source via apps/orchestrator/cluster/ (docker-compose), not the public droplet. The deployed https://orchestrator.dslcore.net has the internal API mounted but the engine dormant (no ORCHESTRATOR_ENGINE / token env) — production activation is a separate exercise (see Notes → Running in production).

0. The cast

All four run on one private Docker network (dslcore-net) and reach each other by service name (e.g. http://epa:5048). The published ports above are for the human-facing UI only; the /_orchestrator internal API rides the same ports but is meant for control-plane traffic.

Part A — Start the cluster

From apps/orchestrator/cluster/:

A.1 Secrets (once)

cp .env.example .env

Set both values in .env (compose fails closed if either is unset). Generate real ones:

python -c "import secrets; print(secrets.token_urlsafe(32))"   # run twice

ORCHESTRATOR_SERVICE_TOKEN=<generated>   # shared bearer token, same across the cluster
SECRET_KEY=<generated>                   # Flask session secret

.env is git-ignored.

A.2 Generate models (first run, or after a schema change)

docker compose --profile setup run --rm codegen

Builds the dsl-dev image (pip install — a few minutes the first time) and runs codegen.cli all for orchestrator + eis + epa + elwpm. generated/ is git-ignored, so this is required on a clean checkout.

A.3 (Optional) Re-seed

The in-repo SQLite DBs ship already seeded, and the seed is idempotent, so this is usually unnecessary. To reset a scenario:

docker compose --profile setup run --rm seed
# full wipe of one app:  rm ../data/orchestrator.db  then re-run seed

A.4 Bring it up

docker compose up -d
docker compose ps            # all four should be "healthy"

UIs: orchestrator http://localhost:5050 · eis :5047 · epa :5048 · elwpm :5049.

A.5 Confirm the engine is running

The engine is armed only on the orchestrator (ORCHESTRATOR_ENGINE=1, 30-second tick):

docker compose logs -f orchestrator | grep -E "\[engine\]"

You want to see:

[engine] started (interval=30s)
[engine] tick: schedules={...} dispatched={...} retries={...}

Each tick runs run_due_schedules → dispatch_pending → process_retries.

Part B — The internal API (GETs and PUTs)

Base prefix /_orchestrator on every app. Every call needs the bearer token:

TOKEN=$(grep '^ORCHESTRATOR_SERVICE_TOKEN=' .env | cut -d= -f2-)
# header on every request:  -H "Authorization: Bearer $TOKEN"

Fail-closed: a missing/unset token → 503 (API refuses to mount); a wrong token → 401.

Which entities an app exposes is declared per-app in apps/<app>/schema/canonical_config.yaml (default OFF; identity field-mapping by default). Reads/writes go through base_controller, so denormalized *_name columns and rule cascades are maintained — never raw SQL.

The event envelope (POST /events)

camelCase; mapped to EventMessage columns. Required: eventId, eventType. If canonicalEntity is set, canonicalKey is required.

{
  "eventId": "EVT-DEMO-001",
  "eventType": "screening.opportunity.approved",
  "source": "eis",
  "schemaVersion": "1.0",
  "canonicalEntity": "ExplorationProject",
  "canonicalKey": "PRJ-DEMO",
  "payload": { "projectCode": "PRJ-DEMO", "projectName": "Demo IOCG",
               "jurisdictionCode": "AU-WA", "primaryCommodity": "Cu-Au" }
}

payload may be a JSON object (or pass payloadJson as a string). Dedup is by eventId — replaying the same id returns 200 duplicate and does nothing.

Quick smoke

curl -s http://localhost:5050/_orchestrator/health -H "Authorization: Bearer $TOKEN"
# a peer entity read (eis exposes ExplorationProject/Tenement/Jurisdiction):
curl -s http://localhost:5047/_orchestrator/entities/ExplorationProject/PRJ-COPPERFIELD \
     -H "Authorization: Bearer $TOKEN"

Part C — How information flows

REGISTER            CONNECT IDENTITY         MOVE                    SCHEDULE
apps + connectors → canonical entities   →  events → routes →     jobs → schedules →
                    + ownership + maps       field maps → delivery  executions + checkpoints
                                             (→ dead-letter)
                                                         │
                                                         ▼
                                          ASSURE                    RECORD
                                       controls → executions →   health checks +
                                       governance exceptions →   audit trail
                                       actions (→ Verified)

The engine tick (every 30s) is what turns this static configuration into live behaviour:

1. run_due_schedules(now) — fires any Schedule whose cron/interval is due, running its JobDefinition (health_check_all, controls.run.*, or an action on a peer) and recording a JobExecution.
2. dispatch_pending(now) — picks up every EventMessage left at status Received by the ingest endpoint, matches enabled RouteDefinitions, applies FieldMappings, delivers to the target app's /entities or /actions, and records a DeliveryAttempt per route. (This is what makes event-driven flows autonomous — before it, ingested events just sat at Received.)
3. process_retries(now) — re-attempts failed deliveries whose back-off has elapsed; dead-letters on max-attempts exhaustion.

Part D — The four acceptance flows

Flow 1 — Screening approval fans out (event-driven; the easiest to demo live)

Trigger: POST /events with screening.opportunity.approved (source eis). Moves: two routes upsert the project into epa and elwpm.

Do it:

TOKEN=$(grep '^ORCHESTRATOR_SERVICE_TOKEN=' .env | cut -d= -f2-)
curl -s -X POST http://localhost:5050/_orchestrator/events \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"eventId":"EVT-DEMO-001","eventType":"screening.opportunity.approved",
       "source":"eis","schemaVersion":"1.0","canonicalEntity":"ExplorationProject",
       "canonicalKey":"PRJ-DEMO",
       "payload":{"projectCode":"PRJ-DEMO","projectName":"Demo IOCG",
                  "jurisdictionCode":"AU-WA","primaryCommodity":"Cu-Au"}}'
# -> 202 {"status":"Received"}

Observe — within one tick (≤30s) the orchestrator log shows:

[engine] tick: ... dispatched={'dispatched': 1, 'results':
  [{'event_id': 'EVT-DEMO-001', 'status': 'Delivered', 'routes': 2,
    'delivered': 2, 'failed': 0, 'skipped': 0}]} ...

Verify it landed (note the ValueMap on epa):

curl -s http://localhost:5048/_orchestrator/entities/ExplorationProject/PRJ-DEMO \
     -H "Authorization: Bearer $TOKEN"     # epa: primary_commodity == "Copper-Gold"
curl -s http://localhost:5049/_orchestrator/entities/ExplorationProject/PRJ-DEMO \
     -H "Authorization: Bearer $TOKEN"     # elwpm: jurisdiction_code == "AU-WA"

Delivery trail on the orchestrator:

docker compose exec -T orchestrator python - <<'PY'
import sqlite3
c=sqlite3.connect('/app/apps/orchestrator/data/orchestrator.db')
for r in c.execute("SELECT route_code,target_application_code,status FROM delivery_attempt "
                   "WHERE event_id='EVT-DEMO-001' ORDER BY id"): print(r)
PY

Flow 2 — Licence risk propagates (conditional route)

Trigger: licence.obligation.at_risk (source elwpm). Route: RTE-LICENCE-RISK-EPA → epa ProjectRisk via project_risk.upsert, condition complianceRisk in [High,Critical]. Mappings: obligationCode→risk_code, complianceRisk→residual_rating, constant title.

# Delivers (High matches the condition):
curl -s -X POST http://localhost:5050/_orchestrator/events \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"eventId":"EVT-DEMO-002","eventType":"licence.obligation.at_risk",
       "source":"elwpm","schemaVersion":"1.0","canonicalEntity":"ProjectRisk",
       "canonicalKey":"RISK-DEMO",
       "payload":{"obligationCode":"RISK-DEMO","complianceRisk":"High"}}'

Send another with "complianceRisk":"Low" and watch it be Skipped (status: Ignored, condition not met) — nothing lands in epa.

Flow 3 — Reconciliation finds a mismatch (schedule-driven)

Control: CTL-TENEMENT-RECON (Reconciliation) — authoritative elwpm vs compare epa on Tenement (match_key_field=tenement_code, compares fields_json). It lists authoritative tenements, GETs each from the compare app, diffs the fields, and raises a GovernanceException on mismatch (deduped: one open exception per control+key).

Trigger: the scheduler runs it via SCH-RECON (run_tenement_reconciliation, cron 0 7 * * * Australia/Brisbane) — so it will not fire inside a short demo window. To run it on demand, drive the control directly (see Notes → On-demand control runs). Expected outcome against the seed: the seeded elwpm and epa tenement copies disagree, so:

Observe: a ControlExecution with status=Fail and one open GovernanceException (the seed's EXC-0001 narrative — licence_status / expiry_date disagree).

Flow 4 — Correction is verified (rerun-and-pass)

After epa's tenement is corrected to match elwpm, verification = rerun the same control. If it now passes, the exception is driven to Verified and an ExceptionAction of type Verified is recorded. Drive it on demand with verify_exception(exc, ...) (see Notes).

Part E — Observing & verifying

• Engine activity: docker compose logs -f orchestrator | grep "\[engine\]"
• Health flow runs autonomously via SCH-HEALTH (interval 300s); each run writes HealthCheck rows and updates ApplicationInstance.operational_status. You'll see health-checked 5: {'eis':'Healthy', ...} in a tick.
• Peek at any table (bypasses the app bootstrap — reliable): bash docker compose exec -T orchestrator python - <<'PY' import sqlite3 c=sqlite3.connect('/app/apps/orchestrator/data/orchestrator.db') for t in ('event_message','delivery_attempt','governance_exception','health_check'): n=c.execute(f"SELECT count(*) FROM {t}").fetchone()[0] print(f"{t}: {n}") PY
• In the UI: Integration → Event Messages / Delivery Attempts; Governance → Governance Exceptions; Scheduling → Job Executions.

Part F — Teardown & reset

docker compose down                     # stop + remove containers and the network

Data persists (the SQLite DBs are bind-mounted in the repo), so down does not clear a scenario. To restore the pristine seeded DBs after a demo:

# from repo root — discards demo writes:
git checkout -- apps/orchestrator/data/orchestrator.db apps/eis/data/eis.db \
                apps/epa/data/epa.db apps/elwpm/data/elwpm.db

Lock gotcha: if git checkout reports "unable to unlink … Invalid argument", a process still holds the DB open. Stop all app servers that use it first — including any local python server/run.py <app> you started outside the cluster. A Flask dev server spawns a reloader pair, so kill both PIDs (netstat -ano | grep :<port> to find them). Then re-run the checkout.

Notes

On-demand control runs (Flows 3 & 4)

The reconciliation/verify controls are schedule-driven (daily cron). To exercise them immediately, call the engine functions in an orchestrator container shell. The app's DB bootstrap needs SCHEMA_NAME set before import and server/ on the path; create_app() takes a config name, not the schema:

docker compose exec -T orchestrator python - <<'PY'
import sys, os
sys.path.insert(0, '/app/server')
os.environ['SCHEMA_NAME'] = 'orchestrator'          # must precede the import
os.environ['USE_APP_REGISTRY'] = 'true'
from datetime import datetime
from server.app import create_app
app = create_app()                                   # reads SCHEMA_NAME
with app.app_context():
    import apps.orchestrator.generated.models.orchestrator as m
    from server.database import db
    from server.lib.orchestrator_engine import OrchestratorClient, run_reconciliation
    ctl = m.ControlDefinition.query.filter_by(control_code='CTL-TENEMENT-RECON').first()
    print(run_reconciliation(ctl, m, OrchestratorClient(), datetime.utcnow()))
    db.session.commit()
PY

For Flow 4, load the open GovernanceException and call verify_exception(exc, m, OrchestratorClient(), datetime.utcnow()) after correcting epa's tenement.

The engine double-start (dev only)

The dev server's reloader spawns two processes, so you may see [engine] started twice — two engine threads (the single-worker guard is process-local). It's mostly self-correcting here (idempotent upserts + Received→terminal dedup mean the second thread finds nothing), but scheduled jobs like the health poll can double-run. Not an issue for the demo; production must run a single worker.

Running in production

The deployed orchestrator has the internal API mounted but dormant. To arm it you would set ORCHESTRATOR_SERVICE_TOKEN and ORCHESTRATOR_ENGINE=1 on the container and solve the sleeping-peer problem: on the droplet, peer apps sleep under the on-demand controller, so the client must reach them via a wake path (public URL → nginx → controller), not direct container-name calls to stopped containers. Keep it a single worker.

Outstanding milestones to turn this from a demo into a live production control plane — arm the engine (env), the sleeping-peer wake-path, single-worker discipline, and the on-demand-controller decision for the orchestrator itself — are tracked in docs/implementation/orchestrator_control_plane.md §9. Start there when picking this back up.