Application Overview

What this app is for

The DSLCore Orchestrator is the integration, scheduling and governance control plane for a fleet of DSLCore applications and the external / legacy systems wired alongside them. It exists so that cross-application behaviour — "when EIS approves an opportunity, create the project in EPA and ELWPM", "every night evaluate ELWPM obligations", "prove the tenement state in EPA still matches ELWPM" — lives in one place with a full, auditable history, instead of being scattered as point-to-point integrations inside each app.

Its defining principle is a boundary:

Domain apps own domain data and business logic. The orchestrator owns execution and integration logic. It stores mappings, events, routes, schedules, controls and execution/audit state — never the master copy of a business record.

The operating chain

REGISTER            CONNECT IDENTITY         MOVE                    SCHEDULE
apps + connectors → canonical entities   →  events → routes →     jobs → schedules →
                    + ownership + maps       field maps → delivery  executions + checkpoints
                                             (→ dead-letter)
                                                         │
                                                         ▼
                                          ASSURE                    RECORD
                                       controls → executions →   health checks +
                                       governance exceptions →   audit trail
                                       actions (→ Verified)

Read left to right, that is exactly the sidebar: Registry → Integration → Scheduling → Governance → Administration.

The five modules

1. Registry — who is out there and how do we reach them

  • ApplicationInstance — every participating app: type (DSLCore / External / Legacy / Infrastructure), environment, base URL, health endpoint, connector, schema version and operational status.
  • Connector — how to communicate with a system: type (DSLCoreHTTP / REST / Webhook / JSON / …), direction, auth type and a credential reference (never a raw secret), timeout and retry policy.
  • CanonicalEntity — the shared business entities and their canonical key field (projectCode, tenementCode, …). This is the vocabulary that lets different apps talk about the same thing.
  • EntityOwnership — which application is authoritative for an entity, or for a single field of one (ELWPM owns a tenement's licence_status; the external ERP owns a budget period's actual_cost).
  • EntityMapping — the crosswalk: canonical key ↔ each app's local key, with a sync status.
  • HealthCheck — technical liveness history per application.

2. Integration — what moves, where, and did it arrive

  • EventDefinition — the catalogue of event types, their producer and materiality.
  • EventMessage — a received event envelope: id, type, source, correlation, canonical key, payload, status and a duplicate flag (dedup by event_id).
  • RouteDefinition — what a trigger causes: source event/app → target app/entity/action, an operation (Create / Update / Upsert / Invoke / …), an optional condition and a retry policy.
  • FieldMapping — per-route field transforms: Direct, ValueMap, Expression, TypeCast, UnitConversion, Lookup or Constant.
  • DeliveryAttempt — one row per attempt to deliver an event down a route: timing, request/response summary, status.
  • DeadLetterItem — where deliveries go when their retries are exhausted; worked to Resolved / Ignored.

3. Scheduling — what runs, when, and what happened

  • ActionDefinition — an invokable operation on an app (endpoint, method, idempotency).
  • JobDefinition — a schedulable unit that runs an action, with concurrency and retry policy.
  • Schedule — Cron / Interval / OneTime, timezone, next/last run and a misfire policy.
  • JobExecution — the run history: trigger, timing, attempt, status, result.
  • SyncCheckpoint — how far an incremental sync has progressed (timestamp / cursor / sequence) and whether it is Current, Stale or Failed.
  • RetryPolicy — max attempts, initial delay, back-off multiplier, max delay, dead-letter-after-failure.

4. Governance — catch absence, inconsistency and staleness — not just events

  • ControlDefinition — an assurance control of one of ten types: Existence, Absence, Reconciliation, Freshness, Timeliness, Threshold, Completeness, Cardinality, State, Sequence — with scope, authoritative vs compare app, match key, the fields/tolerance it checks, severity, owner role and schedule.
  • ControlExecution — each run of a control: Pass / Fail / Error, records evaluated and exceptions created.
  • GovernanceException — a persistent finding worked through the lifecycle Open → Acknowledged → Investigating → Remediation → Resolved → Verified → Closed (or Accepted), carrying expected vs actual state, owner, due date and escalation level.
  • ExceptionAction — the audit-grade action log on an exception (who did what, and the resulting status).

5. Administration

  • AuditRecord — the integration/governance audit trail: actor, action, object, before/after and correlation id.

Two ideas that make it trustworthy

Verification is rerun-and-pass. A governance exception cannot jump to Verified on someone's say-so. The originating control is rerun; only if it now passes is the exception verified and eligible to close. In the demo, CTL-TENEMENT-RECON fails and raises EXC-0001; after EPA is corrected the same control is rerun (a second ControlExecution that passes) and only then does EXC-0001 move Verified → Closed.

Delivery is at-least-once; targets must be idempotent. Routes can retry, so a target may see the same upsert twice. Every action is modelled as idempotent (create-or-update by canonical key). Events are deduplicated by event_id so a replay (EVT-0001-DUP in the demo) is ignored rather than re-delivered.

Build status (updated 2026-08-28)

The framework renders the whole control plane as AI-Safe CRUD — register apps, define routes and controls, read the full history — and the runtime engines are now built (server/lib/orchestrator_engine/, with the peer-facing internal API in server/lib/internal_api/):

  • Event dispatcher — event → matching routes → condition → field transform → delivery, recording a DeliveryAttempt and dead-lettering exhausted failures (Flows 1–2).
  • Transform evaluator — Direct / Constant / ValueMap / TypeCast (Expression is deliberately deferred as a code-execution risk).
  • Control evaluators — Reconciliation (authoritative vs compare app → raises a GovernanceException) and Freshness (checkpoint age). Existence/Absence record a Skipped execution until the richer require: target config (dropped in the flattened DSL) is restored.
  • Verification — rerun-and-pass: an exception only reaches Verified when its originating control is rerun and passes (Flow 4).
  • Scheduler + retry/dead-letter — cron/interval is_due, run_due_schedules, process_retries, and a single-worker background loop (runtime.py).

Tested: the four acceptance flows are covered by unit tests that drive the engine against a fake injectable transport — server/tests/test_orchestrator_flows.py (14 tests). The engine is transport-injected, so no peer servers are needed.

Not yet done: the engine does not auto-run — its background loop is gated behind ORCHESTRATOR_ENGINE=1 (single-worker only) and the orchestrator app is not deployed; there is no multi-app integration test yet (real peer /_orchestrator blueprints in-process via make_flask_transport, or the cluster/ compose) — only the fake-transport unit tests above. The internal API contract, at-least-once delivery over Docker DNS, and the DSLCoreHTTP/REST/Webhook/JSON connectors are specified in the source package's rules/routes/controls/schedules/ workflows/integrations YAMLs.

(Phase note: some code labels this the "Phase 3" runtime; the app schema/seed were "Phase 1". The numbering has drifted — treat the bullets above as ground truth.)