Skip to main content

COG-12: Orchestrations

Status:      Draft (Work in Progress)
Version:     0.1
Created:     2026-04-01
Updated:     2026-04-01
Authors:     Mike Anderson
Work in Progress

This specification is under active development. Structure and details may change significantly based on implementation experience and community feedback.

This standard specifies Orchestrations — multi-step workflow operations on the Covia Grid that compose other operations with automatic dependency resolution, parallel execution, and result aggregation.

Purpose​

Orchestrations enable complex workflows to be defined declaratively as a single asset:

  • Composition: Chain multiple operations into a pipeline with data flowing between steps
  • Parallelism: Independent steps execute concurrently without explicit threading
  • Determinism: The same orchestration definition always produces the same execution graph
  • Auditability: The entire workflow runs as a single job with per-step status tracking
  • Federation: Individual steps can execute on different venues across the Grid

Orchestrations transform isolated operations into coordinated workflows — from simple two-step chains to complex multi-agent pipelines with parallel fan-out and sequential dependencies.

Terminology​

See COG-1: Architecture for definitions of Grid terminology. See COG-7: Operations for the operation metadata format and COG-8: Jobs for job lifecycle.

An Orchestration is an Operation (per COG-7) whose adapter is orchestrator. It contains a steps array defining the workflow and a result specification defining the final output.

Core Principles​

Declarative Workflows​

Orchestrations are defined as immutable JSON metadata — not imperative code. The orchestrator adapter interprets the definition at runtime:

  • No control flow: No loops, conditionals, or branching. Each step runs exactly once.
  • Data-driven dependencies: Steps declare what data they need; the orchestrator infers execution order.
  • Static analysis: The dependency graph can be computed before execution begins.

This declarative model ensures reproducibility and makes orchestrations safe to store, share, and federate.

Automatic Dependency Resolution​

The orchestrator scans all input specifications recursively at construction time to build a dependency graph:

  • A step that references output from step N depends on step N.
  • A step with no step references has no dependencies and is immediately eligible for execution.
  • Dependencies are transitive — if step 2 depends on step 1, and step 3 depends on step 2, step 3 waits for both.

This eliminates manual dependency declaration. The data flow is the dependency graph.

Parallel by Default​

Steps without mutual dependencies execute concurrently:

Step 0: covia:read (no deps)     ─┐
Step 1: covia:read (no deps)     ─┤── run in parallel
Step 2: http:get   (no deps)     ─┘
Step 3: agent:request (deps: 0, 1, 2) ── waits for all three

The orchestrator uses virtual threads for concurrent step execution — parallelism is automatic and lightweight.

Immutable Definitions​

Like all assets (COG-5), orchestration metadata is immutable and content-addressed. The Asset ID is the Value ID (SHA3-256) of the metadata. This ensures:

  • Reproducibility: Invoking an orchestration by Asset ID guarantees the same workflow definition
  • Verifiability: The definition can be inspected before execution
  • Safe sharing: Orchestrations can be federated across venues without modification

Specification​

Metadata Format​

An orchestration is an Operation asset (COG-7) with adapter set to "orchestrator":

{
  "name": "My Pipeline",
  "description": "What this orchestration does",
  "type": "orchestration",
  "operation": {
    "adapter": "orchestrator",
    "input": {
      "type": "object",
      "properties": { ... },
      "required": [ ... ]
    },
    "steps": [ ... ],
    "result": { ... }
  }
}

Required Fields​

FieldTypeDescription
operation.adapterstringMUST be "orchestrator"
operation.stepsarrayOrdered array of step definitions
operation.resultobject or arrayOutput assembly specification
FieldTypeDescription
namestringHuman-readable name
descriptionstringPurpose and usage
typestringSHOULD be "orchestration" for discoverability
operation.inputobjectJSON Schema for orchestration input
operation.outputobjectJSON Schema for orchestration output

Step Definition​

Each element in the steps array MUST be an object with the following fields:

FieldRequiredTypeDescription
opYesstringOperation identifier — adapter reference, Asset ID, or registered name
inputYesspecInput specification (see Input Mapping)
nameNostringHuman-readable step name
venueNostringRemote venue DID or URL for federated execution
{
  "op": "agent:request",
  "name": "Extract Invoice",
  "input": {
    "agentId": ["const", "Alice"],
    "input": ["input", "invoice_text"],
    "wait": ["const", true]
  }
}

Input Mapping​

Input specifications define where each value comes from. They are evaluated recursively at step execution time.

Source Types​

Orchestration Input — ["input", ...]​

References values from the orchestration's own input:

SyntaxResolves to
["input"]Entire orchestration input
["input", "field"]input.field
["input", "config", "model"]input.config.model
Constants — ["const", value]​

Literal values embedded in the definition:

SyntaxResolves to
["const", "Alice"]String "Alice"
["const", true]Boolean true
["const", {"key": "val"}]Object {"key": "val"}
Step Output — [index, ...]​

References output from a previously completed step by zero-based index. This creates a dependency.

SyntaxResolves to
[0]Entire output of step 0
[0, "output"]step0.output
[1, "result", "name"]step1.result.name

Composition​

Input specs can be nested in maps to build structured inputs:

{
  "agentId": ["const", "Bob"],
  "input": {
    "extraction": [0, "output"],
    "metadata": ["input", "metadata"]
  },
  "wait": ["const", true]
}

Each leaf value in the map is resolved independently. The resulting map is passed as the step's input.

Constraints​

  • Step output references MUST refer to steps with a lower index (no forward references, no self-references).
  • Implementations MUST reject orchestrations with invalid step references at construction time.

Result Specification​

The result field defines how the orchestration's final output is assembled from step outputs. It uses the same reference syntax as input specifications:

{
  "result": {
    "extraction": [0, "output"],
    "enrichment": [1, "output"],
    "decision": [2, "output"],
    "original_input": ["input", "invoice_text"]
  }
}

The result specification is evaluated after all steps have completed. It MAY reference any step's output or the orchestration input.

Dependency Resolution Algorithm​

  1. Construction: For each step, recursively scan its input specification for step index references. Collect all referenced indices as the step's dependency set.
  2. Validation: Verify all step references point to lower indices. Reject the orchestration if any reference is invalid.
  3. Scheduling: Maintain a set of pending steps. In each iteration: a. Find all pending steps whose dependency sets are empty (all dependencies satisfied). b. Start those steps concurrently. c. Wait for at least one step to complete. d. Remove the completed step's index from the dependency sets of all subsequent steps. e. Repeat until all steps are complete or a step fails.

This algorithm guarantees maximum parallelism while respecting data dependencies.

Execution Model​

Orchestrations execute as a single Job (COG-8):

  1. Job creation: A new job is created for the orchestration with status STARTED.
  2. Step execution: Each step invokes its operation as a sub-job. The orchestrator manages the lifecycle of all sub-jobs.
  3. Progress tracking: The orchestration job's data includes a steps array containing per-step job records (status, output, timing).
  4. Result assembly: Once all steps complete, the result specification is evaluated and the orchestration job completes with the assembled output.
  5. Error propagation: If any step fails, the orchestration job fails with a reference to the failed step.

Step Status Tracking​

The orchestration job record includes a steps field — a vector of per-step job data:

{
  "status": "COMPLETE",
  "output": { "extraction": {...}, "enrichment": {...}, "decision": {...} },
  "steps": [
    { "op": "agent:request", "status": "COMPLETE", "output": {...} },
    { "op": "agent:request", "status": "COMPLETE", "output": {...} },
    { "op": "agent:request", "status": "COMPLETE", "output": {...} }
  ]
}

This enables inspection of individual step outcomes for debugging and auditing.

Federated Execution​

Steps MAY include a venue field to route execution to a remote venue:

{
  "op": "agent:request",
  "input": { ... },
  "venue": "did:web:remote-venue.example.com"
}

When a venue field is present:

  1. The orchestrator connects to the remote venue via the Grid protocol.
  2. The step's operation is invoked on the remote venue with the resolved input.
  3. The result is returned to the local orchestrator and made available to dependent steps.

Federated steps participate in the same dependency graph as local steps. The orchestrator handles cross-venue communication transparently.

Examples​

Simple Two-Step Pipeline​

Fetch data and process it:

{
  "name": "Fetch and Process",
  "type": "orchestration",
  "operation": {
    "adapter": "orchestrator",
    "steps": [
      {
        "op": "http:get",
        "name": "Fetch",
        "input": { "url": ["input", "url"] }
      },
      {
        "op": "langchain:openai",
        "name": "Summarise",
        "input": {
          "messages": ["const", [{"role": "user", "content": "Summarise this data"}]],
          "model": ["const", "gpt-4o"]
        }
      }
    ],
    "result": {
      "raw": [0, "body"],
      "summary": [1, "content"]
    }
  }
}

Steps 0 and 1 have no mutual dependency — they run in parallel.

Parallel Fan-Out​

Read multiple data sources concurrently, then aggregate:

{
  "name": "Multi-Source Aggregation",
  "type": "orchestration",
  "operation": {
    "adapter": "orchestrator",
    "steps": [
      { "op": "covia:read", "input": { "path": ["const", "w/vendors"] } },
      { "op": "covia:read", "input": { "path": ["const", "w/orders"] } },
      { "op": "covia:read", "input": { "path": ["const", "w/invoices"] } },
      {
        "op": "agent:request",
        "input": {
          "agentId": ["const", "Analyser"],
          "input": {
            "vendors": [0, "value"],
            "orders": [1, "value"],
            "invoices": [2, "value"]
          },
          "wait": ["const", true]
        }
      }
    ],
    "result": { "analysis": [3, "output"] }
  }
}

Steps 0-2 run in parallel. Step 3 waits for all three.

Agent Pipeline (AP Invoice Demo)​

Three-agent pipeline where each agent processes the previous agent's structured output:

{
  "name": "AP Invoice Pipeline",
  "type": "orchestration",
  "description": "Alice extracts, Bob enriches, Carol approves",
  "operation": {
    "adapter": "orchestrator",
    "input": {
      "type": "object",
      "properties": {
        "invoice_text": { "type": "string", "description": "Raw invoice text" }
      },
      "required": ["invoice_text"]
    },
    "steps": [
      {
        "op": "agent:request",
        "name": "Alice — Extract",
        "input": {
          "agentId": ["const", "Alice"],
          "input": ["input"],
          "wait": ["const", true]
        }
      },
      {
        "op": "agent:request",
        "name": "Bob — Enrich",
        "input": {
          "agentId": ["const", "Bob"],
          "input": {
            "source_agent": ["const", "Alice"],
            "extraction": [0, "output"]
          },
          "wait": ["const", true]
        }
      },
      {
        "op": "agent:request",
        "name": "Carol — Approve",
        "input": {
          "agentId": ["const", "Carol"],
          "input": {
            "source_pipeline": ["const", "Alice -> Bob -> Carol"],
            "extraction": [0, "output"],
            "enrichment": [1, "output"]
          },
          "wait": ["const", true]
        }
      }
    ],
    "result": {
      "extraction": [0, "output"],
      "enrichment": [1, "output"],
      "decision": [2, "output"]
    }
  }
}

Strictly sequential — each step depends on the previous. The result contains all three stages for a complete audit trail.

Federated Pipeline​

Execute steps across multiple venues:

{
  "name": "Cross-Venue Pipeline",
  "type": "orchestration",
  "operation": {
    "adapter": "orchestrator",
    "steps": [
      {
        "op": "agent:request",
        "input": { "agentId": ["const", "Scanner"], "input": ["input"], "wait": ["const", true] }
      },
      {
        "op": "agent:request",
        "input": { "agentId": ["const", "Validator"], "input": [0, "output"], "wait": ["const", true] },
        "venue": "did:web:compliance.example.com"
      }
    ],
    "result": {
      "scanned": [0, "output"],
      "validated": [1, "output"]
    }
  }
}

Step 0 executes locally. Step 1 executes on a remote compliance venue.

Error Handling​

Step Failure​

If any step's sub-job finishes with a non-COMPLETE status (e.g. FAILED, CANCELLED):

  1. The orchestration job MUST fail.
  2. The error message SHOULD identify the failed step by index.
  3. Steps that have not yet started SHOULD be skipped.
  4. Steps that are already running MAY continue to completion but their results are not used.

Timeout Behaviour​

Orchestrations inherit the timeout behaviour of the enclosing job (COG-8). Individual steps may have their own timeouts based on the operation they invoke. The orchestrator does not impose additional timeout constraints beyond what the sub-jobs enforce.

Invalid Definitions​

Implementations MUST reject orchestration definitions that:

  • Contain forward references (step N referencing step M where M >= N)
  • Contain self-references
  • Have an empty steps array
  • Reference non-existent operations in op fields

Rejection SHOULD occur at construction time (when the orchestration job starts), not during step execution.

Security Considerations​

Resource Limits​

Orchestrations can amplify resource consumption:

  • Each step is a separate job invocation with its own resource usage
  • Parallel steps multiply concurrent resource consumption
  • Nested orchestrations (an orchestration step invoking another orchestration) create recursive resource amplification

Venues SHOULD impose limits on:

  • Maximum number of steps per orchestration
  • Maximum orchestration nesting depth
  • Total concurrent sub-jobs per orchestration

Federated Trust​

When steps execute on remote venues:

  • The local venue trusts the remote venue to execute the operation correctly
  • Step outputs from remote venues are not independently verified
  • Sensitive data in step inputs is transmitted to remote venues

Operators should consider the trust implications of federated steps, especially for orchestrations that handle credentials, personal data, or financial information.

Input Injection​

Step inputs are assembled from orchestration inputs, constants, and prior step outputs. Implementations SHOULD validate assembled inputs against the target operation's schema before invocation to prevent injection through crafted orchestration inputs.