CI Module Reference

Reference for the ci package and the mounted /v1/ci/{compile,validate} control-plane endpoints (#633, superseding the API half of #631).

What this is

.dagnats/ci.yml is a small, declarative CI spec: a set of named checks (each backed by a Dagger function call or a plain task type — see call: vs task: below), an optional deploy step, and an optional durable human-approval gate before deploy. The ci package (github.com/danmestas/dagnats/ci, module root) parses that YAML and compiles it into a dag.WorkflowDef the DagNats engine can run.

ci lives at the module root rather than under internal/ for one concrete reason: the dagnats-ci add-on is a separate Go module (github.com/danmestas/dagnats-ci, replace github.com/danmestas/dagnats => ../) so it can eventually be extracted into its own repository, and Go does not let one module import another module’s internal/ tree. This is the same shape as the openapi/ package’s promotion (#614) — a small, stdlib-leaning public package at the root, used by both the core module and a nested consumer.

Core DagNats stays spec-agnostic. POST /workflows (see REST API Reference) only ever accepts a dag.WorkflowDef — it has no idea ci.yml exists. The two /v1/ci/* endpoints below are the only place .dagnats/ci.yml awareness enters the control plane; they compile a spec into a dag.WorkflowDef and hand it to the exact same RegisterWorkflowWithWarnings path POST /workflows uses. The worker set that actually executes dagger.call and ci.approval steps (the Dagger caller, the Dagster adapter) is out of scope for this package and endpoint pair — see the tracking issue for that follow-up.

The `ci.yml` spec

on:
  pull_request: { branches: [main] }
  push:         { branches: [main] }
  schedule:     { cron: "0 6 * * *" }   # DagNats cron trigger, not GitHub Actions

defaults:
  module: "."          # Dagger module path in the repository
  # engine: ...        # advisory only; accepted and not interpreted

checks:
  test:  { call: "test" }
  lint:  { call: "lint" }
  build: { call: "build", needs: [test, lint], timeout: "20m" }

deploy:
  call: "publish"
  needs: [build]
  approval: required   # durable human gate before deploy

Each check compiles to a dagger.call step; approval: required inserts an approve-deploy step (ci.approval task) ahead of the deploy step so the engine waits for a human signal instead of deploying unattended. branches: under deploy: is rejected — it isn’t implemented yet and a silent no-op would be worse than an explicit diagnostic.

`call:` vs `task:`

Every check and the deploy: step accept exactly one of two mutually exclusive fields (#671):

checks:
  test: { task: "go-test" }        # compiles to Task: "go-test" — any worker
  scan: { call: "security-scan" }  # compiles to dagger.call, exactly as before

call: names a Dagger function. The step compiles to the dagger.call task type with a Dagger-shaped Metadata (module, call) the Dagger worker reads to invoke the function. This is the original, and still default-feeling, shape — Dagger stays fully supported.
task: names a plain task type verbatim. The step’s Task is set to that value with no Metadata at all, so it dispatches to whatever worker in the fleet handles that task type through the ordinary worker protocol — no Dagger involvement required. needs:, timeout: (and, on deploy:, approval:/branches:) behave identically either way.
Setting both, or neither, is a compile-time Diagnostic naming the check (Field: "<check-name>") or "deploy", in the same accumulating style as every other per-check problem — an author sees every mistake in one ci.yml push, not one push per error.
task: values are restricted to the same safe charset run labels use (lowercase letters, digits, _, -, .) because the engine builds the dispatch subject from Task verbatim, with no sanitization — an unsafe value would otherwise compile and register cleanly but silently mint a broken subject only once a run actually tries to dispatch it.

Dagger is one supported runner among any worker that speaks the task protocol — ci.yml no longer implies Dagger by construction. A repo whose workers already handle plain task types (go-test, lint, docker-build, whatever the fleet dispatches) can adopt ci.yml without touching Dagger at all; a repo already using Dagger keeps using call: unchanged.

`retries:` and `retry:`

Every check accepts an optional retry policy, mapped onto the engine’s dag.RetryPolicy (#681). retries: is shorthand for a fixed-delay policy; retry: is the full policy. Setting both is a compile-time Diagnostic — they’re mutually exclusive, the same way call:/task: are.

checks:
  # Shorthand: 3 attempts, fixed 5s delay between each -- the same default
  # a native workflow gets from the legacy StepDef.Retries field.
  clone: { call: "clone", retries: 3 }

  # Full policy: every field explicit.
  test:
    call: "test"
    retry:
      max_attempts: 5
      strategy: exponential   # "fixed" (default), "linear", or "exponential"
      initial_delay: 30s
      max_delay: 2m
      multiplier: 2

initial_delay and max_delay are Go duration strings, parsed the same way timeout: is. Unlike timeout:, an empty or omitted delay maps to zero rather than a package default — the native dag.RetryPolicy format has no builtin default for these fields, so ci.yml doesn’t invent one. Negative retries:, retry.max_attempts, or retry.multiplier values are diagnostics, not silently-ignored input; the upper bound on max_attempts and any bound on multiplier come from dag.Validate (#683), same as a native workflow.

`worker_group:`

Every check and deploy: accept an optional worker_group:, mapped onto the compiled step’s WorkerGroup field (#695). It scopes the step to a specific worker group so only a worker (or worker token, over the HTTP bridge) polling that exact group can pick it up — the mechanism a repo owner uses to run CI on their own runners without also being handed every other task in the fleet.

checks:
  test: { task: "go-test", worker_group: "repo-alpha" }

worker_group: works with both call: and task:, including dotted task types. A grouped step publishes to task.{type}.={group}.{runID}; the = before the group keeps it distinct from a dotted task type with no group, so call: (which compiles to the dotted dagger.call task type) can carry a worker group:

checks:
  build: { call: "build", worker_group: "repo-alpha" }
  scan:  { task: "dagger.custom", worker_group: "repo-alpha" }
  test:  { task: "go-test", worker_group: "repo-alpha" }

A worker or worker token must poll that exact group to receive the step. See Wire Protocol: Task Subjects.

worker_group: values follow the same charset rule task types do (A-Za-z0-9_-) but, unlike task:, may never contain a dot or = — dag.ValidWorkerGroup enforces this identically whether the value came from ci.yml or a native StepDef.WorkerGroup.

Unknown fields are diagnosed, not dropped

ci.yml’s YAML decoding used to match yaml.v3’s default (non-strict) behavior: an unrecognized key was silently ignored. That turned a typo (retres: for retries:) or a stale key (run:, left over from another CI system) into a silent behavior change instead of a caught mistake (#681). Parse now diagnoses unrecognized keys at every level that has a known field set — the top level (on, defaults, checks, deploy), defaults:, each check (including its nested retry: block), and deploy:.

checks:
  test:
    call: "test"
    retreis: 3   # typo -- diagnosed as "checks.test: unknown field \"retreis\""

Extension keys. A top-level key prefixed with x- is exempt: it, and everything nested under it, is never scanned or diagnosed. This is the same convention docker-compose and OpenAPI use for vendor extensions, and it’s also where a YAML anchor referenced by a <<: *name merge key should live, so it doesn’t read as an unrecognized real field:

x-defaults: &shared
  call: "build"
  timeout: "20m"

checks:
  build:
    <<: *shared
  build-arm64:
    <<: *shared
    call: "build-arm64"

A merge key (<<:) itself is never diagnosed as an unknown field at any level, but the scan does not resolve what it merges in — an unrecognized field on the anchor’s own body is only caught if that body is itself scanned (i.e. lives directly under a known field, not under x-).

Diagnostics, not fail-fast errors

Unlike the pre-#633 compiler (which returned the first error it hit), ci.Parse and ci.Compile accumulate every problem they find into a []Diagnostic and keep going, so an author fixes every mistake in one pass instead of one ci.yml push per error:

type Diagnostic struct {
	Line    int    `json:"line,omitempty"`
	Column  int    `json:"column,omitempty"`
	Field   string `json:"field,omitempty"`
	Message string `json:"message"`
}

Line / Column are 1-based YAML source positions. Parse decodes via yaml.Node first specifically to keep these — they’re 0 for a diagnostic raised later, inside Compile, once the original bytes are no longer available (an unknown needs target, an invalid timeout, a dag.Validate failure). That’s the “best available” position, not a missing feature.
Field names the offending top-level field (Parse) or check/deploy step (Compile) when known.
Accumulation is bounded by DiagnosticsMax (100): once reached, one final "too many errors" diagnostic is appended and every further problem is silently dropped, so a pathological spec can’t produce an unbounded response body.

Three entry points:

func Parse(spec []byte) (Spec, []Diagnostic)
func Compile(name string, s Spec) (dag.WorkflowDef, []Diagnostic)
func CompileYAML(name string, spec []byte) (dag.WorkflowDef, []Diagnostic)

CompileYAML is Parse + Compile for callers holding raw bytes (the dagnats-ci compile CLI and both endpoints below use it). The returned dag.WorkflowDef is only valid — and only non-zero — when the diagnostics slice is empty.

The `jobs:` shape

A spec with a top-level jobs: key is the GitHub-Actions-shaped spec: on:, jobs: with needs:, and steps: of shell run: commands. A spec with checks: compiles exactly as described above. A spec with both is a diagnostic: “a spec uses either jobs: or checks:, not both”.

on:
  push:
    branches: [trunk]
env:
  CI: "1"
jobs:
  test:
    timeout-minutes: 30
    steps:
      - run: npm test
  deploy:
    needs: [test]
    environment: production
    steps:
      - name: publish
        run: npx --yes wrangler@4.138.0 deploy
        env:
          CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}

Allowed keys:

top level: on, env, jobs, name
job: name, needs, timeout-minutes, retries, env, environment, working-directory, steps
step: name, run, env, working-directory

How it compiles

One step per job. The step ID is the job id (letters, digits, _ and -, not starting with a digit or -), DependsOn is needs (a single job id or a list; repeats are kept once), and Timeout comes from timeout-minutes (a positive integer up to 360; a check’s default when absent). retries is the same fixed-delay shorthand a check has (at most 100,000).
The task type comes from the caller. Every job compiles to task <task_namespace>.job in worker group <task_namespace>, where task_namespace is supplied with the compile request. A jobs: spec compiled without one is a diagnostic. The namespace must be a single subject token (no dots). CompileYAML has no namespace, so a jobs: spec passed to it is always diagnosed; use CompileYAMLWith.
The job travels in Metadata["ci.job"], as a JSON string with exactly these keys:
```
{
  "name": "", "env": {"CI": "1"}, "working-directory": "",
  "environment": "production", "secrets": [],
  "steps": [{"name": "publish", "run": "npx ...", "env": {},
             "working-directory": "", "secrets": ["CLOUDFLARE_API_TOKEN"]}]
}
```
The job’s env is the workflow’s env with the job’s layered on top (the job wins). A step’s env stays on that step for the worker to apply over the job’s. A job whose encoded JSON exceeds 64 KiB is a diagnostic, because it travels in every task message for that job, and so is a spec whose compiled workflow exceeds 768 KiB of JSON, since the workflow env is copied into every job and the whole def is stored as one NATS message.
on: is returned, not interpreted. The compile and validate responses carry it as structured JSON, normalised so a single event or a list of events becomes a map of event to filter (on: push becomes {"push":{}}). The caller decides which events exist and whether a workflow runs; the compiler only checks that on: is well formed and names at least one event.

Secrets

${{ secrets.NAME }} is accepted only as the entire value of an env entry, at workflow, job, or step level, and only bound to an env var of the same name:

env:
  CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}   # accepted
  API_KEY: ${{ secrets.PROD_TOKEN }}                            # diagnosed

The compiler removes the entry from env and records NAME in a secrets list. It never sees or stores a secret value, only a name. Secrets are scoped as in GitHub Actions: one bound in the workflow’s or the job’s env goes in the job’s secrets list, for the worker to export to every step; one bound in a step’s env goes in that step’s own secrets list and reaches that step alone.

Because the lists carry names only, a worker can export each secret under exactly one variable name, its own. A different key would silently export the secret under the wrong name, so the compiler rejects it. For the same reason, a variable is either a secret or a plain value within a job: if it is bound to a secret at any level (workflow, job, or any of the job’s steps), a plain value for it anywhere in that job is a diagnostic, so a worker is never handed both.

Any other ${{ anywhere is a diagnostic: “expressions are not supported; only ${{ secrets.NAME }} as an env value”.

Unsupported Actions keys

These GitHub Actions keys are reported as not supported, each with a reason, rather than as typos: uses, if, strategy, matrix, container, services, runs-on, outputs, permissions, concurrency, defaults, shell, continue-on-error.

YAML rules

A jobs: spec is stricter YAML than a checks: spec:

No aliases (*name), and so no merge keys (<<:) either. The compiler reads the YAML tree directly, where the YAML library’s own alias-expansion limit does not apply, so a small spec of nested aliases could otherwise expand into gigabytes of job JSON.
No x- extension keys. Their usual purpose is holding anchors, which have nothing to refer to them here.
No duplicate keys in any mapping. The repeat is reported, not silently merged or overwritten.
Env var names must match ^[A-Za-z_][A-Za-z0-9_]*$. A key such as A=B would otherwise reach the worker’s process environment as A=B=value and set A.

A spec with both jobs: and checks: gets a single diagnostic saying so, and nothing else about the spec is reported until one of them is removed.

Diagnostics

As with checks:, every problem is reported in one pass with a line and column: unknown and unsupported keys at every level, a needs: naming no job, needs: cycles (reported at the needs: of a job on the cycle, with the path spelled out), a job with no steps, a step with no run, and a bad timeout-minutes or retries.

Worker scope and job input

A worker token scopes task types and worker groups independently. A token claims <ns>.job in group <ns> when its task-type scope admits <ns>.job (the prefix <ns> does, as a namespace) and its group scope admits <ns>. Pairing both scopes on one name is a convention of the caller that mints the token, not an engine rule.

A step with exactly one dependency receives that step’s output as its input, not the run’s input (dag.ResolveInput). So for a check-in or other run input to flow through needs:, a job runner should return its input as its output.

`POST /v1/ci/compile`

Compiles a ci.yml spec and, optionally, registers the result.

Request:

{
  "name": "ci:myrepo",
  "spec": "checks:\n  test: { call: \"test\" }\n",
  "register": false,
  "task_namespace": "myrepo"
}

task_namespace is required for a jobs: spec and ignored for a checks: spec. name is required (400 if empty). spec must be at most 256 KiB — the body is read through a bounded io.LimitReader before the YAML decoder ever sees it, so an oversized spec is rejected (413) rather than truncated.

Response, 200 OK (no diagnostics):

{
  "workflow": { "name": "ci:myrepo", "version": "1.0.0", "steps": [...] },
  "def_hash": "3f1a...c2",
  "registered": false,
  "warnings": [],
  "on": {"push": {"branches": ["trunk"]}}
}

on is present only for a jobs: spec: its normalised, uninterpreted on: block. POST /v1/ci/validate reports it the same way for a valid jobs: spec. A spec with diagnostics has no on on either endpoint.

With "register": true, the compiled definition is persisted through the same RegisterWorkflowWithWarnings call POST /workflows uses — registered is true and warnings carries that call’s respond-reachability warnings (see Workflow Re-registration and def_hash for def_hash’s meaning).

Response, 422 Unprocessable Entity (one or more diagnostics):

{
  "diagnostics": [
    { "field": "build", "message": "check \"build\": unknown needs target \"lint\"" }
  ]
}

Nothing is registered on a 422, even with "register": true — diagnostics short-circuit before registration is ever attempted.

`POST /v1/ci/validate`

Same request body, minus register (it is accepted but ignored — validate never registers, by design, regardless of what the body sends). Always 200 OK. A valid spec returns valid: true:

{ "valid": true, "diagnostics": [] }

An invalid spec returns valid: false with its diagnostics:

{
  "valid": false,
  "diagnostics": [
    { "line": 2, "column": 12, "field": "defaults", "message": "..." }
  ]
}

Use this to let an author check a ci.yml (e.g. in a pre-commit hook or editor integration) without any risk of a side effect.

`dagnats-ci compile` CLI

The dagnats-ci module’s CLI wraps ci.CompileYAML directly:

dagnats-ci compile .dagnats/ci.yml --name ci:myrepo

On success it prints the compiled dag.WorkflowDef JSON to stdout. On diagnostics it prints one file:line:col: field: message line per diagnostic to stderr and exits 1 — nothing is written to stdout.

CI Module Reference

What this is

The ci.yml spec

call: vs task:

retries: and retry:

worker_group: