Skip to content

protocol

import "github.com/danmestas/dagnats/protocol"

protocol/queue.go QueueSnapshot is the shared payload for two surfaces (#632): the synchronous GET /v1/queue response and the periodic event.queue.snapshot notification on the EVENTS stream. Both surfaces answer the same question -- “how deep is each task-type queue right now” -- so they share one wire schema instead of drifting apart. See docs/wire-protocol.md “Consumer contract: run lifecycle events” for the event.queue.snapshot cadence/dedup contract.

protocol/run_event.go RunEvent is the payload for the run-lifecycle consumer contract published to the EVENTS stream (event.run.{workflow}.{runID}.{status}), distinct from protocol.Event on the WORKFLOW_HISTORY stream (history.{runID}). history.* carries the full per-step timeline; event.run.* carries ONLY the reliable “this run reached a terminal state, and it is durably persisted” signal a forge/poller needs instead of GET /runs/{id} polling. See docs/wire-protocol.md “Consumer contract: run lifecycle events”.

Index

Constants

Annotation severities. These are the only three levels forges commonly distinguish; consumers should treat an unrecognized value as AnnotationSeverityNotice rather than rejecting the annotation.

const (
    AnnotationSeverityError   = "error"
    AnnotationSeverityWarning = "warning"
    AnnotationSeverityNotice  = "notice"
)

Bounds for the BUILD_LOGS hot lane (#624). dagnats owns the JetStream hot lane only — no S3, no offloader, no cache. Retention past the hot TTL is a consumer’s job, the same way run history and telemetry already work (docs/wire-protocol.md, “Consumer contract: build logs”).

const (
    // LogChunkBytesMax bounds a single LogChunk's Data payload. A
    // worker Write() larger than this is split into multiple chunks
    // (worker/log_writer.go) rather than growing one chunk unbounded.
    LogChunkBytesMax = 64 * 1024

    // LogStepBytesMax bounds the total bytes (both out and err
    // streams combined) captured for one step. Once hit, exactly one
    // LogMarkerTruncated chunk is emitted and further writes for that
    // step are dropped — bounded storage beats an unbounded log flood
    // from a runaway process.
    LogStepBytesMax = 64 * 1024 * 1024

    // LogReadChunksMax bounds a single non-follow GET /runs/{id}/logs
    // response.
    LogReadChunksMax = 1024

    // LogFollowDurationMax bounds a single SSE follow connection.
    LogFollowDurationMax = 1 * time.Hour

    // LogFollowConcurrentMax bounds concurrent SSE follow connections
    // per server process.
    LogFollowConcurrentMax = 256
)

Stream tags a LogChunk’s origin.

const (
    LogStreamOut    = "out"
    LogStreamErr    = "err"
    LogStreamMarker = "marker"
)

Marker values, carried in LogChunk.Data when Stream == LogStreamMarker. Every path that ends a task attempt/iteration (worker/context.go’s Complete, Fail, FailPermanent, FailRetryAfter, Continue, Pause) emits exactly one of these as the LAST chunk on that logs.{runID}.{stepID}.{attempt}.{iteration} subject (#624 review) — the drain-before-resolve invariant guarantees no write for that attempt/iteration lands after it. GET …/logs’s follow mode and from=failure both depend on this: a marker is a reliable end-of- attempt/iteration signal, not one a consumer has to infer from a separate history stream.

const (
    // LogMarkerCompleted is emitted by Complete.
    LogMarkerCompleted = "completed"
    // LogMarkerFailed is emitted by Fail, FailPermanent, and
    // FailRetryAfter — all three represent this attempt ending in
    // failure, whether or not the engine will retry. BEFORE the
    // resolution publish, so a consumer following the lane sees the
    // marker before (or with) the terminal step.failed event and
    // GET .../logs?from=failure has a recorded position to resolve via
    // GetLastMsgForSubject rather than a scan.
    LogMarkerFailed = "failed"
    // LogMarkerContinued is emitted by Continue (agent-loop iteration
    // boundary) — this attempt's subject is done; the next iteration
    // gets a new attempt-scoped subject.
    LogMarkerContinued = "continued"
    // LogMarkerPaused is emitted by Pause — this attempt is done; a
    // resumed step (LoadCheckpoint) gets a new attempt-scoped subject.
    LogMarkerPaused = "paused"
    // LogMarkerFenced is emitted when a journal step's delivery is
    // fenced by a later claim (ADR-026): this attempt's subject is done
    // and its successor's is already open; nothing failed.
    LogMarkerFenced = "fenced"
    // LogMarkerTruncated is emitted at most once, the moment
    // LogStepBytesMax is reached, BEFORE the attempt-ending marker
    // (which still lands last) — no out/err chunk follows it, but the
    // terminal marker always does.
    LogMarkerTruncated = "truncated"
)

AnnotationsMax is the documented ceiling on Annotations.Annotations that consumers (forge integrations) may rely on when sizing their own buffers or API batch calls. The engine does not enforce this bound -- it never parses Data -- so a worker that emits more is not rejected; it is simply outside the contract a well-behaved consumer promises to honor.

const AnnotationsMax = 1000

CancelSubjectPrefix is the core NATS subject family the engine publishes on when a run is cancelled (#802). Core, not JetStream: it is a best-effort nudge to workers holding a handler for the run, and the durable truth stays the run state (workers re-check it on Heartbeat).

const CancelSubjectPrefix = "dagnats.cancel."

LoopStateMaxBytesDefault is the default cap on the state a step.continue event may carry (#801). The whole run snapshot is one KV value bounded by the server’s max_payload, so the state a loop commits per iteration must stay well inside it.

const LoopStateMaxBytesDefault = 256 << 10

func CancelSubject

func CancelSubject(runID string) string

CancelSubject returns the cancel signal subject for runID.

type Annotation

Annotation pins one finding onto a specific file/line, mirroring the shape most forge check-run APIs (GitHub Checks, GitLab CI, etc.) expect for inline diff annotations.

type Annotation struct {
    Path     string `json:"path"`
    Line     int    `json:"line"`
    Column   int    `json:"column,omitempty"`
    Severity string `json:"severity"`
    Message  string `json:"message"`
}

type Annotations

Annotations is the top-level shape a worker places into TaskResolution.Data when it wants to surface per-line findings.

type Annotations struct {
    Annotations []Annotation `json:"annotations"`
}

type ContinueWait

ContinueWait is what a step.continue asks the engine to wait for before sending the next iteration (ADR-027). DelayMs and Until are a time to wake at (Until wins when both are set, as the later of the two is never wanted); Input wakes the step as soon as an input is queued for it. A time and Input together wake on whichever comes first.

type ContinueWait struct {
    DelayMs int64      `json:"delay_ms,omitempty"`
    Until   *time.Time `json:"until,omitempty"`
    Input   bool       `json:"input,omitempty"`
}

func (*ContinueWait) IsZero

func (w *ContinueWait) IsZero() bool

IsZero reports whether w asks for no wait at all.

func (*ContinueWait) WakeAt

func (w *ContinueWait) WakeAt(now time.Time) (time.Time, bool)

WakeAt resolves the wait’s time to wake at relative to now; false when the wait has no time component.

type Event

Event is the core communication primitive published to the history stream. Payload carries event-specific data as raw JSON to keep the type schema-agnostic.

type Event struct {
    Type          EventType       `json:"type"`
    RunID         string          `json:"run_id"`
    StepID        string          `json:"step_id,omitempty"`
    Timestamp     time.Time       `json:"timestamp"`
    Payload       json.RawMessage `json:"payload,omitempty"`
    TraceParent   string          `json:"trace_parent,omitempty"`
    TraceState    string          `json:"trace_state,omitempty"`
    WorkerID      string          `json:"worker_id,omitempty"`
    AttemptNumber int             `json:"attempt_number,omitempty"`
    // Iteration and DispatchNonce, with AttemptNumber, identify the
    // dispatch a step.continue, step.completed or step.failed answers
    // (#801). The engine accepts such an event only while the step is
    // Running on that iteration and attempt, with that nonce. Iteration
    // is a pointer because iteration 0 must be distinguishable from an
    // event an older worker sent without the field: an event without
    // Iteration carries no identity (see HasIdentity), and the engine
    // accepts it only while it is configured to accept legacy events.
    Iteration     *int   `json:"iteration,omitempty"`
    DispatchNonce string `json:"dispatch_nonce,omitempty"`
    // JournalSeq is the AGENT_JOURNAL sequence of the commit record a
    // journal step wrote before this step.continue, step.completed or
    // step.failed (ADR-026). Zero for a step without a journal.
    JournalSeq uint64 `json:"journal_seq,omitempty"`
    // InputRequestID is the submission a step.input.queued event
    // announces (ADR-027); it keys the event's msg-id so every
    // submission is its own event.
    InputRequestID string `json:"input_request_id,omitempty"`
}

func NewInputQueuedEvent

func NewInputQueuedEvent(runID, stepID, requestID string) Event

NewInputQueuedEvent constructs the step.input.queued event the API publishes after creating a submission (ADR-027). Its msg-id names the request, so a resubmission is one event.

func NewStepEvent

func NewStepEvent(eventType EventType, runID string, stepID string, payload []byte) Event

NewStepEvent constructs an Event for a step lifecycle transition. Panics on empty runID or stepID — these are programmer errors, not runtime errors.

func NewWorkflowEvent

func NewWorkflowEvent(eventType EventType, runID string, payload []byte) Event

NewWorkflowEvent constructs an Event for a workflow lifecycle transition. Panics on empty runID — programmer error.

func UnmarshalEvent

func UnmarshalEvent(data []byte) (Event, error)

UnmarshalEvent deserializes a NATS message body into an Event.

func (Event) HasIdentity

func (e Event) HasIdentity() bool

HasIdentity reports whether the event names the dispatch it answers (#801): Iteration is present and AttemptNumber is set. An event from a worker predating the identity fields has neither.

func (Event) Marshal

func (e Event) Marshal() ([]byte, error)

Marshal serializes the event to JSON for publishing to NATS.

func (Event) NATSMsgID

func (e Event) NATSMsgID() string

NATSMsgID returns the deduplication ID for JetStream idempotent publishing. Composed of run, step, and event type so replays are safe. For workflow events (empty StepID), omits the step segment to avoid double dots. An event that carries its dispatch identity (#801) gets the stable {run}.{step}.{type}.{attempt}.{iteration}: a redelivery inside the dedup window is dropped by JetStream, one outside it by the engine’s identity check, and no two iterations or attempts share an ID.

func (Event) NATSSubject

func (e Event) NATSSubject() string

NATSSubject returns the JetStream subject for publishing this event. All events for a run share one subject so consumers get ordered history. Panics on empty RunID — subjects with empty segments are invalid in NATS.

func (*Event) SetIdentity

func (e *Event) SetIdentity(iteration, attempt int, nonce string)

SetIdentity stamps the dispatch identity on the event (#801). nonce may be empty when the dispatch carried none (a legacy task payload); attempt must be the 1-based attempt the event belongs to.

type EventType

EventType identifies the kind of workflow lifecycle event. Using string constants makes events self-describing over the wire.

type EventType string

const (
    EventWorkflowStarted          EventType = "workflow.started"
    EventStepQueued               EventType = "step.queued"
    EventStepStarted              EventType = "step.started"
    EventStepCompleted            EventType = "step.completed"
    EventStepFailed               EventType = "step.failed"
    EventStepContinue             EventType = "step.continue"
    EventStepMapStarted           EventType = "step.map.started"
    EventStepMapCompleted         EventType = "step.map.completed"
    EventStepMapInstanceCompleted EventType = "step.map.instance.completed"
    EventStepSleepStarted         EventType = "step.sleep.started"
    EventStepSleepCompleted       EventType = "step.sleep.completed"
    EventStepWaitStarted          EventType = "step.wait.started"
    EventStepWaitMatched          EventType = "step.wait.matched"
    EventStepWaitTimeout          EventType = "step.wait.timeout"
    EventWorkflowCompleted        EventType = "workflow.completed"
    EventWorkflowFailed           EventType = "workflow.failed"
    EventWorkflowSpawn            EventType = "workflow.spawn"
    EventWorkflowChildCompleted   EventType = "workflow.child.completed"
    EventWorkflowChildFailed      EventType = "workflow.child.failed"
    EventWorkflowCancelled        EventType = "workflow.cancelled"
    EventStepCancelled            EventType = "step.cancelled"
    EventCompensateStarted        EventType = "compensate.started"
    EventCompensateStepCompleted  EventType = "compensate.step.completed"
    EventCompensateFailed         EventType = "compensate.failed"
    EventCompensateCompleted      EventType = "compensate.completed"
    EventApprovalRequested        EventType = "approval.requested"
    EventApprovalGranted          EventType = "approval.granted"
    EventApprovalRejected         EventType = "approval.rejected"
    EventApprovalExpired          EventType = "approval.expired"
    EventPlannerMaterialized      EventType = "planner.materialized"
    // EventStepInputQueued is published by the API when an input is
    // submitted to a step (ADR-027): it wakes a step waiting for input
    // and lets the engine settle inputs for a finished step. Its
    // msg-id carries the request id, so each submission is one event.
    EventStepInputQueued EventType = "step.input.queued"
    // EventStepWaitElapsed is published by the loop_wait timer when a
    // step's Delay or Until wait is over (ADR-027). It names the
    // dispatch it wakes, so a late fire cannot wake a later wait.
    EventStepWaitElapsed EventType = "step.wait.elapsed"
)

type FailureType

FailureType distinguishes how the engine handles a step failure.

type FailureType string

const (
    FailureTypeRetriable    FailureType = "retriable"
    FailureTypeNonRetriable FailureType = "non_retriable"
    FailureTypeRetryAfter   FailureType = "retry_after"
)

type LogChunk

LogChunk is a single unit of captured step output on the BUILD_LOGS stream, subject logs.{runID}.{stepID}.{attempt}.{iteration} (#624 review round 3: iteration is a SECOND dimension of dispatch identity, distinct from attempt — an agent-loop step’s Continue re-dispatches the SAME attempt with a new iteration, never bumping Attempts (doing so would consume retry budget the step never spent), so iteration must be part of the subject/Msg-Id too or iteration 2+’s chunks collide with iteration 0’s within the dedup window exactly the way un-scoped attempts collided before #624 review round 2. iteration is 0 for a non-loop step. Seq is monotonic per (ATTEMPT, ITERATION) pair and shared across the out/err streams (worker/log_writer.go assigns it from one counter), so ordering by Seq reconstructs write order even though out and err are independently buffered.

Data marshals to a base64 string over JSON (encoding/json’s default []byte handling) so both text and arbitrary binary output round-trip safely.

type LogChunk struct {
    Seq       uint64    `json:"seq"`
    Attempt   int       `json:"attempt"`
    Iteration int       `json:"iteration"`
    TS        time.Time `json:"ts"`
    Stream    string    `json:"stream"`
    Data      []byte    `json:"data"`
}

type QueueGroup

QueueGroup is the pending-task count for one task type on the TASK_QUEUES stream (task.{taskType} subject). OldestWaitMs is omitted when the server could not read the oldest pending message for this subject (a best-effort direct-get failure never fails the whole response/snapshot -- see internal/api/queue.go).

type QueueGroup struct {
    TaskType     string `json:"task_type"`
    Pending      int64  `json:"pending"`
    OldestWaitMs *int64 `json:"oldest_wait_ms,omitempty"`
}

type QueueSnapshot

QueueSnapshot is the wire payload for GET /v1/queue and event.queue.snapshot. Groups is always a non-nil slice, sorted by TaskType, so JSON serializes it as [] rather than null when there are no pending tasks. Truncated is set when the TASK_QUEUES stream carries more distinct task-type subjects than internal/api.QueueGroupsMax -- Groups is capped at that bound rather than growing unbounded.

type QueueSnapshot struct {
    Groups     []QueueGroup `json:"groups"`
    SnapshotAt time.Time    `json:"snapshot_at"`
    Truncated  bool         `json:"truncated,omitempty"`
}

type RunEvent

RunEvent is the wire payload published to event.run.*. Status carries the precise dag.RunStatus string (e.g. “compensated”) even when Type has coalesced to a coarser bucket. Labels is copied from the run’s dag.WorkflowRun.Labels at finalization time (see copyLabels in internal/engine/run_event.go) — omitted from the wire payload when the run has none.

Status is a plain string, not dag.RunStatus: protocol is the wire schema package and stays free of a dependency on dag’s internal types, matching Event’s existing payload-as-raw-JSON discipline.

type RunEvent struct {
    Type        RunEventType      `json:"type"`
    RunID       string            `json:"run_id"`
    WorkflowID  string            `json:"workflow_id"`
    Status      string            `json:"status"`
    CreatedAt   time.Time         `json:"created_at"`
    CompletedAt *time.Time        `json:"completed_at,omitempty"`
    Labels      map[string]string `json:"labels,omitempty"`
    TraceParent string            `json:"trace_parent,omitempty"`
    // TriggerDepth is the source run's dag.WorkflowRun.TriggerDepth,
    // copied here at finalization time (#634) so a run_terminal
    // trigger consuming this event can compute the depth of the run
    // it is about to start (source depth + 1) WITHOUT a second lookup
    // of the source run. Additive: omitempty, older events decode to
    // 0 which is also the correct depth for every manual/HTTP/cron
    // -started run.
    TriggerDepth int `json:"trigger_depth,omitempty"`
}

type RunEventType

RunEventType discriminates the three terminal outcomes a run can reach. Coarser than dag.RunStatus (which also has Compensated / CompensateFailed) — compensation outcomes are reported as RunEventFailed since compensation only runs after the workflow itself failed; the exact status still rides the Status field.

type RunEventType string

const (
    RunEventCompleted RunEventType = "run.completed"
    RunEventFailed    RunEventType = "run.failed"
    RunEventCancelled RunEventType = "run.cancelled"
)

type StepContinuePayload

StepContinuePayload is the structured payload for EventStepContinue (#801). Output is what the handler passed to Continue; State is the state it checkpointed during the iteration, committed by the engine in the same snapshot save that advances the iteration, so a crash between checkpoint and continue cannot repeat a turn.

type StepContinuePayload struct {
    Output json.RawMessage `json:"output,omitempty"`
    State  []byte          `json:"state,omitempty"`
    // StateSet marks State as the state to commit even when it is
    // empty: a handler that checkpointed nil clears the saved state,
    // where an absent State (an event without it) leaves it alone.
    StateSet bool `json:"state_set,omitempty"`
    // Wait is the durable wait the handler asked for before the next
    // iteration (ADR-027): the engine schedules it and no worker holds
    // a claim meanwhile. Absent: the next iteration is sent at once
    // (after any loop_delay).
    Wait *ContinueWait `json:"wait,omitempty"`
}

type StepFailedPayload

StepFailedPayload is the structured payload for EventStepFailed. FailureType defaults to retriable when empty (backward compat). RetryAfterMs is only used with FailureTypeRetryAfter.

type StepFailedPayload struct {
    Error        string      `json:"error"`
    FailureType  FailureType `json:"failure_type,omitempty"`
    RetryAfterMs int64       `json:"retry_after_ms,omitempty"`
}

type TaskPayload

TaskPayload is the message body published to a task subject when the engine dispatches a step for execution. Workers unmarshal this to build a TaskContext. Iteration is the agent-loop iteration index (0 for the first execution); workers include it in Continue event MsgIds to prevent JetStream deduplication across cycles.

type TaskPayload struct {
    TaskID    string          `json:"task_id"` // {runID}.{stepID}
    RunID     string          `json:"run_id"`
    StepID    string          `json:"step_id"`
    Iteration int             `json:"iteration,omitempty"`
    Attempt   int             `json:"attempt,omitempty"`
    Input     json.RawMessage `json:"input,omitempty"`
    // Metadata is static per-step config copied from StepDef.Metadata,
    // delivered to workers via TaskContext.Metadata(). Nil when the step
    // declared no metadata.
    Metadata map[string]string `json:"metadata,omitempty"`
    // RequiredCapabilities is copied verbatim from StepDef.RequiredCapabilities
    // at enqueue time. The worker reads it to gate capability handles (e.g.
    // the control plane). Nil = no capabilities declared (today's behavior).
    RequiredCapabilities []string `json:"required_capabilities,omitempty"`
    // DispatchNonce is the per-dispatch run-binding token (#380, ADR-021
    // Phase A). The engine stamps the same value on StepState.DispatchNonce
    // and here. The worker carries it on control-plane requests so the
    // server can verify the caller received this exact dispatch. Additive,
    // omitempty: legacy payloads deserialize to "" and the server rejects an
    // empty/mismatched nonce on the control-plane path.
    DispatchNonce string `json:"dispatch_nonce,omitempty"`
    // WorkflowName is the workflow definition's name (#503), stamped by the
    // engine at every publish site so SigNoz can filter/group task spans by
    // workflow instead of every run collapsing into one undifferentiated
    // span name. Additive, omitempty: legacy payloads deserialize to "" and
    // the worker simply omits the workflow_name span attribute in that case.
    WorkflowName string `json:"workflow_name,omitempty"`
    // TraceParent/TraceState carry the W3C trace context the HTTP bridge
    // stamps on each polled task (#534, #537) so a worker's execution
    // span parents onto bridge.dispatch instead of orphaning.
    //
    // The tags are the W3C wire names, deliberately UN-underscored —
    // unlike protocol.Event's persisted trace_parent/trace_state a few
    // types below. Renaming either to match the other silently drops the
    // field on decode: the poll path uses plain json.Decode with no
    // DisallowUnknownFields, so a mismatch is invisible at runtime.
    // Additive and omitempty: pre-#537 responses decode to "".
    //
    // Use observe.TraceContextFromTask to turn these into a parent
    // context rather than parsing them by hand.
    TraceParent string `json:"traceparent,omitempty"`
    TraceState  string `json:"tracestate,omitempty"`
    // State is the agent-loop state the step last committed through a
    // step.continue event (#801). The engine stores it in the run
    // snapshot alongside the iteration it belongs to and hands it to
    // every later dispatch of the step: the next iteration, a retry of
    // the current one, and a dispatch sent again after an engine crash.
    // Opaque bytes; TaskContext.LoadCheckpoint returns them. Additive:
    // legacy payloads deserialize to nil.
    State []byte `json:"state,omitempty"`
    // StateMaxBytes is the engine's cap on State, stamped on every
    // dispatch so a worker can refuse an oversized Continue before
    // publishing it, with the same number the engine would enforce.
    // Zero (a legacy payload) means LoopStateMaxBytesDefault.
    StateMaxBytes int `json:"state_max_bytes,omitempty"`
}

type TaskResolution

TaskResolution is the wire format for HTTP bridge resolve actions. The action field discriminates between complete/fail/pause/checkpoint. Workers POST this to /v1/tasks/{id}/resolve to report task outcomes.

type TaskResolution struct {
    Action     string          `json:"action"`
    Output     json.RawMessage `json:"output,omitempty"`
    Error      string          `json:"error,omitempty"`
    Name       string          `json:"name,omitempty"`
    DurationMs int64           `json:"duration_ms,omitempty"`
    Checkpoint json.RawMessage `json:"checkpoint,omitempty"`
    Data       json.RawMessage `json:"data,omitempty"`
}

type WorkerStatusSnapshot

WorkerStatusSnapshot is the per-worker counter snapshot stored in the worker_status KV bucket. Workers update their entry as counters change; the CLI reads and aggregates entries to surface drain progress (issue #182). Last-write-wins on the WorkerID key.

type WorkerStatusSnapshot struct {
    WorkerID              string `json:"worker_id"`
    CancelledTasksSkipped uint64 `json:"cancelled_tasks_skipped"`
}

Generated by gomarkdoc