Retry Policies

A retry policy defines how DagNats handles transient step failures – how many times to retry, how long to wait between attempts, and which backoff strategy to use.

RetryPolicy Configuration

Field	JSON Key	Type	Description
MaxAttempts	`max_attempts`	`int`	Number of retries after the first attempt, so `3` allows up to 4 attempts in all; 0 means no retries
Strategy	`strategy`	`RetryStrategy`	Backoff algorithm: `fixed`, `linear`, or `exponential`
InitialDelay	`initial_delay`	`duration`	Base delay between attempts
MaxDelay	`max_delay`	`duration`	Delay cap (0 = uncapped)
Multiplier	`multiplier`	`float64`	Multiplier for `exponential` strategy only

Backoff Strategies

DagNats supports three backoff strategies, each computing the delay before the next retry attempt (1-based):

Strategy	Formula	Example (initial: 2s)
`fixed`	`initial_delay`	2s, 2s, 2s, 2s
`linear`	`initial_delay * attempt`	2s, 4s, 6s, 8s
`exponential`	`initial_delay * multiplier^(attempt-1)`	2s, 4s, 8s, 16s

All strategies respect MaxDelay – if the computed delay exceeds the cap, MaxDelay is used instead.

Setting Retry Policies

Per-Step

The builder has no method for a full policy, so set the Retry field on the built step. WithRetries(n) on a step is the legacy shortcut: n retries at a fixed 5-second delay.

wf := dag.NewWorkflow("llm-pipeline")
wf.Task("call-llm", "llm.chat").WithTimeout(60 * time.Second)

def, err := wf.Build()
if err != nil {
    return err
}
def.Steps[0].Retry = &dag.RetryPolicy{
    MaxAttempts:  5,
    Strategy:     dag.RetryExponential,
    InitialDelay: 2 * time.Second,
    MaxDelay:     30 * time.Second,
    Multiplier:   2.0,
}

In a JSON workflow definition the same policy is the step’s retry object, with durations as strings:

{
  "id": "call-llm",
  "task": "llm.chat",
  "timeout": "60s",
  "retry": {
    "max_attempts": 5,
    "strategy": "exponential",
    "initial_delay": "2s",
    "max_delay": "30s",
    "multiplier": 2.0
  }
}

Workflow Default

Set DefaultRetry on the definition (default_retry in JSON) to apply a policy to all steps unless overridden:

def.DefaultRetry = &dag.RetryPolicy{
    MaxAttempts:  3,
    Strategy:     dag.RetryFixed,
    InitialDelay: 5 * time.Second,
    MaxDelay:     5 * time.Second,
}

Resolution Order

The engine resolves the effective retry policy for each step using this precedence:

Step-level Retry field (highest priority)
Workflow-level DefaultRetry
Legacy Retries field (converted to fixed-delay policy with 5s delay)
No retries (if none of the above are set)

NATS Implementation

The engine owns retries. The worker acks the failed task and reports step.failed; the engine computes the delay and schedules a durable retry_backoff timer on the SLEEP_TIMERS stream (itself a NakWithDelay). When the timer fires, the engine dispatches a fresh task for the next attempt, which gets the full step timeout. There is no separate timer service or retry queue.

    sequenceDiagram
    participant Worker
    participant NATS as TASK_QUEUES
    participant Engine

    participant Timers as SLEEP_TIMERS

    Worker->>Engine: step.failed (retriable)
    Engine->>Engine: calculate delay (attempt 2)
    Engine->>Timers: schedule retry_backoff (4s)
    Note over Timers: waits 4 seconds
    Timers->>Engine: timer fires
    Engine->>NATS: dispatch attempt 2
    NATS->>Worker: deliver task

Non-Retryable Errors

Workers can signal that a failure should not be retried by calling FailPermanent():

w.Handle("validate", func(ctx worker.TaskContext) error {
    input := ctx.Input()
    if !isValid(input) {
        return ctx.FailPermanent(fmt.Errorf("invalid input: %s", input))
    }
    return ctx.Complete([]byte(`{"valid":true}`))
})

FailPermanent() sets the failure type to non_retriable, causing the engine to skip all remaining retries and immediately proceed to on-failure handling or workflow failure.

See Error Handling for the full FailureType taxonomy.

Retrying transient LLM API failures. LLM providers frequently return 429 (rate limit) and 503 (overloaded) errors. Use exponential backoff with a generous max_delay (30-60s) and 5+ max attempts to ride out transient API unavailability without manual intervention.

Error Handling – failure types and on-failure handlers
Timeouts – per-step and per-workflow deadlines
Rate Limiting – preventing upstream overload

Timeouts