Retries

When a Step or a handler fails, Runcoil tries again after a delay. You choose how many tries and how long the delays are.

Default policy

The server applies a default policy to every Task.

Setting	Default	Server variable
Tries in total, the first one included	4	ENGINE__RETRY__MAX_ATTEMPTS
Delay before each retry	5s,10s,30s,1m,3m,10m,30m,1h	ENGINE__RETRY__DELAYS
Jitter	0.2	ENGINE__RETRY__JITTER

The delay for the first retry is the first value, for the second retry the second value, and so on. The last value repeats. With the defaults, a Step is tried four times and waits about 5 seconds, then 10 seconds, then 30 seconds. Each delay varies by up to 20% either way.

Set a policy for a Task

Pass runcoil.Retry when you add the Task. The policy applies to every Step of the Task. This code is from the onboarding example:

worker.NewTask("create-account", runcoil.HandlerFunc(handler.createAccount),
	runcoil.Retry(runcoil.RetryPolicy{MaxAttempts: 3, Delays: []time.Duration{time.Second}}),
)

MaxAttempts counts the first try. Delays is the wait before each retry, and its last value repeats. A field left at zero takes the default. Each delay must be greater than zero.

A failure outside any Step, such as an error that your handler returns, also uses the policy of the Task.

Set a policy for a Step

Pass step.Retry to step.Do to override the policy for one Step. It takes the same RetryPolicy.

messageID, err := step.Do(ctx, "send-welcome-email", func(ctx context.Context) (string, error) {
	return o.emails.SendWelcome(ctx, created)
}, step.Retry(step.RetryPolicy{MaxAttempts: 5}))

Each Step counts its own tries. When a Step has used all of its tries, it has failed for good, and step.Do returns a *step.Error.

Fail at once

A plain error is retryable. To stop retrying, wrap the error in runcoil.Permanent. A Step whose function returns it fails at once. A handler that returns it fails the attempt without a retry.

This code is from the onboarding example. A request with missing fields cannot succeed on a second try:

if strings.TrimSpace(in.AccountID) == "" || strings.TrimSpace(in.Name) == "" || strings.TrimSpace(in.Email) == "" {
	return onboardingOutput{}, runcoil.Permanent(errors.New("account_id, name, and email are required"))
}

A Step function that panics also fails at once. So does an output that cannot be encoded to JSON.

Choose the next delay

To replace the policy's delay for the next try, wrap the error in runcoil.RetryAfter. A rate-limited API often tells you how long to wait.

return "", runcoil.RetryAfter(err, 30*time.Second)

A failure marked this way still counts as a try.

When a worker dies

A worker renews a lease on each Run it holds. By default it sends a heartbeat every 10 seconds, and a lease lasts 30 seconds. If the worker dies, the lease expires and the server takes the attempt back.

The server counts the lost attempt like a failed try. If tries are left, the Run waits for its next attempt, and another worker runs it. Steps that were recorded before the worker died return their stored results. If no tries are left, the Run fails with the code attempt_lost.

If a worker never acknowledged the dispatch, the server gives the attempt back and counts no try.

After a Step fails for good

When a Step has used all of its tries, step.Do returns a *step.Error. A handler that returns it fails the Run. The onboarding example does this when the welcome email fails for good:

messageID, err := step.Do(ctx, "send-welcome-email", func(ctx context.Context) (string, error) {
	return o.emails.SendWelcome(ctx, created)
})
if err != nil {
	// Keep the account when delivery fails finally; fail the Run.
	return onboardingOutput{}, err
}

The Run fails and the account stays created, because the handler does not undo the first Step. If you want a fallback or an undo instead, check the error with errors.As for *step.Error and run more Steps before you return. Each of those Steps is recorded like any other.

Quick reference

You want to	Use
Change the tries or delays of every Step in a Task	runcoil.Retry on NewTask
Change them for one Step	step.Retry on step.Do
Stop retrying	runcoil.Permanent(err)
Pick the next delay	runcoil.RetryAfter(err, d)