Notifications

A scheduler that keeps firing a broken function fails quietly until someone looks. Notifications tell you instead. By default, every project emails its owner once a scheduler's runs have failed 3 times in a row, and again when a run succeeds. Add a webhook to route the same alerts into your own tooling.

Check the current settings

curl "https://api.volcano.dev/projects/$PROJECT_ID/notifications" \
  -H "Authorization: Bearer $VOLCANO_TOKEN"

A project that never saved settings returns the defaults:

{
  "project_id": "7f2b3c1e-0d4a-4c8e-9a51-2f6d8b0e1c34",
  "email": { "enabled": true, "recipients": [], "status": "delivering" },
  "webhook": { "secret_configured": false },
  "scheduler_failures": {
    "notify_on": "consecutive_failures",
    "threshold": 3,
    "notify_on_recovery": true
  }
}

Alerts are emailed only to the project owner's verified account email address, so recipients stays empty. email.status says whether email is delivering:

While the owner's address is unverified, email does not count as a channel: a project with no webhook sends no alerts, and a failing streak is announced on the first failure after the owner verifies. Changing the account email address needs a new verification, and alerts are not emailed in between.

Add a webhook

curl -X PUT "https://api.volcano.dev/projects/$PROJECT_ID/notifications" \
  -H "Authorization: Bearer $VOLCANO_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "webhook": { "url": "https://ops.example.com/hooks/volcano" } }'

Adding the first webhook generates its signing secret. The response is the only place it appears:

{
  "project_id": "7f2b3c1e-0d4a-4c8e-9a51-2f6d8b0e1c34",
  "email": { "enabled": true, "recipients": [], "status": "delivering" },
  "webhook": {
    "url": "https://ops.example.com/hooks/volcano",
    "secret_configured": true,
    "secret": "whsec_0000000000000000000000000000000000000000000"
  },
  "scheduler_failures": {
    "notify_on": "consecutive_failures",
    "threshold": 3,
    "notify_on_recovery": true
  },
  "updated_at": "2026-10-19T15:04:05Z"
}

Store the secret where your receiver can read it. Every later response reports only secret_configured. If you lose it, rotate it.

Omitted fields keep their values, so a PUT changes only what it names. Changing the URL keeps the secret, and "url": "" removes the webhook and its secret. The URL must be http or https and reachable on the public internet: a URL that names a private or loopback address is refused when it is saved, and a delivery to a host name that resolves to one fails. Put a routing token in the path or query string if your receiver needs one. Credentials in the URL are refused.

Choose when to alert

curl -X PUT "https://api.volcano.dev/projects/$PROJECT_ID/notifications" \
  -H "Authorization: Bearer $VOLCANO_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "scheduler_failures": { "notify_on": "first_failure" } }'

Any other address is refused, as is any address while the owner's is unverified:

{
  "error": "email.recipients: only the project owner's verified account email address is supported; leave recipients empty to email the owner"
}

To send alerts somewhere else, such as a team channel or an on-call tool, add a webhook.

Either way, a failing streak is announced once. Further failures in the same streak send nothing more, and the streak ends at the next successful run. A scheduler is announced failing at most once an hour: a streak that meets the policy sooner after the last alert is announced by its first failure once the hour is up, so a scheduler that alternates between failing and succeeding does not flood you. Recoveries are never held back. The settings apply to every scheduler in the project.

A full-scope project access token can read and change the settings. A read-only token is refused both with 403, because the webhook URL can carry your receiver's token. For the same reason, config export leaves the webhook out.

What counts as a failed run

• Durable functions. A run fails when the execution it started fails, times out, or is stopped. The run finishes when its execution does, so an alert can arrive some time after the tick that started it.
• Standard functions. A run fails when Volcano cannot invoke the function, for example because it is not deployed in the scheduler's region. An error the function returns or throws from its handler does not count. Volcano retries the tick every minute and each attempt is a failed run, so a function missing from its region meets the default threshold within about three minutes.
• Both. A tick that Volcano cannot deliver, such as during an outage, is retried every minute. Its attempts count as failed runs once 15 minutes have passed since its first attempt. A durable function that is being redeployed, a rate limit or a spent plan allowance, a function whose deployment leaves it unable to start, and a standard function that runs past the invoke timeout reach the same point. A tick Volcano refuses for a reason of its own never counts, however long it lasts.

A scheduler runs in one region at a time. Moving it to another region starts a new count of failed runs, but an announced streak stays open: the first successful run in the new region sends the recovery. Runs still finishing in the region it left are recorded there and send nothing. See Scheduler health for how the same state shows in the schedulers API.

A disabled scheduler keeps the health it had when it stopped, apart from durable runs already started then: they still count when their executions finish, and can still send an alert. A scheduler that does not fire, such as one on a HOBBY project, records no runs and sends no alerts.

Send a test

curl -X POST "https://api.volcano.dev/projects/$PROJECT_ID/notifications/test" \
  -H "Authorization: Bearer $VOLCANO_TOKEN"

The test goes out on every channel that delivers right away, signed like an alert, and the response reports each delivery:

{
  "deliveries": [
    { "channel": "email", "delivered": true },
    { "channel": "webhook", "delivered": false, "status_code": 500, "error": "webhook responded 500" }
  ]
}

A failed test delivery is not retried. One test per project per minute: another within the minute answers 429. With email on but the owner's address unverified, the email entry reports delivered: false and nothing is emailed. With no webhook and email off or unverified there is nowhere to send, and the request answers 409.

Webhook requests

Each alert is a POST with a JSON body and these headers:

A scheduler.failing event:

{
  "id": "1c0f8a4e-7f61-5b8d-9a3e-2f4b6d8c0e1a",
  "type": "scheduler.failing",
  "created_at": "2026-10-19T15:00:04Z",
  "project": { "id": "7f2b3c1e-0d4a-4c8e-9a51-2f6d8b0e1c34", "name": "my-app" },
  "function": { "id": "6f1c0f6e-6b0e-4a1d-9f8a-2c3d4e5f6a7b", "name": "hourly-sync", "kind": "durable" },
  "scheduler": {
    "id": "2c9d1e0f-3a4b-4c5d-8e9f-0a1b2c3d4e5f",
    "name": "hourly-sync",
    "health": "failing",
    "consecutive_failures": 3,
    "last_success_at": "2026-10-19T11:00:05Z",
    "last_failure_at": "2026-10-19T15:00:04Z"
  },
  "run": {
    "id": "9b8a7c6d-5e4f-4a3b-8c2d-1e0f9a8b7c6d",
    "region": "us-east-1",
    "status": "failed",
    "started_at": "2026-10-19T15:00:00Z",
    "finished_at": "2026-10-19T15:00:04Z",
    "error": { "type": "TypeError", "message": "Cannot read properties of undefined (reading 'rows')" }
  }
}

A scheduler.recovered event has the same shape, with scheduler.health set to healthy, consecutive_failures set to 0, and the successful run, which has no error. A notification.test event carries only id, type, created_at, and project.

last_success_at is absent until Volcano records a successful run, including for a scheduler whose last success came before scheduler health was tracked. error.type is absent when the failure has no type.

Verify the signature

Compute the HMAC over the raw request body, before parsing it. The key is the whole secret, including its whsec_ prefix. Reject a request whose timestamp is too old so that a captured request cannot be replayed.

import crypto from 'node:crypto';
import http from 'node:http';

const SECRET = process.env.VOLCANO_WEBHOOK_SECRET;
const TOLERANCE_SECONDS = 5 * 60;

function verify(body, headers) {
  const timestamp = headers['volcano-webhook-timestamp'];
  const signature = headers['volcano-webhook-signature'];
  if (!timestamp || !signature) return false;
  if (Math.abs(Date.now() / 1000 - Number(timestamp)) > TOLERANCE_SECONDS) return false;

  const expected = 'v1=' + crypto.createHmac('sha256', SECRET).update(`${timestamp}.`).update(body).digest('hex');
  const received = Buffer.from(signature);
  return received.length === expected.length && crypto.timingSafeEqual(received, Buffer.from(expected));
}

http
  .createServer((req, res) => {
    const chunks = [];
    req.on('data', (chunk) => chunks.push(chunk));
    req.on('end', () => {
      const body = Buffer.concat(chunks);
      if (!verify(body, req.headers)) {
        res.writeHead(401).end();
        return;
      }
      const event = JSON.parse(body);
      console.log(event.type, event.scheduler?.name, event.run?.error?.message);
      res.writeHead(204).end();
    });
  })
  .listen(3000);

Delivery and retries

Answer with any 2xx status within 10 seconds. Any other status, a timeout, or a connection error is retried with a backoff that starts at 30 seconds and doubles up to 15 minutes, for 8 attempts over about 45 minutes. Redirects are not followed, and a redirect fails the delivery without a retry.

Deliveries are at least once, so a receiver can see one event more than once. Deduplicate on the body's id, after verifying the signature. Volcano-Webhook-Id carries the same value, but the signature does not cover headers, so a replayed request could change it. A retried alert can arrive after a later one for the same scheduler, such as scheduler.failing after scheduler.recovered; order a scheduler's events by created_at.

Email retries the same way, and each attempt goes to the owner's current account address. An email fails without a retry when the owner has not verified that address, or when the mail server refuses it for good, such as an unknown mailbox. In rare cases, such as a restart in the middle of an attempt, the same email can arrive twice.

Each attempt reads the current settings. Removing the webhook drops alerts still waiting to be delivered to it, turning email off drops queued emails, and after a rotation, retries of earlier alerts are signed with the new secret.

Rotate the signing secret

curl -X POST "https://api.volcano.dev/projects/$PROJECT_ID/notifications/webhook-secret" \
  -H "Authorization: Bearer $VOLCANO_TOKEN"

The response carries the new secret in webhook.secret, this once. The old secret stops working immediately, so update your receiver right after rotating. Without a webhook the request answers 409.

Manage notifications from the CLI and the manifest

The CLI wraps these endpoints in volcano cloud notifications get, set, rotate-webhook-secret, and test. See Notifications in the CLI.

The configuration manifest takes a notifications section with the same fields. The signing secret is never part of it.

Local development

In local development, the same settings and endpoints work. Email goes to the local inbox at http://localhost:8025, and the webhook URL may name a private address. The local server runs in a Docker container, where localhost is the server itself, so point the webhook at a receiver on your machine through host.docker.internal (Docker Desktop) or your machine's network address.

Related

• Scheduled invocations
• Durable functions
• Function Schedulers API
• Notifications API