Notifications

What it is

Notifications tell you when a scheduled job stops working. When a function scheduler's runs keep failing, Volcano alerts the project by email, by a signed webhook, or both, and again when the scheduler recovers.

A new project emails its owner after 3 failed scheduled runs in a row, with no further setup. Alert emails go only to the owner's verified account email address; to alert anyone else, add a webhook.

How it relates

• Belongs to a project. The settings cover every scheduler in it, for standard and durable functions.
• Driven by schedulers. For a durable function, a run fails when the execution it started fails, times out, or is stopped. For a standard function, a run fails when Volcano cannot invoke it; an error the function itself returns does not count. schedulers list shows each scheduler's health and how many runs in a row have failed.
• Declared in the project configuration under notifications:, except the webhook signing secret, which is never written to the manifest. See project-configuration.md.

CLI operations

Drop cloud to run the same commands against local development. Locally, a webhook may name a private address. The local server runs in a Docker container, where localhost is the server itself, so point the webhook at a receiver on your machine through host.docker.internal (Docker Desktop) or your machine's network address.

Show the settings

volcano cloud notifications get

Email: on, to the project owner
Webhook: not configured
Alert on: 3 failed scheduled runs in a row
Recovery: notify when a failing scheduler succeeds again

Until the owner verifies their account email address, email delivers nothing and the settings say so:

Email: on, to the project owner (not delivering: owner email unverified)

Email then does not count as a channel: without a webhook, no alert is sent. Changing the account email address needs a new verification. In --json output, email.status is delivering, disabled, or owner_email_unverified.

Add a webhook

volcano cloud notifications set --webhook-url https://example.com/hooks/volcano

✓ Notification settings saved
Email: on, to the project owner
Webhook: https://example.com/hooks/volcano
Signing secret: configured
Alert on: 3 failed scheduled runs in a row
Recovery: notify when a failing scheduler succeeds again

Signing secret: whsec_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Warning: Copy this secret now. It is shown once and cannot be retrieved again.

The first webhook URL generates a signing secret, printed only by this command. Changing the URL later keeps the secret. Use --json to capture it in a script. See Verify a webhook for the payload and how to check its signature.

Outside local development, the URL must reach the public internet. A URL naming a private or loopback address is refused.

Change when alerts are sent

volcano cloud notifications set --notify-on first_failure
volcano cloud notifications set --notify-on consecutive_failures --threshold 5
volcano cloud notifications set --notify-on-recovery=false

Flags you leave out keep their current values. A failing streak is announced once, however long it lasts, and a scheduler is announced failing at most once an hour.

Rotate the signing secret

volcano cloud notifications rotate-webhook-secret --yes

Every delivery from then on, including retries of earlier alerts, is signed with the new secret, so update your receiver right away. Without --yes the command asks first.

Send a test

volcano cloud notifications test

✓ Test email delivered
✓ Test webhook delivered (HTTP 200)

The test goes to every configured channel and reports whether each one arrived. Email to an owner without a verified address is reported as not delivered and is not sent. The command fails when one did not:

✓ Test email delivered
Error: test webhook not delivered (HTTP 500): webhook responded 500

One test per project per minute.

See which schedulers are failing

volcano cloud functions schedulers list nightly-export
volcano cloud durable schedulers list order-pipeline

ID                                    Name                      State      Health    Fails  Cron             Next Run              Last Run
---------------------------------------------------------------------------------------------------------------------------------------------------
4f6c1d2e-8a3b-4c5d-9e7f-0a1b2c3d4e5f  nightly-export            enabled    failing   49     0 * * * *        2026-10-19T15:00:00Z  12m ago
  last error: Runtime.ImportModuleError: Cannot find module 'pg'

Fails counts the failed runs in a row, and resets when a run succeeds. Pausing a scheduler keeps its count.