$ aoox

DocsInfrastructure

Notifications

Send word of deploys, backups, jobs, dead containers, low disk, failed certificates, and DNS issues to Telegram, Slack, Discord, a webhook, or email.

Channels
Telegram · Slack · Discord · Webhook · Email
Events
12 toggles per channel
Scope
Platform-wide, owner/admin

Flow

  1. 01eventdeploy, backup, job, container down, disk, certificate, DNS, log errors
  2. 02filterchannels with that event's toggle enabled
  3. 03formata per-platform payload: HTML, attachments, embeds, JSON, email
  4. 04sendfetch with a 10-second timeout; failures are only logged

Supported channels

TypeWhat to fill inHow to get it
TelegramBot token + Chat IDCreate a bot via @BotFather (token like 123456789:AAH…). Chat ID: message the bot, then open https://api.telegram.org/bot<token>/getUpdates; groups/channels usually start with -100…. The bot must already be in that group.
SlackWebhook URLSlack app → Incoming Webhooks → Add New Webhook to Workspace → pick a channel.
DiscordWebhook URLChannel → Edit → Integrations → Webhooks → New Webhook → Copy URL.
WebhookA webhook URL (https://…) + an optional SecretAny endpoint that accepts a JSON POST — n8n, Zapier, your own app. The secret enables HMAC signing.
EmailSMTP host, port, username, password, sender, recipientsRecipients are comma-separated. Port 465 = implicit TLS, 587 = STARTTLS. For Gmail, use an app password.

Adding a channel

  1. Integrations → Notifications → New notification channel (owner/admin)

    Fill in a Name, pick a Type, fill in its fields, and enable the toggles for the events you want. The token/URL/password is stored encrypted and never shown again — the channel list only shows a hint of the target (e.g. chat id, domain).

  2. Send a test

    The test button sends a sample message to that channel. Provider errors are shown as-is — e.g. Telegram's chat not found (the bot isn't in the group / wrong chat id) or SMTP's 535 Authentication failed.

  3. Trigger a real event

    Deploy once, or run a job that's meant to fail (exit 1) to make sure the toggle works.

Events

ToggleFires whenMessage contents
Deploy started (off by default)A new application deployment starts (build, webhook, auto-update, or rollback — not a runtime-only change like a replica/limit update).Application, project, trigger (manual/webhook/auto-update, plus the commit & pusher for webhook), a link to the application page.
Deploy successAn application deployment ends in success (including rollback).Application, project, image, duration, a link to the application page.
Deploy failureAn application or compose stack deployment fails.Same, plus a snippet of the error message (500 characters).
Backup failureA database/volume backup (manual or scheduled) fails, including a failed S3 upload.Database/application, trigger (manual/scheduled), error.
Job failureA scheduled job ends up failed or timeout.Job name, command (200 characters), the last 500 characters of output.
Container downAn application/database container stops unexpectedly — see the rules under Monitoring. Also applies to applications on a remote server.Container name, exit code, whether it's restarting; the server name for a remote server.
Disk almost fullDocker filesystem usage crosses a threshold (default 90%, DISK_ALERT_PERCENT); checked once a day.Percentage used, threshold, space remaining.
Certificate failureTraefik fails to issue/renew a domain's certificate (checked every 10 minutes).The domain and the ACME error message.
DNS domain issueA domain's DNS no longer points at the right IP — checked every 15 minutes, fires after two consecutive failed checks (mismatch/unresolved).Domain, status (mismatch/unresolved), an explanatory message.
Application error (off by default)A running application's container log matches a common error pattern (traceback, panic, exception, level=error, etc — checked every minute).Application, project, error count, up to 3 example lines (redacted), a link to the application page.
Remote server unreachable (on by default)A remote server stops answering (SSH/Docker) on two consecutive checks (checked every minute). One recovery message is sent when it is reachable again; reminders at most once an hour while down.Server name, address, error; the downtime on the recovery message.
HTTP monitor failing (on by default)An application&apos;s HTTP check fails N times in a row (an optional per-application monitor, see Monitoring). One recovery message when healthy again; reminders at most every 6 hours while down.Application, project, target, status code/error, how many checks failed; the downtime on the recovery message; a link to the application page.
CPU or memory alert (on by default)An application or database keeps CPU or memory above a threshold you set for a set time (an optional per-resource rule, off until enabled; see Monitoring). One message when it starts, a reminder every 6 hours while it stays high, and one “back to normal” message.Application or database, project, CPU or memory, the current value, threshold and duration; a link to the application page.
  • Deploy started is off by default (unlike the other events) — deploys can happen often (auto-update, webhooks), so a channel has to opt in if it actually wants to know. It also never fires for a kind: config deployment (a swarm mode/ replica/resource-limit change on the same image) — that isn't a new deploy worth watching for.
  • A link to the application page is included when WEB_ORIGIN is set.
  • A failure to deliver to one channel is only logged by the API and doesn't affect other channels or the process that triggered it (a deploy never fails because of a notification).
  • Container down: at most 1 notification per container every 10 minutes.
  • Remote server unreachable and HTTP monitor failing are on by default, but only do anything when you use the feature (adding a remote server / turning on a monitor for an application), so they are not noisy for anyone who doesn't. Each uses a single toggle for the failure and the recovery message (told apart by the level and by event in the webhook payload: server.down/server.recovered, http.down/http.recovered).
  • DNS domain issue: two consecutive failed checks (to dampen transient DNS blips) before the first notification, then at most once a day per domain while the status stays bad. A domain with nothing to compare against (no expected IP) never sends.
  • Application error is off by default — detection from plain-text logs is prone to false positives. At most 1 notification per application every 15 minutes, except an error shape never seen since the API last started, which is always sent right away. Can be disabled per application with the Ignore error logs switch on the application's Settings tab — see Applications.

Generic webhook payload

The Webhook channel sends a POST with Content-Type: application/json. Common fields: title, level (success / failure / info), url, plus event-specific fields:

deployment.started
{
  "title": "Deployment started: shop",
  "level": "info",
  "url": "https://panel.example.com/applications/…",
  "event": "deployment.started",
  "deploymentId": "…",
  "kind": "build",
  "applicationId": "…",
  "application": "shop",
  "project": "Toko",
  "trigger": "webhook",
  "commitSha": "a1b2c3d4e5f6",
  "commitMessage": "fix: checkout bug",
  "triggeredBy": "octocat"
}
deployment.success / deployment.failure
{
  "title": "Deployment succeeded: shop",
  "level": "success",
  "url": "https://panel.example.com/applications/…",
  "event": "deployment.success",
  "deploymentId": "…",
  "kind": "deploy",
  "applicationId": "…",
  "application": "shop",
  "imageRef": "localhost:5000/toko/shop:a1b2c3d4e5f6",
  "error": null,
  "finishedAt": "2026-09-22T04:30:00.000Z"
}
container.down
{
  "title": "Container down: aoox-app-shop",
  "level": "failure",
  "event": "container.down",
  "container": "aoox-app-shop",
  "exitCode": 137,
  "restarting": true
}
app.error
{
  "title": "Application error detected: shop",
  "level": "failure",
  "url": "https://panel.example.com/applications/…",
  "event": "app.error",
  "applicationId": "…",
  "count": 2,
  "fingerprints": ["TypeError: Cannot read <n>", "FATAL: <n>"]
}

Example receiver (Node.js)

app.post("/hooks/aoox", express.json(), (req, res) => {
  const { event, level, title } = req.body
  if (event === "deployment.failure") alertOnCall(title)
  res.sendStatus(204)
})

Headers & signature

HeaderContents
X-Aoox-EventThe event name, e.g. deployment.success.
X-Aoox-DeliveryA unique delivery ID — use it for idempotency.
X-Aoox-SignatureOnly when a Secret is set: sha256=HMAC-SHA256(secret, raw body), the same scheme GitHub uses.
Verifying on the receiver (Node.js)
const sig = req.get("X-Aoox-Signature")
const mine = "sha256=" + createHmac("sha256", SECRET).update(rawBody).digest("hex")
if (!sig || !timingSafeEqual(Buffer.from(sig), Buffer.from(mine))) return res.sendStatus(401)
Compute the HMAC over the raw request body bytes, not a re-serialized JSON.stringify. Without a secret, secure the endpoint with a hard-to-guess URL instead. Outgoing webhooks can reach internal addresses (SSRF by design) — that's why only owner/admin can add a channel.

Common pitfalls

SymptomCause & fix
The test succeeds but the real event never arrivesThe event's toggle isn't enabled on that channel, or the event genuinely didn't happen (e.g. a dead container on a remote server isn't detected).
Telegram: chat not found / bot was blockedThe bot hasn't been added to the group, the chat id is missing the -100 prefix for a supergroup, or the user hasn't hit /start.
Discord/Slack 4xxThe webhook URL has expired or was deleted on the provider. Create a new one and update the channel.
Email doesn't arrive, test succeedsIt landed in spam, or the sender isn't authorized for the SMTP domain (SPF/DKIM). Use a sender address on the SMTP domain.
TimeoutThe endpoint is slow (> 10 seconds) or unreachable from the API container. Reply fast (204) and process asynchronously.
Not available yet
Per-project or per-application notifications (channels are platform-wide); events for compose stacks beyond deploy failure; a daily digest.

Next steps

Found a mistake? Edit this page on GitHub ↗