DocsInfrastructure
Notifications
Send word of deploys, backups, jobs, dead containers, low disk, failed certificates, and DNS issues to Telegram, Slack, Discord, a webhook, or email.
- Channels
- Telegram · Slack · Discord · Webhook · Email
- Events
- 12 toggles per channel
- Scope
- Platform-wide, owner/admin
Flow
- 01eventdeploy, backup, job, container down, disk, certificate, DNS, log errors
- 02filterchannels with that event's toggle enabled
- 03formata per-platform payload: HTML, attachments, embeds, JSON, email
- 04sendfetch with a 10-second timeout; failures are only logged
Supported channels
| Type | What to fill in | How to get it |
|---|---|---|
| Telegram | Bot token + Chat ID | Create a bot via @BotFather (token like 123456789:AAH…). Chat ID: message the bot, then open https://api.telegram.org/bot<token>/getUpdates; groups/channels usually start with -100…. The bot must already be in that group. |
| Slack | Webhook URL | Slack app → Incoming Webhooks → Add New Webhook to Workspace → pick a channel. |
| Discord | Webhook URL | Channel → Edit → Integrations → Webhooks → New Webhook → Copy URL. |
| Webhook | A webhook URL (https://…) + an optional Secret | Any endpoint that accepts a JSON POST — n8n, Zapier, your own app. The secret enables HMAC signing. |
| SMTP host, port, username, password, sender, recipients | Recipients are comma-separated. Port 465 = implicit TLS, 587 = STARTTLS. For Gmail, use an app password. |
Adding a channel
Integrations → Notifications → New notification channel (owner/admin)
Fill in a Name, pick a Type, fill in its fields, and enable the toggles for the events you want. The token/URL/password is stored encrypted and never shown again — the channel list only shows a hint of the target (e.g. chat id, domain).
Send a test
The test button sends a sample message to that channel. Provider errors are shown as-is — e.g. Telegram's
chat not found(the bot isn't in the group / wrong chat id) or SMTP's535 Authentication failed.Trigger a real event
Deploy once, or run a job that's meant to fail (
exit 1) to make sure the toggle works.
Events
| Toggle | Fires when | Message contents |
|---|---|---|
Deploy started (off by default) | A new application deployment starts (build, webhook, auto-update, or rollback — not a runtime-only change like a replica/limit update). | Application, project, trigger (manual/webhook/auto-update, plus the commit & pusher for webhook), a link to the application page. |
| Deploy success | An application deployment ends in success (including rollback). | Application, project, image, duration, a link to the application page. |
| Deploy failure | An application or compose stack deployment fails. | Same, plus a snippet of the error message (500 characters). |
| Backup failure | A database/volume backup (manual or scheduled) fails, including a failed S3 upload. | Database/application, trigger (manual/scheduled), error. |
| Job failure | A scheduled job ends up failed or timeout. | Job name, command (200 characters), the last 500 characters of output. |
| Container down | An application/database container stops unexpectedly — see the rules under Monitoring. Also applies to applications on a remote server. | Container name, exit code, whether it's restarting; the server name for a remote server. |
| Disk almost full | Docker filesystem usage crosses a threshold (default 90%, DISK_ALERT_PERCENT); checked once a day. | Percentage used, threshold, space remaining. |
| Certificate failure | Traefik fails to issue/renew a domain's certificate (checked every 10 minutes). | The domain and the ACME error message. |
| DNS domain issue | A domain's DNS no longer points at the right IP — checked every 15 minutes, fires after two consecutive failed checks (mismatch/unresolved). | Domain, status (mismatch/unresolved), an explanatory message. |
Application error (off by default) | A running application's container log matches a common error pattern (traceback, panic, exception, level=error, etc — checked every minute). | Application, project, error count, up to 3 example lines (redacted), a link to the application page. |
Remote server unreachable (on by default) | A remote server stops answering (SSH/Docker) on two consecutive checks (checked every minute). One recovery message is sent when it is reachable again; reminders at most once an hour while down. | Server name, address, error; the downtime on the recovery message. |
HTTP monitor failing (on by default) | An application's HTTP check fails N times in a row (an optional per-application monitor, see Monitoring). One recovery message when healthy again; reminders at most every 6 hours while down. | Application, project, target, status code/error, how many checks failed; the downtime on the recovery message; a link to the application page. |
CPU or memory alert (on by default) | An application or database keeps CPU or memory above a threshold you set for a set time (an optional per-resource rule, off until enabled; see Monitoring). One message when it starts, a reminder every 6 hours while it stays high, and one “back to normal” message. | Application or database, project, CPU or memory, the current value, threshold and duration; a link to the application page. |
- Deploy started is off by default (unlike the other events) — deploys can happen often (auto-update, webhooks), so a channel has to opt in if it actually wants to know. It also never fires for a
kind: configdeployment (a swarm mode/ replica/resource-limit change on the same image) — that isn't a new deploy worth watching for. - A link to the application page is included when
WEB_ORIGINis set. - A failure to deliver to one channel is only logged by the API and doesn't affect other channels or the process that triggered it (a deploy never fails because of a notification).
- Container down: at most 1 notification per container every 10 minutes.
- Remote server unreachable and HTTP monitor failing are on by default, but only do anything when you use the feature (adding a remote server / turning on a monitor for an application), so they are not noisy for anyone who doesn't. Each uses a single toggle for the failure and the recovery message (told apart by the level and by
eventin the webhook payload:server.down/server.recovered,http.down/http.recovered). - DNS domain issue: two consecutive failed checks (to dampen transient DNS blips) before the first notification, then at most once a day per domain while the status stays bad. A domain with nothing to compare against (no expected IP) never sends.
- Application error is off by default — detection from plain-text logs is prone to false positives. At most 1 notification per application every 15 minutes, except an error shape never seen since the API last started, which is always sent right away. Can be disabled per application with the Ignore error logs switch on the application's Settings tab — see Applications.
Generic webhook payload
The Webhook channel sends a POST with Content-Type: application/json. Common fields: title, level (success / failure / info), url, plus event-specific fields:
{
"title": "Deployment started: shop",
"level": "info",
"url": "https://panel.example.com/applications/…",
"event": "deployment.started",
"deploymentId": "…",
"kind": "build",
"applicationId": "…",
"application": "shop",
"project": "Toko",
"trigger": "webhook",
"commitSha": "a1b2c3d4e5f6",
"commitMessage": "fix: checkout bug",
"triggeredBy": "octocat"
}{
"title": "Deployment succeeded: shop",
"level": "success",
"url": "https://panel.example.com/applications/…",
"event": "deployment.success",
"deploymentId": "…",
"kind": "deploy",
"applicationId": "…",
"application": "shop",
"imageRef": "localhost:5000/toko/shop:a1b2c3d4e5f6",
"error": null,
"finishedAt": "2026-09-22T04:30:00.000Z"
}{
"title": "Container down: aoox-app-shop",
"level": "failure",
"event": "container.down",
"container": "aoox-app-shop",
"exitCode": 137,
"restarting": true
}{
"title": "Application error detected: shop",
"level": "failure",
"url": "https://panel.example.com/applications/…",
"event": "app.error",
"applicationId": "…",
"count": 2,
"fingerprints": ["TypeError: Cannot read <n>", "FATAL: <n>"]
}Example receiver (Node.js)
app.post("/hooks/aoox", express.json(), (req, res) => {
const { event, level, title } = req.body
if (event === "deployment.failure") alertOnCall(title)
res.sendStatus(204)
})Headers & signature
| Header | Contents |
|---|---|
X-Aoox-Event | The event name, e.g. deployment.success. |
X-Aoox-Delivery | A unique delivery ID — use it for idempotency. |
X-Aoox-Signature | Only when a Secret is set: sha256=HMAC-SHA256(secret, raw body), the same scheme GitHub uses. |
const sig = req.get("X-Aoox-Signature")
const mine = "sha256=" + createHmac("sha256", SECRET).update(rawBody).digest("hex")
if (!sig || !timingSafeEqual(Buffer.from(sig), Buffer.from(mine))) return res.sendStatus(401)JSON.stringify. Without a secret, secure the endpoint with a hard-to-guess URL instead. Outgoing webhooks can reach internal addresses (SSRF by design) — that's why only owner/admin can add a channel.Common pitfalls
| Symptom | Cause & fix |
|---|---|
| The test succeeds but the real event never arrives | The event's toggle isn't enabled on that channel, or the event genuinely didn't happen (e.g. a dead container on a remote server isn't detected). |
| Telegram: chat not found / bot was blocked | The bot hasn't been added to the group, the chat id is missing the -100 prefix for a supergroup, or the user hasn't hit /start. |
| Discord/Slack 4xx | The webhook URL has expired or was deleted on the provider. Create a new one and update the channel. |
| Email doesn't arrive, test succeeds | It landed in spam, or the sender isn't authorized for the SMTP domain (SPF/DKIM). Use a sender address on the SMTP domain. |
| Timeout | The endpoint is slow (> 10 seconds) or unreachable from the API container. Reply fast (204) and process asynchronously. |
Next steps
Found a mistake? Edit this page on GitHub ↗