$ aoox

DocsInfrastructure

Monitoring

Live host metrics, per-container metrics for applications/databases, disk usage, and resource limits.

Host
CPU, RAM, storage — every 2 seconds, 5-minute window
Container
CPU, memory, network — 1h live, 24h/7d/30d stored
Limits
CPU millicores & memory MiB per app/database

Where to find it

  1. 01DashboardLive host CPU/RAM/Storage + Host & Docker disk cards (owner/admin)
  2. 02Application → DeployCPU / memory / network KPIs + sparkline
  3. 03DatabaseThe same KPIs above the panel
  4. 04Docker → disk stripImage/volume/container/cache breakdown + cleanup (Details)

Dashboard: live host

For owner/admin, the Dashboard shows three charts of the last 5 minutes (sampled every 2 seconds, polled only while the tab is visible):

ChartSourceNotes
Host CPUIdle/total delta from os.cpus() — all coresThe caption also shows the share used by aoox containers (divided by core count).
Host RAMtotalmem − freememIncludes OS cache — a higher number than htop is expected.
Host storagestatfs(STORAGE_PATH), default /In Docker this is the host disk behind /var/lib/docker. Set STORAGE_PATH if Docker's data lives on another partition. Filesystem unreadable means the path doesn't exist in the container.

Below the charts is a Recent activity card (owner/admin only): the 10 latest audit log entries with the time, who did it, a readable action (e.g. “Deploy application”, “Delete database”), the status, and how it was done (session, API token, webhook). It refreshes every 30 seconds while the tab is visible; View all opens the Audit log page. Request contents are not shown on this card.

Inside a Docker container, /proc/stat and /proc/meminfo belong to the host, so the numbers reflect the host with no extra Docker call per tick. History is 150 points kept in the API's memory (lost on restart).

Host & Docker disk cards

  • Host: Docker version, running/total containers, and the CPU and memory used by aoox containers (from docker info + the sampler).
  • Docker disk: a stacked meter for image / volume / container / build cache (from docker system df) — not the whole host filesystem; see the Storage chart for that.
  • Project summary: how many projects have something running (an application, database, or stack).

The Docker page

Infrastructure → Docker (owner/admin) lists containers, images, volumes and networks on the host or on a remote server (pick the server above the tabs; the Docker item is in the Platform group of the sidebar). Anything aoox owns is badged and shown read-only: aoox containers (panel, proxy, registry, applications, databases, stacks) cannot be stopped or removed from here — use their own pages.

  • Containers: all of them, stopped ones included. For containers aoox does not own: logs (a one-time tail, up to 500 lines), start, stop, restart, and remove (owner only, with a confirmation).
  • Images: remove an image no container uses, or clean up dangling ones. Images in use are refused.
  • Volumes and Networks: lists only, with size, users and a marker for aoox's orphaned volumes.

Leftover containers. Projects and applications deleted on older versions of aoox left their containers running, with their Traefik labels, so a leftover could still answer on its old domain. The disk strip's Details dialog lists them (name, kind, state) when there are any, for containers on the aoox host whose project, application, database or stack no longer exists. The owner can remove one with its trash button, or tick Also remove leftover containers before Clean now (off by default, also in the nightly cleanup). The panel itself, the proxy, registry, build and job helpers and anything aoox cannot tie to a deleted resource are never listed. Volumes are not removed by this; use the orphan-volume option for them. Remote servers are not covered.

Per-container metrics

An application's Deploy tab and database pages have a CPU / Memory / Network KPI row with a sparkline: sampled every 15 seconds. The range picker above the chart decides where the data comes from:

RangeSourceResolution
1hLive samples in the API's memory (240 points)15 seconds
24hRollup stored in the database1 minute
7d / 30dRollup stored in the database1 hour
  • 15-second samples are rolled up into one row per minute per application/database (swarm tasks are summed), then averaged into hourly rows.
  • Minute rows are pruned after 48 hours; hourly rows after METRICS_RETENTION_DAYS days (default 30).
  • Because it's stored in the database, history survives an API restart — only the last 1-hour window is live.
MetricFormulaNotes
CPU %Δcpu_total / Δsystem × online_cpus × 100 — same as docker statsTwo stats calls one second apart per sample, since non-stream mode doesn't populate precpu.
Memoryusage − inactive_file (cgroup v2) / − cache (v1)100% = the container limit if set, otherwise host RAM (memoryLimitBytes).
NetworkCumulative rx/tx bytes across all interfacesShown as a rate between samples.
  • current: null when the container is stopped or has no sample yet (just started < 15 seconds ago).
  • Per-container disk I/O isn't available on Docker Desktop, so it isn't shown.
  • Containers labeled as an aoox component (app, database) are sampled via one stream; compose stack containers via a second stream identified by the compose project name (aoox-<slug>) — jobs and build helpers are still never sampled.
  • Applications on a remote server aren't sampled — the sampler only reads the aoox host's daemon.
  • For applications in service mode, the metric is the sum of tasks running on the host — see Docker Swarm.

Resource limits

Applications (Settings) and databases (the Resource limits card) have a CPU (millicores) and Memory (MiB) field. 1000 millicores = 1 core; empty = unlimited.

Example
CPU 500    → max half a core (NanoCpus 0.5e9)
Memory 512 → hard limit of 512 MiB, no swap (MemorySwap = Memory)
ActionEffect
Setting / changing a valueApplied straight to the running container (docker update), no restart.
Clearing itThe container is recreated — Docker can't remove a limit through update. Applications: recreated from the current image; databases: reprovisioned (data on the volume is safe).
  • A container that exceeds its memory limit gets OOM-killed by the kernel → restarted by Docker → a container-down notification if it keeps happening. Check the memory sparkline before lowering a limit.
  • PR previews and separate container jobs inherit their application's limit.
  • Compose stacks: set deploy.resources in the compose file.

Detecting a dead container

The API keeps a docker events stream (die) for aoox-labeled containers, plus a second stream for compose stack containers (identified by the project name). After a 5-second grace period it decides:

ConditionDecision
The container is already gone (replaced by a deploy / deleted)Ignored
The application/database status is stopped (an intentional stop)Ignored
Running again with exit code 0 (a clean restart)Ignored
Anything else (crash, OOM, exit ≠ 0)Send a containerDown notification, max once per container every 10 minutes

Enable it in Notifications → the Container down toggle. Applies to the local host and to applications on a remote server (the message names the server).

HTTP monitor (optional)

A running container isn't necessarily healthy: an application can return 5xx or hang without ever dying. Turn on an HTTP check per application on the Monitor tab (developers and up edit, viewers only read).

SettingNotes
PathDefault /, e.g. /health. The only thing you choose — it must start with a single /, no spaces/control characters, no .., at most 200 characters.
Interval1–60 minutes (default 5).
Timeout5–30 seconds (default 10).
Healthy statusDefault 200-399; can be a list of codes/ranges, e.g. 200,204.
Consecutive failuresHow many failures in a row before an alert (default 2).
Internal addressSkip the public domain and check through the host port / container name.

What is checked, and from where

  • You never type the host. The target is derived from the application itself: its first domain (the public URL through the proxy, so the proxy path is tested too); if there is none, its host port (host:port, the server's address for a remote server); if there is none either, the container's name on the aoox network (only when the API runs in that network, i.e. the distribution stack). The target in use is shown on the tab. That way the panel cannot be pointed at scanning an internal network or cloud metadata.
  • Redirects are followed at most 3 times and only to the same host and port; a redirect anywhere else is not followed (its 3xx status is what is judged). The response body is read up to a small cap and dropped — it is never stored.
  • The check runs from the panel machine: it does not prove the application is reachable from the wider internet, and with a public domain the result can be affected by hairpin NAT.

When it checks, when it stays quiet

  • Only running applications. A stopped application is not "down": its monitor just goes back to "not checked".
  • No checks while a deploy runs and for 60 seconds after, and a streak of failures that spans a deploy starts over — so a deploy doesn't trigger an alert.
  • The last status is stored in the database, so an API restart in the middle of an outage doesn't announce it again and doesn't send a false "down" for a healthy application.

Notifications, uptime and history

Once the consecutive-failure count is reached, one HTTP monitor failing notification is sent (reminders at most every 6 hours while it stays down) and one recovery message with the downtime when it is healthy again — see Notifications. The Monitor tab shows the status, 24-hour and 7-day uptime, average and p95 latency, a 24-hour latency chart, and the latest down incidents. Results are kept for 7 days; the response body is never stored. The monitor's settings travel with project export/import (without history).

Resource alerts (optional)

Get a notification when an application or database keeps using too much CPU or memory. Each has a Resource alerts card: on an application it sits under the HTTP monitor on the Monitor tab, on a database on the Overview tab. CPU and memory are separate rules.

SettingNotes
ActiveOff by default. The switch saves at once.
AboveThe threshold in percent. Default 90%. Memory is capped at 100%.
ForHow long the value must stay above the threshold. Default 5 minutes. The two selects need Save.
  • CPU is the percent of one core, the same number the charts show, so it can go above 100 on several cores (it is not divided by the CPU limit).
  • Memory is the percent of the container's memory limit, or of the host's RAM when no limit is set (see Resource limits).
  • The check runs once a minute on the stored per-minute metrics, so it also covers applications on a remote server. The alert arrives about a minute after the value has been above the threshold for the whole duration; one missing minute of data is tolerated, a gap is never treated as high.
  • A rule alerts once, reminds every 6 hours while the value stays high, and sends one back to normal message when it drops. Each rule shows a badge (normal / checking / alerting) and the last reading.
  • Nothing is sent while the application or database is stopped or while the application is deploying, and readings from before the last finished deployment do not count. Changing a rule resets its state without a message.
  • The state is stored, so restarting the API neither repeats an alert nor forgets a breach.
  • Messages go to every channel with the CPU or memory alert toggle on (on by default) and to the notification bell — see Notifications. Developers and up edit the rules; viewers see them read-only. The rules travel with project export/import.
Not covered
Compose stacks have no resource alerts, and there is no alert for the disk usage of volumes.

Low disk warning

Once a day (at 07:00 API time) aoox measures usage on the filesystem where Docker's data lives. Past a threshold — default 90%, change it with DISK_ALERT_PERCENT — a diskLow event is sent to any channel with the toggle enabled, at most once a day while it stays above the threshold.

The usual follow-up is Clean now in the disk strip at the top of the Docker page, lowering deployment retention, or pruning old backups — see Registry.

Common pitfalls

SymptomCause & fix
Host storage: Filesystem unreadableSTORAGE_PATH points to a path that doesn't exist in the API container. Leave it blank (default /) or mount that path.
Host CPU 100% while htop looks lowThe 2-second sample is sensitive to bursts; look at the trend, not one point.
Container metrics emptyThe container just started (< 15 seconds), is stopped, or isn't an app/database container (compose isn't sampled).
Memory at 100% after setting a limitThe application genuinely needs more; raise the limit or check for a leak. 100% means the limit, not host RAM.
1h history gone after a restartThe 1h range is live in memory by design; 24h and up are read from the database and persist.
Not available yet
Metrics for applications on a remote server; alerts based on CPU/RAM thresholds; export to Prometheus; per-container disk I/O.

Next steps

Found a mistake? Edit this page on GitHub ↗