DocsInfrastructure
Troubleshooting
The most common problems, grouped by area, with a quick way to diagnose each one.
- Start with
- API logs + docker ps
- Source of truth
- Deployment logs on the Deploy tab
- Still stuck?
- Open an issue with the info below
Quick diagnosis
On the server, from the aoox-api folder
# status of the three core containers
docker compose -f docker-compose.dist.yml --env-file .env.dist ps
# API logs (deploy, backup, notifications, SSH) and web
docker compose -f docker-compose.dist.yml --env-file .env.dist logs -f --tail 200 api
docker compose -f docker-compose.dist.yml --env-file .env.dist logs -f --tail 100 web
# every container managed by aoox
docker ps -a --filter label=com.docker.compose.project=aoox
# proxy & registry
docker logs -f --tail 100 aoox-proxy
docker logs -f --tail 100 aoox-registry
# is the API alive?
curl -s http://localhost:3001/auth/setup-status- Deploy problems: the deployment log on the Deploy tab is the most complete source — tokens and passwords are already redacted, so it's safe to share.
- Application runtime problems: the container log on the same tab, or
docker logs aoox-app-<slug>. - Who changed what: the audit log (owner/admin).
Installation & accessBuild & deployDomain & HTTPSDatabase & backupTerminal & remote serversNotifications, jobs, disk
Installation & access
| Symptom | Cause | Fix |
|---|---|---|
aoox install stops with “Docker already has data volumes from an earlier aoox installation” | The install folder is empty (or was deleted) but the aoox_postgres_data volume is still there. A fresh install would generate a new POSTGRES_PASSWORD that cannot open the old database, so the installer refuses before changing anything. | Keep the data: put the old .env.dist back (/opt/aoox/backups/<timestamp>/ if the installer saved one) and run sudo aoox reinstall. Start clean: sudo aoox install --force --new-secrets --wipe-data (deletes that volume and the stack's containers; it asks you to type the folder, or pass --confirm-wipe=<dir>). |
Fixes/new features don't show up after a panel update (e.g. the webhook PUBLIC_API_URL is still localhost) | aoox update panel and the “Apply update” button only run pull + up -d: the compose file and new variables in .env.dist are not updated. | Run sudo aoox reinstall (--dry-run first to see what changes). Without the CLI: edit .env.dist / use the Environment page, or adjust the compose file manually per the release notes — see Upgrading. aoox install --force also keeps your secrets and backs up the old files first; only --force --new-secrets --wipe-data deletes the data. |
| The web app loads, but sign-in / the API fails (network error) | PUBLIC_API_URL isn't reachable from the browser (wrong IP/port, firewall blocking 3001). | Match it to an address the browser can actually open; curl <PUBLIC_API_URL>/auth/setup-status from your laptop should respond. |
Terminal: origin not allowed | WEB_ORIGIN ≠ the browser's URL (scheme/host/port). | Match them exactly, then up -d again. See Domain for the panel. |
| Rate limits (sign-in) or the audit log treat every user as one address, or show a proxy IP instead of the client's | The API only trusts X-Forwarded-For from hops on private networks (Traefik, the web container). A proxy on a public address, or a chain of several proxies, is not trusted by default. | Set TRUST_PROXY in .env.dist: a hop count (2), or a comma list of the proxy IPs/CIDRs. true trusts every hop, so a client can fake its IP if the API port is reachable directly; false ignores the header. Then up -d again. |
| Registry/Proxy card: Docker unreachable / permission denied on the socket | DOCKER_GID is wrong or the socket isn't mounted. | stat -c %g /var/run/docker.sock → set DOCKER_GID → recreate the API. |
| Everyone gets logged out suddenly | JWT_SECRET changed. | Expected after a rotation; restore the old value if it wasn't intentional. |
| Registry/Git/notification/S3 credentials can't be read | ENCRYPTION_KEY changed. | Restore the old value; if it's lost, re-enter all credentials. |
/setup doesn't show up / 404 | A user already exists. | Sign in normally; the owner can reset a member's password. To start from scratch: down -v. |
Build & deploy
| Symptom | Cause | Fix |
|---|---|---|
| Deploy fails: No self-hosted registry | The local registry hasn't been provisioned yet. | Provision the registry first. |
| Build: repository not found / 128 | A private repo without a Git credential, an expired token, or the wrong branch. | Add a credential in Integrations → Git credentials and select it on the application; double-check the branch name. |
| Build: unknown flag --mount / heredoc error | BuildKit syntax in the Dockerfile; a classic builder is used. | Replace it with a plain step — see How builds work. |
| Health check never turns healthy | The container port ≠ the port it's listening on, it only listens on 127.0.0.1, the path doesn't return 2xx, or the image has no wget/curl/node/python3. | Listen on 0.0.0.0, match the port, and provide a fast 200 endpoint. |
| New env isn't picked up by the application | Env is applied when the container is created. | Deploy or Rollback once more. |
400: unknown reference ${{database.x.url}} | The wrong slug, or the database is in another project. | References only work within the same project. |
| A webhook push doesn't trigger a deploy | A different branch, a non-push event, a deployment already running (busy), or a 401 secret mismatch. | Check Recent Deliveries on the provider — see the response table. |
| Rollback fails: image not found | The tag was pruned by retention or deleted manually. | Redeploy that commit; raise the deployment history limit. |
| Nixpacks/static: the first build takes forever | The base image (~350 MB) plus the helper is built once. | Wait it out; use a Dockerfile for faster builds. |
| Deploy succeeded (Running) but the app isn't reachable | No host port and no domain are configured — both used to default to empty and lived in edit-only tabs, so a freshly deployed app could silently end up unreachable. | Add either one from the Settings or Domain tab. For new applications, pick IP+port or Domain in the Access step when creating it, so it doesn't need fixing afterward. |
Domain & HTTPS
| Symptom | Cause | Fix |
|---|---|---|
| Domain tab: Proxy isn't running | Traefik hasn't been provisioned. | Settings → Reverse proxy → provision (owner). |
Panel domain saved, DNS & ufw are fine, browser still says unable to connect | The proxy wasn't running at the time (now started automatically when the domain is saved), or ports 80/443 are blocked by your VPS provider's firewall/security group — ufw inactive ≠ ports open to the public. | docker ps --filter name=aoox-proxy should show Up; open 80/443 in your VPS provider's panel. See Domain for the panel → Common pitfalls. |
Panel unreachable after turning on Only allow access through the domain; IP:3000 no longer answers | The option publishes ports 3000/3001 on 127.0.0.1 only, so the domain is now the only way in, and it is broken (DNS, firewall/security group, certificate). | Over SSH, in the install folder: delete the two ports: lines in docker-compose.override.yml, then docker compose -f docker-compose.dist.yml -f docker-compose.override.yml --env-file .env.dist up -d. See Domain for the panel → Only allow access through the domain. |
| 404 page not found from Traefik | The container isn't healthy yet, or the domain was added before the container was recreated. | Wait for it to become healthy; check the deploy log; make sure the health check passes. |
| Bad Gateway (502) from a domain, although the container is running | Traefik can't reach the app: Container port is not the port the app listens on (e.g. an nginx image listens on 80, the port is 3000). | When the image declares EXPOSE ports that exclude the Container port, the Settings and Domain tabs show a warning (and the deploy log a note) with a one-click Use port 80 button. Otherwise set Container port to the port the app really listens on in Settings. |
| Self-signed certificate / TRAEFIK DEFAULT CERT | ACME failed: DNS isn't correct yet, port 80 is closed, a rate limit, or the email was empty at provision time. | docker logs -f aoox-proxy; test with PROXY_ACME_STAGING=true. |
| Redirect loop | Cloudflare's Flexible mode, or another proxy terminating TLS in front. | Switch to Full (strict), or disable HTTPS in aoox. |
DNS check: ? | The server's public IP couldn't be detected (NAT/no outbound internet). | Set PUBLIC_IP in .env.dist. |
| PR preview doesn't resolve | No wildcard DNS, or the wrong preview domain. | *.preview.example.com A <ip>. |
Database & backup
| Symptom | Cause | Fix |
|---|---|---|
Application: ECONNREFUSED / ENOTFOUND to the database | Using an external/localhost URL from inside the container, or an app on a remote server using an internal host. | Use a reference (internal host) for a local app; use an external connection + host port for a remote server. |
| Database status is error right after creation | The image tag doesn't exist, or the host port is already in use. | Delete it, recreate with a different tag/port. |
| Backup is 20 bytes / fails | Credentials/DB weren't ready yet, or the disk is full. | Read the error message on the backup row; check the disk strip at the top of the Docker page. |
| Testing an S3 destination: NoSuchBucket / fails against a local S3 server | The bucket doesn't exist, the region is wrong, or the endpoint uses localhost from inside the container. | Create the bucket; use the S3 container's name (e.g. RustFS) + :9000 as the endpoint. |
| Data browser: 400, only one statement allowed | A trailing ; or multiple statements. | Run them one at a time, or use SQL import. |
| Member: 403 while running a query | A write statement. | Owner/admin only; a member's session is read-only at the engine level. |
Terminal & remote servers
| Symptom | Cause | Fix |
|---|---|---|
| Terminal: auth failed + an authorization command | The platform key isn't in that user's authorized_keys yet (automatic authorization refuses a symlinked ~/.ssh or authorized_keys). | Run the command shown, on the host, as that same user. |
Terminal: SSH authentication failed for root@host.docker.internal | The terminal logs in as root (the default), while on a cloud VPS (Ubuntu and the like) root login is usually refused by sshd and your login is another user. | Set User in Infrastructure → Environment to the VPS login user (e.g. ubuntu), run the authorization command on the VPS as that user, and check that user's ~/.ssh/authorized_keys and PermitRootLogin in /etc/ssh/sshd_config. Details in Web terminal. |
Terminal: TERMINAL_SSH_USER is not set (older versions) | Older installs have no such line in .env.dist, and updates never touch that file. Recent versions fall back to root when it's empty. | Run sudo aoox reinstall (adds the missing line without touching data). Without the CLI: add TERMINAL_SSH_USER=root to .env.dist or use the Environment page. If the login must be another user, set it from Settings → Infrastructure → Environment (owner, no SSH needed), or manually in .env.dist then restart the stack. |
Terminal: ECONNREFUSED to host.docker.internal | sshd is down/wrong port, or Linux without extra_hosts. | Use the docker0 gateway IP (e.g. 172.17.0.1) as TERMINAL_SSH_HOST. |
| Key directory not writable (uid 1000) | ./secrets is owned by root. | sudo chown -R 1000:1000 ./secrets. |
| Remote server: docker: command not found / permission denied | Docker isn't on that user's PATH, or the user isn't in the docker group. | curl -fsSL https://get.docker.com | sh, usermod -aG docker <user>. |
| Remote deploy succeeds but isn't reachable | The host port is empty, or the server's firewall is blocking the port. | Fill in a host port; open the port in ufw/a security group. Domains don't work the same on a remote server. |
| A port is reachable from the internet although ufw denies it | Docker publishes ports through its own firewall rules, which ufw does not filter. | Close it in the provider firewall, or set the port's Listens on to This server only. The Open ports tab on the Docker page lists every published port and whether it is open. |
Notifications, jobs, disk
| Symptom | Cause | Fix |
|---|---|---|
| The notification test succeeds, but the real event never arrives | The event's toggle isn't enabled, or the event genuinely didn't happen (e.g. a dead container on a remote server isn't detected). | Enable the toggle; trigger a real event. |
| A teammate can't see a project | They haven't been added as a project member yet; instance members only see projects they're assigned to. | Add them on the project's Members tab — see Project. |
| Viewer: the button is there but the action fails with 403 | The project's viewer role is read-only, and it's enforced by the API. | Promote them to developer if they genuinely need to make changes. |
| Swarm: task pending / image pull failed on another node | The constraint doesn't match, or REGISTRY_PUBLIC_HOST is still localhost so other nodes can't pull the image. | See Docker Swarm. |
| A job runs at the wrong time | The API's timezone is UTC unless TZ is set. | Set TZ=Asia/Jakarta (your zone) in .env.dist and recreate the api container, or adjust the cron's hour. |
| Job: container not running | Target is In container but the application is stopped. | Use Separate container, or start the application. |
| Disk full (or a low-disk notification) | Old deployment images, dangling images, build cache, local backups. | Docker (disk strip at the top) → Clean now; lower deployment history & backup retention counts; run registry GC. |
| Compose stack: a service can't see the env | Stack env is only used for interpolation. | Add an environment: block to the service. |
Starting from scratch (testing)
This wipes all of aoox's own data — not the application/database containers already created from the dashboard:
docker compose -f docker-compose.dist.yml --env-file .env.dist down -v
docker compose -f docker-compose.dist.yml --env-file .env.dist up -dApp/db/registry/proxy containers stay put and become orphaned after a reset. Delete them from the dashboard first, or:
docker ps -aq --filter label=com.docker.compose.project=aoox | xargs docker rm -f (and their volumes too, if you really want a clean slate).Port & env reference
| Port | For | Env |
|---|---|---|
| 3000 | Web (dashboard) | WEB_PORT |
| 3001 | API | API_PORT |
| 5000 | Local registry | REGISTRY_PORT |
| 80 / 443 | Reverse proxy (ACME needs 80) | PROXY_HTTP_PORT / PROXY_HTTPS_PORT |
| 22 | SSH to the host (terminal) | TERMINAL_SSH_PORT |
| 9000 / 9001 | RustFS from the template (S3 API / Console) | — |
Reporting an issue
If it's still not fixed, open an issue on GitHub aoox-api ↗. All bugs go to this one repository, whether they concern the dashboard, the API, the CLI or the documentation (existing issues). Include:
- The API/web image version (the Host card on the Dashboard, or the tag in
.env.dist) and the Docker version. - The steps taken and what you expected.
- A snippet of the API log around when it happened (secrets are already redacted, but double-check).
- For deploys: the deployment log from the Deploy tab and which build type was used.
Next steps
Found a mistake? Edit this page on GitHub ↗