Operations & Troubleshooting
Day-two operations for self-hosted Dynamiq: health checks, logs, scaling, backup and restore, and fixes for the failures you'll actually hit.
Once the platform is installed, most of running it is checking health, scaling to your workload, backing up state, and diagnosing the occasional failed pod. This page collects those day-two tasks for the canonical install — release dynamiq in namespace dynamiq. It assumes you've been through Install on Kubernetes (Helm); the Troubleshooting section links back to the exact install steps a failure traces to.
Health and logs
The chart runs five Deployments in the release namespace. In a healthy install every pod is Ready:
kubectl -n dynamiq get podsYou should see one pod per Deployment at steady state — nexus, synapse, catalyst, runtime, and ui — all Running and Ready (more per service if you've raised replicas or enabled autoscaling).
Each backend service serves a liveness and readiness probe on /health at container port 8080; the ui probes / on its http port instead. Hit the public API health endpoint from outside the cluster the same way the install verifies it:
curl https://api.dynamiq.example.com/healthFor logs, address each Deployment by name:
kubectl -n dynamiq logs deploy/nexus # or synapse, catalyst, runtime, ui
kubectl -n dynamiq logs deploy/catalyst -f # follow live
kubectl -n dynamiq logs deploy/nexus --previous # the crashed container, after a restartnexus is the place to start for platform-level problems — it owns authentication, projects, and scheduling — while catalyst and runtime carry the execution workloads.
Scaling
Every service block accepts replicaCount and an autoscaling block. Set a static count, or turn on autoscaling and let an HPA own the replica count between minReplicas and maxReplicas — the two are mutually exclusive per service. The full pattern (requests, limits, HPA targets, scheduling) is in Sizing, scheduling, autoscaling.
- All five services scale horizontally. They're stateless — state lives in Postgres, S3, and NATS — so you can run several replicas of each. Keep at least 2 replicas of the public services for zero-downtime rollouts.
- catalyst and runtime are the resource-heaviest. They ship the largest default requests and limits because they run the execution workloads (sizing baseline); scale and size those first as load grows.
- Workload namespaces scale independently. The inference and managed-database pods that run in the feature namespaces are scheduled by nexus, not the chart, and follow the resource settings of the deployed resource — see Deploy & Integrate.
Backup and restore
Self-hosted state lives in the services you provide, not in the chart. Back up each on its own schedule:
| State | Lives in | Back up with |
|---|---|---|
| Orgs, projects, workflows, deployment metadata, auth | PostgreSQL (nexus, synapse, catalyst databases) | pg_dump per database, or your managed provider's automated snapshots |
| Files, run artifacts, Knowledge Base documents | S3 bucket | Bucket versioning and cross-region replication |
| In-flight jobs and streams | NATS JetStream | Treat as transient; back up its persistent volumes only if you provisioned them |
| Signing keys, database and provider credentials, the license | Kubernetes Secrets | Keep the source manifests under version control — you created these, the chart doesn't |
Postgres is the system of record: orgs, projects, workflows, and every deployment's metadata live there. Losing it loses the platform, so verify your dumps restore cleanly.
Restore the database before rolling the app back. Because Atlas migrations are forward-only, recovering from a bad schema-migrating upgrade means restoring the Postgres snapshot first, then rolling the release back to the version that matches it. See Upgrades & Rollback → Roll back.
Troubleshooting
Getting help
If a problem outlasts the fixes above, contact support@getdynamiq.ai. Include:
- The chart version —
helm list -n dynamiq. - Pod states —
kubectl -n dynamiq get pods. - Relevant logs — the output of
kubectl -n dynamiq logs deploy/<svc>(orjob/dynamiq) for the failing component.