QWED deployment guide

Deploy QWED to production using Docker, Kubernetes, or bare metal. Includes environment variables reference, production checklist, and monitoring setup.

This guide provides instructions for deploying QWED in various environments.

​

Table of contents

1. Docker deployment
2. Kubernetes deployment
3. Manual / bare metal deployment
4. Environment variables reference
5. Production checklist
6. Troubleshooting
7. Operations and monitoring

---

​

Docker deployment

The easiest way to run QWED locally or on a single server is using Docker Compose.

​

Dockerfile

The Dockerfile in the repository root has two targets:

• server: The API server. Runs uvicorn on port 8000 as a non-root user (UID 1000), installs every server dependency from a hash-pinned lock file, and has a health check on /health. Docker Hub publishes this target as qwedai/qwed-verification.
• action: The QWED GitHub Action. It stays the last, default target, so build the server with --target server.

# Build the API server image manually
docker build --target server -t qwed-core .

The server image uses /var/lib/qwed as its working directory, which holds the default SQLite database. UVICORN_HOST and UVICORN_PORT set the bind address and port, and the health check follows UVICORN_PORT.

Docker Hub tags before 8.0.0 ran the GitHub Action, not the API server. latest tracks main, and versioned tags exist from 8.0.0. If you ran the image as a repository scanner, use the GitHub Action instead.

​

Docker Compose

deploy/docker-compose.yml orchestrates:

• qwed-core: The main API server, built from the server target. Its data lives in the qwed-data volume.
• postgres: Primary database.
• redis: Cache and rate limiting.
• jaeger: Distributed tracing.
• prometheus: Metrics collection. Scrapes the API at qwed-core:8000.
• grafana: Observability dashboards, with Prometheus and Jaeger data sources provisioned.

Before starting the stack, copy the settings template and fill in every required value:

cp deploy/.env.example deploy/.env

# deploy/.env

# QWED API server (required)
QWED_CORS_ORIGINS=https://app.yourcompany.com
# Generate each secret separately and keep it:
#   python -c "import secrets; print(secrets.token_urlsafe(48))"
API_KEY_SECRET=
# Changing them invalidates issued tokens and API keys, or breaks the signed
# audit chain. The lookup secret must differ from the JWT secret.
QWED_JWT_SECRET_KEY=
QWED_API_KEY_LOOKUP_SECRET=
QWED_AUDIT_SECRET_KEY=

# Optional. User IDs allowed to read /metrics. Empty denies every caller.
QWED_METRICS_OPERATOR_USER_IDS=

# Postgres (required)
POSTGRES_USER=qwed
POSTGRES_DB=qwed_db
POSTGRES_PASSWORD=
# Must match the three values above. Build it as a postgresql:// URL with the
# user, POSTGRES_PASSWORD, the host "postgres" (the Compose service), port 5432
# and qwed_db, the same shape as deploy/.env.example shows.
DATABASE_URL=

# Grafana (required)
GF_SECURITY_ADMIN_USER=admin
# Generate a separate password the same way:
#   python -c "import secrets; print(secrets.token_urlsafe(24))"
GF_SECURITY_ADMIN_PASSWORD=

# Optional overrides
REDIS_URL=redis://redis:6379/0
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317

Docker Compose refuses to start while QWED_CORS_ORIGINS, API_KEY_SECRET, QWED_JWT_SECRET_KEY, QWED_API_KEY_LOOKUP_SECRET, QWED_AUDIT_SECRET_KEY, or DATABASE_URL is empty. The Postgres container refuses to start while POSTGRES_PASSWORD is empty. The Grafana container refuses to start while GF_SECURITY_ADMIN_PASSWORD is empty, instead of falling back to admin/admin. Generate each secret separately with:

python -c "import secrets; print(secrets.token_urlsafe(48))"

Start the stack from the repository root:

docker compose -f deploy/docker-compose.yml up -d

The qwed-core service passes no provider settings to the container, so a model or key written only in deploy/.env doesn’t reach the server, and requests that use a provider fail. Pass them with an override file next to deploy/docker-compose.yml; Compose fills ${...} from deploy/.env, so the values stay there:

# deploy/docker-compose.override.yml
services:
  qwed-core:
    environment:
      - ACTIVE_PROVIDER=${ACTIVE_PROVIDER}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - OPENAI_MODEL=${OPENAI_MODEL}

Add ACTIVE_PROVIDER, the provider’s key, and its model (here OPENAI_API_KEY and OPENAI_MODEL) to deploy/.env: an unset one is passed empty. Then start with docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.override.yml up -d, using the variables of the provider you run.

Access the services:

• API: http://localhost:8000
• API Docs: http://localhost:8000/docs
• Grafana: http://localhost:3000
• Prometheus: http://localhost:9090
• Jaeger: http://localhost:16686

​

Run the image without Compose

Pass the required secrets to docker run from your shell, so no value is written into the command:

# Generate each secret once and keep it. Changing the lookup or audit key later
# invalidates existing API keys or audit signatures.
for name in API_KEY_SECRET QWED_JWT_SECRET_KEY QWED_API_KEY_LOOKUP_SECRET QWED_AUDIT_SECRET_KEY; do
  export "$name=$(python -c 'import secrets; print(secrets.token_urlsafe(48))')"
done
# For Postgres, also export DATABASE_URL from your secrets manager. Left unset,
# docker run passes nothing and the server uses SQLite in the qwed-data volume.

docker run -d -p 8000:8000 \
  -v qwed-data:/var/lib/qwed \
  -e QWED_CORS_ORIGINS="https://app.yourcompany.com" \
  -e API_KEY_SECRET \
  -e QWED_JWT_SECRET_KEY \
  -e QWED_API_KEY_LOOKUP_SECRET \
  -e QWED_AUDIT_SECRET_KEY \
  -e DATABASE_URL \
  qwedai/qwed-verification:9.1.0

Omit DATABASE_URL to use the SQLite database in the qwed-data volume. The server names no model. For requests that use a provider (natural-language, logic, statistics, and fact), also pass the provider’s settings, for example -e ACTIVE_PROVIDER -e OPENAI_API_KEY -e OPENAI_MODEL. The container doesn’t read the qwed init settings on your host. The same applies to the signing-key command below.

​

Attestation signing key

Without a signing key, the server generates one per process, so its attestations stop verifying after a restart or on another replica. If you run more than one replica, or need tokens to survive restarts, generate an EC P-256 key once and mount it:

python -c "from cryptography.hazmat.primitives.asymmetric import ec; from cryptography.hazmat.primitives import serialization as s; print(ec.generate_private_key(ec.SECP256R1()).private_bytes(s.Encoding.PEM, s.PrivateFormat.PKCS8, s.NoEncryption()).decode())" > attestation-key.pem

docker run -p 8000:8000 \
  -v qwed-data:/var/lib/qwed \
  -v "$PWD/attestation-key.pem:/run/secrets/attestation-key.pem:ro" \
  -e QWED_SIGNING_KEY_FILE=/run/secrets/attestation-key.pem \
  -e DATABASE_URL \
  -e API_KEY_SECRET \
  -e QWED_JWT_SECRET_KEY \
  -e QWED_API_KEY_LOOKUP_SECRET \
  -e QWED_AUDIT_SECRET_KEY \
  -e QWED_CORS_ORIGINS="https://app.yourcompany.com" \
  qwedai/qwed-verification:9.1.0

Keep the key file out of version control. See Signing keys.

​

Docker requirement for code execution

Both the Stats Engine and the Consensus Engine (in high/maximum mode) execute model-generated Python inside Docker containers. Docker is required — there are no in-process fallbacks. If Docker is unavailable, these endpoints return HTTP 503.

---

​

Kubernetes deployment

For production environments, QWED ships Kubernetes manifests in deploy/kubernetes/.

​

Prerequisites

• A running Kubernetes cluster (v1.24+ recommended).
• kubectl configured.
• A PostgreSQL database and Redis instance (managed services recommended for production).

​

Deployment steps

1. Create Namespace

   kubectl create namespace qwed
   

2. Configure Secrets & ConfigMaps Edit deploy/kubernetes/secret.yaml and fill in every value. The file ships with empty values for DATABASE_URL, API_KEY_SECRET, QWED_JWT_SECRET_KEY, QWED_API_KEY_LOOKUP_SECRET, QWED_AUDIT_SECRET_KEY, and your LLM provider key. The server refuses to start while the JWT or lookup secret is empty, and fails closed on the first audit write while the audit secret is empty. The DATABASE_URL is stored in the Secret (not the ConfigMap) because it contains credentials.

   kubectl apply -f deploy/kubernetes/configmap.yaml
   kubectl apply -f deploy/kubernetes/secret.yaml
   
   For production, use Sealed Secrets or External Secrets Operator instead of storing plain-text values in secret.yaml.

3. Deploy Application deploy/kubernetes/deployment.yaml pins docker.io/qwedai/qwed-verification:9.1.0. The first tag of the API server image is 8.0.0. Don’t point the Deployment at an earlier tag, because earlier tags ran the GitHub Action.

   kubectl apply -f deploy/kubernetes/deployment.yaml
   kubectl apply -f deploy/kubernetes/service.yaml
   

4. Verify Deployment

   kubectl get pods -n qwed
   

​

Horizontal pod autoscaling (HPA)

For high-traffic environments, enable HPA (requires Metrics Server):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: qwed-core-hpa
  namespace: qwed
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: qwed-core
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Multi-replica audit writes are not supported. The audit logger keeps each organization’s chain head in memory, so replicas sharing one database can make audit appends fail with 500 audit persistence failed. See Chain continuity across restarts.

---

​

Manual / bare metal deployment

If you prefer to run the application directly on a host or VM:

​

1. Prerequisites

Docker Installation (Required for Secure Code Execution) QWED requires Docker for all model-generated code execution in the Stats Engine and Consensus Engine. Without Docker, these verification endpoints return HTTP 503.

​

Linux (Ubuntu/Debian)

sudo apt-get update
sudo apt-get install docker.io
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER

​

2. Python dependencies

# Install QWED with the API server extra
pip install -e ".[server]"

The server extra installs the Postgres drivers, the Redis and Prometheus clients, the OpenTelemetry packages, and the docker client (docker>=7.0.0). The code executor imports the docker client when the server loads, so install the extra even if Docker is not running.

​

3. Database setup

Ensure PostgreSQL and Redis are running. Set the DATABASE_URL and REDIS_URL environment variables.

Initialize the database:

python -c "from qwed_new.core.database import create_db_and_tables; create_db_and_tables()"

​

4. Running the API

# Production Mode with Gunicorn (Linux/macOS)
gunicorn qwed_new.api.main:app -w 1 -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000

Run one worker per database. Every verification request appends an audit row, and the audit logger supports one writer per database. With several workers, each appends audit rows independently, and a conflicting append makes the verification request fail with 500 audit persistence failed. See Chain continuity across restarts.

---

​

Environment variables reference

​

Infrastructure

Variable	Description	Default	Required
DATABASE_URL	Postgres connection string	sqlite:///./qwed.db	Yes (Prod)
REDIS_URL	Redis connection string	redis://localhost:6379/0	Yes (Prod)
API_KEY_SECRET	Secret key for signing JWTs	None (must be set)	YES
QWED_CORS_ORIGINS	Comma-separated list of allowed CORS origins	None (must be set)	YES
QWED_SKIP_ENV_INTEGRITY_CHECK	Set to true to bypass startup environment integrity check	false	No
QWED_ALLOWED_STARTUP_PTH_FILES	Comma-separated .pth startup hooks to allow at startup. A plain name (my_hook.pth) is allowed but still scanned for suspicious patterns. A digest pin (my_hook.pth:sha256:<64 hex>) is allowed only while the file’s SHA-256 matches, and isn’t scanned. A malformed pin stops the server at startup. See Startup environment integrity	Unset	No
QWED_API_KEY_LOOKUP_SECRET	Secret for HMAC-SHA256 API-key lookup digests and agent token digests. Must differ from QWED_JWT_SECRET_KEY. Changing it invalidates existing API keys and agent tokens	None	Yes
QWED_JWT_SECRET_KEY	Secret for signing JWTs. Keep it stable: changing it invalidates issued tokens	None	Yes
QWED_AUDIT_SECRET_KEY	HMAC key that signs the per-organization audit hash chain. The server refuses to start without it. Rotating it invalidates the signatures on existing rows, so verify and archive the trail first. See Audit trail setup	None (must be set)	Yes
QWED_SIGNING_KEY	From v8.0.0. The ES256 attestation signing key as unencrypted EC P-256 PEM text. In an env file, write newlines as \n. Set this or QWED_SIGNING_KEY_FILE, not both. Callers verify attestations against the public key at /.well-known/jwks.json. See Signing keys	Unset (a key is generated per process)	Recommended
QWED_SIGNING_KEY_FILE	From v8.0.0. Path to a file holding the signing key PEM, such as a mounted secret. Setting both key variables, or a key that is unreadable, not P-256, or encrypted, stops the server at startup	Unset	Recommended
QWED_ATTESTATION_TTL_SECONDS	From v8.0.0. How long an attestation stays valid, in seconds, from 60 to 2592000 (30 days). Any other value stops the server at startup	86400 (24 hours)	No
QWED_ENABLE_AUDIT_SEED	Set to 1 to enable POST /audit/seed, which writes sample rows into the signed audit trail. Development only. Leave unset in production	Unset (route returns 404)	No
QWED_METRICS_OPERATOR_USER_IDS	Comma-separated user IDs allowed to read /metrics. Empty denies every caller	None	No
UVICORN_HOST	Bind address of the API server in the Docker image	0.0.0.0	No
UVICORN_PORT	Port of the API server in the Docker image. The image’s health check follows it	8000	No

API_KEY_SECRET no longer has a default value. The server will refuse to start if it is not set. This prevents accidental deployment with a weak default secret.

​

Security configuration

Variable	Description	Default
MAX_INPUT_LENGTH	Max query length (chars)	2000
SIMILARITY_THRESHOLD	Prompt injection detection threshold	0.6
DOCKER_TIMEOUT	Code execution timeout (seconds)	10
DOCKER_MEMORY_LIMIT	Container memory limit	512m
QWED_SANDBOX_IMAGE	From v8.0.0. Alternative image for the stats and consensus code sandbox. QWED accepts it only when pinned by digest (name@sha256:<64 hex>). Any other value is refused, logged, and replaced by the pinned default image	amancevice/pandas@sha256:9257b889…
QWED_SANDBOX_UID_RANGE	From v8.0.0. Operator-reserved uid range (FIRST-LAST, within 65536–2147483646) that QWED hands sandbox workspaces to when the server process runs as root. An invalid value is logged and the default is used	1879048192-2147483646
QWED_CROSSHAIR_IMAGE	From v8.0.0. Local Docker image the Symbolic engine runs CrossHair in. Build it from docker/crosshair-sandbox/Dockerfile. QWED never pulls it at runtime	qwed-crosshair-sandbox:0.0.101
QWED_SKIP_ENV_INTEGRITY_CHECK	Skip .pth file verification on startup	false

​

Maths limits

From v9.0.0, /verify/math and the maths items of /verify/batch run SymPy in a pool of worker processes. These settings size that pool. Unset or blank, each keeps its default.

Variable	Description	Default
QWED_MATH_WORKERS	Worker processes. A whole number, at least 1	2
QWED_MATH_TIMEOUT_SECONDS	Time limit in seconds, from 0.001 to 3600. It applies separately to the wait for a free worker and to the run, so a job that has to wait can take up to twice this. A job that waits longer is BLOCKED (math.capacity). A job that runs longer is UNVERIFIABLE (math.timeout), and its worker is killed and replaced. Starting a new worker is not counted: the job waits up to 120 seconds for it, and a worker that does not start is BLOCKED (math.worker_error)	10
QWED_MATH_MAX_WAITING	Jobs that may wait for a free worker. A whole number, at least 0. A job that finds every worker busy and the queue full is BLOCKED (math.capacity) at once	32
QWED_MATH_WORKER_MEMORY_MB	Memory limit of each worker in MB, on Linux only. A whole number, at least 0. 0 turns the limit off. A job that runs out of memory is BLOCKED (math.out_of_memory)	2048
QWED_BATCH_MATH_BUDGET_SECONDS	Time budget in seconds for the maths items of one batch job, at least 0. It is wall-clock time from when the job starts, so time spent on the job’s other items counts too. Each maths item keeps the time limit above, cut to what is left of the budget when it reaches a worker. Starting a new worker isn’t counted against the item’s own limit, which is set before the start, so that item can finish after the budget ran out and return its result. The start still uses the job’s wall-clock budget, so later items can get batch.math_budget, and it counts toward the batch item limit QWED_BATCH_ITEM_TIMEOUT (60 seconds by default), so a slow start can fail the item. An item reached with no budget left is not run (batch.math_budget). 0 means no maths item of a batch runs	120

A value these settings cannot use stops the server at startup with an error naming the setting. That includes a value that is not a number, inf or nan, a value below its minimum, a fraction for a count, and a time limit over 3600 seconds. QWED never puts another value in its place.

From v9.1.0, the server also reads these settings from .env (the working directory’s, then ~/.qwed/.env), and a value set in the process environment wins. In v9.0.0, a server started with uvicorn qwed_new.api.main:app read the four QWED_MATH_* settings before .env loaded: a value only in .env was ignored, and one the server couldn’t use made every maths job BLOCKED (math.worker_error) instead of stopping the start. Before you upgrade to v9.1.0, check .env for maths settings the process environment doesn’t set: the server now uses them, and one it can’t use stops the start.

From v9.1.0, deploy/.env.example and deploy/kubernetes/configmap.yaml list the five settings, commented out. deploy/docker-compose.yml passes them to the server by name, so one you don’t set stays unset and keeps its default. Size the pool for your load.

​

AI providers

Variable	Description	Default
ACTIVE_PROVIDER	Selected LLM provider: openai, openai_direct, openai_compat, azure_openai, anthropic, claude_opus, gemini, ollama, or auto. Any other value stops each request that needs a provider with an error naming ACTIVE_PROVIDER	ollama
AZURE_OPENAI_API_KEY	API key for Azure OpenAI	-
AZURE_OPENAI_ENDPOINT	Endpoint URL	-
AZURE_OPENAI_DEPLOYMENT	Azure OpenAI deployment to run. Required for Azure OpenAI	-
ANTHROPIC_API_KEY	API key for Anthropic	-
ANTHROPIC_MODEL	Anthropic model to run. ANTHROPIC_DEPLOYMENT wins when set. Required for Anthropic	-
GOOGLE_API_KEY	API key for Google Gemini (or GEMINI_API_KEY)	-
GEMINI_MODEL	Gemini model to run. Required for Gemini	-

From v9.0.0, QWED names no default model. Set the model variable of the provider you use (OPENAI_MODEL, AZURE_OPENAI_DEPLOYMENT, ANTHROPIC_MODEL, CLAUDE_OPUS_DEPLOYMENT, GEMINI_MODEL, OLLAMA_MODEL, or CUSTOM_MODEL). The server still starts without one, because it builds a provider on first use. Logic, statistics, and fact requests that need the provider then fail with an error that names the variable. The natural-language routes answer UNVERIFIABLE. See LLM configuration.

​

Observability

Variable	Description	Default
OTEL_EXPORTER_OTLP_ENDPOINT	Jaeger/OTLP Endpoint	http://localhost:4317

---

​

Production checklist

Before going to production, ensure the following:

​

1. Database setup

• Use a managed PostgreSQL instance (e.g., AWS RDS, Azure Database for PostgreSQL).
• Enable automated backups.
• Run database migrations.

​

2. Redis configuration

• Use a managed Redis instance (e.g., AWS ElastiCache).
• Configure eviction policy (LRU).
• Enable persistence (RDB/AOF).

​

3. Security

• Set API_KEY_SECRET: This is mandatory — generate a secure value with python -c "import secrets; print(secrets.token_urlsafe(48))".
• Set QWED_CORS_ORIGINS: Explicitly list your allowed origins (e.g., https://app.yourcompany.com). The server will not start without this.
• Set QWED_AUDIT_SECRET_KEY: Required. The server will not start without it. Store it in a secrets manager and don’t rotate it without archiving the existing audit trail.
• Leave QWED_ENABLE_AUDIT_SEED unset: Keeps POST /audit/seed disabled so sample rows never enter the production audit trail.
• Rotate Keys: Change all default passwords and secrets.
• SSL/TLS: Ensure the API is behind a Load Balancer with a valid SSL certificate.
• Network Policies: Restrict access to database/Redis.
• Docker Security: Ensure the Docker socket is protected or use a secure container runtime (gVisor) for the Stats Verification engine if possible.

​

4. Observability

• Configure alert rules in Prometheus/Grafana.
• Ensure logs are shipped to a centralized logging system.

---

​

Troubleshooting

​

1. Docker permission denied

Error: docker: Got permission denied while trying to connect to the Docker daemon socket Solution: Ensure the user running the app is in the docker group, or (for Docker Compose) ensure the socket is mounted correctly and the container user has permissions.

​

2. Stats or consensus verification returning 503

Error: Service temporarily unavailable on /verify/stats or /verify/consensus Cause: The secure Docker sandbox is unreachable. QWED does not fall back to in-process execution. Solution:

1. Check Docker is running: docker ps
2. Verify the Docker daemon responds to pings: docker info
3. Check the pinned sandbox image exists: docker pull amancevice/pandas@sha256:9257b88909bddf60c8a85b06d572fc6dfc708ea9c8a3c3dc2149e312d57f91e3 (or the digest you set in QWED_SANDBOX_IMAGE).
4. Check the daemon’s user mapping. QWED reports a userns-remap daemon, a rootless daemon used by a root server process, or a daemon whose security options QWED can’t read as SECURE_RUNTIME_UNAVAILABLE. Run QWED as a non-root user with a rootless daemon that the same user runs, or as any user with a default (non-remapped) daemon.

---

​

Operations and monitoring

​

View recent security events

# Python script
python -c "
from qwed_new.core.database import get_session
from qwed_new.core.models import SecurityEvent
from sqlmodel import select

with get_session() as session:
    events = session.exec(
        select(SecurityEvent)
        .where(SecurityEvent.event_type == 'BLOCKED')
        .order_by(SecurityEvent.timestamp.desc())
        .limit(10)
    ).all()
    for e in events:
        print(f'{e.timestamp}: {e.reason}')
"

​

Security metrics dashboard

# Block rate by hour
sqlite3 qwed_v2.db "
SELECT strftime('%Y-%m-%d %H:00', timestamp) as hour, COUNT(*) as blocks
FROM security_event
WHERE event_type = 'BLOCKED'
GROUP BY hour
ORDER BY hour DESC
LIMIT 24;
"

For detailed architecture documentation, see:

• Architecture