> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qwedai.tech/llms.txt
> Use this file to discover all available pages before exploring further.

# QWED deployment guide

> Deploy QWED to production using Docker, Kubernetes, or bare metal. Includes environment variables reference, production checklist, and monitoring setup.

This guide provides instructions for deploying QWED in various environments.

## Table of contents

1. [Docker deployment](#docker-deployment)
2. [Kubernetes deployment](#kubernetes-deployment)
3. [Manual / bare metal deployment](#manual-%2F-bare-metal-deployment)
4. [Environment variables reference](#environment-variables-reference)
5. [Production checklist](#production-checklist)
6. [Troubleshooting](#troubleshooting)
7. [Operations and monitoring](#operations-and-monitoring)

***

## Docker deployment

The easiest way to run QWED locally or on a single server is using Docker Compose.

### Dockerfile

The `Dockerfile` in the repository root has two targets:

* **`server`**: The API server. Runs uvicorn on port 8000 as a non-root user (UID 1000), installs every server dependency from a hash-pinned lock file, and has a health check on `/health`. Docker Hub publishes this target as `qwedai/qwed-verification`.
* **`action`**: The QWED GitHub Action. It stays the last, default target, so build the server with `--target server`.

```bash theme={null}
# Build the API server image manually
docker build --target server -t qwed-core .
```

The server image uses `/var/lib/qwed` as its working directory, which holds the default SQLite database. `UVICORN_HOST` and `UVICORN_PORT` set the bind address and port, and the health check follows `UVICORN_PORT`.

> [!WARNING]
>
> Docker Hub tags before 8.0.0 ran the GitHub Action, not the API server. `latest` tracks `main`, and versioned tags exist from 8.0.0. If you ran the image as a repository scanner, use the [GitHub Action](https://docs.qwedai.tech/advanced/github-action.md) instead.

### Docker Compose

`deploy/docker-compose.yml` orchestrates:

* **qwed-core**: The main API server, built from the `server` target. Its data lives in the `qwed-data` volume.
* **postgres**: Primary database.
* **redis**: Cache and rate limiting.
* **jaeger**: Distributed tracing.
* **prometheus**: Metrics collection. Scrapes the API at `qwed-core:8000`.
* **grafana**: Observability dashboards, with Prometheus and Jaeger data sources provisioned.

Before starting the stack, copy the settings template and fill in every required value:

```bash theme={null}
cp deploy/.env.example deploy/.env
```

```bash theme={null}
# deploy/.env

# QWED API server (required)
QWED_CORS_ORIGINS=https://app.yourcompany.com
# Generate each secret separately and keep it:
#   python -c "import secrets; print(secrets.token_urlsafe(48))"
API_KEY_SECRET=
# Changing them invalidates issued tokens and API keys, or breaks the signed
# audit chain. The lookup secret must differ from the JWT secret.
QWED_JWT_SECRET_KEY=
QWED_API_KEY_LOOKUP_SECRET=
QWED_AUDIT_SECRET_KEY=

# Optional. User IDs allowed to read /metrics. Empty denies every caller.
QWED_METRICS_OPERATOR_USER_IDS=

# Postgres (required)
POSTGRES_USER=qwed
POSTGRES_DB=qwed_db
POSTGRES_PASSWORD=
# Must match the three values above. Build it as a postgresql:// URL with the
# user, POSTGRES_PASSWORD, the host "postgres" (the Compose service), port 5432
# and qwed_db, the same shape as deploy/.env.example shows.
DATABASE_URL=

# Grafana (required)
GF_SECURITY_ADMIN_USER=admin
# Generate a separate password the same way:
#   python -c "import secrets; print(secrets.token_urlsafe(24))"
GF_SECURITY_ADMIN_PASSWORD=

# Optional overrides
REDIS_URL=redis://redis:6379/0
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317
```

> [!WARNING]
>
> Docker Compose refuses to start while `QWED_CORS_ORIGINS`, `API_KEY_SECRET`, `QWED_JWT_SECRET_KEY`, `QWED_API_KEY_LOOKUP_SECRET`, `QWED_AUDIT_SECRET_KEY`, or `DATABASE_URL` is empty. The Postgres container refuses to start while `POSTGRES_PASSWORD` is empty. The Grafana container refuses to start while `GF_SECURITY_ADMIN_PASSWORD` is empty, instead of falling back to `admin`/`admin`. Generate each secret separately with:
>
> ```bash theme={null}
> python -c "import secrets; print(secrets.token_urlsafe(48))"
> ```

Start the stack from the repository root:

```bash theme={null}
docker compose -f deploy/docker-compose.yml up -d
```

The `qwed-core` service passes no provider settings to the container, so a model or key written only in `deploy/.env` doesn't reach the server, and requests that use a provider fail. Pass them with an override file next to `deploy/docker-compose.yml`; Compose fills `${...}` from `deploy/.env`, so the values stay there:

```yaml theme={null}
# deploy/docker-compose.override.yml
services:
  qwed-core:
    environment:
      - ACTIVE_PROVIDER=${ACTIVE_PROVIDER}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - OPENAI_MODEL=${OPENAI_MODEL}
```

Add `ACTIVE_PROVIDER`, the provider's key, and its model (here `OPENAI_API_KEY` and `OPENAI_MODEL`) to `deploy/.env`: an unset one is passed empty. Then start with `docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.override.yml up -d`, using the variables of the provider you run.

Access the services:

* **API**: [http://localhost:8000](http://localhost:8000)
* **API Docs**: [http://localhost:8000/docs](http://localhost:8000/docs)
* **Grafana**: [http://localhost:3000](http://localhost:3000)
* **Prometheus**: [http://localhost:9090](http://localhost:9090)
* **Jaeger**: [http://localhost:16686](http://localhost:16686)

### Run the image without Compose

Pass the required secrets to `docker run` from your shell, so no value is written into the command:

```bash theme={null}
# Generate each secret once and keep it. Changing the lookup or audit key later
# invalidates existing API keys or audit signatures.
for name in API_KEY_SECRET QWED_JWT_SECRET_KEY QWED_API_KEY_LOOKUP_SECRET QWED_AUDIT_SECRET_KEY; do
  export "$name=$(python -c 'import secrets; print(secrets.token_urlsafe(48))')"
done
# For Postgres, also export DATABASE_URL from your secrets manager. Left unset,
# docker run passes nothing and the server uses SQLite in the qwed-data volume.

docker run -d -p 8000:8000 \
  -v qwed-data:/var/lib/qwed \
  -e QWED_CORS_ORIGINS="https://app.yourcompany.com" \
  -e API_KEY_SECRET \
  -e QWED_JWT_SECRET_KEY \
  -e QWED_API_KEY_LOOKUP_SECRET \
  -e QWED_AUDIT_SECRET_KEY \
  -e DATABASE_URL \
  qwedai/qwed-verification:9.1.0
```

Omit `DATABASE_URL` to use the SQLite database in the `qwed-data` volume. The server names no model. For requests that use a provider (natural-language, logic, statistics, and fact), also pass the provider's settings, for example `-e ACTIVE_PROVIDER -e OPENAI_API_KEY -e OPENAI_MODEL`. The container doesn't read the `qwed init` settings on your host. The same applies to the signing-key command below.

### Attestation signing key

Without a signing key, the server generates one per process, so its attestations stop verifying after a restart or on another replica. If you run more than one replica, or need tokens to survive restarts, generate an EC P-256 key once and mount it:

```bash theme={null}
python -c "from cryptography.hazmat.primitives.asymmetric import ec; from cryptography.hazmat.primitives import serialization as s; print(ec.generate_private_key(ec.SECP256R1()).private_bytes(s.Encoding.PEM, s.PrivateFormat.PKCS8, s.NoEncryption()).decode())" > attestation-key.pem

docker run -p 8000:8000 \
  -v qwed-data:/var/lib/qwed \
  -v "$PWD/attestation-key.pem:/run/secrets/attestation-key.pem:ro" \
  -e QWED_SIGNING_KEY_FILE=/run/secrets/attestation-key.pem \
  -e DATABASE_URL \
  -e API_KEY_SECRET \
  -e QWED_JWT_SECRET_KEY \
  -e QWED_API_KEY_LOOKUP_SECRET \
  -e QWED_AUDIT_SECRET_KEY \
  -e QWED_CORS_ORIGINS="https://app.yourcompany.com" \
  qwedai/qwed-verification:9.1.0
```

Keep the key file out of version control. See [Signing keys](https://docs.qwedai.tech/advanced/attestations.md#signing-keys).

### Docker requirement for code execution

Both the Stats Engine and the Consensus Engine (in `high`/`maximum` mode) execute model-generated Python inside Docker containers. Docker is **required** — there are no in-process fallbacks. If Docker is unavailable, these endpoints return HTTP 503.

***

## Kubernetes deployment

For production environments, QWED ships Kubernetes manifests in `deploy/kubernetes/`.

### Prerequisites

* A running Kubernetes cluster (v1.24+ recommended).
* `kubectl` configured.
* A PostgreSQL database and Redis instance (managed services recommended for production).

### Deployment steps

1. **Create Namespace**
   ```bash theme={null}
   kubectl create namespace qwed
   ```

2. **Configure Secrets & ConfigMaps**
   Edit `deploy/kubernetes/secret.yaml` and fill in every value. The file ships with empty values for `DATABASE_URL`, `API_KEY_SECRET`, `QWED_JWT_SECRET_KEY`, `QWED_API_KEY_LOOKUP_SECRET`, `QWED_AUDIT_SECRET_KEY`, and your LLM provider key. The server refuses to start while the JWT or lookup secret is empty, and fails closed on the first audit write while the audit secret is empty. The `DATABASE_URL` is stored in the Secret (not the ConfigMap) because it contains credentials.

   ```bash theme={null}
   kubectl apply -f deploy/kubernetes/configmap.yaml
   kubectl apply -f deploy/kubernetes/secret.yaml
   ```

   > [!TIP]
   >
   > For production, use [Sealed Secrets](https://sealed-secrets.netlify.app/) or [External Secrets Operator](https://external-secrets.io/) instead of storing plain-text values in `secret.yaml`.

3. **Deploy Application**
   `deploy/kubernetes/deployment.yaml` pins `docker.io/qwedai/qwed-verification:9.1.0`. The first tag of the API server image is 8.0.0. Don't point the Deployment at an earlier tag, because earlier tags ran the GitHub Action.
   ```bash theme={null}
   kubectl apply -f deploy/kubernetes/deployment.yaml
   kubectl apply -f deploy/kubernetes/service.yaml
   ```

4. **Verify Deployment**
   ```bash theme={null}
   kubectl get pods -n qwed
   ```

### Horizontal pod autoscaling (HPA)

For high-traffic environments, enable HPA (requires Metrics Server):

```yaml theme={null}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: qwed-core-hpa
  namespace: qwed
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: qwed-core
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
```

> [!NOTE]
>
> Multi-replica audit writes are not supported. The audit logger keeps each organization's chain head in memory, so replicas sharing one database can make audit appends fail with `500 audit persistence failed`. See [Chain continuity across restarts](https://docs.qwedai.tech/advanced/compliance.md#chain-continuity-across-restarts).

***

## Manual / bare metal deployment

If you prefer to run the application directly on a host or VM:

### 1. Prerequisites

**Docker Installation (Required for Secure Code Execution)**
QWED requires Docker for all model-generated code execution in the Stats Engine and Consensus Engine. Without Docker, these verification endpoints return HTTP 503.

#### Linux (Ubuntu/Debian)

```bash theme={null}
sudo apt-get update
sudo apt-get install docker.io
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER
```

### 2. Python dependencies

```bash theme={null}
# Install QWED with the API server extra
pip install -e ".[server]"
```

The `server` extra installs the Postgres drivers, the Redis and Prometheus clients, the OpenTelemetry packages, and the `docker` client (`docker>=7.0.0`). The code executor imports the `docker` client when the server loads, so install the extra even if Docker is not running.

### 3. Database setup

Ensure PostgreSQL and Redis are running. Set the `DATABASE_URL` and `REDIS_URL` environment variables.

Initialize the database:

```bash theme={null}
python -c "from qwed_new.core.database import create_db_and_tables; create_db_and_tables()"
```

### 4. Running the API

```bash theme={null}
# Production Mode with Gunicorn (Linux/macOS)
gunicorn qwed_new.api.main:app -w 1 -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000
```

> [!WARNING]
>
> Run one worker per database. Every verification request appends an audit row, and the audit logger supports one writer per database. With several workers, each appends audit rows independently, and a conflicting append makes the verification request fail with `500 audit persistence failed`. See [Chain continuity across restarts](https://docs.qwedai.tech/advanced/compliance.md#chain-continuity-across-restarts).

***

## Environment variables reference

### Infrastructure

| Variable | Description | Default | Required |
| - | - | - | - |
| `DATABASE_URL` | Postgres connection string | `sqlite:///./qwed.db` | Yes (Prod) |
| `REDIS_URL` | Redis connection string | `redis://localhost:6379/0` | Yes (Prod) |
| `API_KEY_SECRET` | Secret key for signing JWTs | None (must be set) | **YES** |
| `QWED_CORS_ORIGINS` | Comma-separated list of allowed CORS origins | None (must be set) | **YES** |
| `QWED_SKIP_ENV_INTEGRITY_CHECK` | Set to `true` to bypass startup environment integrity check | `false` | No |
| `QWED_ALLOWED_STARTUP_PTH_FILES` | Comma-separated `.pth` startup hooks to allow at startup. A plain name (`my_hook.pth`) is allowed but still scanned for suspicious patterns. A digest pin (`my_hook.pth:sha256:<64 hex>`) is allowed only while the file's SHA-256 matches, and isn't scanned. A malformed pin stops the server at startup. See [Startup environment integrity](https://docs.qwedai.tech/advanced/security-hardening.md#llm05-supply-chain-vulnerabilities) | Unset | No |
| `QWED_API_KEY_LOOKUP_SECRET` | Secret for HMAC-SHA256 API-key lookup digests and agent token digests. Must differ from `QWED_JWT_SECRET_KEY`. Changing it invalidates existing API keys and agent tokens | None | **Yes** |
| `QWED_JWT_SECRET_KEY` | Secret for signing JWTs. Keep it stable: changing it invalidates issued tokens | None | **Yes** |
| `QWED_AUDIT_SECRET_KEY` | HMAC key that signs the per-organization audit hash chain. The server refuses to start without it. Rotating it invalidates the signatures on existing rows, so verify and archive the trail first. See [Audit trail setup](https://docs.qwedai.tech/advanced/compliance.md#3-audit-trail-setup) | None (must be set) | **Yes** |
| `QWED_SIGNING_KEY` | From v8.0.0. The ES256 attestation signing key as unencrypted EC P-256 PEM text. In an env file, write newlines as `\n`. Set this or `QWED_SIGNING_KEY_FILE`, not both. Callers verify attestations against the public key at `/.well-known/jwks.json`. See [Signing keys](https://docs.qwedai.tech/advanced/attestations.md#signing-keys) | Unset (a key is generated per process) | Recommended |
| `QWED_SIGNING_KEY_FILE` | From v8.0.0. Path to a file holding the signing key PEM, such as a mounted secret. Setting both key variables, or a key that is unreadable, not P-256, or encrypted, stops the server at startup | Unset | Recommended |
| `QWED_ATTESTATION_TTL_SECONDS` | From v8.0.0. How long an attestation stays valid, in seconds, from `60` to `2592000` (30 days). Any other value stops the server at startup | `86400` (24 hours) | No |
| `QWED_ENABLE_AUDIT_SEED` | Set to `1` to enable `POST /audit/seed`, which writes sample rows into the signed audit trail. Development only. Leave unset in production | Unset (route returns `404`) | No |
| `QWED_METRICS_OPERATOR_USER_IDS` | Comma-separated user IDs allowed to read `/metrics`. Empty denies every caller | None | No |
| `UVICORN_HOST` | Bind address of the API server in the Docker image | `0.0.0.0` | No |
| `UVICORN_PORT` | Port of the API server in the Docker image. The image's health check follows it | `8000` | No |

> [!WARNING]
>
> `API_KEY_SECRET` no longer has a default value. The server will refuse to start if it is not set. This prevents accidental deployment with a weak default secret.

### Security configuration

| Variable | Description | Default |
| - | - | - |
| `MAX_INPUT_LENGTH` | Max query length (chars) | 2000 |
| `SIMILARITY_THRESHOLD` | Prompt injection detection threshold | 0.6 |
| `DOCKER_TIMEOUT` | Code execution timeout (seconds) | 10 |
| `DOCKER_MEMORY_LIMIT` | Container memory limit | 512m |
| `QWED_SANDBOX_IMAGE` | From v8.0.0. Alternative image for the stats and consensus code sandbox. QWED accepts it only when pinned by digest (`name@sha256:<64 hex>`). Any other value is refused, logged, and replaced by the pinned default image | `amancevice/pandas@sha256:9257b889…` |
| `QWED_SANDBOX_UID_RANGE` | From v8.0.0. Operator-reserved uid range (`FIRST-LAST`, within `65536`–`2147483646`) that QWED hands sandbox workspaces to when the server process runs as root. An invalid value is logged and the default is used | `1879048192-2147483646` |
| `QWED_CROSSHAIR_IMAGE` | From v8.0.0. Local Docker image the Symbolic engine runs CrossHair in. Build it from `docker/crosshair-sandbox/Dockerfile`. QWED never pulls it at runtime | `qwed-crosshair-sandbox:0.0.101` |
| `QWED_SKIP_ENV_INTEGRITY_CHECK` | Skip `.pth` file verification on startup | `false` |

### Maths limits

From v9.0.0, `/verify/math` and the maths items of `/verify/batch` run SymPy in a pool of worker processes. These settings size that pool. Unset or blank, each keeps its default.

| Variable | Description | Default |
| - | - | - |
| `QWED_MATH_WORKERS` | Worker processes. A whole number, at least `1` | `2` |
| `QWED_MATH_TIMEOUT_SECONDS` | Time limit in seconds, from `0.001` to `3600`. It applies separately to the wait for a free worker and to the run, so a job that has to wait can take up to twice this. A job that waits longer is `BLOCKED` (`math.capacity`). A job that runs longer is `UNVERIFIABLE` (`math.timeout`), and its worker is killed and replaced. Starting a new worker is not counted: the job waits up to 120 seconds for it, and a worker that does not start is `BLOCKED` (`math.worker_error`) | `10` |
| `QWED_MATH_MAX_WAITING` | Jobs that may wait for a free worker. A whole number, at least `0`. A job that finds every worker busy and the queue full is `BLOCKED` (`math.capacity`) at once | `32` |
| `QWED_MATH_WORKER_MEMORY_MB` | Memory limit of each worker in MB, on Linux only. A whole number, at least `0`. `0` turns the limit off. A job that runs out of memory is `BLOCKED` (`math.out_of_memory`) | `2048` |
| `QWED_BATCH_MATH_BUDGET_SECONDS` | Time budget in seconds for the maths items of one batch job, at least `0`. It is wall-clock time from when the job starts, so time spent on the job's other items counts too. Each maths item keeps the time limit above, cut to what is left of the budget when it reaches a worker. Starting a new worker isn't counted against the item's own limit, which is set before the start, so that item can finish after the budget ran out and return its result. The start still uses the job's wall-clock budget, so later items can get `batch.math_budget`, and it counts toward the batch item limit `QWED_BATCH_ITEM_TIMEOUT` (60 seconds by default), so a slow start can fail the item. An item reached with no budget left is not run (`batch.math_budget`). `0` means no maths item of a batch runs | `120` |

> [!WARNING]
>
> A value these settings cannot use stops the server at startup with an error naming the setting. That includes a value that is not a number, `inf` or `nan`, a value below its minimum, a fraction for a count, and a time limit over 3600 seconds. QWED never puts another value in its place.

From v9.1.0, the server also reads these settings from `.env` (the working directory's, then `~/.qwed/.env`), and a value set in the process environment wins. In v9.0.0, a server started with `uvicorn qwed_new.api.main:app` read the four `QWED_MATH_*` settings before `.env` loaded: a value only in `.env` was ignored, and one the server couldn't use made every maths job `BLOCKED` (`math.worker_error`) instead of stopping the start. Before you upgrade to v9.1.0, check `.env` for maths settings the process environment doesn't set: the server now uses them, and one it can't use stops the start.

From v9.1.0, `deploy/.env.example` and `deploy/kubernetes/configmap.yaml` list the five settings, commented out. `deploy/docker-compose.yml` passes them to the server by name, so one you don't set stays unset and keeps its default. Size the pool for your load.

### AI providers

| Variable | Description | Default |
| - | - | - |
| `ACTIVE_PROVIDER` | Selected LLM provider: `openai`, `openai_direct`, `openai_compat`, `azure_openai`, `anthropic`, `claude_opus`, `gemini`, `ollama`, or `auto`. Any other value stops each request that needs a provider with an error naming `ACTIVE_PROVIDER` | `ollama` |
| `AZURE_OPENAI_API_KEY` | API key for Azure OpenAI | - |
| `AZURE_OPENAI_ENDPOINT` | Endpoint URL | - |
| `AZURE_OPENAI_DEPLOYMENT` | Azure OpenAI deployment to run. Required for Azure OpenAI | - |
| `ANTHROPIC_API_KEY` | API key for Anthropic | - |
| `ANTHROPIC_MODEL` | Anthropic model to run. `ANTHROPIC_DEPLOYMENT` wins when set. Required for Anthropic | - |
| `GOOGLE_API_KEY` | API key for Google Gemini (or `GEMINI_API_KEY`) | - |
| `GEMINI_MODEL` | Gemini model to run. Required for Gemini | - |

From v9.0.0, QWED names no default model. Set the model variable of the provider you use (`OPENAI_MODEL`, `AZURE_OPENAI_DEPLOYMENT`, `ANTHROPIC_MODEL`, `CLAUDE_OPUS_DEPLOYMENT`, `GEMINI_MODEL`, `OLLAMA_MODEL`, or `CUSTOM_MODEL`). The server still starts without one, because it builds a provider on first use. Logic, statistics, and fact requests that need the provider then fail with an error that names the variable. The natural-language routes answer `UNVERIFIABLE`. See [LLM configuration](https://docs.qwedai.tech/getting-started/llm-configuration.md#supported-llm-providers).

### Observability

| Variable | Description | Default |
| - | - | - |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Jaeger/OTLP Endpoint | `http://localhost:4317` |

***

## Production checklist

Before going to production, ensure the following:

### 1. Database setup

* [ ] Use a managed PostgreSQL instance (e.g., AWS RDS, Azure Database for PostgreSQL).
* [ ] Enable automated backups.
* [ ] Run database migrations.

### 2. Redis configuration

* [ ] Use a managed Redis instance (e.g., AWS ElastiCache).
* [ ] Configure eviction policy (LRU).
* [ ] Enable persistence (RDB/AOF).

### 3. Security

* [ ] **Set `API_KEY_SECRET`**: This is mandatory — generate a secure value with `python -c "import secrets; print(secrets.token_urlsafe(48))"`.
* [ ] **Set `QWED_CORS_ORIGINS`**: Explicitly list your allowed origins (e.g., `https://app.yourcompany.com`). The server will not start without this.
* [ ] **Set `QWED_AUDIT_SECRET_KEY`**: Required. The server will not start without it. Store it in a secrets manager and don't rotate it without archiving the existing audit trail.
* [ ] **Leave `QWED_ENABLE_AUDIT_SEED` unset**: Keeps `POST /audit/seed` disabled so sample rows never enter the production audit trail.
* [ ] **Rotate Keys**: Change all default passwords and secrets.
* [ ] **SSL/TLS**: Ensure the API is behind a Load Balancer with a valid SSL certificate.
* [ ] **Network Policies**: Restrict access to database/Redis.
* [ ] **Docker Security**: Ensure the Docker socket is protected or use a secure container runtime (gVisor) for the Stats Verification engine if possible.

### 4. Observability

* [ ] Configure alert rules in Prometheus/Grafana.
* [ ] Ensure logs are shipped to a centralized logging system.

***

## Troubleshooting

### 1. Docker permission denied

**Error:** `docker: Got permission denied while trying to connect to the Docker daemon socket`
**Solution:** Ensure the user running the app is in the `docker` group, or (for Docker Compose) ensure the socket is mounted correctly and the container user has permissions.

### 2. Stats or consensus verification returning 503

**Error:** `Service temporarily unavailable` on `/verify/stats` or `/verify/consensus`
**Cause:** The secure Docker sandbox is unreachable. QWED does not fall back to in-process execution.
**Solution:**

1. Check Docker is running: `docker ps`
2. Verify the Docker daemon responds to pings: `docker info`
3. Check the pinned sandbox image exists: `docker pull amancevice/pandas@sha256:9257b88909bddf60c8a85b06d572fc6dfc708ea9c8a3c3dc2149e312d57f91e3` (or the digest you set in `QWED_SANDBOX_IMAGE`).
4. Check the daemon's user mapping. QWED reports a `userns-remap` daemon, a rootless daemon used by a root server process, or a daemon whose security options QWED can't read as `SECURE_RUNTIME_UNAVAILABLE`. Run QWED as a non-root user with a rootless daemon that the same user runs, or as any user with a default (non-remapped) daemon.

***

## Operations and monitoring

### View recent security events

```bash theme={null}
# Python script
python -c "
from qwed_new.core.database import get_session
from qwed_new.core.models import SecurityEvent
from sqlmodel import select

with get_session() as session:
    events = session.exec(
        select(SecurityEvent)
        .where(SecurityEvent.event_type == 'BLOCKED')
        .order_by(SecurityEvent.timestamp.desc())
        .limit(10)
    ).all()
    for e in events:
        print(f'{e.timestamp}: {e.reason}')
"
```

### Security metrics dashboard

```bash theme={null}
# Block rate by hour
sqlite3 qwed_v2.db "
SELECT strftime('%Y-%m-%d %H:00', timestamp) as hour, COUNT(*) as blocks
FROM security_event
WHERE event_type = 'BLOCKED'
GROUP BY hour
ORDER BY hour DESC
LIMIT 24;
"
```

For detailed architecture documentation, see:

* [`Architecture`](https://docs.qwedai.tech/architecture)


## Related topics

- [QWED + Ollama integration guide](https://docs.qwedai.tech/advanced/ollama.md)
- [QWED production deployment checklist](https://docs.qwedai.tech/integration/production.md)
- [QWED A2A deployment for zero-trust agent gateways](https://docs.qwedai.tech/a2a/deployment.md)
- [QWED compliance guide](https://docs.qwedai.tech/advanced/compliance.md)
- [QWED integration guide](https://docs.qwedai.tech/advanced/integration-guide.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.