# Kubernetes

Deploy Sandbox sandboxes with the Quor Helm chart

Use the Quor Helm chart to deploy Sandbox with Kubernetes sandboxes. The chart creates the sandbox namespace, PodTemplate,
RBAC, egress proxy, NetworkPolicies, and Scheduled Task worker used by Sandbox.

## Requirements

- Kubernetes 1.33 or later
- A full Quor Helm deployment with the vector database enabled
- Nodes with enough CPU, memory, and ephemeral storage for active sandboxes
- A URL that sandbox pods can use to reach the Quor API

The chart fails the install or upgrade when Sandbox is enabled on an older Kubernetes version,
when the sandbox backend is not `kubernetes`, when `SANDBOX_API_SERVER_URL` is empty,
or when sandbox push authentication is not configured.

## Prepare sandbox nodes

By default, sandbox pods select nodes with this label:

```text
quor.app/workload=sandbox
```

Label each node that can run sandboxes:

```bash
kubectl label node <node-name> quor.app/workload=sandbox
```

The default sandbox toleration also supports nodes dedicated with this taint:

```bash
kubectl taint node <node-name> workload=sandbox:NoSchedule
```

The taint is optional.
The node label is required unless you replace `sandboxPod.nodeSelector` with a selector that matches your cluster.

## Create the sandbox push Secret

Sandbox uses an Ed25519 key to authenticate file and history pushes into sandbox pods.
Generate a key and store it in the Quor namespace:

```bash
SANDBOX_PUSH_PRIVATE_KEY="$(python3 -c 'import base64; from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey; from cryptography.hazmat.primitives.serialization import Encoding, NoEncryption, PrivateFormat; key = Ed25519PrivateKey.generate(); print(base64.b64encode(key.private_bytes(Encoding.Raw, PrivateFormat.Raw, NoEncryption())).decode())')"

kubectl create namespace onyx --dry-run=client -o yaml | kubectl apply -f -
kubectl -n onyx create secret generic onyx-sandbox-push-secret \
  --from-literal=private_key="$SANDBOX_PUSH_PRIVATE_KEY"
unset SANDBOX_PUSH_PRIVATE_KEY
```

Keep this Secret stable across upgrades.
Replacing the key while sandboxes are running interrupts authenticated pushes until the affected components and
sandboxes are recreated.

## Configure Helm

Add the following settings to the values file used by your existing Quor deployment.
If `configMap` or `auth` is already present,
merge these entries under the existing keys rather than adding a second block.

```yaml values.yaml
configMap:
  ENABLE_CRAFT: "true"
  SANDBOX_BACKEND: "kubernetes"
  SANDBOX_API_SERVER_URL: "https://onyx.example.com"

auth:
  sandboxPushSecret:
    enabled: true
    existingSecret: "onyx-sandbox-push-secret"
```

`SANDBOX_API_SERVER_URL` can be the public Quor URL or a cluster-internal URL.
Use a scheme and hostname that resolve from the sandbox proxy and route to the Quor API.

Install or upgrade Quor with your complete deployment values:

```bash
helm upgrade --install onyx onyx/onyx \
  --namespace onyx \
  --create-namespace \
  --values onyx-values.yaml
```

Replace `onyx-values.yaml` with the path to the values file used by your deployment.

Alternatively, save only the Sandbox settings in a separate `craft-values.yaml` overlay.
Pass your existing deployment values first and the Sandbox overlay second so Helm combines them:

```bash
helm upgrade --install onyx onyx/onyx \
  --namespace onyx \
  --create-namespace \
  --values onyx-values.yaml \
  --values craft-values.yaml
```

## Plan sandbox capacity

Each active user receives a sandbox pod. The main sandbox container has these defaults:

| Resource | Request | Limit |
| --- | --- | --- |
| CPU | `1000m` | `2000m` |
| Memory | `2Gi` | `10Gi` |
| Ephemeral storage | `5Gi` | `20Gi` |

The pod also includes initialization and sidecar containers for network setup, file transfer, snapshots, and restore.
To change the main container resources, merge these `configMap` entries into your deployment values or Sandbox overlay:

```yaml
configMap:
  SANDBOX_POD_CPU_REQUEST: "1000m"
  SANDBOX_POD_CPU_LIMIT: "2000m"
  SANDBOX_POD_MEMORY_REQUEST: "2Gi"
  SANDBOX_POD_MEMORY_LIMIT: "10Gi"
  SANDBOX_POD_EPHEMERAL_STORAGE_REQUEST: "5Gi"
  SANDBOX_POD_EPHEMERAL_STORAGE_LIMIT: "20Gi"
```

Use `sandboxPod.nodeSelector`, `sandboxPod.tolerations`, and `sandboxPod.affinity` to control placement.
The proxy and Scheduled Task worker have separate resource settings under `sandboxProxy.resources` and
`celery_worker_scheduled_tasks.resources`.

## Verify the deployment

Confirm the proxy, sandbox template, and Scheduled Task worker exist:

```bash
kubectl -n onyx get deployment -l app.kubernetes.io/component=sandbox-proxy
kubectl -n onyx-sandboxes get podtemplate sandbox-pod
kubectl -n onyx get deployment -l app=celery-worker-scheduled-tasks
```

After configuring a model and user access, send a prompt in Sandbox. A sandbox pod should appear:

```bash
kubectl -n onyx-sandboxes get pods -l app.kubernetes.io/component=sandbox
```

## Network configuration

Sandbox pods can send traffic only to DNS and the sandbox proxy. The proxy handles external requests, App policies,
and credential injection.

Most clusters work with the chart defaults. Two cluster layouts require additional values:

| Cluster layout | Configuration |
| --- | --- |
| NodeLocal DNS or another non-pod DNS listener | Add its CIDR to `craft.dnsExtraCIDRs`. |
| An external or unlabeled sandbox proxy | Add the proxy CIDR to `craft.proxyExtraCIDRs`. |
| Dual-stack networking | Set `sandboxProxy.egressAllowIPv6: true`. |

<Warning>
  Keep the sandbox egress NetworkPolicies enabled. They restrict sandbox pods to the proxy and DNS,
  and block link-local cloud metadata access from the proxy path.
</Warning>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Helm rejects the Sandbox configuration">
    Read the render error first. Confirm Kubernetes 1.33 or later, `SANDBOX_BACKEND: "kubernetes"`,
    a non-empty `SANDBOX_API_SERVER_URL`, and `auth.sandboxPushSecret.enabled: true` with a populated Secret.
  </Accordion>

  <Accordion title="Sandbox pods remain Pending">
    Run `kubectl -n onyx-sandboxes describe pod <pod-name>`.
    Confirm at least one node matches `sandboxPod.nodeSelector`, accepts the configured taints and tolerations,
    and has enough CPU, memory, and ephemeral storage.
  </Accordion>

  <Accordion title="A sandbox fails during initialization">
    Inspect the `sandbox-init` container logs and the proxy pods.
    DNS must resolve the proxy before the sandbox firewall is installed.
    Clusters using NodeLocal DNS usually need its listener CIDR under `craft.dnsExtraCIDRs`.
  </Accordion>

  <Accordion title="Sandbox cannot reach the Quor API">
    Verify `SANDBOX_API_SERVER_URL`, DNS, TLS trust, and routing from the proxy.
    Do not point it at a Service name from another cluster or an address the proxy cannot resolve.
  </Accordion>

  <Accordion title="The sandbox proxy is not Ready">
    Run `kubectl -n onyx logs -l app.kubernetes.io/component=sandbox-proxy --tail=200` and check its access to
    PostgreSQL, Redis, the Quor API, and external destinations. On dual-stack clusters,
    enable `sandboxProxy.egressAllowIPv6`.
  </Accordion>
</AccordionGroup>

<CardGroup cols={2}>
  <Card title="Sandbox deployment overview" icon="hammer" href="/deployment/local/craft">
    Compare the Kubernetes and Docker Compose paths.
  </Card>

  <Card title="Managing Sandbox" icon="sliders" href="/admins/managing_features/craft">
    Configure models and user access after deployment.
  </Card>
</CardGroup>