#Container Provider: Docker vs Kubernetes

The A&D and KotH engine runs on top of GZ::CTF's existing container abstraction (IContainerManager + IContainerProvider), so it works on either backend. But the two providers are not feature-equal: the Docker path delivers the full A&D toolkit (precise filesystem diffs, image snapshots, in-place container moves, the KotH leader-cooldown), while the Kubernetes path is a leaner, exec-light implementation built to survive on locked-down nodes. This page explains how to choose a backend, exactly what each one gives the A&D engine, and the operational limits you'll hit at scale.

#Choosing the provider

The backend is selected by a single config key — ContainerProvider.Type in compose/appsettings.json — a string enum with two values, Docker and Kubernetes. If you leave it at the shipped default you get Docker: the provider/manager pair is bound once at process start and never both, so there is no runtime fallback from one to the other.

Both deploy paths live in the platform template. Pick one before first boot:

• Docker — the default. The wizard renders compose/appsettings.json with ContainerProvider.Type already set to Docker, and make platform-up brings up gzctf, the DB, cache, Traefik, WireGuard, and ssh-jump from the published image. This is the path the rest of the docs assume.
• Kubernetes — apply the manifests under the template's k8s/ directory in order (00-namespace → 10-postgres → 20-redis → 30-gzctf-config → 40-gzctf → 50-ingress), then kubectl -n gzctf rollout status deploy/gzctf. The appsettings.json baked into 30-gzctf-config.yaml already sets ContainerProvider.Type to Kubernetes. See k8s/README.md in the template for the full walkthrough.

The config is plain JSON (the template puts every setting in compose/appsettings.json, rendered from compose/appsettings.example.json). The Docker default looks like this:

// compose/appsettings.json
"ContainerProvider": {
  "Type": "Docker",
  "PortMappingType": "PlatformProxy",
  "EnableTrafficCapture": true,
  "PublicEntry": "PUBLIC_ENTRY",
  "DockerConfig": {
    "SwarmMode": false,
    "ChallengeNetwork": "challenges",
    "Uri": "unix:///var/run/docker.sock"
  },
  "KubernetesConfig": {
    "Namespace": "gzctf-challenges",
    "AllowCIDR": [ "10.0.0.0/8" ],
    "DNS": [ "8.8.8.8", "1.1.1.1" ]
  }
}

The KubernetesConfig block is read only when Type is Kubernetes; the DockerConfig block only when Type is Docker. To run on Kubernetes you flip Type to "Kubernetes" (already done for you in k8s/30-gzctf-config.yaml).

See /config/appsettings for the full ContainerProvider, DockerConfig, and KubernetesConfig schemas.

#What A&D gets on Docker

Docker is the reference backend and the one the template's compose/ path runs by default. The provider (DockerProvider) eagerly creates the challenge bridges at boot via EnsureNetworkCreated().

#Networks ("per-team" bridges via Open / Isolated modes)

DockerProvider derives bridge names from DockerConfig.ChallengeNetwork (default prefix gzctf, set to challenges here) and one bridge per NetworkMode:

{prefix}-open       bridge, ip_masquerade on   → can reach the internet
{prefix}-isolated   bridge, ip_masquerade off  → no NAT, cannot reach the internet
{prefix}-custom     attached only if it already exists

A&D and KotH containers land on the open or isolated bridge according to the challenge's AdAllowEgress flag (in LaunchOneAsync / LaunchKothTargetAsync):

var networkMode = challenge.AdAllowEgress ? NetworkMode.Open : NetworkMode.Isolated;

The isolated bridge is created with com.docker.network.bridge.enable_ip_masquerade=false so isolated challenges have no NAT path off-host. Note that gzctf attaches itself to the isolated bridge with GwPriority = -100 so Docker never promotes that no-NAT gateway to gzctf's own default route (which would silently break gzctf's outbound git clones, mirrors, and webhooks).

#/flag read-only bind mount

A&D flags rotate every tick, so the engine never bakes the flag into an env var (a baked GZCTF_FLAG would freeze at create time and go stale after the first rotation). Instead the flag is a host-backed file bind-mounted read-only at /flag.

AdFlagMountService keeps one host file per (participationId, challengeId) under /app/ad-flags (its in-container view), resolves the corresponding host path, warms up the file (Docker bind-mounts a missing source as an empty directory, which would break /flag), and hands the path to DockerManager.GetCreateContainerParameters as a read-only mount:

Mounts = new List<Mount>
{
    new()
    {
        Type = "bind",
        Source = config.FlagBindSource,   // host path of the per-team flag file
        Target = config.FlagFilePath ?? "/flag",
        ReadOnly = true
    }
}

The read-only mount returns EROFS on write/unlink even to container-root (CAP_DAC_OVERRIDE does not bypass a mount-layer block, and remount/unmount needs CAP_SYS_ADMIN, dropped by default). gzctf rewrites the file in place each tick and the container sees the new flag through the shared inode. The in-container path is surfaced as GZCTF_FLAG_FILE so challenge code reads the live flag from there.

KotH hills get no flag mount — teams plant their own rotating token into the /koth/king marker, so there is no platform-planted flag to deliver.

#Precise live diff + snapshot tarballs

Because Docker exposes a layer-diff API, the "what did this team change" view is exact. ComputeLiveChangesAsync calls docker diff (InspectChangesAsync) against the baseline image, then:

• collapses ancestor directories to the changed leaf files (Docker reports every parent dir of a change), and
• filters runtime/churn noise via IsNoiseChangePath — the flag mount, __pycache__, /tmp, /run, /var/log, /proc, /sys, /dev, etc.

This gives add/modify/delete classification on the running container. At game end (when Game.AdAllowSnapshotDownload is set), TrySnapshotAsync additionally:

1. captures the docker diff into SnapshotChanges,
2. docker commits the container to an image,
3. docker saves it, gzips it, and uploads the tarball to blob storage at ad-snapshots/{gameId}/{participationId}-{challengeId}.tar.gz,
4. deletes the local image (the tarball is the deliverable).

File-level inspection (ReadCurrentFileBytesAsync) reads bytes straight from the container filesystem via the daemon's tar archive API — no shell or coreutils needed in the image.

#In-place retain-restart and network moves

On Docker the engine can change a container's network without destroying it. When an operator toggles AdAllowEgress mid-game, TryMoveContainerNetworkAsync connects the container to the new bridge, disconnects the old one (connect-before-disconnect so it's never on zero networks), re-reads the new IP, and the team keeps all its patches. A full recreate happens only if the move fails.

The KotH leader-cooldown is also Docker-only: at each refresh boundary the recent leader's VPN /32s are dropped to the hill IP via iptables in the WireGuard sidecar's netns (per-hill chain KOTH_CD_<challengeId>).

#What A&D gets on Kubernetes

The K8s path (KubernetesManager / KubernetesProvider) is deliberately exec-light so it can run on managed/virtual nodes that forbid exec, initContainers, and subPath. It launches the same challenge images and the marker-based scoring works, but flag delivery, isolation, and inspection are implemented differently.

#Flag-pull sidecar → /gzctf-flag/flag

There is no bind mount. Instead K8s uses a pull model: a tiny busybox:stable sidecar (gzctf-flag-writer) polls a flag URL every 5 s and writes the flag into a shared emptyDir, which the challenge container mounts read-only at the flag directory.

AdContainerManager.LaunchOneAsync builds the pull URL from Ad:FlagPullBaseUrl and the per-pod token:

flagFilePath = "/gzctf-flag/flag";
var baseUrl = config["Ad:FlagPullBaseUrl"]?.TrimEnd('/');
// ...
flagPullUrl =
    $"{baseUrl}/api/Game/{gameId}/Ad/PodFlag/{participationId}/{challengeId}/{podToken}";

KubernetesManager then renders the writer sidecar alongside the challenge:

# challenge container mounts the shared volume read-only at the flag's *directory*
volumeMounts:
  - name: ad-flag
    mountPath: /gzctf-flag
    readOnly: true
---
# sidecar (gzctf-flag-writer) re-pulls every 5s into the shared emptyDir
containers:
  - name: gzctf-flag-writer
    image: busybox:stable
    command: ["sh", "-c", "while true; do wget -qO /flagdir/flag \"$GZCTF_FLAG_URL\"; sleep 5; done"]
    env:
      - name: GZCTF_FLAG_URL
        value: "http://<ip>:8080/api/Game/.../Ad/PodFlag/..."
    volumeMounts:
      - name: ad-flag
        mountPath: /flagdir
volumes:
  - name: ad-flag
    emptyDir: {}

There are no initContainers (virtual nodes don't support them) — the sidecar seeds the flag on its first poll, so the challenge may briefly see a missing file at startup. On real nodes the read-only mount protects the file; on virtual nodes that don't enforce readOnly, container-root can delete it, but the next poll (≤ 5 s) re-plants it.

if (Uri.TryCreate(configuration["Ad:FlagPullBaseUrl"], UriKind.Absolute, out var fp)
    && System.Net.IPAddress.TryParse(fp.Host, out _))
{
    _flagPullHost = fp.Host;
    _flagPullPort = fp.Port;
}

#Egress-style challenges only; live-diff is an mtime heuristic

The K8s path is best suited to egress-style challenges (network services reachable over the pod network). Because there is no layer-diff API:

• Live diff (ComputeLiveChangesAsync, K8s branch) execs find / -xdev … -newer /proc/1 -type f in the pod — an mtime heuristic that lists files modified since the container started. It has no add/modify/delete classification and misses deletions.
• Snapshots are Docker-only. TrySnapshotAsync returns null (logged) when no Docker provider is registered; only SnapshotChanges (the filtered file list) is captured at game end, not an image tarball.

#Retain-restart limitation

#Egress isolation

Both providers expose the same two network modes per challenge, driven by AdAllowEgress:

The key point is that open is not "wide open." The open default still denies the private + link-local ranges defined once in AdEgressBaseline.PrivateAndLinkLocal and shared by both providers:

public static readonly string[] PrivateAndLinkLocal =
    ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "169.254.0.0/16"];

169.254.0.0/16 is the important one — it blocks the cloud metadata service, so an RCE/SSRF challenge can't steal node IAM credentials. The other three are RFC1918 (cluster pod/service + private LAN).

• Kubernetes enforces this with two egress NetworkPolicies selected by the pod label gzctf.gzti.me/NetworkMode. The open policy allows 0.0.0.0/0 except the baseline + auto-detected node/control-plane addresses + any operator AllowCidr; the isolated policy denies all egress except the flag-pull endpoint and cluster DNS. AllowCidr augments the baseline — it can never re-open the private ranges. (A startup probe, VerifyNetworkPolicy, confirms the CNI actually enforces policy.)
• Docker has no egress-policy API, so AdEgressIsolationService installs equivalent DOCKER-USER rules (chain GZCTF_AD_ISO) keyed on an ipset of the live challenge-container IPs from the DB. It drops challenge→challenge (the lateral pivot) and challenge→169.254/16, 10/8, 172.16/12, 192.168/16, while leaving the checker, the WG sidecar, and container→internet allowed. It re-applies every 30 s (and on-demand right after a launch/move) so it survives a dockerd restart that flushes DOCKER-USER. Toggle with DockerConfig.EnforceEgressIsolation (default true).

The result is the same containment guarantee on both backends: a fully-popped challenge container cannot pivot team→team or reach cloud metadata, while flag delivery and the intended checker/VPN paths keep working.

#Capacity guidance

On Docker, A&D scale is bounded by the shared challenge bridges:

Because every A&D challenge spawns one container per accepted team (KotH spawns one shared hill per challenge), container count grows as teams × A&D-challenges. The first wall you hit is usually IP exhaustion on a /24 egress bridge (~254), then the ~1023-interface bridge limit.

To scale past these, shard the bridges: split challenges across multiple challenge networks (a wider subnet per bridge raises the ~254 IP ceiling; multiple bridges raise the ~1023-interface ceiling), or move to Kubernetes where pods are spread across nodes and the per-bridge interface limit doesn't apply. Per-container resource limits (CPUCount, MemoryLimit, StorageLimit) are set per challenge — see /guide/authoring/challenge-yaml. Provider-level network and capacity knobs (ChallengeNetwork, AllowCidr, EnforceEgressIsolation, Ad:FlagPullBaseUrl) are documented in /config/appsettings; scoring behaviour that drives container churn (resets, SLA ticks) is covered in /guide/features/scoring.