Container Image Delivery

DynamoAI worker images are large, and the delivery path for them is the most common cause of a stalled first deployment or upgrade. Work through this page after the Environment Dependencies checks and before Deploy.

The short version: host DynamoAI images in a registry that stores them, size nodes for a multi-gigabyte image, and pre-pull for workers that scale from zero.

Image Characteristics

Evaluation worker images are the largest the platform ships, and one layer dominates every transfer:

Property	Typical value
Compressed size	Above 5 GiB
Largest single layer	Above 4 GiB
Extracted size on disk	Roughly 1.5 to 2 times compressed
Layer reuse between releases	Minimal

Two consequences follow, and both are easy to plan for and expensive to discover late:

1. Pull time is bounded by one layer. A single layer above 4 GiB cannot be split across connections, so additional bandwidth and parallelism do not shorten it.
2. An upgrade transfers close to a whole image. Budget the full image size for every version change rather than a delta, including patch releases.

Registry

The Container registry and artifact access requirements are defined in the dependencies registry. This section covers the delivery behaviour that requirement implies.

Registry Readiness0/4

• Images are copied into a registry you host, not proxied on demandA pull-through or remote proxy repository fetches from upstream on a cache miss. A single layer above 4 GiB can exceed the proxy socket timeout before any bytes reach the node, and the pull fails with context canceled, which reads as a network fault but is a timeout.
• If a proxy repository is unavoidable, its timeouts are raised firstSee the settings table below, and check any reverse proxy or load balancer in front of the registry as well.
• A registry administrator is identified and reachableProxy and timeout settings require administrative access, which deployment teams do not always hold.
• Every required image and tag is present before the upgrade windowVerify by digest, not by the tag appearing in a registry UI. A cached manifest is a few kilobytes and caches instantly; it does not mean the layers are cached.

Where a proxying remote repository cannot be avoided, raise its timeouts before validating. In JFrog Artifactory these live on the remote repository under Advanced:

Setting	Default	Needed
Store Artifacts Locally	Enabled	Must stay enabled, or nothing ever caches
Socket Timeout (ms)	15000	Raise to 300000; 15 seconds cannot move a 4 GiB layer
Missed Retrieval Cache Period (Sec)	1800	Caches negative lookups; set to 0 while validating
Unused Artifacts Cleanup Period (Hr)	0	A non-zero value evicts cached layers and the pull is paid again

A reverse proxy or load balancer in front of the registry has its own proxy_read_timeout and proxy_send_timeout, and those override the repository setting.

Node Capacity

Node Capacity0/3

• Nodes have headroom for the extracted image plus working filesBudget 10 to 15 GiB of ephemeral storage per node for the image alone, on top of what the pods request.
• Worker ephemeral-storage requests are accounted for in node sizingWorker pods request ephemeral storage in their own right, commonly 20 GiB each, which limits how many fit on a node regardless of actual usage.
• DiskPressure is clear on the target node poolA node that cannot hold the extracted image fails the pull after downloading it in full.

Version Upgrades

Version Upgrades0/3

• The full image size is budgeted for every upgrade, not a deltaLayer reuse between releases is minimal, so a patch release is not a reason to expect a fast pull.
• The digest is bumped whenever the tag is bumpedIf values pin both, the rendered reference is repository:tag@digest, and container runtimes resolve by digest and ignore the tag. Update both together, or pin only the tag.
• The running image is confirmed after the upgrade, not the declared oneA partial upgrade, with the API on the new release and workers still on the old one, is difficult to attribute from the UI.

List every DynamoAI image running in the namespace

kubectl -n <namespace> get deploy,statefulset,scaledjob -o json \
  | grep -oE '"image": "[^"]*"' | sort -u

Workers That Scale From Zero

Evaluation workers run as KEDA ScaledJobs with minReplicaCount: 0. A node with no cached layers pays the full pull before the first job runs, and node replacement or autoscaling resets that cost.

Scale-From-Zero Readiness0/3

• The worker image is pre-pulled onto the worker node poolA DaemonSet whose init container references the image places the layers on every node ahead of time, removing both first-test latency and the context canceled failure mode.
• Only one pull is in flight per image during validationConcurrent cold pulls of the same large layer split available bandwidth and make each more likely to time out.
• maxReplicaCount reflects real cluster capacityKEDA creates a pod per queued job regardless of whether the cluster can run it. A high ceiling against a small node pool produces a queue of Pending pods rather than throughput.

Pause the ScaledJob while warming the cache, and resume it afterwards:

Pause a ScaledJob during cache warming

kubectl -n <namespace> annotate scaledjob <scaledjob-name> \
  autoscaling.keda.sh/paused="true" --overwrite

Network

Network0/3

• Nodes can reach the registry endpointThe Docker registry hostname is often a different host from the registry web UI and REST API. Reaching one does not prove the other is reachable.
• Image pull credentials are configured, or the repository genuinely allows anonymous pullsA registry can allow anonymous Docker pulls while still requiring authentication on its REST API.
• Egress restrictions are documented before installAir-gapped and allow-listed clusters are supported, but components that reach external services must be identified in advance.

Verify Before Declaring the Environment Ready

Pull the worker image onto a node with a plain pod and confirm it completes:

prepull-check.yaml

apiVersion: v1
kind: Pod
metadata:
  name: prepull-check
  namespace: <namespace>
spec:
  restartPolicy: Never
  containers:
    - name: hold
      image: <registry>/<project>/images/pen-testing:<tag>
      command: ["sleep", "600"]
      resources:
        requests: { cpu: "100m", memory: "256Mi" }

Apply the check and read its events

kubectl apply -f prepull-check.yaml
kubectl -n <namespace> describe pod prepull-check | tail -20

A Successfully pulled image ... in Xm event is the pass condition. Use a plain pod rather than a Job or ScaledJob so that nothing deletes it mid-pull: a deletion cancels the transfer and reports context canceled, which looks identical to a timeout.

To confirm that bytes are moving, without needing a privileged pod:

Sample image filesystem usage on the node

kubectl get --raw "/api/v1/nodes/<node-name>/proxy/stats/summary" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['node']['runtime']['imageFs']['usedBytes']/1024**3, 'GiB')"

Sample it twice, a minute apart. A flat value while a pull is in flight means the transfer is stalled, and the problem is the registry rather than the cluster.

Symptoms that appear after deployment, including context canceled on worker image pulls, are covered in Troubleshooting DynamoEval Workers.