Skip to main content

Object Storage Availability

Scope

Applies to every DynamoAI product. Covers the bundled object store only; a managed S3-compatible store is the responsibility of its provider.

This page covers making the in-cluster object store redundant, so that losing a pod, a node, or an availability zone does not take object storage down.

It applies only to deployments using the bundled in-cluster object store, the In-cluster MinIO pattern in Model Storage. If you use an external object store or cloud object storage, availability is a property of that service and this page does not apply.

Availability of other platform components is configured separately and is not covered here.

Single-instance mode has no redundancy​

By default the bundled object store runs in single-instance mode: one pod, one PersistentVolume. There is no erasure coding and no second copy.

This means:

  • Any pod restart, node drain, eviction, or cluster upgrade is a full object-storage outage for its duration.
  • Locally redundant block storage is zonal. The volume is bound to one availability zone and the pod cannot be rescheduled outside it.
  • Cloud block storage is also typically single-attach. A forced node move requires a detach and reattach cycle, during which the service has no endpoints.

Guardrail and evaluation model servers stream artifacts from this store when they start. An outage blocks model pods from starting, as well as uploads.

Single-instance mode is appropriate for evaluation. It is not appropriate where object storage availability matters.

Distributed mode​

Distributed mode runs the object store as a StatefulSet across multiple pods, each with its own volume, and applies erasure coding across them.

Four drives is the practical minimum, and it is a development default. MinIO's production guidance is parity EC:3 or higher, which needs six drives. The four-drive configuration below tolerates one drive loss for writes.

At four drives the default parity splits the set evenly, which gives:

Drives neededEffect
Read quorum2 of 4Reads succeed with two drives lost
Write quorum3 of 4Writes succeed with one drive lost

Parity that is exactly half the erasure set raises the write quorum by one to prevent a split-brain scenario where a network partition isolates exactly half the drives. So a four-drive deployment tolerates one instance being unavailable while remaining writable, and a second loss leaves it readable but read-only.

Usable capacity is roughly half of raw capacity. Four 50Gi volumes provide about 100Gi usable.

To keep accepting writes through the loss of an availability zone that holds two instances, use six drives across three zones.

Configuration​

minio:
  mode: distributed
  statefulset:
    replicaCount: 4
    zones: 1
    drivesPerNode: 1
    podManagementPolicy: Parallel
  pdb:
    create: true
    maxUnavailable: 1
  podAntiAffinityPreset: hard
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app.kubernetes.io/name: minio
  persistence:
    enabled: true
    size: 50Gi
    storageClass: <your storage class>

Each setting is chosen against the quorum above, and they are not independent:

  • maxUnavailable: 1 matches the three-of-four write quorum. A larger value permits a voluntary eviction that makes the store read-only.
  • podAntiAffinityPreset: hard keeps one instance per node. Without it two drives can share a node, which turns that single node into a quorum-loss event.
  • whenUnsatisfiable: DoNotSchedule on zone spread is deliberate. ScheduleAnyway reduces the constraint to a hint the scheduler may silently ignore, which can place every instance in one zone while appearing correctly configured. On a single-zone node pool this constraint is a no-op, so it is safe to set either way.

Prerequisites​

  • At least four schedulable nodes available to the object store. With hard anti-affinity, fewer nodes than replicas leaves pods Pending.
  • A node pool spanning three availability zones, for zone-level tolerance. On a single-zone pool, distributed mode still provides pod-level and node-level redundancy but cannot survive the loss of the zone.
  • Sufficient storage quota for the full replica count. Capacity is now replicaCount × size.

Migrating from single-instance mode​

This is not an in-place upgrade. MinIO does not support expanding or converting a single-drive deployment, so the data must be mirrored out and back in.

Migration requires:

  1. Mirroring the existing buckets out to a location reachable from the cluster.
  2. Applying the distributed configuration, which replaces the Deployment with a StatefulSet.
  3. Mirroring the data back into the new store.
  4. Reclaiming the original PersistentVolumeClaim once the new store is verified. It is orphaned by the switch, and its data is retained but no longer used.

Plan a maintenance window. Object storage is unavailable to the platform for the duration.

Verifying​

# All instances Running, and on distinct nodes
kubectl --context <ctx> -n <ns> get pods -l app.kubernetes.io/name=minio \
  -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase'

# Which zone each node is in
kubectl --context <ctx> get nodes \
  -o custom-columns='NODE:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone'

# The disruption budget permits exactly one disruption
kubectl --context <ctx> -n <ns> get pdb

Confirm ALLOWED DISRUPTIONS reads 1. A budget permitting zero disruptions blocks node drains and cluster upgrades indefinitely.

If instances share a node, or all sit in one zone, the deployment has pod-level redundancy only regardless of replica count.

Upstream status and packaging​

The MinIO community edition was declared unmaintained in February 2026 and its repository was archived on 25 April 2026, so the bundled store has no upstream patch path. Distributed mode is driven by the Bitnami build of the image, whose versioned tags are published only in Bitnami's frozen legacy catalog. Mirror that build into your registry and keep it; an upstream MinIO image ignores the distributed-mode settings. Where you already operate an S3-compatible object store, pointing the platform at it removes this component from your failure domain.

Scope​

This page covers availability of the object store within a single Kubernetes cluster. It does not provide backup, and it does not provide recovery from the loss of a region. Distributed mode protects against instance and infrastructure failure; it is not a substitute for a backup of the object data. For backup and regional recovery, see Disaster Recovery.

Last updated on