The first time you need your API Connect backup to work is not the time to discover it hasn’t been succeeding for the past three months. I learned this the hard way during a Bank of Montreal Sev-1 during a DR exercise — the backup infrastructure was nominally configured, the job was running daily, and the backups were all going to an S3 bucket whose bucket policy had silently been tightened by a security team update. The backup job reported success. The actual data was 0 bytes. When they tried to restore to the DR site, they had nothing.

This article is the DR runbook I wish I’d had before that incident — the things you need to verify, the things that go wrong, and the exact steps to recover when everything is on fire.

1. APIC Backup Architecture: What Is Backed Up and Where

API Connect’s backup architecture has multiple layers, each with different failure modes:

Layer 1: Cloud Native PostgreSQL (CNPG) — Management and Analytics databases

On OpenShift, the Management and Analytics databases run on CNPG. CNPG uses barman under the hood for WAL archiving and base backups. The backup chain:

CNPG Pod → WAL segments written continuously → barman archives WAL to barmanObjectStore (S3)
                                   ↓
Base backup runs daily → barmanObjectStore (S3)

The critical configuration is the barmanObjectStore in the CNPG cluster spec. This is what makes backups durable. Without it, base backups go to the pod’s ephemeral storage — and if the pod dies, you have no database backup.

Layer 2: Management plane configuration backup

The management plane backs up its own configuration (API definitions, user accounts, org settings) separately from the database. This backup goes to an NFS share or S3 bucket configured in the management plane’s backup settings.

Layer 3: DataPower configuration backup

DataPower appliances (both standalone and B2B) should have configuration backups scheduled. For B2B Gateway, this includes the transaction data on the RAID volume.

Most common misconfiguration — missing barmanObjectStore:

The barmanObjectStore section is not always present in the CNPG cluster spec by default, or it can be accidentally removed during operator upgrades. When it’s missing, CNPG reports backup “success” (the base backup completes to local storage) but the off-cluster archive silently fails.

# What the correct barmanObjectStore configuration looks like in the CNPG cluster spec:
spec:
  backup:
    barmanObjectStore:
      destinationPath: "s3://backup-bucket/apic-backups/<cluster-name>/"
      s3Credentials:
        accessKeyId:
          name: <secret-name>
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: <secret-name>
          key: SECRET_ACCESS_KEY
      endpointURL: "https://s3.<region>.amazonaws.com"  # Optional for AWS
      wal:
        compression: gzip
        maxParallel: 4

If destinationPath is empty or s3Credentials point to a secret that doesn’t exist, your backups are broken.

2. Verifying APIC Backups Are Actually Succeeding

CNPG backup status:

# Check last successful backup
oc get backup -n <apic-namespace> -o wide

# Example output:
# NAME                 CLUSTER           LAST BACKUP   ...
# apic-mgmt-backup-1   apic-mgmt-cluster   Completed   2026-07-12T02:00:00Z

# Get detailed backup information
oc describe backup <backup-name> -n <apic-namespace>

# Check barman backup list from the CNPG pod
oc exec -n <apic-namespace> <cnpg-pod-name> -- barman list-backups <cluster-name>
# Each backup should show: size, wal_size, compress_ratio

# Verify backup exists in S3
oc exec -n <apic-namespace> <cnpg-pod-name> -- \
  aws s3 ls s3://<backup-bucket>/apic-backups/<cluster-name>/ --recursive | tail -10

If Last Successful Backup is more than 26 hours old for a daily backup schedule, something is wrong. The barmanObjectStore kubectl output for Banca Transilvania showed an empty S3 bucket during their analytics shard failure — the backup cron job was running but pointing to the wrong bucket path.

3. Analytics Index Recovery: no_valid_shard_copy

When the analytics Elasticsearch cluster has a red shard — particularly the no_valid_shard_copy error — it means the data that should exist on that shard is not actually present on any node. This is the most serious analytics failure mode.

The error:

"reason":"[summary-api-2026.06][3] – failed to create shard,
failure: Failed to create replica shard [summary-api-2026.06][3],
reason: [3] total missing shards; ...
allocate_explanation: Cannot allocate because all copies of the shard
are missing or have been discarded"

What it means: Elasticsearch uses sharding — data is divided across multiple shards, each replicated to multiple nodes. no_valid_shard_copy means that for one or more shards, no replica exists with valid data. The primary shard may also be missing. There is no clean copy of this data anywhere in the cluster.

Causes:

  • Node loss during upgrade (a node’s persistent volume was corrupted or lost)
  • Upgrade interruption mid-index-write (the index was partially written)
  • Persistent Volume Claim (PVC) failure on a storage node
  • Cluster was already in yellow/red state before the upgrade started

Step-by-step recovery:

# Step 1: Get full cluster health and unallocated shards
curl -k -u admin:<password> \
  https://<analytics-host>:9200/_cluster/health?pretty&level=shards

# Step 2: Get details of unallocated shards
curl -k -u admin:<password> \
  https://<analytics-host>:9200/_cluster/allocation/explain?pretty

# Step 3: For each unallocated shard, look at the explanation
# The output will say: "all copies of the shard are missing" or describe the specific corruption

# Step 4: If data is truly lost (no valid copy exists), the shard must be force-allocated as empty
# This means accepting data loss for that shard's data
# DANGER: Only do this if you're certain the data cannot be recovered

# Force-allocate a shard as empty (loses that data):
curl -k -u admin:<password> -X POST \
  "https://<analytics-host>:9200/_cluster/reroute" \
  -H 'Content-Type: application/json' \
  -d '{
    "commands": [{
      "allocate_empty_primary": {
        "index": "summary-api-2026.06",
        "shard": 3,
        "node": "<node-name>",
        "accept_data_loss": true
      }
    }]
  }'

Prevention is better than recovery: The Banca Transilvania case showed that analytics shard failures are almost always preceded by a yellow cluster health state that was ignored. Monitor Elasticsearch cluster health actively — a yellow cluster should be treated as a warning, not acceptable.

4. DataPower B2B Gateway RAID Full

The DataPower B2B Gateway stores transaction data on a RAID volume. Over time — particularly in high-volume B2B environments like payment networks or logistics integrations — this volume fills up. The EmblemHealth case involved a DataPower B2B appliance that was relocated, and during the relocation the RAID was inadvertently knocked into a degraded state, causing transaction data to stop being written.

How B2B persistence fills the RAID:

  • Inbound AS2/MQ transactions are written to the RAID before processing
  • Completed transactions are acknowledged and purged from the RAID on a schedule
  • If the purge schedule is too infrequent, or if partner acknowledgements are not being received (which prevents purge), the RAID fills

Monitoring RAID consumption:

# In DataPower CLI — check RAID/volume status
show raid
# Shows: volume name, total size, used space, free space, status (Online/Degraded/Failed)

# Check B2B transaction store usage
show b2b-store-usage

5. DataPower Appliance Physical Relocation/Replacement

Pre-move checklist:

  1. Export DataPower configuration backup to external storage (USB or SFTP)
  2. Document current network configuration (IP addresses, VLANs, host routes)
  3. For HA pair: note which appliance is primary vs. secondary
  4. For B2B: export all trading partner configuration
  5. Confirm the RAID status on both appliances before powering down
  6. Note firmware version — firmware differences between HA pair appliances can cause unexpected behaviour after a replacement

Bringing a replaced appliance back into an HA pair:

# On the replacement DataPower (at same IP as failed unit):
# 1. Load the configuration backup exported in step 1
# 2. Register the appliance with the API Connect management plane
# 3. Verify HA status
show ha
# Both appliances should show: State: Online, Role: Primary/Secondary

# 4. Verify synchronization
show ha sync
# Should show: Configuration synchronized, Transaction state synchronized

6. APIC Restore Runbook: Subsystem Restore Sequence

The restore sequence matters. Restoring subsystems out of order can cause data loss.

VMware Restore Order:

  1. Restore Management plane first — this is the source of truth
  2. Restore Analytics (if separate) — will re-index from gateway offload data
  3. Restore Gateway Directors — re-register with restored management plane
  4. Restore Portal — re-sync with restored management plane

OpenShift Restore Order:

  1. Restore CNPG (Management database) — using barman restore command
  2. Restart Management plane pods — to pick up restored database
  3. Verify Management plane is healthy before restoring anything else
  4. Restore Analytics CNPG if analytics data is critical
  5. Recycle Gateway Director pods — they will re-register with restored management
  6. Trigger portal site re-sync

Critical ordering mistake: Restoring the Gateway Director before the Management plane. The gateway will re-register with a stale or partially-restored management database, and you will spend hours debugging why configuration changes are not persisting or why old API definitions are reappearing.

# CNPG restore command (OpenShift)
oc exec -n <apic-namespace> <cnpg-pod-name> -- \
  barman recover <cluster-name> <backup-id> /var/lib/postgresql/data \
  --target-time "2026-07-12 03:00:00"

# After restore, restart the management plane to pick up the restored data
oc rollout restart deployment/<management-deployment-name> -n <apic-namespace>

Next: “API Connect Analytics: Diagnosing and Recovering a Broken Analytics Subsystem” — deep dive into Elasticsearch cluster health, shard allocation failures, and the analytics ingestion pipeline.