Upgrading IBM API Connect is one of those tasks that looks deceptively simple on paper — run the installer, watch the progress bars, done — until something goes sideways at 2am on a Friday. After hundreds of upgrades across financial institutions, payment networks, and large enterprises, a clear pattern emerges: the upgrade itself almost never fails. What fails is everything you didn’t check beforehand.

This article is the pre-flight checklist I wish every platform engineer had before touching a production API Connect upgrade.

1. Why Upgrades Fail — Most Common Root Causes

Before diving into the checklist, it helps to understand where upgrades actually break.

APICOPS_1002E: The OAuth Provider Blocker

The APICOPS_1002E error is the upgrade blocker I see most often in enterprises running older API Connect versions (typically 10.0.5.x and below). It fires when the upgrade validation routine encounters an OAuth provider configured with an invalid or malformed security / securityDefinitions section in the OpenAPI definition. The upgrade pre-check halts before any subsystem is touched, which sounds harsh — but it’s protecting you from a worse outcome: a partially-upgraded environment where OAuth flows silently break for production APIs.

Expired Certificates Blocking Subsystem Communication

API Connect subsystems communicate over mutual TLS. Each subsystem — Management, Gateway, Analytics, Portal — presents a certificate to its peers. During an upgrade, as old components shut down and new ones start, these certificate chains are re-evaluated. If even one certificate in the chain has expired, the new component will refuse to start, typically with a cryptic connection-rejected error in the subsystem logs.

Incomplete Backup Verification

“I’ve got backups” is not the same as “I’ve verified those backups are restorable.” The most common backup misconfiguration I encounter: the barmanObjectStore configuration in the Cloud Native PostgreSQL (CNPG) cluster spec points to an S3 bucket whose credentials have rotated, or the bucket policy has changed. The backup runs daily and reports success, but the actual WAL segments and base backups are going nowhere. When you need to restore, you get an empty bucket.

DataPower Gateway Director Re-registration

On both VMware and OpenShift, the Gateway Director’s registration with the Management plane is stateful. If the Management subsystem is upgraded before the Gateway Director, the gateway can lose its registration — effectively becoming invisible to the management plane even though it’s still processing traffic. Existing API calls continue to work; new publishes queue indefinitely.

taskmanager/nat-cluster Pod Failures on Kubernetes

On OpenShift, the taskmanager and nat-cluster pods are the publish pipeline’s workhorses. If they crash or become CrashLoopBackOff during an upgrade, API publishes will queue but never complete. The symptom is developers reporting that their API changes “aren’t taking effect,” while the Cloud Manager UI shows the gateway as disconnected.

2. Mandatory Pre-Upgrade Steps

Run apicops version:pre-upgrade

The apicops tool is IBM’s own pre-upgrade validation utility. Run it against your current subsystem before touching anything:

# Download and install apicops if not already available
apicops version:pre-upgrade --server https://<management-host> --username <admin-user>

This validates:

  • OAuth provider definitions for schema validity
  • TLS certificate expiry dates across all subsystems
  • Subsystem health status
  • Backup availability and recency

[VERIFY: Exact flag syntax for apicops version:pre-upgrade may vary by version — confirm against your apicops release notes]

Any ERROR-level findings from apicops must be resolved before proceeding. WARNING-level findings should be reviewed and understood.

Validate Subsystem Health

# OpenShift — check all subsystem pods are Running and healthy
oc get pods -n <apic-namespace> | grep -E "management|gateway|analytics|portal"

# Check pod readiness gates
oc get pods -n <apic-namespace> -o wide | grep Ready

# Verify no pods in CrashLoopBackOff or Error state
oc get pods -n <apic-namespace> | grep -v Running

All pods should show 1/1 or 2/2 Ready before you begin the upgrade.

Confirm PostgreSQL/CNPG Backups

# Check CNPG backup status
oc get backup -n <apic-namespace>
# Look for lastSuccessfulBackup timestamp and status: Completed

# Verify barmanObjectStore is configured
oc get cluster <cluster-name> -n <apic-namespace> -o jsonpath='{.spec.backup.barmanObjectStore}'

The barmanObjectStore must return a valid configuration object including destinationPath, s3Credentials, and endpointURL. If it’s empty or missing, your backups are writing to ephemeral storage and will be lost on pod rescheduling.

Check TLS Certificate Expiry Across All Subsystems

# Use apicops to audit certificates
apicops certificates:list --server https://<management-host>

# Manual check — list subsystem endpoints and their cert details
apicops debug:info --server https://<management-host>

For each subsystem, note the certificate expiry date. Any cert expiring within 30 days of your planned upgrade date should be renewed before you start the upgrade. Certificate renewal mid-upgrade is significantly harder.

3. VMware-Specific Gotchas

OVA Pre-Upgrade Prep

When upgrading a VMware-based API Connect deployment:

  1. Snapshot every subsystem VM before starting. Not just the Management plane — snapshot Gateway Director, Analytics, and Portal VMs too.
  2. Verify free disk space on every VM’s datastores. Upgrades unpack large archives; a full datastore will fail mid-upgrade with an I/O error that is hard to diagnose.
  3. Confirm VMware tools version is current on each VM. Outdated VMware tools can cause virtual hardware version mismatches during the upgrade’s VM reconfiguration phase.
  4. Disable HA admissions control temporarily — if a VM restarts during upgrade and HA tries to restart it on a different host before the new software is fully installed, you can end up with a split-version situation.

Subsystem Upgrade Sequence on VMware

The correct upgrade order on VMware is:

  1. Management (always first)
  2. Gateway Director
  3. Analytics
  4. Portal

This order matters because each subsequent subsystem must register with the Management plane. Upgrading them out of order risks stale registration state.

Gateway Loses Registration Mid-Upgrade

If you see the Gateway Director go “disconnected” in the Cloud Manager UI during or immediately after an upgrade, do not panic. This is often recoverable without a full re-registration. The Dynamic Re-Registration (DRR) procedure (covered in Section 5) is designed exactly for this scenario.

4. OpenShift-Specific Gotchas

Operator Upgrade Order

On OpenShift, API Connect is installed and managed via Operators. The upgrade order must respect operator dependency chains:

# Check current operator versions
oc get operators -n <apic-namespace>

# Typically: cert-manager → platform-operators → api-connect-operator → subsystem operators
# Always upgrade cert-manager FIRST
oc get clusterserviceversion -n <apic-namespace>

Always upgrade the cert-manager Operator before upgrading API Connect. Many API Connect components depend on cert-manager for certificate management. An outdated cert-manager can cause TLS handshake failures during the API Connect upgrade.

CNPG Cluster Health

# Check CNPG cluster is healthy
oc get cnpg -n <apic-namespace>
# Each pod should be Running with 1/1 Ready

# Check for any barmanObjectStore errors in CNPG operator logs
oc logs -n <apic-namespace> deployment/cnpg-operator | grep -i barman

Pod Readiness Gates

OpenShift uses readiness gates to ensure pods are fully ready before receiving traffic. During an upgrade, old pods are terminated and new ones started. If the readiness gates don’t pass (typically due to a health check failing), the pod is marked Unready and traffic is not routed to it. This can cause brief service interruptions during rolling upgrades — plan for this in your maintenance window.

5. Dynamic Re-Registration (DRR) Recovery Procedure

The Dynamic Re-Registration (DRR) procedure re-establishes the trust relationship between a Gateway Director and the Management plane without requiring a full subsystem reinstallation.

When to use DRR:

  • Gateway Director shows “disconnected” in Cloud Manager UI after upgrade
  • New API publishes are queuing but never arriving at the gateway
  • apicops debug:info shows gateway: disconnected

When NOT to use DRR:

  • If the gateway is processing live traffic successfully and only new publishes are affected — try recycling the gateway service pod first.

DRR Step-by-Step:

# Step 1: Identify the gateway service that needs re-registration
apicops debug:info --server https://<management-host> | grep -A5 "Gateway"

# Step 2: Delete the existing gateway registration from the management plane
# This removes the stale registration record — the gateway will re-register on restart
apic login --server <management-host> --username <admin> --password <pwd> --realm admin/default-idp-1
apic gateway-services:delete <gateway-service-name> --scope catalog --server <management-host>

# Step 3: Restart the gateway service pod (OpenShift) or gateway VM (VMware)
# OpenShift:
oc delete pod -n <apic-namespace> -l app.kubernetes.io/name=gateway-service

# Step 4: Wait for re-registration (typically 2-5 minutes)
# Watch the gateway service come back online
watch oc get pods -n <apic-namespace> -l app.kubernetes.io/name=gateway-service

# Step 5: Verify re-registration
apicops debug:info --server https://<management-host> | grep -A3 "Gateway"
# Should now show: gateway: connected

[VERIFY: The exact apic CLI and apicops commands for DRR may vary by API Connect version — confirm against IBM documentation for your specific version]

6. Post-Upgrade Verification Checklist

Run through this checklist after every API Connect upgrade, regardless of how smooth the upgrade appeared:

# 1. All subsystem pods/VMs are Running
# OpenShift:
oc get pods -n <apic-namespace>
# VMware: Check all subsystem VMs are running and responding

# 2. Management plane API responds
curl -k https://<management-host>/api/platform/home

# 3. Gateway Director is registered and processing
apicops debug:info --server https://<management-host> | grep -E "gateway|Gateway"

# 4. Analytics ingestion is working
# Look for recent events in the Analytics dashboard
# Or check the ingestion endpoint health
curl -k https://<analytics-host>/_health

# 5. Portal is accessible and syncing
# Log into the portal admin UI and confirm site sync status shows "Synced"
# Check portal connectivity to management plane
oc logs -n <apic-namespace> deployment/<portal-deployment> | grep -i sync

# 6. Test an actual API publish
# Create or update a test API and publish it
# Confirm it appears in the gateway's API list within 2 minutes

# 7. Check for any new ERROR-level events in Cloud Manager UI
# Navigate to Dashboard → Events → filter by Severity: Error, Time: Last 1 hour

# 8. Verify TLS certs have not changed unexpectedly
apicops certificates:list --server https://<management-host>
# Compare thumbprints against your pre-upgrade baseline

A clean post-upgrade verification means you’ve caught the 80% of issues that are silent until you test them under load. Never skip it — and never schedule an upgrade without a rollback plan and an adequate maintenance window.

Next in this series: “TLS Certificates in API Connect and DataPower: How to Stop Them Killing Your Production APIs” — covering the most dangerous TLS behaviour changes across API Connect versions and how to audit your entire certificate estate before disaster strikes.