API Connect Analytics is the subsystem that turns your API gateway from a black box into something you can understand. When it’s working, you get visibility into API usage, latency distributions, error rates, and top clients. When it’s broken — when indices go red, when data stops flowing, when the Analytics tab in the Cloud Manager UI shows nothing — you lose that visibility at exactly the moment you probably need it most: during incidents and upgrades.

The Banca Transilvania incident was instructive here. The analytics index summary-api-2026.06 went red during a routine upgrade. The upgrade completed successfully. The Analytics tab in Cloud Manager showed nothing. Post-upgrade, the operations team had no visibility into API traffic — which made it impossible to confirm whether their APIs were actually working or whether they were silently failing. They spent three hours trying to understand why their APIs were returning errors before someone thought to check the Analytics subsystem directly.

This article is about understanding the Analytics architecture, diagnosing failures fast, and recovering from the most common analytics breakages.

1. Analytics Architecture in Two Minutes

The API Connect Analytics pipeline has four components:

DataPower Gateway
    ↓ (offloads API events over TLS)
Analytics Ingestion Endpoint
    ↓
Elasticsearch Cluster (index: summary-api-YYYY.MM)
    ↓
Cloud Manager Analytics UI ← API calls for dashboard data

Failure modes:

  • DataPower can’t reach analytics ingestion → data never leaves the gateway (gateway still works, you just lose visibility)
  • Analytics ingestion endpoint is down → data queues or drops on DataPower
  • Elasticsearch cluster is red/yellow → data may be written but queries return errors or partial results
  • Cloud Manager can’t reach Elasticsearch → UI shows no data even when data exists

The key insight: a red Elasticsearch cluster does not necessarily mean your APIs are failing. It means you can’t see your APIs. Your gateway is still processing traffic.

2. Red and Yellow Elasticsearch Cluster Health

Elasticsearch cluster health has three states:

  • Green: All primary and replica shards are allocated. Cluster is fully functional.
  • Yellow: All primary shards are allocated, but some replica shards are not. The cluster is functional but at reduced resilience — if a node fails, data loss is possible.
  • Red: One or more primary shards are unallocated. The cluster is partially functional — queries on affected indices may return partial results or errors.

Getting full cluster status:

# Overall cluster health with shard-level detail
curl -k -u admin:<password> \
  "https://<analytics-host>:9200/_cluster/health?pretty&level=shards"

# Example green response:
# {
#   "cluster_name": "analytics",
#   "status": "green",
#   "number_of_nodes": 3,
#   "active_primary_shards": 150,
#   "relocating_shards": 0,
#   "active_shards": 300
# }

# Example red response:
# {
#   "cluster_name": "analytics",
#   "status": "red",
#   "number_of_nodes": 3,
#   "active_primary_shards": 147,
#   "relocating_shards": 0,
#   "unassigned_shards": 3
# }

Getting the list of unallocated shards:

curl -k -u admin:<password> \
  "https://<analytics-host>:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason"

# Example output showing unassigned shards:
# summary-api-2026.06   3   p   UNASSIGNED   ALLOCATION_FAILED
# summary-api-2026.06   7   r   UNASSIGNED   NODE_LEFT

Interpreting the allocate_explanation:

# Get detailed explanation for why a specific shard can't be allocated
curl -k -u admin:<password> \
  "https://<analytics-host>:9200/_cluster/allocation/explain?pretty" \
  -H 'Content-Type: application/json' \
  -d '{
    "index": "summary-api-2026.06",
    "shard": 3,
    "primary": true
  }'

# Example output for Banca Transilvania case:
# {
#   "index": "summary-api-2026.06",
#   "shard": 3,
#   "primary": true,
#   "unassigned_info": {
#     "reason": "ALLOCATION_FAILED",
#     "at": "2026-07-10T14:32:11.234Z",
#     "last_allocation_status": "no_valid_shard_copy"
#   },
#   "allocate_explanation": "Cannot allocate because all copies of the shard
#     are missing or have been discarded."
# }

The allocate_explanation tells you exactly why the shard can’t be allocated. no_valid_shard_copy means the data doesn’t exist anywhere. NODE_LEFT means the node that held the replica is temporarily unavailable and the shard will likely allocate automatically when the node rejoins.

3. no_valid_shard_copy: Causes and Recovery

no_valid_shard_copy is the most serious shard failure state — it means the data for that shard is gone. It cannot be recovered from another node because no other node has it.

Causes:

  1. Node loss during write — a node went down while a write was in progress and the primary was on that node
  2. PVC failure — the persistent volume holding the shard data was corrupted or lost
  3. Upgrade interruption — the upgrade process killed a node mid-index-write
  4. Manual error — someone accidentally deleted the Elasticsearch data directory

Step-by-step recovery:

# Step 1: Confirm the shard is truly in no_valid_shard_copy state
# Use the allocation/explain call from Section 2

# Step 2: Check if the index has any other shards with data
curl -k -u admin:<password> \
  "https://<analytics-host>:9200/_cat/shards/summary-api-2026.06?v&h=shard,prirep,state,unassigned.reason"

# Step 3: If other shards of this index are healthy, the missing data is limited to
# the specific shard. Force-allocation as empty accepts data loss for that shard only.

curl -k -u admin:<password> -X POST \
  "https://<analytics-host>:9200/_cluster/reroute" \
  -H 'Content-Type: application/json' \
  -d '{
    "commands": [{
      "allocate_empty_primary": {
        "index": "summary-api-2026.06",
        "shard": 3,
        "node": "<node-with-free-space>",
        "accept_data_loss": true
      }
    }]
  }'

# Step 4: Verify the cluster goes yellow (or green if all replicas are present)
curl -k -u admin:<password> \
  "https://<analytics-host>:9200/_cluster/health?pretty"

# Step 5: Confirm the affected index is now writeable
curl -k -u admin:<password> \
  "https://<analytics-host>:9200/summary-api-2026.06/_health"

About that accept_data_loss: true: This is honest. When you force-allocate an empty primary, you’re telling Elasticsearch “I know this shard has no data, create it empty.” The data that was supposed to be in that shard is gone. Accept it, document it, and move on. The cluster will start receiving new data and rebuilding the index over time.

4. Analytics Data Not Appearing After Upgrade

Three common causes:

Cause 1: Ingestion endpoint not restarted after upgrade

During an API Connect upgrade, the analytics ingestion endpoint may not restart automatically. New events from DataPower are being generated, but they’re hitting a closed door.

# Check ingestion endpoint health
curl -k -u admin:<password> \
  "https://<analytics-host>:9200/_cluster/health?pretty" | grep status

# Check if ingestion is receiving data by looking at the latest index document count
curl -k -u admin:<password> \
  "https://<analytics-host>:9200/_cat/indices/summary-api-2026.07?v&h=index,docs.count,store.size"

Cause 2: TLS mismatch between DataPower and analytics ingestion

If the analytics ingestion endpoint’s TLS certificate was regenerated during the upgrade, DataPower’s configured TLS credential for the analytics endpoint may no longer match.

# On DataPower — check the analytics offload configuration
# Via DataPower CLI:
show analytics-settings
# Look for: endpoint URL, TLS credential object name

# Verify the TLS credential object's cert matches what's on the analytics ingestion endpoint
show crypto certificate <credential-name>

Cause 3: Wrong offload configuration after upgrade

# On DataPower — verify analytics offload is pointing to the right endpoint
show analytics
# Check: URL, TLS profile, and that "enabled" is true

5. Analytics Offload Configuration

Correct settings for OpenShift:

// DataPower multi-protocol gateway or analytics offload configuration
// OpenShift: analytics ingestion endpoint is typically internal to the cluster
{
  "name": "analytics-offload",
  "url": "https://analytics-ingestion.<apic-namespace>.svc.cluster.local:9200",
  "tlsProfile": "analytics-tls",
  "enabled": true,
  "batchSize": 1000,
  "flushInterval": 5
}

Verifying offload end-to-end:

  1. Generate some test API traffic through the gateway
  2. Wait 30-60 seconds for batch to flush
  3. Query Elasticsearch directly for recent documents: 
    curl -k -u admin:<password> \
  "https://<analytics-host>:9200/summary-api-2026.07/_search?q=*&sort=@timestamp:desc&size=5"

6. Analytics Mustgather: Exact Commands for Support Case

When you need to open an IBM support case for analytics issues, gather these before calling:

# 1. Cluster health
curl -k -u admin:<password> "https://<analytics-host>:9200/_cluster/health?pretty&level=shards" > cluster-health.txt

# 2. Index list with document counts
curl -k -u admin:<password> "https://<analytics-host>:9200/_cat/indices?v&h=index,docs.count,size,health" > indices.txt

# 3. All unassigned shard explanations
curl -k -u admin:<password> "https://<analytics-host>:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason" > unassigned-shards.txt

# 4. Node stats (for memory/disk issues)
curl -k -u admin:<password> "https://<analytics-host>:9200/_nodes/stats?pretty" > nodes-stats.json

# 5. DataPower analytics offload config
# From DataPower CLI:
# show analytics > datapower-analytics-config.txt
# show crypto certificate > datapower-crypto-certs.txt

# Package all:
tar -czf analytics-mustgather-$(date +%Y%m%d).tar.gz cluster-health.txt indices.txt unassigned-shards.txt nodes-stats.json datapower-analytics-config.txt datapower-crypto-certs.txt

Next: “OAuth 2.0 and JWT in API Connect and DataPower: Five Mistakes That Break Your Upgrade” — covering the APICOPS_1002E blocker, POST-based auth-code flows, JWT validation mistakes, and OAuth introspection configuration.