API Connect Developer Portal: A Troubleshooting Field Guide
Draft!!
The Developer Portal is the public face of your API program. When it’s down, your external developers can’t register, can’t see your API documentation, and can’t get API keys. For organizations where the portal is the primary channel for developer onboarding, even a few hours of portal unavailability means lost developer productivity and, depending on your industry, potential SLA breaches.
This field guide covers the most common portal issues I encounter in the field — from install failures through to post-upgrade sync problems — with enough specificity to help you actually fix things rather than just understand them.
1. Most Common Portal Install Failure: ISO and Portplan-out Directory Issues
A fresh portal install on OpenShift or VMware fails more often than it should, and the most common culprit is the ISO file and directory structure around the portplan-out directory.
What happens: The portal installer extracts files from the ISO and writes them to portplan-out. If the directory doesn’t exist, or if it exists but the installer user doesn’t have write permissions, the install fails with an error that often doesn’t clearly point to the directory as the cause.
Correct directory structure before starting a fresh portal install:
/opt/
├── apic-portal/ # Installer working directory
│ ├── portplan-out/ # MUST exist and be writable by installer user
│ │ ├── portal-www/ # Created by installer
│ │ ├── portal-db/ # Created by installer
│ │ └── portal-datapower/ # Created by installer
│ ├── apic-portal-installer.iso # The ISO image
│ └── install.sh # Installer script
Before starting the install:
# Verify the portplan-out directory exists and is writable
ls -la /opt/apic-portal/
mkdir -p /opt/apic-portal/portplan-out
chmod 755 /opt/apic-portal/portplan-out
# Verify the ISO file integrity (checksum if documented)
sha256sum /path/to/apic-portal-installer.iso
# Confirm ISO is not truncated
ls -lh /path/to/apic-portal-installer.iso
On OpenShift, the equivalent check is ensuring the persistent volume claims are bound and the installer pod has the correct security context to write to them.
2. The Page-Editor Crash Bug in API Connect 10.0.8.9
Multiple case numbers from a Japanese customer pointed to a consistent pattern: the portal’s page editor crashes when editing certain page templates. The issue manifests as:
- The page editor loads normally
- User makes a small edit (adding a paragraph, changing a heading)
- User clicks “Save” — the page editor freezes, then crashes with an HTTP 500 error
- The change is not saved
Affected versions: IBM confirmed this bug affects API Connect 10.0.8.9. [VERIFY: IBM APAR reference for the page-editor crash in 10.0.8.9 — confirm via IBM support portal for exact APAR number and fix availability]
Trigger conditions: The crash appears to be triggered by specific character sequences in page content — particularly double-byte character sequences (Japanese text, but also certain Unicode characters) in the page metadata or body. If your portal uses Japanese character sets or has pages with complex Unicode content, you are more likely to hit this.
Workaround: The page editor crash does not affect the REST API for page management. You can work around it by:
- Exporting the affected page content via the portal REST API
- Editing the content in an external editor
- Re-uploading via the REST API
Confirming fix availability: Check the IBM API Connect support portal for the latest available fix pack after 10.0.8.9. IBM typically includes fix availability in the release notes for each subsequent fix pack. If your organisation can tolerate a portal downtime window, upgrading to a fixed version is the correct resolution.
3. Password Reset Failures
Password reset is a deceptively complex flow involving several systems: the portal itself, the management plane, and potentially an external identity provider (LDAP or OIDC). When it breaks, it typically fails in one of two ways — silent failure (user never gets an email) or error (user gets the email but the reset link doesn’t work).
Logs to gather from both management and portal:
# On the portal subsystem — gather portal application logs
oc logs -n <apic-namespace> deployment/<portal-www-pod> --tail=500 > portal-www.log
oc logs -n <apic-namespace> deployment/<portal-db-pod> --tail=500 > portal-db.log
# On the management plane — gather management logs related to password reset
oc logs -n <apic-namespace> deployment/<management-pod> --tail=500 | grep -i "password\|reset\|email" > mgmt-password-reset.log
# Check the portal database for reset token state
# Log into the portal DB pod
oc exec -it -n <apic-namespace> <portal-db-pod> -- mysql -u<user> -p<password> portal
mysql> SELECT user_id, email, reset_token, reset_token_expiry FROM users WHERE reset_token IS NOT NULL;
LDAP integration points that break this flow:
If your portal uses LDAP for authentication (common at enterprises — the Ujjivan Small Finance Bank pattern involved LDAP misconfiguration preventing new user creation), the password reset flow goes through your LDAP directory. If:
- The LDAP service account used by the portal has insufficient rights to perform password modifications
- The LDAP attribute mapping for password is incorrect
- The LDAP server’s password policy rejects the generated reset password (e.g., too short, doesn’t meet complexity requirements)
Tracing password reset end-to-end:
- User submits email → portal creates reset token in portal DB, sends email via SMTP
- User clicks link → portal validates token (checks DB + expiry)
- Portal submits new password to management plane (LDAP proxy mode) or directly to LDAP
- LDAP validates password against password policy
- Success/failure returned to portal
Step 4 is where most failures occur with LDAP. Check your LDAP server’s password policy constraints and compare against what the portal is sending.
4. Adding Users via LDAP: Config Checklist
LDAP integration with the portal is powerful but sensitive to configuration. Here’s the checklist I run through:
LDAP attribute mapping (critical fields):
| Portal Field | Standard LDAP Attribute | Common Mistakes |
|---|---|---|
| Username | uid or sAMAccountName |
Mapping to cn (common name) which includes spaces |
mail or userPrincipalName |
Mapping to emailAddress which may not exist |
|
| Display Name | cn or displayName |
Not mapping at all |
| Group/Membership | memberOf |
Not configuring group search base |
Testing LDAP connectivity from the portal subsystem:
# From the portal www pod, test LDAP connectivity
oc exec -it -n <apic-namespace> <portal-www-pod> -- /bin/bash
# Test LDAP bind (authenticating as the portal's LDAP service account)
ldapsearch -H ldap://<ldap-host>:<ldap-port> \
-D "<service-account-dn>" \
-w "<service-account-password>" \
-b "<user-search-base>" \
"(uid=<test-user>)"
# Test user search
ldapsearch -H ldap://<ldap-host>:<ldap-port> \
-D "<service-account-dn>" \
-w "<service-account-password>" \
-b "<user-search-base>" \
"(&(objectClass=person)(uid=*))" \
uid mail memberOf
The portal’s LDAP configuration in the management plane UI must use the correct LDAP attribute names exactly as they appear in your directory. Case matters.
5. Portal Unavailability After Upgrade
After upgrading the portal subsystem, you may find the portal returns 503s or shows an empty/incomplete page even though the pods are running. This is usually one of three things.
Check 1: Portal sites need re-syncing with the management plane
After any portal subsystem upgrade, the portal sites must be explicitly re-synced with the management plane. This is a post-upgrade step that is sometimes skipped:
# Via CLI — list portal sites
apic portal-services:list --server <management-host>
# Trigger a site sync for each portal service
apic portal-services:sync --server <management-host> <portal-service-id>
# Or via the management plane REST API
curl -k -X POST \
-H "Authorization: Bearer $TOKEN" \
https://<management-host>/api/portal-services/<portal-id>/actions/publish
Check 2: The portal’s TLS certificate has changed
If during the upgrade the portal’s TLS certificate was regenerated (e.g., self-signed cert changed), the management plane may have the old certificate pinned. The management plane will reject the portal’s webhook callback. Regenerate and re-register the portal’s TLS identity.
Check 3: Ingress/Route misconfiguration on OpenShift
Portal upgrades on OpenShift sometimes result in the ingress route or OpenShift Route object having an outdated target. Verify:
oc get route -n <apic-namespace> | grep portal
oc describe route -n <apic-namespace> <portal-route-name> | grep -E "Hostname|Path|TLS"
6. Portal Health Checklist: Five Commands to Run Every Time
Run these before and after any portal maintenance:
# 1. Portal www pod is Running and Ready
oc get pods -n <apic-namespace> -l app.kubernetes.io/name=portal-www
# 2. Portal DB pod is Running and Ready
oc get pods -n <apic-namespace> -l app.kubernetes.io/name=portal-db
# 3. Portal site sync status is "Synced" (via Cloud Manager UI or API)
# Check via API:
curl -k -H "Authorization: Bearer $TOKEN" \
https://<management-host>/api/portal-services | jq '.[] | {name, siteState}'
# 4. Portal SMTP is working (send a test email)
# The simplest check: trigger a password reset and verify email is sent
# Check SMTP relay is not blocked post-upgrade:
oc logs -n <apic-namespace> deployment/<portal-www-pod> | grep -i smtp
# 5. Portal can reach the management plane (webhook health)
oc logs -n <apic-namespace> deployment/<portal-www-pod> | grep -i "management\|webhook\|api-connect" | tail -20
Portal issues are often mistaken for management plane issues because the portal is the visible symptom-bearer for deeper problems. Use this checklist to narrow down where the actual problem lies.
Next: “API Connect and DataPower Backup and DR: What Actually Works at 2am” — covering backup verification, analytics index recovery, DataPower RAID, and restore runbooks.
