How to Diagnose a DataPower Restart: Reading error-report and Backtrace Files
Draft!!
DataPower restarted. Now what? The first instinct is to open a support ticket — but before you do, the answer is almost certainly already on the appliance. DataPower writes a detailed post-mortem every time it restarts, and knowing how to read it will either resolve the issue immediately or cut hours off your support case.
1. First: Know Which Type of Restart Occurred
Not all restarts are equal. Before reading any file, identify which category you are dealing with:
| Restart type | What it means | Where to confirm |
|---|---|---|
| Throttler Reload | Memory fell below the Throttle At or Terminate At threshold. Deliberate, controlled restart. | system.log — look for throttler-reload |
| Firmware crash (panic) | An unhandled internal error caused the firmware to abort. | error-report backtrace files |
| Manual restart | Operator ran shutdown restart or restarted the pod/container. |
system.log — look for system shutdown initiated |
| OCP pod eviction | Kubernetes killed the pod due to node pressure or OOM. | OCP event log, not DataPower logs |
| Domain restart | A single application domain restarted, not the full firmware. | system.log in the affected domain |
A Throttler Reload is not a crash — it is a deliberate protective action. Do not treat it like a firmware panic. The diagnostic approach differs.
2. Generating the Error Report
If the appliance is still running, generate the error report immediately — before any further restarts wash the state:
# From the CLI (top-level, not inside a domain)
top
generate error-report
The report is written to temporary:///error-report-<timestamp>.zip and also to logtemp:///error-report-<timestamp>.zip. Download it from the WebGUI under Administration → Main → Error Report or via the REST Management Interface:
curl -k -u admin:password \
https://<datapower-host>:5554/mgmt/filestore/temporary/ \
| grep error-report
Note: On a containerised DataPower (OCP/Kubernetes), the
error-reportis written inside the pod’s ephemeral filesystem. Retrieve it before the pod restarts again, or configure a persistent log target to ship it externally.
3. Error Report Archive Structure
The .zip contains many files. For restart diagnosis, focus on these:
error-report-<timestamp>.zip
├── system.log ← Start here — the timeline of events
├── backtrace.txt ← Present only on a firmware crash/panic
├── backtrace-<n>.txt ← Additional threads at time of crash
├── MemoryStatus2.xml ← System memory at time of report
├── SystemUsage.xml ← CPU load at time of report
├── ErrorReportSummary.xml ← High-level summary, useful for quick triage
├── GatewayPeeringStatus.xml ← Peering state at time of restart
└── <domain>/
└── log ← Per-domain log (look here for domain-level events)
4. Reading system.log
system.log is your timeline. Open it and search backwards from the end for the restart event. Key patterns:
# Throttler Reload
0x8100003d ... throttler-reload
mgmt(notice): ... System firmware reload due to memory threshold
# Controlled shutdown (manual or OCP signal)
mgmt(notice): System shutdown initiated
# Firmware panic (will be followed by backtrace files)
mgmt(error): Firmware panic: <reason>
mgmt(error): Generating crash report
Once you find the restart line, scroll up 60–120 seconds to see what was happening immediately before. Look for:
- Repeated
connection refusedorconnection pool full— traffic overload document-cache fullorstylesheet-cache full— cache exhaustion driving memory growthGatewayScript runtime failure— a runaway GatewayScript consuming memory/CPUgateway-peeringerrors — peering instability preceding a restart
5. Backtrace File
Backtrace files are only present after a firmware panic (unhandled error). Before 10.6.5.0 they are not present after a Throttler Reload. These are only for IBM Support and Engineering to review and cannot be read by those outside of IBM.
6. Diagnosing a Throttler Reload (No Backtrace)
A Throttler Reload leaves no backtrace. Diagnose it through memory data:
Step 1 — Confirm it was a Throttler Reload:
# In system.log, search for:
throttler-reload
# or
memory threshold
Step 2 — Establish the memory trend before the reload:
Open MemoryStatus2.xml from the error report. The AvailableMemory field shows the state at time of report. Compare against any previously collected show memory snapshots or monitoring data to establish the trend.
Step 3 — Identify the memory consumer:
Run these commands on the recovered appliance to understand what is using memory:
top; co; show memory
top; co; show statistics
Then open DomainsMemoryStatus2.xml from the error report — the ServicesLifetime counter per domain indicates which domain has processed the most data. The largest consumers are the most likely candidates, though per-domain figures will not pinpoint a genuine leak (see the memory post for detail on why).
Step 4 — Check the throttle thresholds:
Navigate to WebGUI → Administration → Device → Memory Management and note the configured thresholds. If Available memory is consistently hovering near the Throttle At threshold, the thresholds need adjusting or the appliance is genuinely under-resourced.
7. What to Send IBM Support
A support case opened with the right data upfront resolves significantly faster. For a restart:
| Item | How to get it |
|---|---|
Full error-report.zip |
generate error-report from CLI, then download |
show version output |
From CLI — include firmware build number |
show system output |
System identity, uptime, serial |
| Timeline of events | What changed in the 30 minutes before the restart |
system.log excerpt |
200 lines either side of the restart event |
| Backtrace files | Already inside the error-report.zip — confirm they are present |
| Memory trend graph | 48-hour graph from your monitoring tool if available |
Do not paraphrase the logs in the case description. Attach the files. Support engineers read log files faster than case descriptions and it eliminates ambiguity.
8. Quick Triage Checklist
□ Was there a backtrace?
YES → Firmware panic → open support case with error-report + show version
NO → Continue below
□ Does system.log show "throttler-reload"?
YES → Memory-triggered restart → check MemoryStatus2.xml trend + thresholds
NO → Check for "shutdown initiated" → manual or OCP-driven restart
□ Is this repeating on a schedule?
YES → Likely correlated with a batch job, cert rotation, or scheduled task → check cron/automation
□ Did it affect only one node in an HA pair?
YES → Hardware or network issue on that node more likely → check OCP node events
□ Can you reproduce it?
YES → Capture error-report immediately after next occurrence
