netPark System Outage

Postmortem

Title: netPark Outage – DoS Event

When: August 27th, 2025, from 3:45 PM to 11:30 PM EST (America/New York). Outage windows occurred at 3:45–4:32 PM, 5:22–5:23 PM, 8:11–8:24 PM, and 9:44–9:46 PM.

What Happened: Customers experienced excessive loading times and intermittent server connection issues across all netPark‑hosted platforms.

Why it Happened: The immediate cause was a malicious scan which triggered a bug in a client’s website code, leading to a DoS event. This overwhelmed our caching servers, causing them to exceed the network allowance for their size and begin denying requests, which led to service disruptions.

Steps Taken to Resolve:

  • At 4:30 PM, the caching server node was rebooted, temporarily restoring services.
  • At 5:22 PM, when the issue resurfaced, the node was failed over and replaced, as hardware was initially suspected.
  • At 8:11 PM, alerts triggered again. All servers were recycled between 8:11 PM and 8:24 PM to restore stability.
  • At 8:51 PM, a Level 1 Severity ticket was opened with AWS. By 9:37 PM, AWS provided insights that helped the team identify the true root cause: a malicious scan against a client’s website, which triggered a DoS event.
  • The offending IP was blocked, and additional preventative measures were implemented.

Preventative Actions Taken:

  • Completed: Offending IP blocked; GeoIP databases updated
  • Scheduled (by 9/5/2025): Patch to resolve the client website bug; batch script to purge oversized cache entries automatically.
  • Researching: Separation of netPark and WordPress session handling for caching.

Additional Questions: Please reach out to support@netpark.us for assistance.

Resolved
Assessed

This afternoon, netPark experienced a server outage that impacted several areas of system functionality. Our Server Team is actively investigating the root cause, and a detailed retrospective will be shared with clients once the review is complete.

9 Affected Services: