netPark Certificate Outage

Postmortem

Around 11-11:30am EST we experienced a major outage due to the expiration of a root CA certificate with Lets Encrypt. This issue mostly affected Self Park locations, but also affected many clients who had not performed recent Windows Updates or had legacy Credit Card EMV terminals.

The root CA certificate associated with Lets Encrypt, used to reach our servers, expired today. Normally this would not be a problem - as long as the computers and devices are up-to-date they should have the latest certificates. However, many of the on-site NIMs (used for communicating with Self Park devices) did not have the newest certificate and went offline.After a bit of research, we started a two-pronged approach to work through this:

  1. Use non-Lets Encrypt certificates for our main domains. Unfortunately it took some time to generate these certificates and configure them for the server. Eventually we’ll switch back to Lets Encrypt, but not before thoroughly reviewing all existing devices to verify they’ll connect without issues.

  2. Manually apply the certificate to each NIM that was offline to bring it back online. This process took some time to research and implement as well, but once we figured it out we were able to start hitting each NIM and making the change manually.

We’ll be doing a thorough review of this whole process and all devices to see what we can implement to help prevent or eliminate this as an issue in the future.

Löst

All connections should be working at this point. If there are any outstanding issues please attempt to reboot the computer, NIM or exit station first, then contact netPark support if it is still on-going.

This issue was initially brought to our attention around 11-11:30am EST. After a bit of review we were able to determine that the issue was due to the Lets Encrypt certificates that we use for our server’s TLS configuration. The root CA certificate associated with Lets Encrypt, used to reach our servers, expired today. Normally this would not be a problem - as long as the computers and devices are up-to-date they should have the latest certificates. However, many of the on-site NIMs (used for communicating with Self Park devices) did not have the newest certificate and went offline.

After a bit of research, we started a two-pronged approach to work through this:

  1. Use non-Lets Encrypt certificates for our main domains. Unfortunately it took some time to generate these certificates and configure them for the server. Eventually we’ll switch back to Lets Encrypt, but not before thoroughly reviewing all existing devices to verify they’ll connect without issues.

  2. Manually apply the certificate to each NIM that was offline to bring it back online. This process took some time to research and implement as well, but once we figured it out we were able to start hitting each NIM and making the change manually.

We’ll be doing a thorough review of this whole process and all devices to see what we can implement to help prevent or eliminate this as an issue in the future.

Bedömd

One of the certificates in our server’s certificate chain expired today. Computers that are out-of-date or that do not have the latest certificates installed may not be able to reach netPark. We’re currently working on a mitigation for this problem to bring everything back online as soon as possible.

3 berörda tjänster: