Public web portal and secure web portal suffering from major outage

Incident Report for AfterShip Tracking

Postmortem

With regard to disruptions of our services on 2017-11-30, namely * www.aftership.com * www.postmen.com * secure.aftership.com * secure.postmen.com

which were caused by issues with Google Cloud Networking, we had been informed by Google:

"A system that is responsible for throttling individual instances in cases where there is networking congestion received data from one of its data sources that were implying that there are multiple choke points on our network. This caused throttling for a few VMs while the network had no real choke points in the path of these instances. There are enough action items on our internal post mortem to detect similar issues faster and not have this exact issue happening again."

We will follow instructions from Google to minimize future recurrence.

Posted Dec 17, 2017 - 20:09 EST

Resolved

The outage was caused by an issue with Google Cloud Networking. It had been resolved by Google at 18:46 HKT. We will work with Google to prevent or minimize future recurrence.

Posted Nov 30, 2017 - 07:43 EST

Update

We have not received any status updates from Google Cloud by 18:30 HKT. We continue to monitor the situation.

Posted Nov 30, 2017 - 05:59 EST

Monitoring

Normal operations appear to have resumed for the www.aftership.com and secure.aftership.com web sites. The current outage was caused by an issue with Google Cloud Network, which is being looked at by Google at this point. We continue to monitor the situation.

Posted Nov 30, 2017 - 05:33 EST

Identified

The problem was identified to be an issue with Google Cloud Network. We expect to get an update from Google on this by 18:30 HKT.

Posted Nov 30, 2017 - 05:04 EST

Investigating

Since 17:14 HKT Nov 30, 2017, www.aftership.com and secure.aftership.com web sites have been suffering from major outage. We are investigating the cause of the outage and try to get them back in service as soon as we can.

Posted Nov 30, 2017 - 04:57 EST