Cloud services affected by CloudFlare Outage

Incident Report for IT Portal

Resolved

This incident has been resolved.
Posted May 27, 2022 - 00:14 EDT

Update

We have worked with Cloudflare and made changes to our setup of DNS records and the issue now seems resolved. We speculate an issue on amazon's DNS servers. We switched our cluster DNS records from amazon CNAMEs to the IP of the cluster and the issue had resolved itself.

BEFORE
www.siportal.net is a CNAME pointing to us-active.siportal.net
us-active.siportal.net is a CNAME pointing to wc04.siportal.net
wc04.siportal.net was a CNAME pointing to ec2-34-232-208-181.compute-1.amazonaws.com

AFTER
www.siportal.net is a CNAME pointing to us-active.siportal.net
us-active.siportal.net is a CNAME pointing to wc04.siportal.net
wc04.siportal.net is now pointing to A 34.232.208.181
Posted May 27, 2022 - 00:12 EDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted May 27, 2022 - 00:09 EDT

Identified

Access to multiple cloud regions is being affected by an Issue at Cloudflare. The issue seems to be sporadic and access is limited at times. See the video below for a detailed explanation.
https://www.loom.com/share/9c7e6ff845034fbcae8786414b74cd54
Posted May 26, 2022 - 23:15 EDT
This incident affected: US (US API, US Cloud), Support (Portal KBs), UK (UK Cloud, UK API), CA (CA Cloud, CA API), EU (EU Cloud, EU API), and AU (AU Cloud, AU API).