Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

At approximately 14:30 EDT on September 8, 2022, the eduroam.us national proxy servers stopped responding to requests.  Service was restored by 14:55 EDT, and eduroam administrators were notified by 16:50 EDT.  Since that time additional checks and monitoring were instituted to allow a quicker recovery in the event of a similar incident.

...

The outage was caused by the load balancing system’s health testing processes removing all of the IPv4 addresses from the load balancer due to a transient network failure of the Virtual Private Network (VPN) between the six geographically distributed AWS (Amazon Web Services) sites running the two Top Level RADIUS Servers (tlrs1.eduroam.us and tlrs2.eduroam.us). 

The cause of the network failure isn’t known. The data centers running these tasks communicate via the open Internet, which Internet2 does not control.  IPv6 traffic was not impacted, which prevented the monitoring system from detecting the issue.

...

eduroam operations staff will continue to improve monitoring capabilities, identify single points-of-failure and mitigate where possible.

...