View Events

Event Update From: 07/28/2022 01:18 PM
Title:
AWS Network Connectivity Issues Recoverying
Summary of Work:
			AWS states that they are seeing the recovery of resources in the affected Availability Zone/Region.

10:49 AM PDT We continue to see recovery of EC2 instances that were affected by the loss of power in a single Availability Zone in the US-EAST-2 Region. At this stage, the vast majority of affected EC2 instances and EBS volumes have returned to a healthy state and we continue to work on the remaining EC2 instances and EBS volumes. Elastic Load Balancing has shifted traffic away from the affected Availability Zone. Single-AZ RDS databases were also affected and will recover as the underlying EC2 instance recovers. Multi-AZ RDS databases would have mitigated impact by failing away from the affected Availability Zone. While the vast majority of Lambda functions continue operating normally, some functions are experiencing invocation failures and latencies, but we expect this to improve over the next 30 minutes. Power has been restored to all affected resources and remains stable. We expect the recovery of EC2 instances and EBS volumes to continue to improve over the next 45 minutes. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.


Affected AWS Services:
AWS Amplify
AWS Cloud9
AWS CodeCommit
AWS CodePipeline
AWS Greengrass
AWS IoT Events
AWS Lambda
AWS RoboMaker
AWS Service Catalog
AWS Transit Gateway
Amazon API Gateway
Amazon AppStream 2.0
Amazon CloudWatch
Amazon DynamoDB
Amazon ElastiCache
Amazon Elastic Container Registry
Amazon Elastic Kubernetes Service
Amazon Elastic Load Balancing
Amazon Forecast
Amazon Kinesis Analytics
Amazon Location Service
Amazon Lookout for Metrics
Amazon MQ
Amazon Managed Service for Prometheus
Amazon MemoryDB for Redis
Amazon OpenSearch Service
Amazon Personalize
Amazon Relational Database Service
Amazon Route 53 Resolver
Amazon Timestream
			
Event Start:
07/28/2022 12:43 PM
Event End:


Original Event From: 07/28/2022 12:43 PM
Resolution:
AWS states that all affected services have recovered. Starting at 9:57 AM PDT some EC2 instances and EBS volumes experienced a loss of power within a single Availability Zone in the US-EAST-2 Region. Power was restored at 10:19 AM PDT and EC2 instances and EBS volumes began to recover. By 10:23 AM PDT, the vast majority of EC2 instances and EBS volumes has fully recovered and by 11:37 AM PDT, all but a very small number of EC2 instances and EBS volumes had recovered. Elastic Load Balancing shifted traffic away from the affected Availability Zone, which has now been shifted back. RDS impact for single-AZ databases, which have also been recovered. Other services (tagged below) saw impact during the event, but most have fully recovered. Those that are still seeing impact, will provide updates via the Personal Health Dashboard as they work towards full recovery. The issue has been resolved and the service is operating normally.
Title:
AWS Network Connectivity Issues
Summary of Work:
		Some users of AWS are experiencing connectivity to EC2 instances within the US-EAST-2 (Ohio) region. AWS stated that a single availability zone was affected and, if possible, move resources to another availability zone.
		
Impacted Audience:
Users of AWS and Users of systems in AWS
Impact of Work:
		10:25 AM PDT We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the majority of the affected EC2 instances have recovered. We expect to recover the vast majority of EC2 instances within the next hour. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.
		
Event Start:
07/28/2022 12:43 PM
Event End:
Affected Systems/Services:
Not listed - please see Impact of Work
Affected Campus:
Chicago,Springfield,Urbana-Champaign