DNS issues

Downtime

Aug 29 at 02:05pm PDT

Affected services

limacharlie.io

app.limacharlie.io

jwt.limacharlie.io

API Canada Telemetry Ingestion

API Canada Insight

API Canada External Telemetry

API Canada Cloud Adapters

API Canada Services

API Canada Enrollment

API Canada Management

API Canada Modules

API Canada Directory

API Europe Telemetry Ingestion

API Europe Insight

API Europe External Telemetry

API Europe Cloud Adapters

API Europe Services

API Europe Enrollment

API Europe Management

API Europe Modules

API Europe Directory

API India Telemetry Ingestion

API India Insight

API India External Telemetry

API India Cloud Adapters

API India Services

API India Enrollment

API India Management

API India Modules

API India Directory

API UK Telemetry Ingestion

API UK Insight

API UK External Telemetry

API UK Cloud Adapters

API UK Services

API UK Enrollment

API UK Management

API UK Modules

API UK Directory

API USA Telemetry Ingestion

API USA InsightAPI USA Insight

API USA External Telemetry

API USA Cloud Adapters

API USA Services

API USA Enrollment

API USA Management

API USA Modules

API USA Directory

Resolved
Sep 09 at 07:07am PDT

Summary Root Cause Analysis Report - Post-Mortem

Incident Overview:
On August 29 2024 the limacharlie.io domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken.

Timeline of the Events:

August 29, 2024
- 2:05pm PT: DNS issues began.
- 2:10pm PT: Reports received of customer receiving errors and no longer being able to access orgs.
- 2:21pm PT: Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved.
- 2:30pm PT: Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution.
- 3:38pm PT: Confirmed full resolution at all global root nameservers.

Root Cause:
1. Human Error at upstream provider: LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue.
2. System Status: All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set.

Actions Take:
- Vendor Communication: The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue.

Lessons Learned / Future Recommendations:
- Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented.
- Request written confirmation of proposed manual changes by upstream providers prior to execution.

Updated
Aug 29 at 03:38pm PDT

DNS issues have been confirmed resolved. All services are operational.

Updated
Aug 29 at 02:30pm PDT

The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation.

You may notice intermittent issues as DNS propagation completes.

Created
Aug 29 at 02:05pm PDT

The limacharlie.io domain experienced DNS issues which caused unavailability of services.