Back to overview
Downtime

DNS issues

Aug 29 at 02:05pm PDT
Affected services
limacharlie.io
app.limacharlie.io
jwt.limacharlie.io
API Canada Telemetry Ingestion
API Canada Insight
API Canada External Telemetry
API Canada Cloud Adapters
API Canada Services
API Canada Enrollment
API Canada Management
API Canada Modules
API Canada Directory
API Europe Telemetry Ingestion
API Europe Insight
API Europe External Telemetry
API Europe Cloud Adapters
API Europe Services
API Europe Enrollment
API Europe Management
API Europe Modules
API Europe Directory
API India Telemetry Ingestion
API India Insight
API India External Telemetry
API India Cloud Adapters
API India Services
API India Enrollment
API India Management
API India Modules
API India Directory
API UK Telemetry Ingestion
API UK Insight
API UK External Telemetry
API UK Cloud Adapters
API UK Services
API UK Enrollment
API UK Management
API UK Modules
API UK Directory
API USA Telemetry Ingestion
API USA InsightAPI USA Insight
API USA External Telemetry
API USA Cloud Adapters
API USA Services
API USA Enrollment
API USA Management
API USA Modules
API USA Directory

Status Report Update State Resolved
Sep 09 at 07:07am PDT

Summary Root Cause Analysis Report - Post-Mortem

Incident Overview:
On August 29 2024 the limacharlie.io domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken.

Timeline of the Events:

August 29, 2024
- 2:05pm PT: DNS issues began.
- 2:10pm PT: Reports received of customer receiving errors and no longer being able to access orgs.
- 2:21pm PT: Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved.
- 2:30pm PT: Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution.
- 3:38pm PT: Confirmed full resolution at all global root nameservers.

Root Cause:
1. Human Error at upstream provider: LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue.
2. System Status: All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set.

Actions Take:
- Vendor Communication: The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue.

Lessons Learned / Future Recommendations:
- Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented.
- Request written confirmation of proposed manual changes by upstream providers prior to execution.

Status Report Update State Updated
Aug 29 at 03:38pm PDT

DNS issues have been confirmed resolved. All services are operational.

Status Report Update State Updated
Aug 29 at 02:30pm PDT

The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation.

You may notice intermittent issues as DNS propagation completes.

Status Report Update State Created
Aug 29 at 02:05pm PDT

The limacharlie.io domain experienced DNS issues which caused unavailability of services.