Incidents reported on status page for LimaCharlie https://status.limacharlie.io/ https://d1lppblt9t2x15.cloudfront.net/logos/15198662a2b4adfb9330b87b5786b222.png https://status.limacharlie.io/ en https://status.limacharlie.io/ Wed, 01 Oct 2025 17:43:08 +0000 https://status.limacharlie.io/#f1610180d6a5c815074e8c8bc517f1c95d1b3ce8344fdfc8d943d877c23d2a2b API Europe Telemetry Ingestion recovered https://status.limacharlie.io/ Wed, 01 Oct 2025 17:33:15 +0000 https://status.limacharlie.io/#f1610180d6a5c815074e8c8bc517f1c95d1b3ce8344fdfc8d943d877c23d2a2b API Europe Telemetry Ingestion went down https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 14:06:00 -0000 https://status.limacharlie.io/incident/734948#14668d9fae64ba622d346c0864a128c3a4c444754e02e33e143f5d0289b2c991 Root cause has been identified, and all systems have normalized. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 14:06:00 -0000 https://status.limacharlie.io/incident/734948#14668d9fae64ba622d346c0864a128c3a4c444754e02e33e143f5d0289b2c991 Root cause has been identified, and all systems have normalized. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 14:06:00 -0000 https://status.limacharlie.io/incident/734948#14668d9fae64ba622d346c0864a128c3a4c444754e02e33e143f5d0289b2c991 Root cause has been identified, and all systems have normalized. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 14:06:00 -0000 https://status.limacharlie.io/incident/734948#14668d9fae64ba622d346c0864a128c3a4c444754e02e33e143f5d0289b2c991 Root cause has been identified, and all systems have normalized. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 14:06:00 -0000 https://status.limacharlie.io/incident/734948#14668d9fae64ba622d346c0864a128c3a4c444754e02e33e143f5d0289b2c991 Root cause has been identified, and all systems have normalized. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 14:06:00 -0000 https://status.limacharlie.io/incident/734948#14668d9fae64ba622d346c0864a128c3a4c444754e02e33e143f5d0289b2c991 Root cause has been identified, and all systems have normalized. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 14:06:00 -0000 https://status.limacharlie.io/incident/734948#14668d9fae64ba622d346c0864a128c3a4c444754e02e33e143f5d0289b2c991 Root cause has been identified, and all systems have normalized. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 14:06:00 -0000 https://status.limacharlie.io/incident/734948#14668d9fae64ba622d346c0864a128c3a4c444754e02e33e143f5d0289b2c991 Root cause has been identified, and all systems have normalized. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 14:06:00 -0000 https://status.limacharlie.io/incident/734948#14668d9fae64ba622d346c0864a128c3a4c444754e02e33e143f5d0289b2c991 Root cause has been identified, and all systems have normalized. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 13:23:00 -0000 https://status.limacharlie.io/incident/734948#3bcd8924508560c2603c28112aa3f424d5f19876cdb8c8da9cba2f3d0208f511 Systems are stabilizing, still investigating root cause of the issue. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 13:23:00 -0000 https://status.limacharlie.io/incident/734948#3bcd8924508560c2603c28112aa3f424d5f19876cdb8c8da9cba2f3d0208f511 Systems are stabilizing, still investigating root cause of the issue. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 13:23:00 -0000 https://status.limacharlie.io/incident/734948#3bcd8924508560c2603c28112aa3f424d5f19876cdb8c8da9cba2f3d0208f511 Systems are stabilizing, still investigating root cause of the issue. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 13:23:00 -0000 https://status.limacharlie.io/incident/734948#3bcd8924508560c2603c28112aa3f424d5f19876cdb8c8da9cba2f3d0208f511 Systems are stabilizing, still investigating root cause of the issue. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 13:23:00 -0000 https://status.limacharlie.io/incident/734948#3bcd8924508560c2603c28112aa3f424d5f19876cdb8c8da9cba2f3d0208f511 Systems are stabilizing, still investigating root cause of the issue. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 13:23:00 -0000 https://status.limacharlie.io/incident/734948#3bcd8924508560c2603c28112aa3f424d5f19876cdb8c8da9cba2f3d0208f511 Systems are stabilizing, still investigating root cause of the issue. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 13:23:00 -0000 https://status.limacharlie.io/incident/734948#3bcd8924508560c2603c28112aa3f424d5f19876cdb8c8da9cba2f3d0208f511 Systems are stabilizing, still investigating root cause of the issue. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 13:23:00 -0000 https://status.limacharlie.io/incident/734948#3bcd8924508560c2603c28112aa3f424d5f19876cdb8c8da9cba2f3d0208f511 Systems are stabilizing, still investigating root cause of the issue. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 13:23:00 -0000 https://status.limacharlie.io/incident/734948#3bcd8924508560c2603c28112aa3f424d5f19876cdb8c8da9cba2f3d0208f511 Systems are stabilizing, still investigating root cause of the issue. https://status.limacharlie.io/ Tue, 30 Sep 2025 12:26:07 +0000 https://status.limacharlie.io/#3492b6badddec8ee067c984ab4129b337908c6a199d8cc4715adfc98b8857f52 API Europe Telemetry Ingestion recovered https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 11:43:00 -0000 https://status.limacharlie.io/incident/734948#56f84afb29dcc4eea68b45af17998f42d378e2abbceebde077551703fdefc363 We are currently experiencing degraded performance of systems in region Europe. We are investigating these issues, and will update as soon as we have more information. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 11:43:00 -0000 https://status.limacharlie.io/incident/734948#56f84afb29dcc4eea68b45af17998f42d378e2abbceebde077551703fdefc363 We are currently experiencing degraded performance of systems in region Europe. We are investigating these issues, and will update as soon as we have more information. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 11:43:00 -0000 https://status.limacharlie.io/incident/734948#56f84afb29dcc4eea68b45af17998f42d378e2abbceebde077551703fdefc363 We are currently experiencing degraded performance of systems in region Europe. We are investigating these issues, and will update as soon as we have more information. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 11:43:00 -0000 https://status.limacharlie.io/incident/734948#56f84afb29dcc4eea68b45af17998f42d378e2abbceebde077551703fdefc363 We are currently experiencing degraded performance of systems in region Europe. We are investigating these issues, and will update as soon as we have more information. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 11:43:00 -0000 https://status.limacharlie.io/incident/734948#56f84afb29dcc4eea68b45af17998f42d378e2abbceebde077551703fdefc363 We are currently experiencing degraded performance of systems in region Europe. We are investigating these issues, and will update as soon as we have more information. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 11:43:00 -0000 https://status.limacharlie.io/incident/734948#56f84afb29dcc4eea68b45af17998f42d378e2abbceebde077551703fdefc363 We are currently experiencing degraded performance of systems in region Europe. We are investigating these issues, and will update as soon as we have more information. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 11:43:00 -0000 https://status.limacharlie.io/incident/734948#56f84afb29dcc4eea68b45af17998f42d378e2abbceebde077551703fdefc363 We are currently experiencing degraded performance of systems in region Europe. We are investigating these issues, and will update as soon as we have more information. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 11:43:00 -0000 https://status.limacharlie.io/incident/734948#56f84afb29dcc4eea68b45af17998f42d378e2abbceebde077551703fdefc363 We are currently experiencing degraded performance of systems in region Europe. We are investigating these issues, and will update as soon as we have more information. https://status.limacharlie.io/incident/734948 Tue, 30 Sep 2025 11:43:00 -0000 https://status.limacharlie.io/incident/734948#56f84afb29dcc4eea68b45af17998f42d378e2abbceebde077551703fdefc363 We are currently experiencing degraded performance of systems in region Europe. We are investigating these issues, and will update as soon as we have more information. https://status.limacharlie.io/ Tue, 30 Sep 2025 11:36:41 +0000 https://status.limacharlie.io/#3492b6badddec8ee067c984ab4129b337908c6a199d8cc4715adfc98b8857f52 API Europe Telemetry Ingestion went down https://status.limacharlie.io/incident/731217 Wed, 24 Sep 2025 18:04:00 -0000 https://status.limacharlie.io/incident/731217#ac7ab3cb2ebeb97c8909023ca52289bbca13ab44b974477c751ee1c77316baec All systems have returned to normal operation. Thank you for your patience as we worked through this issue. https://status.limacharlie.io/incident/731217 Wed, 24 Sep 2025 18:04:00 -0000 https://status.limacharlie.io/incident/731217#ac7ab3cb2ebeb97c8909023ca52289bbca13ab44b974477c751ee1c77316baec All systems have returned to normal operation. Thank you for your patience as we worked through this issue. https://status.limacharlie.io/incident/731217 Wed, 24 Sep 2025 14:43:00 -0000 https://status.limacharlie.io/incident/731217#1b7f5eb9b2e463d2aa6cdc0b86d91fde212c82c58820a0111930c0ed696d3f13 We are seeing partial recovery in the system. We are continuing to monitor and work toward full resolution. https://status.limacharlie.io/incident/731217 Wed, 24 Sep 2025 14:43:00 -0000 https://status.limacharlie.io/incident/731217#1b7f5eb9b2e463d2aa6cdc0b86d91fde212c82c58820a0111930c0ed696d3f13 We are seeing partial recovery in the system. We are continuing to monitor and work toward full resolution. https://status.limacharlie.io/ Wed, 24 Sep 2025 14:06:16 +0000 https://status.limacharlie.io/#ec71b1cd8ec1e57d5637ccda18047654693fd61165511edab9a9939fac4526bb API Europe Telemetry Ingestion recovered https://status.limacharlie.io/incident/731217 Wed, 24 Sep 2025 13:40:00 -0000 https://status.limacharlie.io/incident/731217#5d28155f412e2075504da8814b307c12a2730dce30b83ca634cc9cbe8cd788f5 We are experiencing issues which impact sensors within our European data centers. Users may see issues with sensors in this region (e.g. sensors attempting to enroll, sensor telemetry ingestion, sensor tasking, etc.) We are actively investigating the issue. https://status.limacharlie.io/incident/731217 Wed, 24 Sep 2025 13:40:00 -0000 https://status.limacharlie.io/incident/731217#5d28155f412e2075504da8814b307c12a2730dce30b83ca634cc9cbe8cd788f5 We are experiencing issues which impact sensors within our European data centers. Users may see issues with sensors in this region (e.g. sensors attempting to enroll, sensor telemetry ingestion, sensor tasking, etc.) We are actively investigating the issue. https://status.limacharlie.io/ Wed, 24 Sep 2025 13:11:20 +0000 https://status.limacharlie.io/#ec71b1cd8ec1e57d5637ccda18047654693fd61165511edab9a9939fac4526bb API Europe Telemetry Ingestion went down https://status.limacharlie.io/ Tue, 23 Sep 2025 22:22:35 +0000 https://status.limacharlie.io/#cdadfbf509350f06b78e330eadace70fb844436602d799110b774fd47ff650d5 API India Telemetry Ingestion recovered https://status.limacharlie.io/ Tue, 23 Sep 2025 22:17:56 +0000 https://status.limacharlie.io/#cdadfbf509350f06b78e330eadace70fb844436602d799110b774fd47ff650d5 API India Telemetry Ingestion went down https://status.limacharlie.io/ Thu, 04 Sep 2025 11:35:51 +0000 https://status.limacharlie.io/#ce613173435405df5f165eb51dd32818242b7427415944dd894069fa0d305323 API Europe Management recovered https://status.limacharlie.io/ Thu, 04 Sep 2025 11:30:46 +0000 https://status.limacharlie.io/#ce613173435405df5f165eb51dd32818242b7427415944dd894069fa0d305323 API Europe Management went down https://status.limacharlie.io/incident/709200 Wed, 20 Aug 2025 11:08:00 -0000 https://status.limacharlie.io/incident/709200#7bfeb9a44c49ff490baf65a716c1e579dc88749f95239482163f8541b5361dda At approx 5:01AM MST, we experienced a short window of outage to Firebase that impacted our web app, and API. System have since recovered. 6:13AM MST - Systems are fully recovered, and nominal. https://status.limacharlie.io/incident/709200 Wed, 20 Aug 2025 11:08:00 -0000 https://status.limacharlie.io/incident/709200#7bfeb9a44c49ff490baf65a716c1e579dc88749f95239482163f8541b5361dda At approx 5:01AM MST, we experienced a short window of outage to Firebase that impacted our web app, and API. System have since recovered. 6:13AM MST - Systems are fully recovered, and nominal. https://status.limacharlie.io/ Thu, 14 Aug 2025 19:50:32 +0000 https://status.limacharlie.io/#ee3d9140cdf11969e15c94f0dd5921044eefb054b8f9094444795ce8fc466259 API Europe Insight recovered https://status.limacharlie.io/ Thu, 14 Aug 2025 19:45:30 +0000 https://status.limacharlie.io/#ee3d9140cdf11969e15c94f0dd5921044eefb054b8f9094444795ce8fc466259 API Europe Insight went down https://status.limacharlie.io/ Tue, 12 Aug 2025 19:45:44 +0000 https://status.limacharlie.io/#398e17051e289d9eb05814f7c4143a75dec418fde284a6db1baa8b181feb5039 API Europe Insight recovered https://status.limacharlie.io/ Tue, 12 Aug 2025 19:40:34 +0000 https://status.limacharlie.io/#398e17051e289d9eb05814f7c4143a75dec418fde284a6db1baa8b181feb5039 API Europe Insight went down https://status.limacharlie.io/ Sun, 27 Jul 2025 19:46:04 +0000 https://status.limacharlie.io/#169999326a18d1abd637bd08bb0ea0400241e1630d56a852c8d0177199588a2c API Europe Directory recovered https://status.limacharlie.io/ Sun, 27 Jul 2025 19:45:30 +0000 https://status.limacharlie.io/#db9a19cae3d2c2a8ad634fb99180b9423941c74ead104461164b323ebac56655 API Europe Enrollment recovered https://status.limacharlie.io/ Sun, 27 Jul 2025 19:45:24 +0000 https://status.limacharlie.io/#ecd8baacb258aa9a107f359d3855294f347b3a8cf54f95d74f7d75eb473953a6 API Europe Cloud Adapters recovered https://status.limacharlie.io/ Sun, 27 Jul 2025 19:41:05 +0000 https://status.limacharlie.io/#169999326a18d1abd637bd08bb0ea0400241e1630d56a852c8d0177199588a2c API Europe Directory went down https://status.limacharlie.io/ Sun, 27 Jul 2025 19:40:42 +0000 https://status.limacharlie.io/#ecd8baacb258aa9a107f359d3855294f347b3a8cf54f95d74f7d75eb473953a6 API Europe Cloud Adapters went down https://status.limacharlie.io/ Sun, 27 Jul 2025 19:40:31 +0000 https://status.limacharlie.io/#db9a19cae3d2c2a8ad634fb99180b9423941c74ead104461164b323ebac56655 API Europe Enrollment went down https://status.limacharlie.io/ Sat, 26 Jul 2025 19:39:24 +0000 https://status.limacharlie.io/#94ad5669ce6822f63a3b7f22c706d85f4b13228352b78c444ba643d24aab1b11 API Europe Cloud Adapters recovered https://status.limacharlie.io/ Sat, 26 Jul 2025 19:34:24 +0000 https://status.limacharlie.io/#94ad5669ce6822f63a3b7f22c706d85f4b13228352b78c444ba643d24aab1b11 API Europe Cloud Adapters went down https://status.limacharlie.io/ Wed, 16 Jul 2025 12:34:14 +0000 https://status.limacharlie.io/#aadc0862ce5a700225bf54b53a2241271cc0aab32c9c33842628f9d00ce6e6af API USA Cloud Adapters recovered https://status.limacharlie.io/ Wed, 16 Jul 2025 12:29:59 +0000 https://status.limacharlie.io/#aadc0862ce5a700225bf54b53a2241271cc0aab32c9c33842628f9d00ce6e6af API USA Cloud Adapters went down https://status.limacharlie.io/ Wed, 16 Jul 2025 10:55:35 +0000 https://status.limacharlie.io/#01b8f5ebd12e85cdf67a529d27e878a0f5646dfb8b85b8c373ee1640a61b5862 API USA Cloud Adapters recovered https://status.limacharlie.io/ Wed, 16 Jul 2025 10:30:08 +0000 https://status.limacharlie.io/#01b8f5ebd12e85cdf67a529d27e878a0f5646dfb8b85b8c373ee1640a61b5862 API USA Cloud Adapters went down https://status.limacharlie.io/ Wed, 16 Jul 2025 10:24:59 +0000 https://status.limacharlie.io/#292ce72e84e3562a9196e4c20ad5f9a95a1ff4fb70b6f0cb8af3e266c0c8a7b8 API USA Cloud Adapters recovered https://status.limacharlie.io/ Wed, 16 Jul 2025 10:20:22 +0000 https://status.limacharlie.io/#292ce72e84e3562a9196e4c20ad5f9a95a1ff4fb70b6f0cb8af3e266c0c8a7b8 API USA Cloud Adapters went down https://status.limacharlie.io/incident/602673 Fri, 13 Jun 2025 15:35:00 -0000 https://status.limacharlie.io/incident/602673#4e3eeb2cc271cced26bb9c22351a979c73e43e3a2d5a1cb6948339c2732beb33 We experienced degraded systems due to residual impact from upstream provider outages from yesterday. We are in a stabilization phase, and are working towards nominal operations. 8:03 AM MST - All systems nominal at this time. Thanks for your patience. https://status.limacharlie.io/incident/602673 Fri, 13 Jun 2025 15:35:00 -0000 https://status.limacharlie.io/incident/602673#4e3eeb2cc271cced26bb9c22351a979c73e43e3a2d5a1cb6948339c2732beb33 We experienced degraded systems due to residual impact from upstream provider outages from yesterday. We are in a stabilization phase, and are working towards nominal operations. 8:03 AM MST - All systems nominal at this time. Thanks for your patience. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/572987 Tue, 20 May 2025 15:59:00 -0000 https://status.limacharlie.io/incident/572987#03adfa12a0dcce6f1b9744818b82401b26ae55cc90b1f51b6a66a6e8b31da023 The issue is identified and resolved. https://status.limacharlie.io/incident/572987 Tue, 20 May 2025 15:39:00 -0000 https://status.limacharlie.io/incident/572987#fc41c581d5e5d4b20b1bcaf6c5474898c2b1322d6e52ea07a1de111dd7d50088 The Web App can not load. Preliminary investigation indicates possible GCP failure. Investigating more. All API, backend services, and event ingestion are operating normally. This is not impacting any backend services. Only the web app portion of our services. 9:52AM MST - This issue seems to be related to GCP services issues. We are engaging GCP to escalate this issue, and working on other methods to stabilize. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/500894 Thu, 23 Jan 2025 22:20:00 -0000 https://status.limacharlie.io/incident/500894#10fa1ffa36d8d19c3bf9635429c7a3308376dc3228e0d157ff993057caf674e1 Situation : We are experiencing an issue with the reliable-tasking extension. Task : We are effecting hardware upgrade that we expect to stabilize and resolve the issue. Expected Impact: There will be a loss of currently buffered reliable tasks. Jan 24, 2025 10AM - Issue has been stabilized, but we are still root causing the issue. No expected Impact at this time. We will update as we have more information. Jan 24, 2025 5PM - We have implemented a fix, and while we expect resolution, will be monitoring over the next 24hrs. Next update tomorrow. https://status.limacharlie.io/incident/483417 Wed, 11 Dec 2024 15:18:00 -0000 https://status.limacharlie.io/incident/483417#6a756918fd2f30ab79a39e67cdfa148013c46ec06f929c5e8f8589920f330e02 ## Summary Root Cause Analysis Report - Post-Mortem ## Incident Overview: On December 11th, 2024, a few LimaCharlie customers reported Blue Screen of Death (BSOD) issues affecting Windows systems running version 4.32.1 of the LC EDR Agent. The issue only affected a small subset of systems running `lc:latest` that were under high memory pressure. Engineering quickly identified the issue and rolled back the agent version from `4.32.1` to `4.32.0` to mitigate the problem. The incident response was swift, with engineering identifying and diagnosing the problem within approximately one hour of the first customer report. A community-wide announcement was made and remediation was implemented through a version rollback within 2.5 hours of initial report. ## Timeline of Events: - 2024-12-10 @ 9:41am PT: LC EDR Agent version `4.32.1` is released - 2024-12-11 @ 05:02am PT: Multiple customers reported isolated Blue Screen of Death (BSOD) incidents on Windows systems running the LimaCharlie EDR Agent - 2024-12-11 @ 06:06am PT: LC engineering diagnosed a kernel driver defect in `latest` endpoint agent version `4.32.1` - 2024-12-11 @ 07:18am PT: Announcement posted in LC Community Slack - 2024-12-11 @ 07:30am PT: LC Agent `latest` version rolled back to `4.32.0` ## Root Causes: The root cause was identified as a kernel driver bug related to improper IRQL handling when accessing paged memory with a spin lock. The issue manifested when the kernel driver (`tmp_hbs_acq.sys`) attempted to access pageable memory at an interrupt request level (IRQL) that was too high, triggering Windows bug check D1 (`DRIVER_IRQL_NOT_LESS_OR_EQUAL`). This bug was infrequent on test systems where no memory pressure was present. ## Actions Taken: - Implement stricter IRQL validation checks in the kernel driver code to ensure proper memory access patterns - Add comprehensive memory access guards around paged pool operations when holding spin locks - Enhance the driver's memory management to properly handle high-pressure scenarios - Implement additional automated testing specifically for IRQL-related scenarios in the kernel driver - Enhance pre-release testing environment to better simulate high-memory-pressure scenarios and kernel-level interactions across diverse Windows configurations These changes will be implemented and thoroughly tested before being included in a future release. The development team will also review other areas of the kernel driver code for similar potential IRQL issues. ## Lessons Learned: - The critical importance of thorough IRQL validation in kernel driver development to prevent system crashes - The need for enhanced pre-release testing environments that better simulate real-world conditions, particularly high-memory-pressure scenarios - The value of quick rollback capabilities when dealing with kernel-level components - The importance of maintaining detailed version control and change documentation to quickly identify potential causes of issues - The effectiveness of rapid customer communication and transparent incident handling in maintaining trust ## Future Recommendations: - Enhance documentation and communication around the intended purpose of `lc:latest` tag, emphasizing its role as a testing/staging version rather than for production use - Develop best practices documentation for rolling out new agent versions across customer environments, including recommended testing procedures and rollback strategies - Added “Best Practices” to docs: https://docs.limacharlie.io/docs/endpoint-agent-versioning-and-upgrades#updating-endpoint-agents - Update web app to display both `latest` and `stable` as current versions on the Deployed Versions page, removing UI indicators that suggest `stable` to be outdated. - Implement a formal notification system for `lc:latest` version changes to help customers better track and manage testing cycles https://status.limacharlie.io/incident/462214 Fri, 15 Nov 2024 19:03:00 -0000 https://status.limacharlie.io/incident/462214#b8de15150703c6507801266a9d32f7be39e083c06341d67ce1feb900319106f1 Performance issues should be resolved. We continue to monitor systems to ensure sustained performance. https://status.limacharlie.io/incident/462214 Fri, 15 Nov 2024 18:49:00 -0000 https://status.limacharlie.io/incident/462214#232c64165284ce70cf68b6d4fdf9e2fd92837f2ef66dbd2fa0eadda6932424bc Users may see performance issues when loading sensor lists. We are aware of the issue and are working on a resolution. https://status.limacharlie.io/incident/461630 Thu, 14 Nov 2024 18:58:00 -0000 https://status.limacharlie.io/incident/461630#a35d29268a08db294254be07148ffc586b73b970bc1ac5bb0a59cfd04ef02735 Performance issues should be resolved. We continue to monitor systems to ensure sustained performance. https://status.limacharlie.io/incident/461630 Thu, 14 Nov 2024 18:54:00 -0000 https://status.limacharlie.io/incident/461630#083ce016e912e91c8c3eeed4dd6bfd3d11e60f97a2a62268efc542885a04b744 Users may see performance issues when loading sensor lists. We are aware of the issue and are working on a resolution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/420234 Tue, 27 Aug 2024 18:52:00 -0000 https://status.limacharlie.io/incident/420234#79d9171dd674cc4e8adf887c497a5a632abd86c4dfcef2a219ee90c3d8124d15 Most services have returned to normal operation. We are continuing to monitor. https://status.limacharlie.io/incident/420234 Tue, 27 Aug 2024 18:37:00 -0000 https://status.limacharlie.io/incident/420234#450f116623fb6b2e9a697224ec7a2573c6019a194331ad17e4400d9d62cc323b We are investigating reports of issues with the LimaCharlie API. These issues also appear to be impacting the LimaCharlie web interface. Additional details to follow. https://status.limacharlie.io/incident/377074 Thu, 30 May 2024 17:27:00 -0000 https://status.limacharlie.io/incident/377074#5fb679df204a4d644e893b6747b732935aefb33a3550ea915dc798c92bb83cf1 Team, we have had reports of the app.limacharlie.io not loading the first time. While we are investigating this issue in more depth, the quick solution is a hard refresh of the site in browser. We will update with more information as we have it. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/369919 Wed, 15 May 2024 21:47:00 -0000 https://status.limacharlie.io/incident/369919#ddbb8bf0c47c9c4579ca017d4591d9a0c4761f3f1c31001d790243de7e9ac879 We are currently experiencing high latency impact from a Google Cloud DataStore issue, that is impacting our US services. We have engaged GCP directly and have escalated the issue. This issue is ongoing and we expect degraded service quality during this time. We will be providing updates as we have them and are working to remedy this issue as quickly as possible. May 15 2024 7:49PM UPDATE - We are still working the issue with GCP support and will update as we have more information. May 15 2024 11:34PM UPDATE - We are still working with GCP support to bring this issue to resolution. More updates as we have them. May 16 2024 9:34AM MST UPDATE - We have identified a possible root cause and working towards resolution. We expect degraded performance from Cloud Sensors during this issue. May 16 2024 4:54PM MST UPDATE - We have identified the issue and are working towards resolution. We expect minimal impact to Cloud Adapters at this time. May 17 2024 2:21PM MST UPDATE - Latency performance is back with acceptable ranges, and we are calling this incident resolved. All systems are nominal. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/303482 Thu, 28 Dec 2023 23:00:00 -0000 https://status.limacharlie.io/incident/303482#14a46a345d2fdcb395b5f4df22dac0820df2c6cbfdae902791ba1c7577f5ca68 Maintenance completed https://status.limacharlie.io/incident/303482 Thu, 28 Dec 2023 22:30:00 -0000 https://status.limacharlie.io/incident/303482#2bc202b2ca405f3ce8e06b9cd4c8f01477ecfb93cf18f316f166c73a92384601 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in the USA region. During this maintenance, our systems will be in a partially degraded state for a period of up to 30 minutes. Maintenance Window Details: Date: 12-28-2023 Start Time: 3:30PM MST End Time: 4:00PM MST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified timeframe to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/293237 Thu, 14 Dec 2023 03:37:00 -0000 https://status.limacharlie.io/incident/293237#1579251df3a7f350ca675261ffe808f1baaf13bf86599d6f187037de997f7f1a **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On November 28, 2023, an issue was identified with Sensor Live items not functioning properly. The problem was initially reported as an issue with the LimaCharlie console. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of Events:** - **Nov 28, Early Morning:** Customers began reporting the incident via Slack. - **Throughout the Morning:** Various team members engaged to investigate issue. The team validated the issue and observed it appeared to impact both backend and frontend systems. - **Mid-Morning:** The affected systems were identified. Service restarts were necessary to bring about a resolution to the issue. **Root Causes:** 1. **Infrastructure Robustness:** The reliance on some older technologies brought about complications that did not allow for simple troubleshooting of the issue. This indicated a need for a more robust and autonomous infrastructure. 2. **System Monitoring Gaps:** System probes were unable to detect the service degradation due to limitations in their configuration. 3. **Communication Delays:** A subset of configuration was missed during the change of emergency notification tooling which led to a delay in notification to the appropriate parties, causing delays in addressing the issue. **Actions Taken:** 1. **Testing and Infrastructure Improvements:** Initiatives to replace older technologies with more modern solutions and enhance monitoring with improved metrics and tests. 2. **Incident Reporting Process Updates:** An updated email system will be used. 3. **Customer Communication:** Plans to revise customer notifications regarding incidents and include incident response information in our onboarding process. 4. **Internal Documentation:** Creation of documentation on incident reporting best practices and severity levels, ensuring all team members are informed and prepared. **Lessons Learned:** - The importance of robust monitoring systems that can detect not just system uptime but also functional performance. - The need for clear and efficient communication channels both internally and with customers. - The value of having a resilient and autonomous infrastructure to reduce downtime and manual intervention requirements. **Future Recommendations:** - Continue to enhance system monitoring and alerting capabilities. - Streamline communication protocols for faster incident response. - Regularly review and update infrastructure to ensure it meets the evolving needs of the service and customers. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/293237 Tue, 28 Nov 2023 15:02:00 -0000 https://status.limacharlie.io/incident/293237#d60d419c3e538f4afb8a48f1dd95eb8f5568c8488170a343a7347f31a09a3453 Console commands appear to be working as expected. We continue to monitor. For reference, automated taskings continued to work as expected throughout this incident. https://status.limacharlie.io/incident/293237 Tue, 28 Nov 2023 11:52:00 -0000 https://status.limacharlie.io/incident/293237#847949d349d28af83df75c39600152d07ce75b21afd475d007f7bda2be016c2f We are investigating an issue that is impacting the Console functionality. Additional details will be posted to this page once they become available. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/164362 Sat, 14 Jan 2023 06:34:00 -0000 https://status.limacharlie.io/incident/164362#8b64c4a76074c8a52888de2f198c0523574c10242ac3657a2401a40d670b3d6c # 2023-01-13 - Cisco Umbrella blocking access to LimaCharlie cloud On 2023-01-13 we became aware of an issue where Cisco Umbrella was blocking access to the LimaCharlie cloud. ## Status We are awaiting a response from the Cisco Talos team to have the "limacharlie.io" domain added to their false positive list. In the meantime impacted customers may add the following LimaCharlie domains to their Global Allow list by following the steps below: ## Workaround Steps 1. Log into the [Cisco Umbrella portal](https://login.umbrella.com/) 2. Go to Policies -> Policy Components -> Destination Lists. 3. Add the following entries to the Global Allow List: limacharlie.io 9157798c50af372c.lc.limacharlie.io 70182cf634c346bd.lc.limacharlie.io 4d897015b0815621.lc.limacharlie.io b76093c3662d5b4f.lc.limacharlie.io aae67d7e76570ec1.lc.limacharlie.io Your sensors should start resuming connections to the LimaCharlie cloud. ## Requesting your assistance If you have any contacts at Cisco who may be able to get this unblocking expedited, please reach out to us via [email](mailto:answers@limacharlie.io) or on our [community Slack](https://slack.limacharlie.io) Additionally, please submit a [Web Reputation Support Ticket](https://talosintelligence.com/reputation_center/web_reputation) to Cisco Talos. This will help increase the reputation of the domains above. ## Impact Customers who utilize both the Cisco Umbrella network security product may have found they have suddenly lost connectivity with the LimaCharlie cloud. This includes access to the LimaCharlie: - web application, - APIs, - endpoint telemetry servers, - output destination servers ## Mitigation of Risk The LimaCharlie sensor continues to collect telemetry while offline, and transmits it once the connection has been re-established. Further details can be found in our article ["What happens when the host is offline?"](https://help.limacharlie.io/en/articles/5662761-what-happens-when-the-host-is-offline). ## Timeline ### 2023-01-13 8:18 a.m. ET [User reported issue via community Slack](https://lcendpoint.slack.com/archives/C1S96NSGP/p1673615906520759) 1:04 p.m. ET [User reported issue via private Slack channel](https://refractionpoint.slack.com/archives/C02BWLJQZE3/p1673633043017499) 1:08 p.m. ET Opened [Cisco Umbrella support ticket #1448986](https://mail.google.com/mail/u/0/#sent/KtbxLvgxCGNmCrmqmsKfNHZChFrFjcXdxB), per [reporting procedure](https://blog.talosintelligence.com/talosintelligencecom-is-rolling-out-new/) 1:31 p.m. ET Added notice to [#status in community Slack](https://lcendpoint.slack.com/archives/CMS9NA7NK/p1673634585175149) 1:39 p.m. ET Submitted false positive report to [Talos Intelligence](https://talosintelligence.com/reputation_center/sender_domain), submitted the following domains as false positives: [limacharlie.io](https://talosintelligence.com/tickets/sdr/313558) [9157798c50af372c.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313559) [70182cf634c346bd.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313560) [4d897015b0815621.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313561) [b76093c3662d5b4f.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313562) [aae67d7e76570ec1.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313563) 1:45 p.m. ET Signed up for Cisco Umbrella trial to verify steps to add limacharlie.io domains to the allow list. Submitted a [categorization dispute](https://docs.umbrella.com/umbrella-user-guide/docs/dispute-a-categorization#category-request-umbrella) 2:46 p.m. ET Called Cisco TAC via telephone to try to escalate issue; they emailed Umbrella support on our behalf and provided my direct telephone number. 3:30 p.m. ET Opened another support ticket via [Cisco Umbrella support request #1449194](https://support.umbrella.com/hc/en-us/requests/1449194); case ID #1449194 7:29 p.m. ET Submitted escalation request via email on Talos ticket ### 2023-01-14 1:09 a.m. ET Validated issue still persists via [Talos reputation center](https://www.talosintelligence.com/reputation_center) Submitted [Web Reputation Support Ticket](https://talosintelligence.com/tickets/313642) **Included the following in the ticket:** Cisco Umbrella is incorrectly reporting the LimaCharlie domains listed as Phishing. LimaCharlie provides a legitimate security infrastructure-as-a-service offering, which can be validated at these external sites: [GitHub](https://github.com/refractionPOINT/limacharlie) Shows our long standing history, first as an open source project and then as a commercial offering. See link to our public repositories which include links to our domain with authenticated date/time stamps. [Crunchbase](https://www.crunchbase.com/organization/limacharlie) Shows funding for our company and the legitimate backers behind us. Further questions can be directed to Amrik Randhawa, Chief Operating Officer via email at {answers@limacharlie.io -- telephone number provided in the submitted ticket}. 4:11 a.m. ET Outreach to Cisco Talos Intelligence Group via connection 6:47 a.m. ET User reported incident resolution. 7:18 a.m. ET Status of submitted [Talos Tickets](https://talosintelligence.com/tickets) still appears in "New" state. [Cisco Talos Reputation Center](https://talosintelligence.com/reputation_center/lookup?search=limacharlie.io) still shows domains in Untrusted state. 1:53 p.m. ET [Cisco Talos Reputation Center](https://talosintelligence.com/reputation_center/lookup?search=limacharlie.io) now shows domains in Neutral state. Users have started reporting systems returning back to normal operation. 4:22 p.m. ET We continue to work to get the LimaCharlie domains in a Trusted state with Cisco Talos. We have received reports that other Cisco security products such as IronPort are still negatively impacted. ### 2023-01-16 3:51 a.m. ET Received report of Palo Alto XDR flagging limacharlie.io domains as malicious. 7:51 a.m. ET Cisco Talos Support tickets marked as "RESOLVED - Please allow 24h for your system to update" with comment: "This domain was penalized by SDR as it used to have a Untrusted Web Reputation score due to malicious or suspicious behavior. The issue has been rectified and the domain reputation has been adjusted accordingly." 10:44 a.m. ET Posted message to [Palo Alto LIVE community](https://live.paloaltonetworks.com/t5/cortex-xdr-discussions/cortex-xdr-false-positive-detection-of-limacharlie-sensor-edr/m-p/527230#M3399) to report false positive. https://status.limacharlie.io/incident/164362 Sat, 14 Jan 2023 06:34:00 -0000 https://status.limacharlie.io/incident/164362#8b64c4a76074c8a52888de2f198c0523574c10242ac3657a2401a40d670b3d6c # 2023-01-13 - Cisco Umbrella blocking access to LimaCharlie cloud On 2023-01-13 we became aware of an issue where Cisco Umbrella was blocking access to the LimaCharlie cloud. ## Status We are awaiting a response from the Cisco Talos team to have the "limacharlie.io" domain added to their false positive list. In the meantime impacted customers may add the following LimaCharlie domains to their Global Allow list by following the steps below: ## Workaround Steps 1. Log into the [Cisco Umbrella portal](https://login.umbrella.com/) 2. Go to Policies -> Policy Components -> Destination Lists. 3. Add the following entries to the Global Allow List: limacharlie.io 9157798c50af372c.lc.limacharlie.io 70182cf634c346bd.lc.limacharlie.io 4d897015b0815621.lc.limacharlie.io b76093c3662d5b4f.lc.limacharlie.io aae67d7e76570ec1.lc.limacharlie.io Your sensors should start resuming connections to the LimaCharlie cloud. ## Requesting your assistance If you have any contacts at Cisco who may be able to get this unblocking expedited, please reach out to us via [email](mailto:answers@limacharlie.io) or on our [community Slack](https://slack.limacharlie.io) Additionally, please submit a [Web Reputation Support Ticket](https://talosintelligence.com/reputation_center/web_reputation) to Cisco Talos. This will help increase the reputation of the domains above. ## Impact Customers who utilize both the Cisco Umbrella network security product may have found they have suddenly lost connectivity with the LimaCharlie cloud. This includes access to the LimaCharlie: - web application, - APIs, - endpoint telemetry servers, - output destination servers ## Mitigation of Risk The LimaCharlie sensor continues to collect telemetry while offline, and transmits it once the connection has been re-established. Further details can be found in our article ["What happens when the host is offline?"](https://help.limacharlie.io/en/articles/5662761-what-happens-when-the-host-is-offline). ## Timeline ### 2023-01-13 8:18 a.m. ET [User reported issue via community Slack](https://lcendpoint.slack.com/archives/C1S96NSGP/p1673615906520759) 1:04 p.m. ET [User reported issue via private Slack channel](https://refractionpoint.slack.com/archives/C02BWLJQZE3/p1673633043017499) 1:08 p.m. ET Opened [Cisco Umbrella support ticket #1448986](https://mail.google.com/mail/u/0/#sent/KtbxLvgxCGNmCrmqmsKfNHZChFrFjcXdxB), per [reporting procedure](https://blog.talosintelligence.com/talosintelligencecom-is-rolling-out-new/) 1:31 p.m. ET Added notice to [#status in community Slack](https://lcendpoint.slack.com/archives/CMS9NA7NK/p1673634585175149) 1:39 p.m. ET Submitted false positive report to [Talos Intelligence](https://talosintelligence.com/reputation_center/sender_domain), submitted the following domains as false positives: [limacharlie.io](https://talosintelligence.com/tickets/sdr/313558) [9157798c50af372c.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313559) [70182cf634c346bd.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313560) [4d897015b0815621.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313561) [b76093c3662d5b4f.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313562) [aae67d7e76570ec1.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313563) 1:45 p.m. ET Signed up for Cisco Umbrella trial to verify steps to add limacharlie.io domains to the allow list. Submitted a [categorization dispute](https://docs.umbrella.com/umbrella-user-guide/docs/dispute-a-categorization#category-request-umbrella) 2:46 p.m. ET Called Cisco TAC via telephone to try to escalate issue; they emailed Umbrella support on our behalf and provided my direct telephone number. 3:30 p.m. ET Opened another support ticket via [Cisco Umbrella support request #1449194](https://support.umbrella.com/hc/en-us/requests/1449194); case ID #1449194 7:29 p.m. ET Submitted escalation request via email on Talos ticket ### 2023-01-14 1:09 a.m. ET Validated issue still persists via [Talos reputation center](https://www.talosintelligence.com/reputation_center) Submitted [Web Reputation Support Ticket](https://talosintelligence.com/tickets/313642) **Included the following in the ticket:** Cisco Umbrella is incorrectly reporting the LimaCharlie domains listed as Phishing. LimaCharlie provides a legitimate security infrastructure-as-a-service offering, which can be validated at these external sites: [GitHub](https://github.com/refractionPOINT/limacharlie) Shows our long standing history, first as an open source project and then as a commercial offering. See link to our public repositories which include links to our domain with authenticated date/time stamps. [Crunchbase](https://www.crunchbase.com/organization/limacharlie) Shows funding for our company and the legitimate backers behind us. Further questions can be directed to Amrik Randhawa, Chief Operating Officer via email at {answers@limacharlie.io -- telephone number provided in the submitted ticket}. 4:11 a.m. ET Outreach to Cisco Talos Intelligence Group via connection 6:47 a.m. ET User reported incident resolution. 7:18 a.m. ET Status of submitted [Talos Tickets](https://talosintelligence.com/tickets) still appears in "New" state. [Cisco Talos Reputation Center](https://talosintelligence.com/reputation_center/lookup?search=limacharlie.io) still shows domains in Untrusted state. 1:53 p.m. ET [Cisco Talos Reputation Center](https://talosintelligence.com/reputation_center/lookup?search=limacharlie.io) now shows domains in Neutral state. Users have started reporting systems returning back to normal operation. 4:22 p.m. ET We continue to work to get the LimaCharlie domains in a Trusted state with Cisco Talos. We have received reports that other Cisco security products such as IronPort are still negatively impacted. ### 2023-01-16 3:51 a.m. ET Received report of Palo Alto XDR flagging limacharlie.io domains as malicious. 7:51 a.m. ET Cisco Talos Support tickets marked as "RESOLVED - Please allow 24h for your system to update" with comment: "This domain was penalized by SDR as it used to have a Untrusted Web Reputation score due to malicious or suspicious behavior. The issue has been rectified and the domain reputation has been adjusted accordingly." 10:44 a.m. ET Posted message to [Palo Alto LIVE community](https://live.paloaltonetworks.com/t5/cortex-xdr-discussions/cortex-xdr-false-positive-detection-of-limacharlie-sensor-edr/m-p/527230#M3399) to report false positive.