Incidents reported on status page for LimaCharlie https://status.limacharlie.io/ https://d1lppblt9t2x15.cloudfront.net/logos/15198662a2b4adfb9330b87b5786b222.png https://status.limacharlie.io/ en https://status.limacharlie.io/incident/709200 Wed, 20 Aug 2025 11:08:00 -0000 https://status.limacharlie.io/incident/709200#7bfeb9a44c49ff490baf65a716c1e579dc88749f95239482163f8541b5361dda At approx 5:01AM MST, we experienced a short window of outage to Firebase that impacted our web app, and API. System have since recovered. 6:13AM MST - Systems are fully recovered, and nominal. https://status.limacharlie.io/incident/709200 Wed, 20 Aug 2025 11:08:00 -0000 https://status.limacharlie.io/incident/709200#7bfeb9a44c49ff490baf65a716c1e579dc88749f95239482163f8541b5361dda At approx 5:01AM MST, we experienced a short window of outage to Firebase that impacted our web app, and API. System have since recovered. 6:13AM MST - Systems are fully recovered, and nominal. https://status.limacharlie.io/ Thu, 14 Aug 2025 19:50:32 +0000 https://status.limacharlie.io/#ee3d9140cdf11969e15c94f0dd5921044eefb054b8f9094444795ce8fc466259 API Europe Insight recovered https://status.limacharlie.io/ Thu, 14 Aug 2025 19:45:30 +0000 https://status.limacharlie.io/#ee3d9140cdf11969e15c94f0dd5921044eefb054b8f9094444795ce8fc466259 API Europe Insight went down https://status.limacharlie.io/ Tue, 12 Aug 2025 19:45:44 +0000 https://status.limacharlie.io/#398e17051e289d9eb05814f7c4143a75dec418fde284a6db1baa8b181feb5039 API Europe Insight recovered https://status.limacharlie.io/ Tue, 12 Aug 2025 19:40:34 +0000 https://status.limacharlie.io/#398e17051e289d9eb05814f7c4143a75dec418fde284a6db1baa8b181feb5039 API Europe Insight went down https://status.limacharlie.io/ Sun, 27 Jul 2025 19:46:04 +0000 https://status.limacharlie.io/#169999326a18d1abd637bd08bb0ea0400241e1630d56a852c8d0177199588a2c API Europe Directory recovered https://status.limacharlie.io/ Sun, 27 Jul 2025 19:45:30 +0000 https://status.limacharlie.io/#db9a19cae3d2c2a8ad634fb99180b9423941c74ead104461164b323ebac56655 API Europe Enrollment recovered https://status.limacharlie.io/ Sun, 27 Jul 2025 19:45:24 +0000 https://status.limacharlie.io/#ecd8baacb258aa9a107f359d3855294f347b3a8cf54f95d74f7d75eb473953a6 API Europe Cloud Adapters recovered https://status.limacharlie.io/ Sun, 27 Jul 2025 19:41:05 +0000 https://status.limacharlie.io/#169999326a18d1abd637bd08bb0ea0400241e1630d56a852c8d0177199588a2c API Europe Directory went down https://status.limacharlie.io/ Sun, 27 Jul 2025 19:40:42 +0000 https://status.limacharlie.io/#ecd8baacb258aa9a107f359d3855294f347b3a8cf54f95d74f7d75eb473953a6 API Europe Cloud Adapters went down https://status.limacharlie.io/ Sun, 27 Jul 2025 19:40:31 +0000 https://status.limacharlie.io/#db9a19cae3d2c2a8ad634fb99180b9423941c74ead104461164b323ebac56655 API Europe Enrollment went down https://status.limacharlie.io/ Sat, 26 Jul 2025 19:39:24 +0000 https://status.limacharlie.io/#94ad5669ce6822f63a3b7f22c706d85f4b13228352b78c444ba643d24aab1b11 API Europe Cloud Adapters recovered https://status.limacharlie.io/ Sat, 26 Jul 2025 19:34:24 +0000 https://status.limacharlie.io/#94ad5669ce6822f63a3b7f22c706d85f4b13228352b78c444ba643d24aab1b11 API Europe Cloud Adapters went down https://status.limacharlie.io/ Wed, 16 Jul 2025 12:34:14 +0000 https://status.limacharlie.io/#aadc0862ce5a700225bf54b53a2241271cc0aab32c9c33842628f9d00ce6e6af API USA Cloud Adapters recovered https://status.limacharlie.io/ Wed, 16 Jul 2025 12:29:59 +0000 https://status.limacharlie.io/#aadc0862ce5a700225bf54b53a2241271cc0aab32c9c33842628f9d00ce6e6af API USA Cloud Adapters went down https://status.limacharlie.io/ Wed, 16 Jul 2025 10:55:35 +0000 https://status.limacharlie.io/#01b8f5ebd12e85cdf67a529d27e878a0f5646dfb8b85b8c373ee1640a61b5862 API USA Cloud Adapters recovered https://status.limacharlie.io/ Wed, 16 Jul 2025 10:30:08 +0000 https://status.limacharlie.io/#01b8f5ebd12e85cdf67a529d27e878a0f5646dfb8b85b8c373ee1640a61b5862 API USA Cloud Adapters went down https://status.limacharlie.io/ Wed, 16 Jul 2025 10:24:59 +0000 https://status.limacharlie.io/#292ce72e84e3562a9196e4c20ad5f9a95a1ff4fb70b6f0cb8af3e266c0c8a7b8 API USA Cloud Adapters recovered https://status.limacharlie.io/ Wed, 16 Jul 2025 10:20:22 +0000 https://status.limacharlie.io/#292ce72e84e3562a9196e4c20ad5f9a95a1ff4fb70b6f0cb8af3e266c0c8a7b8 API USA Cloud Adapters went down https://status.limacharlie.io/ Mon, 07 Jul 2025 18:34:29 +0000 https://status.limacharlie.io/#4f00e937816570058bfe271eda4c421ccc75923560d64bc6fd13082ec1d43719 docs.limacharlie.io recovered https://status.limacharlie.io/ Mon, 07 Jul 2025 18:19:31 +0000 https://status.limacharlie.io/#4f00e937816570058bfe271eda4c421ccc75923560d64bc6fd13082ec1d43719 docs.limacharlie.io went down https://status.limacharlie.io/ Fri, 13 Jun 2025 19:27:35 +0000 https://status.limacharlie.io/#92ca22d33f53d6f53244727e7be36c9fbdc2d4598419e9fe50c86378debb40ce API Europe Directory recovered https://status.limacharlie.io/ Fri, 13 Jun 2025 19:27:17 +0000 https://status.limacharlie.io/#0462462683de23f80170756c7ec94080833d4464be74d1e197d44e8b878d99c8 API Europe Management recovered https://status.limacharlie.io/ Fri, 13 Jun 2025 19:26:52 +0000 https://status.limacharlie.io/#ba14013dd616f52188a6953e397f298b7359e45708463268b93e79b9ba699618 API Europe Services recovered https://status.limacharlie.io/ Fri, 13 Jun 2025 19:22:36 +0000 https://status.limacharlie.io/#92ca22d33f53d6f53244727e7be36c9fbdc2d4598419e9fe50c86378debb40ce API Europe Directory went down https://status.limacharlie.io/ Fri, 13 Jun 2025 19:22:15 +0000 https://status.limacharlie.io/#0462462683de23f80170756c7ec94080833d4464be74d1e197d44e8b878d99c8 API Europe Management went down https://status.limacharlie.io/ Fri, 13 Jun 2025 19:22:09 +0000 https://status.limacharlie.io/#ba14013dd616f52188a6953e397f298b7359e45708463268b93e79b9ba699618 API Europe Services went down https://status.limacharlie.io/ Fri, 13 Jun 2025 18:00:13 +0000 https://status.limacharlie.io/#6516b25020e49f3e239d2f0361453ed76c0d30534873ecadeb5057abee395097 API USA Directory recovered https://status.limacharlie.io/ Fri, 13 Jun 2025 17:57:05 +0000 https://status.limacharlie.io/#6516b25020e49f3e239d2f0361453ed76c0d30534873ecadeb5057abee395097 API USA Directory went down https://status.limacharlie.io/ Fri, 13 Jun 2025 17:52:05 +0000 https://status.limacharlie.io/#d2e71bc16cc809153311db10956aba433961f38f222d29f00eabe941b4f6b91f API USA Directory recovered https://status.limacharlie.io/ Fri, 13 Jun 2025 17:45:15 +0000 https://status.limacharlie.io/#d2e71bc16cc809153311db10956aba433961f38f222d29f00eabe941b4f6b91f API USA Directory went down https://status.limacharlie.io/ Fri, 13 Jun 2025 16:16:57 +0000 https://status.limacharlie.io/#ca5508a7f1f966e3a827f75836e4b192fa337e23b40492a1e47032b897d9c7c7 API USA Directory recovered https://status.limacharlie.io/incident/602673 Fri, 13 Jun 2025 15:35:00 -0000 https://status.limacharlie.io/incident/602673#4e3eeb2cc271cced26bb9c22351a979c73e43e3a2d5a1cb6948339c2732beb33 We experienced degraded systems due to residual impact from upstream provider outages from yesterday. We are in a stabilization phase, and are working towards nominal operations. 8:03 AM MST - All systems nominal at this time. Thanks for your patience. https://status.limacharlie.io/incident/602673 Fri, 13 Jun 2025 15:35:00 -0000 https://status.limacharlie.io/incident/602673#4e3eeb2cc271cced26bb9c22351a979c73e43e3a2d5a1cb6948339c2732beb33 We experienced degraded systems due to residual impact from upstream provider outages from yesterday. We are in a stabilization phase, and are working towards nominal operations. 8:03 AM MST - All systems nominal at this time. Thanks for your patience. https://status.limacharlie.io/ Fri, 13 Jun 2025 15:19:57 +0000 https://status.limacharlie.io/#ca5508a7f1f966e3a827f75836e4b192fa337e23b40492a1e47032b897d9c7c7 API USA Directory went down https://status.limacharlie.io/ Thu, 12 Jun 2025 20:10:03 +0000 https://status.limacharlie.io/#7900e992eb6148ce4aeb9858d26a19e8b43c591e2baebf255f47f61571f772a5 API USA Telemetry Ingestion recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 19:05:10 +0000 https://status.limacharlie.io/#7900e992eb6148ce4aeb9858d26a19e8b43c591e2baebf255f47f61571f772a5 API USA Telemetry Ingestion went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:59:06 +0000 https://status.limacharlie.io/#832ecf97b494ba3e26f6d249a8192bbdc72c017d70a2d12bd95ddf7d596f6023 API Canada Modules recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:58:06 +0000 https://status.limacharlie.io/#c69e293c861f8a1f9f877ef00368609a15e8fc8e4df5a10ce15877beb9a0ccb0 API Canada Insight recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:55:34 +0000 https://status.limacharlie.io/#6e2a6c590772d3daf4b4d7bf090165502dc8736d0105bc8b8c7c26218feeaacd API USA Management recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:53:12 +0000 https://status.limacharlie.io/#087caceb7ed3b21ad18146f337663834207f2003d420d30696c0951818808471 API Canada Services recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:53:09 +0000 https://status.limacharlie.io/#5ef4203b29c470648b852ed16ba7be29baf7680cf6a98289fbd150ff0fc15156 API Canada Cloud Adapters recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:52:37 +0000 https://status.limacharlie.io/#c7faea95ca10771a52eefaf9d6cb034b2bec2380c3f720a8039be04316f19098 API USA Directory recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:49:47 +0000 https://status.limacharlie.io/#eb9a534e5f1fb1074776975cb19ff580070ca6b19be2764860178f954245e879 API USA Telemetry Ingestion recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:47:27 +0000 https://status.limacharlie.io/#e206b97f3568f43829460640539863239b974dbf6eca760ce2cd8deebb8b8692 API Canada Telemetry Ingestion recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:44:54 +0000 https://status.limacharlie.io/#35e4f302fec50ff41aa2f9352a929331dca7508a3c31a6addf3aff4996c34b69 API USA Cloud Adapters recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:44:38 +0000 https://status.limacharlie.io/#f196c16b8b4a4933a383a794a1415728c35b025eef2e3d25613f64688f0b674f API USA External Telemetry recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:44:12 +0000 https://status.limacharlie.io/#c730b15f8411d159023eac522830663ffffdc1a2256d6d96bd810b6d8b3f7ba0 API Canada Directory recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:40:39 +0000 https://status.limacharlie.io/#c7faea95ca10771a52eefaf9d6cb034b2bec2380c3f720a8039be04316f19098 API USA Directory went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:38:57 +0000 https://status.limacharlie.io/#c38f210a84ba17ae17d27c3f00937703d2e83fc34fabb47232d82ef52ce73632 API Canada Management recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:38:51 +0000 https://status.limacharlie.io/#ba331bf5c144023bb707424c91a4c53580592f26699701e206c569d460dd0161 API Canada Enrollment recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:37:58 +0000 https://status.limacharlie.io/#5d4e7f28e309a264871fccaab598db56743e9d8d5d50ba40164577e9d64a9da7 API Canada External Telemetry recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:37:42 +0000 https://status.limacharlie.io/#c69e293c861f8a1f9f877ef00368609a15e8fc8e4df5a10ce15877beb9a0ccb0 API Canada Insight went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:37:32 +0000 https://status.limacharlie.io/#fb71e5da6f11cd54a0908cd4dc90c664f8522c3d3ff7a0f0be6564762b1b7c7e API USA Modules recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:36:21 +0000 https://status.limacharlie.io/#24d1c820cd625d7c84dd98cc8b8efc516d19b7347e6202ed060e162cfeb8b2ee API USA Enrollment recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:33:17 +0000 https://status.limacharlie.io/#087caceb7ed3b21ad18146f337663834207f2003d420d30696c0951818808471 API Canada Services went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:31:21 +0000 https://status.limacharlie.io/#6e2a6c590772d3daf4b4d7bf090165502dc8736d0105bc8b8c7c26218feeaacd API USA Management went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:29:12 +0000 https://status.limacharlie.io/#c730b15f8411d159023eac522830663ffffdc1a2256d6d96bd810b6d8b3f7ba0 API Canada Directory went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:28:06 +0000 https://status.limacharlie.io/#5ef4203b29c470648b852ed16ba7be29baf7680cf6a98289fbd150ff0fc15156 API Canada Cloud Adapters went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:28:00 +0000 https://status.limacharlie.io/#5d4e7f28e309a264871fccaab598db56743e9d8d5d50ba40164577e9d64a9da7 API Canada External Telemetry went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:27:22 +0000 https://status.limacharlie.io/#e206b97f3568f43829460640539863239b974dbf6eca760ce2cd8deebb8b8692 API Canada Telemetry Ingestion went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:26:52 +0000 https://status.limacharlie.io/#e9d53cd459dcc22435e79b885ff2dccacfc232df74ea8ba4edede8c719d34530 API India External Telemetry recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:26:27 +0000 https://status.limacharlie.io/#76c7c923658f31ce5b1ed259871a1d150e45f4531d59408e4afe37a9ae4460e4 API India Telemetry Ingestion recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:26:09 +0000 https://status.limacharlie.io/#01c16a8e5b84de8c6ba2d6f129d32609522a43dd9f280f0feffa23e4fc426570 API Europe Modules recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:25:57 +0000 https://status.limacharlie.io/#9e3d432b5fcea2539911fb15e7ad2c66e840fb671e228fc3da89a38da9df7246 API Europe Management recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:25:51 +0000 https://status.limacharlie.io/#b9658ebd313c0517b3a52f0626963c2501f65bc389e08612e5c22699674cd504 API Europe Enrollment recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:25:37 +0000 https://status.limacharlie.io/#0144388607657c62ce11fa608d246bad8c30a5cb9f92dc877f25262f3094907d API Europe Services recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:25:29 +0000 https://status.limacharlie.io/#3bd788d4fdbf6fd6260c99f24f132c2911d15b2895263276496e2e2bb90dc040 API USA Services recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:25:09 +0000 https://status.limacharlie.io/#9d6d6423bb77ecebcf58d1d929503d7c161eed956e7dde216c5a4389f9f89824 API Europe Insight recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:24:58 +0000 https://status.limacharlie.io/#aa7a83485a561927de4af68ebb36e732723cf5168a96728517f082618f7aca86 API USA InsightAPI USA Insight recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:24:33 +0000 https://status.limacharlie.io/#d93d0d67462f73fc80d9bd1798624f0b47495efff2dc4ed381e30e6116e36361 API UK Modules recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:24:18 +0000 https://status.limacharlie.io/#ac705f3aeb754d1e04a83005b45c5d7b458cfe49193d00dcdebb0efbd5975d11 API USA Management recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:24:11 +0000 https://status.limacharlie.io/#832ecf97b494ba3e26f6d249a8192bbdc72c017d70a2d12bd95ddf7d596f6023 API Canada Modules went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:24:09 +0000 https://status.limacharlie.io/#cbff483a0b44dc072d6cb52bccf01dd352e53e6a25d2e002dbb4c6f7d0c47de3 API UK Enrollment recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:24:06 +0000 https://status.limacharlie.io/#ba331bf5c144023bb707424c91a4c53580592f26699701e206c569d460dd0161 API Canada Enrollment went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:23:57 +0000 https://status.limacharlie.io/#c38f210a84ba17ae17d27c3f00937703d2e83fc34fabb47232d82ef52ce73632 API Canada Management went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:22:47 +0000 https://status.limacharlie.io/#f9632c9d4bd37eb54cde0e8e7adab0b086125d7722d26942d8db6dd9c364c8a7 API USA Directory recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:22:37 +0000 https://status.limacharlie.io/#f245058badc771665a125bba0d3df7e9e3a2284563060dd6bca6f2466660dd2d API UK Telemetry Ingestion recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:22:26 +0000 https://status.limacharlie.io/#e4b2a1fc3bfda254af0e4b4aefafd01084dcfea40cf75b4a9b3e7b83a780ebe6 API UK Cloud Adapters recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:22:19 +0000 https://status.limacharlie.io/#4987b98456de7ec3fea232e29804756cb0b956816d315d2566b7956c01e1347e API India Directory recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:22:05 +0000 https://status.limacharlie.io/#89fb0dbff02a8d70084a71bd142b5c560e6b6cabcb08d158ce1e8d983dcdeaee API UK Insight recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:22:04 +0000 https://status.limacharlie.io/#0a6122e8cbbab230c11f3333956a9e0d01e73e680b3bc842fc2c4718c3138de6 API India Management recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:21:56 +0000 https://status.limacharlie.io/#b94aaf144d0dd53bce3375b8d7531841d8a0dc15349a9afd734fd19de18f2da0 API India Enrollment recovered https://status.limacharlie.io/ Thu, 12 Jun 2025 18:21:10 +0000 https://status.limacharlie.io/#01c16a8e5b84de8c6ba2d6f129d32609522a43dd9f280f0feffa23e4fc426570 API Europe Modules went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:17:57 +0000 https://status.limacharlie.io/#0a6122e8cbbab230c11f3333956a9e0d01e73e680b3bc842fc2c4718c3138de6 API India Management went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:16:55 +0000 https://status.limacharlie.io/#89fb0dbff02a8d70084a71bd142b5c560e6b6cabcb08d158ce1e8d983dcdeaee API UK Insight went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:16:49 +0000 https://status.limacharlie.io/#e9d53cd459dcc22435e79b885ff2dccacfc232df74ea8ba4edede8c719d34530 API India External Telemetry went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:16:27 +0000 https://status.limacharlie.io/#76c7c923658f31ce5b1ed259871a1d150e45f4531d59408e4afe37a9ae4460e4 API India Telemetry Ingestion went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:16:17 +0000 https://status.limacharlie.io/#e4b2a1fc3bfda254af0e4b4aefafd01084dcfea40cf75b4a9b3e7b83a780ebe6 API UK Cloud Adapters went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:15:57 +0000 https://status.limacharlie.io/#b94aaf144d0dd53bce3375b8d7531841d8a0dc15349a9afd734fd19de18f2da0 API India Enrollment went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:15:56 +0000 https://status.limacharlie.io/#9e3d432b5fcea2539911fb15e7ad2c66e840fb671e228fc3da89a38da9df7246 API Europe Management went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:15:50 +0000 https://status.limacharlie.io/#b9658ebd313c0517b3a52f0626963c2501f65bc389e08612e5c22699674cd504 API Europe Enrollment went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:15:39 +0000 https://status.limacharlie.io/#0144388607657c62ce11fa608d246bad8c30a5cb9f92dc877f25262f3094907d API Europe Services went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:15:31 +0000 https://status.limacharlie.io/#3bd788d4fdbf6fd6260c99f24f132c2911d15b2895263276496e2e2bb90dc040 API USA Services went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:15:09 +0000 https://status.limacharlie.io/#9d6d6423bb77ecebcf58d1d929503d7c161eed956e7dde216c5a4389f9f89824 API Europe Insight went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:14:55 +0000 https://status.limacharlie.io/#aa7a83485a561927de4af68ebb36e732723cf5168a96728517f082618f7aca86 API USA InsightAPI USA Insight went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:14:44 +0000 https://status.limacharlie.io/#f196c16b8b4a4933a383a794a1415728c35b025eef2e3d25613f64688f0b674f API USA External Telemetry went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:14:43 +0000 https://status.limacharlie.io/#eb9a534e5f1fb1074776975cb19ff580070ca6b19be2764860178f954245e879 API USA Telemetry Ingestion went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:14:29 +0000 https://status.limacharlie.io/#d93d0d67462f73fc80d9bd1798624f0b47495efff2dc4ed381e30e6116e36361 API UK Modules went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:14:20 +0000 https://status.limacharlie.io/#cbff483a0b44dc072d6cb52bccf01dd352e53e6a25d2e002dbb4c6f7d0c47de3 API UK Enrollment went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:13:52 +0000 https://status.limacharlie.io/#35e4f302fec50ff41aa2f9352a929331dca7508a3c31a6addf3aff4996c34b69 API USA Cloud Adapters went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:13:34 +0000 https://status.limacharlie.io/#4987b98456de7ec3fea232e29804756cb0b956816d315d2566b7956c01e1347e API India Directory went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:13:31 +0000 https://status.limacharlie.io/#f245058badc771665a125bba0d3df7e9e3a2284563060dd6bca6f2466660dd2d API UK Telemetry Ingestion went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:13:29 +0000 https://status.limacharlie.io/#fb71e5da6f11cd54a0908cd4dc90c664f8522c3d3ff7a0f0be6564762b1b7c7e API USA Modules went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:11:46 +0000 https://status.limacharlie.io/#f9632c9d4bd37eb54cde0e8e7adab0b086125d7722d26942d8db6dd9c364c8a7 API USA Directory went down https://status.limacharlie.io/ Thu, 12 Jun 2025 18:10:23 +0000 https://status.limacharlie.io/#ac705f3aeb754d1e04a83005b45c5d7b458cfe49193d00dcdebb0efbd5975d11 API USA Management went down https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/incident/601750 Thu, 12 Jun 2025 18:09:00 -0000 https://status.limacharlie.io/incident/601750#963667b85b1198c10bb082e4b20ba7d95408a1983b878c3ecc44f6049cca0719 We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues. https://status.limacharlie.io/ Thu, 12 Jun 2025 18:08:16 +0000 https://status.limacharlie.io/#24d1c820cd625d7c84dd98cc8b8efc516d19b7347e6202ed060e162cfeb8b2ee API USA Enrollment went down https://status.limacharlie.io/ Mon, 02 Jun 2025 19:23:47 +0000 https://status.limacharlie.io/#3c14ba762d214e9ced6253971404689ad75066346f0a3ba02b35e32b4a4cca95 API UK Directory recovered https://status.limacharlie.io/ Mon, 02 Jun 2025 19:19:11 +0000 https://status.limacharlie.io/#3c14ba762d214e9ced6253971404689ad75066346f0a3ba02b35e32b4a4cca95 API UK Directory went down https://status.limacharlie.io/incident/572987 Tue, 20 May 2025 15:59:00 -0000 https://status.limacharlie.io/incident/572987#03adfa12a0dcce6f1b9744818b82401b26ae55cc90b1f51b6a66a6e8b31da023 The issue is identified and resolved. https://status.limacharlie.io/incident/572987 Tue, 20 May 2025 15:39:00 -0000 https://status.limacharlie.io/incident/572987#fc41c581d5e5d4b20b1bcaf6c5474898c2b1322d6e52ea07a1de111dd7d50088 The Web App can not load. Preliminary investigation indicates possible GCP failure. Investigating more. All API, backend services, and event ingestion are operating normally. This is not impacting any backend services. Only the web app portion of our services. 9:52AM MST - This issue seems to be related to GCP services issues. We are engaging GCP to escalate this issue, and working on other methods to stabilize. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 19:30:00 -0000 https://status.limacharlie.io/incident/534994#420f318b946ade471a9c8ecdcc31b3a82eae6f7f7491c0e4fe20e072f088aa79 3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/534994 Wed, 26 Mar 2025 16:09:00 -0000 https://status.limacharlie.io/incident/534994#cd07d5b232157793fdf2d96036f72cf1acb426e2325d9d5133f495c171f26e6e We are investigating API errors which are impacting all LimaCharlie services. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/530919 Wed, 19 Mar 2025 17:24:00 -0000 https://status.limacharlie.io/incident/530919#30fca900fe81ab40dbe81a5bdc04b2c7ff777d33a32a4aaa18129230062c44d4 We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved. https://status.limacharlie.io/incident/500894 Thu, 23 Jan 2025 22:20:00 -0000 https://status.limacharlie.io/incident/500894#10fa1ffa36d8d19c3bf9635429c7a3308376dc3228e0d157ff993057caf674e1 Situation : We are experiencing an issue with the reliable-tasking extension. Task : We are effecting hardware upgrade that we expect to stabilize and resolve the issue. Expected Impact: There will be a loss of currently buffered reliable tasks. Jan 24, 2025 10AM - Issue has been stabilized, but we are still root causing the issue. No expected Impact at this time. We will update as we have more information. Jan 24, 2025 5PM - We have implemented a fix, and while we expect resolution, will be monitoring over the next 24hrs. Next update tomorrow. https://status.limacharlie.io/incident/483417 Wed, 11 Dec 2024 15:18:00 -0000 https://status.limacharlie.io/incident/483417#6a756918fd2f30ab79a39e67cdfa148013c46ec06f929c5e8f8589920f330e02 ## Summary Root Cause Analysis Report - Post-Mortem ## Incident Overview: On December 11th, 2024, a few LimaCharlie customers reported Blue Screen of Death (BSOD) issues affecting Windows systems running version 4.32.1 of the LC EDR Agent. The issue only affected a small subset of systems running `lc:latest` that were under high memory pressure. Engineering quickly identified the issue and rolled back the agent version from `4.32.1` to `4.32.0` to mitigate the problem. The incident response was swift, with engineering identifying and diagnosing the problem within approximately one hour of the first customer report. A community-wide announcement was made and remediation was implemented through a version rollback within 2.5 hours of initial report. ## Timeline of Events: - 2024-12-10 @ 9:41am PT: LC EDR Agent version `4.32.1` is released - 2024-12-11 @ 05:02am PT: Multiple customers reported isolated Blue Screen of Death (BSOD) incidents on Windows systems running the LimaCharlie EDR Agent - 2024-12-11 @ 06:06am PT: LC engineering diagnosed a kernel driver defect in `latest` endpoint agent version `4.32.1` - 2024-12-11 @ 07:18am PT: Announcement posted in LC Community Slack - 2024-12-11 @ 07:30am PT: LC Agent `latest` version rolled back to `4.32.0` ## Root Causes: The root cause was identified as a kernel driver bug related to improper IRQL handling when accessing paged memory with a spin lock. The issue manifested when the kernel driver (`tmp_hbs_acq.sys`) attempted to access pageable memory at an interrupt request level (IRQL) that was too high, triggering Windows bug check D1 (`DRIVER_IRQL_NOT_LESS_OR_EQUAL`). This bug was infrequent on test systems where no memory pressure was present. ## Actions Taken: - Implement stricter IRQL validation checks in the kernel driver code to ensure proper memory access patterns - Add comprehensive memory access guards around paged pool operations when holding spin locks - Enhance the driver's memory management to properly handle high-pressure scenarios - Implement additional automated testing specifically for IRQL-related scenarios in the kernel driver - Enhance pre-release testing environment to better simulate high-memory-pressure scenarios and kernel-level interactions across diverse Windows configurations These changes will be implemented and thoroughly tested before being included in a future release. The development team will also review other areas of the kernel driver code for similar potential IRQL issues. ## Lessons Learned: - The critical importance of thorough IRQL validation in kernel driver development to prevent system crashes - The need for enhanced pre-release testing environments that better simulate real-world conditions, particularly high-memory-pressure scenarios - The value of quick rollback capabilities when dealing with kernel-level components - The importance of maintaining detailed version control and change documentation to quickly identify potential causes of issues - The effectiveness of rapid customer communication and transparent incident handling in maintaining trust ## Future Recommendations: - Enhance documentation and communication around the intended purpose of `lc:latest` tag, emphasizing its role as a testing/staging version rather than for production use - Develop best practices documentation for rolling out new agent versions across customer environments, including recommended testing procedures and rollback strategies - Added “Best Practices” to docs: https://docs.limacharlie.io/docs/endpoint-agent-versioning-and-upgrades#updating-endpoint-agents - Update web app to display both `latest` and `stable` as current versions on the Deployed Versions page, removing UI indicators that suggest `stable` to be outdated. - Implement a formal notification system for `lc:latest` version changes to help customers better track and manage testing cycles https://status.limacharlie.io/incident/462214 Fri, 15 Nov 2024 19:03:00 -0000 https://status.limacharlie.io/incident/462214#b8de15150703c6507801266a9d32f7be39e083c06341d67ce1feb900319106f1 Performance issues should be resolved. We continue to monitor systems to ensure sustained performance. https://status.limacharlie.io/incident/462214 Fri, 15 Nov 2024 18:49:00 -0000 https://status.limacharlie.io/incident/462214#232c64165284ce70cf68b6d4fdf9e2fd92837f2ef66dbd2fa0eadda6932424bc Users may see performance issues when loading sensor lists. We are aware of the issue and are working on a resolution. https://status.limacharlie.io/incident/461630 Thu, 14 Nov 2024 18:58:00 -0000 https://status.limacharlie.io/incident/461630#a35d29268a08db294254be07148ffc586b73b970bc1ac5bb0a59cfd04ef02735 Performance issues should be resolved. We continue to monitor systems to ensure sustained performance. https://status.limacharlie.io/incident/461630 Thu, 14 Nov 2024 18:54:00 -0000 https://status.limacharlie.io/incident/461630#083ce016e912e91c8c3eeed4dd6bfd3d11e60f97a2a62268efc542885a04b744 Users may see performance issues when loading sensor lists. We are aware of the issue and are working on a resolution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Mon, 09 Sep 2024 14:07:00 -0000 https://status.limacharlie.io/incident/421455#ccfb97013668d426ad813a02a77d4b0ea67588f22b98ca7edf6e87265f289ed5 **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On August 29 2024 the [limacharlie.io](http://limacharlie.io) domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of the Events:** **August 29, 2024** - **2:05pm PT:** DNS issues began. - **2:10pm PT:** Reports received of customer receiving errors and no longer being able to access orgs. - **2:21pm PT:** Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - **2:30pm PT:** Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - **3:38pm PT:** Confirmed full resolution at all global root nameservers. **Root Cause:** 1. **Human Error at upstream provider:** LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. **System Status:** All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. **Actions Take:** - **Vendor Communication:** The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. **Lessons Learned / Future Recommendations:** - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 22:38:00 -0000 https://status.limacharlie.io/incident/421455#d7e780840a1700a663427920cf67c5aab2538ef978af20cf905e70ab395f2438 DNS issues have been confirmed resolved. All services are operational. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:30:00 -0000 https://status.limacharlie.io/incident/421455#9813ad378d9818ccfd7b15b0514e583f47a1362b5555dc0652586b2e98c54fcc The DNS issue has been resolved. Local DNS propagation time may vary depending on geographic locale. Underlying LimaCharlie services are in operation. You may notice intermittent issues as DNS propagation completes. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/421455 Thu, 29 Aug 2024 21:05:00 -0000 https://status.limacharlie.io/incident/421455#2b51a8c4646c9258f71261ab4fe4d75fe19232a0289becc63343460dd4b5ff4c The limacharlie.io domain experienced DNS issues which caused unavailability of services. https://status.limacharlie.io/incident/420234 Tue, 27 Aug 2024 18:52:00 -0000 https://status.limacharlie.io/incident/420234#79d9171dd674cc4e8adf887c497a5a632abd86c4dfcef2a219ee90c3d8124d15 Most services have returned to normal operation. We are continuing to monitor. https://status.limacharlie.io/incident/420234 Tue, 27 Aug 2024 18:37:00 -0000 https://status.limacharlie.io/incident/420234#450f116623fb6b2e9a697224ec7a2573c6019a194331ad17e4400d9d62cc323b We are investigating reports of issues with the LimaCharlie API. These issues also appear to be impacting the LimaCharlie web interface. Additional details to follow. https://status.limacharlie.io/incident/377074 Thu, 30 May 2024 17:27:00 -0000 https://status.limacharlie.io/incident/377074#5fb679df204a4d644e893b6747b732935aefb33a3550ea915dc798c92bb83cf1 Team, we have had reports of the app.limacharlie.io not loading the first time. While we are investigating this issue in more depth, the quick solution is a hard refresh of the site in browser. We will update with more information as we have it. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 03:00:00 -0000 https://status.limacharlie.io/incident/376520#359dbacc274fa6e90b04c0d100716e5499b855db71a3d634b3ce483bb6c2427e Maintenance completed https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376520 Thu, 30 May 2024 00:00:00 -0000 https://status.limacharlie.io/incident/376520#dec5c320642ae6d8f6237d869409f1dc176b495e7b0b929d22a7753db6b4b8bb We will be performing maintenance on US systems starting at 5PM PST. We expect no impact to systems during this time. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 19:08:35 -0000 https://status.limacharlie.io/incident/376453#0d72398230eeb19afc238f400d710fe613152735ebf502b0aee74e16e29c43c2 Maintenance completed https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/376453 Wed, 29 May 2024 16:38:00 -0000 https://status.limacharlie.io/incident/376453#4c1e90a0b3cc10b8fc04bcceaec161a80f70b903e05458d0b9ff828c4da8e782 We are upgrading/implementing some news systems. While we perform these actions we expect no impact. https://status.limacharlie.io/incident/369919 Wed, 15 May 2024 21:47:00 -0000 https://status.limacharlie.io/incident/369919#ddbb8bf0c47c9c4579ca017d4591d9a0c4761f3f1c31001d790243de7e9ac879 We are currently experiencing high latency impact from a Google Cloud DataStore issue, that is impacting our US services. We have engaged GCP directly and have escalated the issue. This issue is ongoing and we expect degraded service quality during this time. We will be providing updates as we have them and are working to remedy this issue as quickly as possible. May 15 2024 7:49PM UPDATE - We are still working the issue with GCP support and will update as we have more information. May 15 2024 11:34PM UPDATE - We are still working with GCP support to bring this issue to resolution. More updates as we have them. May 16 2024 9:34AM MST UPDATE - We have identified a possible root cause and working towards resolution. We expect degraded performance from Cloud Sensors during this issue. May 16 2024 4:54PM MST UPDATE - We have identified the issue and are working towards resolution. We expect minimal impact to Cloud Adapters at this time. May 17 2024 2:21PM MST UPDATE - Latency performance is back with acceptable ranges, and we are calling this incident resolved. All systems are nominal. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 23:15:00 -0000 https://status.limacharlie.io/incident/311970#665fadf3e5ca083d9e4a7dc58a49092181e9b1ae2f99cb6391585d2ab4e568c3 Maintenance completed https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/311970 Sat, 20 Jan 2024 17:15:00 -0000 https://status.limacharlie.io/incident/311970#96d4136514badff6862d6013a90520191a93017c408eabdd26abb5af394c2c27 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in all regions. During this maintenance, our systems will be in a partially degraded state for a period of up to 120 minutes. Maintenance Window Details: Date: Saturday, January 20, 2024 Start Time: 9:15 AM PST End Time: 3:15 PM PST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified time frame to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/303482 Thu, 28 Dec 2023 23:00:00 -0000 https://status.limacharlie.io/incident/303482#14a46a345d2fdcb395b5f4df22dac0820df2c6cbfdae902791ba1c7577f5ca68 Maintenance completed https://status.limacharlie.io/incident/303482 Thu, 28 Dec 2023 22:30:00 -0000 https://status.limacharlie.io/incident/303482#2bc202b2ca405f3ce8e06b9cd4c8f01477ecfb93cf18f316f166c73a92384601 We want to inform you about an upcoming scheduled maintenance window that will affect our systems in the USA region. During this maintenance, our systems will be in a partially degraded state for a period of up to 30 minutes. Maintenance Window Details: Date: 12-28-2023 Start Time: 3:30PM MST End Time: 4:00PM MST During this maintenance window, you may experience the following issues: UI Delays: Users may encounter delays in refreshing data within the user interface. Please be patient as the system may take longer than usual to display updated information. Inconsistency in data displayed: After creating content (e.g. a D&R rule), the content will in fact be created, however, user interface may not immediately reflect this. We recommend being patient and performing a subsequent refreshes to ensure the correct status is displayed. We understand that these issues may cause some inconvenience, and we apologize for any disruption this may cause to your workflow. However, please rest assured that all telemetry data will continue to be received normally throughout this maintenance window. Your data remains secure, and no telemetry will be lost or compromised during this period. Our team is committed to ensuring the stability and reliability of our systems, and this maintenance is essential to maintaining the high standards of service that you expect from us. We will make every effort to complete the maintenance within the specified timeframe to minimize any inconvenience. If you have any urgent concerns or questions regarding this maintenance, please do not hesitate to contact our support team at answers@limacharlie.io We appreciate your understanding and cooperation during this maintenance period. Thank you for your continued trust in our services. https://status.limacharlie.io/incident/293237 Thu, 14 Dec 2023 03:37:00 -0000 https://status.limacharlie.io/incident/293237#1579251df3a7f350ca675261ffe808f1baaf13bf86599d6f187037de997f7f1a **Summary Root Cause Analysis Report - Post-Mortem** **Incident Overview:** On November 28, 2023, an issue was identified with Sensor Live items not functioning properly. The problem was initially reported as an issue with the LimaCharlie console. This report summarizes the timeline of events, root causes, and corrective actions taken. **Timeline of Events:** - **Nov 28, Early Morning:** Customers began reporting the incident via Slack. - **Throughout the Morning:** Various team members engaged to investigate issue. The team validated the issue and observed it appeared to impact both backend and frontend systems. - **Mid-Morning:** The affected systems were identified. Service restarts were necessary to bring about a resolution to the issue. **Root Causes:** 1. **Infrastructure Robustness:** The reliance on some older technologies brought about complications that did not allow for simple troubleshooting of the issue. This indicated a need for a more robust and autonomous infrastructure. 2. **System Monitoring Gaps:** System probes were unable to detect the service degradation due to limitations in their configuration. 3. **Communication Delays:** A subset of configuration was missed during the change of emergency notification tooling which led to a delay in notification to the appropriate parties, causing delays in addressing the issue. **Actions Taken:** 1. **Testing and Infrastructure Improvements:** Initiatives to replace older technologies with more modern solutions and enhance monitoring with improved metrics and tests. 2. **Incident Reporting Process Updates:** An updated email system will be used. 3. **Customer Communication:** Plans to revise customer notifications regarding incidents and include incident response information in our onboarding process. 4. **Internal Documentation:** Creation of documentation on incident reporting best practices and severity levels, ensuring all team members are informed and prepared. **Lessons Learned:** - The importance of robust monitoring systems that can detect not just system uptime but also functional performance. - The need for clear and efficient communication channels both internally and with customers. - The value of having a resilient and autonomous infrastructure to reduce downtime and manual intervention requirements. **Future Recommendations:** - Continue to enhance system monitoring and alerting capabilities. - Streamline communication protocols for faster incident response. - Regularly review and update infrastructure to ensure it meets the evolving needs of the service and customers. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Wed, 06 Dec 2023 05:55:00 -0000 https://status.limacharlie.io/incident/296350#04e554772ebd3c37346603241bff135e5e9eb431b64fe0ddd20c433bc755d012 While the incident with our upstream provider is on-going, the impact to LimaCharlie services appear to have been resolved. https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/296350 Tue, 05 Dec 2023 16:51:00 -0000 https://status.limacharlie.io/incident/296350#3f821a878a23607d46a4eea268248375e763504759f30deb0c2ee7e0112e98c3 Our upstream provider is currently experiencing network connectivity issues, impacting services in the USA. We continue to monitor the situation. Detailed updates can be found here: https://status.cloud.google.com/incidents/TzTbUF4sxVhj1mcK9j25#RP1d9aZLNFZEJmTBk8e1 https://status.limacharlie.io/incident/293237 Tue, 28 Nov 2023 15:02:00 -0000 https://status.limacharlie.io/incident/293237#d60d419c3e538f4afb8a48f1dd95eb8f5568c8488170a343a7347f31a09a3453 Console commands appear to be working as expected. We continue to monitor. For reference, automated taskings continued to work as expected throughout this incident. https://status.limacharlie.io/incident/293237 Tue, 28 Nov 2023 11:52:00 -0000 https://status.limacharlie.io/incident/293237#847949d349d28af83df75c39600152d07ce75b21afd475d007f7bda2be016c2f We are investigating an issue that is impacting the Console functionality. Additional details will be posted to this page once they become available. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 06:30:00 -0000 https://status.limacharlie.io/incident/171383#8d4ea36a9804e75f89e6bd18c9e5a184860cf6ba902b32ce112e38d38c23e066 Maintenance completed https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/171383 Sat, 04 Feb 2023 00:15:00 -0000 https://status.limacharlie.io/incident/171383#9f96b71f5eb7e2bddcbbda7eec4434327cb9904e6413a3ff291706dd4f8ab903 We are performing cluster upgrade, and expected minimal impact to systems. https://status.limacharlie.io/incident/164362 Sat, 14 Jan 2023 06:34:00 -0000 https://status.limacharlie.io/incident/164362#8b64c4a76074c8a52888de2f198c0523574c10242ac3657a2401a40d670b3d6c # 2023-01-13 - Cisco Umbrella blocking access to LimaCharlie cloud On 2023-01-13 we became aware of an issue where Cisco Umbrella was blocking access to the LimaCharlie cloud. ## Status We are awaiting a response from the Cisco Talos team to have the "limacharlie.io" domain added to their false positive list. In the meantime impacted customers may add the following LimaCharlie domains to their Global Allow list by following the steps below: ## Workaround Steps 1. Log into the [Cisco Umbrella portal](https://login.umbrella.com/) 2. Go to Policies -> Policy Components -> Destination Lists. 3. Add the following entries to the Global Allow List: limacharlie.io 9157798c50af372c.lc.limacharlie.io 70182cf634c346bd.lc.limacharlie.io 4d897015b0815621.lc.limacharlie.io b76093c3662d5b4f.lc.limacharlie.io aae67d7e76570ec1.lc.limacharlie.io Your sensors should start resuming connections to the LimaCharlie cloud. ## Requesting your assistance If you have any contacts at Cisco who may be able to get this unblocking expedited, please reach out to us via [email](mailto:answers@limacharlie.io) or on our [community Slack](https://slack.limacharlie.io) Additionally, please submit a [Web Reputation Support Ticket](https://talosintelligence.com/reputation_center/web_reputation) to Cisco Talos. This will help increase the reputation of the domains above. ## Impact Customers who utilize both the Cisco Umbrella network security product may have found they have suddenly lost connectivity with the LimaCharlie cloud. This includes access to the LimaCharlie: - web application, - APIs, - endpoint telemetry servers, - output destination servers ## Mitigation of Risk The LimaCharlie sensor continues to collect telemetry while offline, and transmits it once the connection has been re-established. Further details can be found in our article ["What happens when the host is offline?"](https://help.limacharlie.io/en/articles/5662761-what-happens-when-the-host-is-offline). ## Timeline ### 2023-01-13 8:18 a.m. ET [User reported issue via community Slack](https://lcendpoint.slack.com/archives/C1S96NSGP/p1673615906520759) 1:04 p.m. ET [User reported issue via private Slack channel](https://refractionpoint.slack.com/archives/C02BWLJQZE3/p1673633043017499) 1:08 p.m. ET Opened [Cisco Umbrella support ticket #1448986](https://mail.google.com/mail/u/0/#sent/KtbxLvgxCGNmCrmqmsKfNHZChFrFjcXdxB), per [reporting procedure](https://blog.talosintelligence.com/talosintelligencecom-is-rolling-out-new/) 1:31 p.m. ET Added notice to [#status in community Slack](https://lcendpoint.slack.com/archives/CMS9NA7NK/p1673634585175149) 1:39 p.m. ET Submitted false positive report to [Talos Intelligence](https://talosintelligence.com/reputation_center/sender_domain), submitted the following domains as false positives: [limacharlie.io](https://talosintelligence.com/tickets/sdr/313558) [9157798c50af372c.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313559) [70182cf634c346bd.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313560) [4d897015b0815621.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313561) [b76093c3662d5b4f.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313562) [aae67d7e76570ec1.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313563) 1:45 p.m. ET Signed up for Cisco Umbrella trial to verify steps to add limacharlie.io domains to the allow list. Submitted a [categorization dispute](https://docs.umbrella.com/umbrella-user-guide/docs/dispute-a-categorization#category-request-umbrella) 2:46 p.m. ET Called Cisco TAC via telephone to try to escalate issue; they emailed Umbrella support on our behalf and provided my direct telephone number. 3:30 p.m. ET Opened another support ticket via [Cisco Umbrella support request #1449194](https://support.umbrella.com/hc/en-us/requests/1449194); case ID #1449194 7:29 p.m. ET Submitted escalation request via email on Talos ticket ### 2023-01-14 1:09 a.m. ET Validated issue still persists via [Talos reputation center](https://www.talosintelligence.com/reputation_center) Submitted [Web Reputation Support Ticket](https://talosintelligence.com/tickets/313642) **Included the following in the ticket:** Cisco Umbrella is incorrectly reporting the LimaCharlie domains listed as Phishing. LimaCharlie provides a legitimate security infrastructure-as-a-service offering, which can be validated at these external sites: [GitHub](https://github.com/refractionPOINT/limacharlie) Shows our long standing history, first as an open source project and then as a commercial offering. See link to our public repositories which include links to our domain with authenticated date/time stamps. [Crunchbase](https://www.crunchbase.com/organization/limacharlie) Shows funding for our company and the legitimate backers behind us. Further questions can be directed to Amrik Randhawa, Chief Operating Officer via email at {answers@limacharlie.io -- telephone number provided in the submitted ticket}. 4:11 a.m. ET Outreach to Cisco Talos Intelligence Group via connection 6:47 a.m. ET User reported incident resolution. 7:18 a.m. ET Status of submitted [Talos Tickets](https://talosintelligence.com/tickets) still appears in "New" state. [Cisco Talos Reputation Center](https://talosintelligence.com/reputation_center/lookup?search=limacharlie.io) still shows domains in Untrusted state. 1:53 p.m. ET [Cisco Talos Reputation Center](https://talosintelligence.com/reputation_center/lookup?search=limacharlie.io) now shows domains in Neutral state. Users have started reporting systems returning back to normal operation. 4:22 p.m. ET We continue to work to get the LimaCharlie domains in a Trusted state with Cisco Talos. We have received reports that other Cisco security products such as IronPort are still negatively impacted. ### 2023-01-16 3:51 a.m. ET Received report of Palo Alto XDR flagging limacharlie.io domains as malicious. 7:51 a.m. ET Cisco Talos Support tickets marked as "RESOLVED - Please allow 24h for your system to update" with comment: "This domain was penalized by SDR as it used to have a Untrusted Web Reputation score due to malicious or suspicious behavior. The issue has been rectified and the domain reputation has been adjusted accordingly." 10:44 a.m. ET Posted message to [Palo Alto LIVE community](https://live.paloaltonetworks.com/t5/cortex-xdr-discussions/cortex-xdr-false-positive-detection-of-limacharlie-sensor-edr/m-p/527230#M3399) to report false positive. https://status.limacharlie.io/incident/164362 Sat, 14 Jan 2023 06:34:00 -0000 https://status.limacharlie.io/incident/164362#8b64c4a76074c8a52888de2f198c0523574c10242ac3657a2401a40d670b3d6c # 2023-01-13 - Cisco Umbrella blocking access to LimaCharlie cloud On 2023-01-13 we became aware of an issue where Cisco Umbrella was blocking access to the LimaCharlie cloud. ## Status We are awaiting a response from the Cisco Talos team to have the "limacharlie.io" domain added to their false positive list. In the meantime impacted customers may add the following LimaCharlie domains to their Global Allow list by following the steps below: ## Workaround Steps 1. Log into the [Cisco Umbrella portal](https://login.umbrella.com/) 2. Go to Policies -> Policy Components -> Destination Lists. 3. Add the following entries to the Global Allow List: limacharlie.io 9157798c50af372c.lc.limacharlie.io 70182cf634c346bd.lc.limacharlie.io 4d897015b0815621.lc.limacharlie.io b76093c3662d5b4f.lc.limacharlie.io aae67d7e76570ec1.lc.limacharlie.io Your sensors should start resuming connections to the LimaCharlie cloud. ## Requesting your assistance If you have any contacts at Cisco who may be able to get this unblocking expedited, please reach out to us via [email](mailto:answers@limacharlie.io) or on our [community Slack](https://slack.limacharlie.io) Additionally, please submit a [Web Reputation Support Ticket](https://talosintelligence.com/reputation_center/web_reputation) to Cisco Talos. This will help increase the reputation of the domains above. ## Impact Customers who utilize both the Cisco Umbrella network security product may have found they have suddenly lost connectivity with the LimaCharlie cloud. This includes access to the LimaCharlie: - web application, - APIs, - endpoint telemetry servers, - output destination servers ## Mitigation of Risk The LimaCharlie sensor continues to collect telemetry while offline, and transmits it once the connection has been re-established. Further details can be found in our article ["What happens when the host is offline?"](https://help.limacharlie.io/en/articles/5662761-what-happens-when-the-host-is-offline). ## Timeline ### 2023-01-13 8:18 a.m. ET [User reported issue via community Slack](https://lcendpoint.slack.com/archives/C1S96NSGP/p1673615906520759) 1:04 p.m. ET [User reported issue via private Slack channel](https://refractionpoint.slack.com/archives/C02BWLJQZE3/p1673633043017499) 1:08 p.m. ET Opened [Cisco Umbrella support ticket #1448986](https://mail.google.com/mail/u/0/#sent/KtbxLvgxCGNmCrmqmsKfNHZChFrFjcXdxB), per [reporting procedure](https://blog.talosintelligence.com/talosintelligencecom-is-rolling-out-new/) 1:31 p.m. ET Added notice to [#status in community Slack](https://lcendpoint.slack.com/archives/CMS9NA7NK/p1673634585175149) 1:39 p.m. ET Submitted false positive report to [Talos Intelligence](https://talosintelligence.com/reputation_center/sender_domain), submitted the following domains as false positives: [limacharlie.io](https://talosintelligence.com/tickets/sdr/313558) [9157798c50af372c.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313559) [70182cf634c346bd.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313560) [4d897015b0815621.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313561) [b76093c3662d5b4f.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313562) [aae67d7e76570ec1.lc.limacharlie.io](https://talosintelligence.com/tickets/sdr/313563) 1:45 p.m. ET Signed up for Cisco Umbrella trial to verify steps to add limacharlie.io domains to the allow list. Submitted a [categorization dispute](https://docs.umbrella.com/umbrella-user-guide/docs/dispute-a-categorization#category-request-umbrella) 2:46 p.m. ET Called Cisco TAC via telephone to try to escalate issue; they emailed Umbrella support on our behalf and provided my direct telephone number. 3:30 p.m. ET Opened another support ticket via [Cisco Umbrella support request #1449194](https://support.umbrella.com/hc/en-us/requests/1449194); case ID #1449194 7:29 p.m. ET Submitted escalation request via email on Talos ticket ### 2023-01-14 1:09 a.m. ET Validated issue still persists via [Talos reputation center](https://www.talosintelligence.com/reputation_center) Submitted [Web Reputation Support Ticket](https://talosintelligence.com/tickets/313642) **Included the following in the ticket:** Cisco Umbrella is incorrectly reporting the LimaCharlie domains listed as Phishing. LimaCharlie provides a legitimate security infrastructure-as-a-service offering, which can be validated at these external sites: [GitHub](https://github.com/refractionPOINT/limacharlie) Shows our long standing history, first as an open source project and then as a commercial offering. See link to our public repositories which include links to our domain with authenticated date/time stamps. [Crunchbase](https://www.crunchbase.com/organization/limacharlie) Shows funding for our company and the legitimate backers behind us. Further questions can be directed to Amrik Randhawa, Chief Operating Officer via email at {answers@limacharlie.io -- telephone number provided in the submitted ticket}. 4:11 a.m. ET Outreach to Cisco Talos Intelligence Group via connection 6:47 a.m. ET User reported incident resolution. 7:18 a.m. ET Status of submitted [Talos Tickets](https://talosintelligence.com/tickets) still appears in "New" state. [Cisco Talos Reputation Center](https://talosintelligence.com/reputation_center/lookup?search=limacharlie.io) still shows domains in Untrusted state. 1:53 p.m. ET [Cisco Talos Reputation Center](https://talosintelligence.com/reputation_center/lookup?search=limacharlie.io) now shows domains in Neutral state. Users have started reporting systems returning back to normal operation. 4:22 p.m. ET We continue to work to get the LimaCharlie domains in a Trusted state with Cisco Talos. We have received reports that other Cisco security products such as IronPort are still negatively impacted. ### 2023-01-16 3:51 a.m. ET Received report of Palo Alto XDR flagging limacharlie.io domains as malicious. 7:51 a.m. ET Cisco Talos Support tickets marked as "RESOLVED - Please allow 24h for your system to update" with comment: "This domain was penalized by SDR as it used to have a Untrusted Web Reputation score due to malicious or suspicious behavior. The issue has been rectified and the domain reputation has been adjusted accordingly." 10:44 a.m. ET Posted message to [Palo Alto LIVE community](https://live.paloaltonetworks.com/t5/cortex-xdr-discussions/cortex-xdr-false-positive-detection-of-limacharlie-sensor-edr/m-p/527230#M3399) to report false positive.