2023-01-13 - Cisco Umbrella blocking access to LimaCharlie cloud

On 2023-01-13 we became aware of an issue where Cisco Umbrella was blocking access to the LimaCharlie cloud.

Status

We are awaiting a response from the Cisco Talos team to have the "limacharlie.io" domain added to their false positive list.

In the meantime impacted customers may add the following LimaCharlie domains to their Global Allow list by following the steps below:

Workaround Steps

Log into the Cisco Umbrella portal
Go to Policies -> Policy Components -> Destination Lists.
Add the following entries to the Global Allow List:

limacharlie.io
9157798c50af372c.lc.limacharlie.io
70182cf634c346bd.lc.limacharlie.io
4d897015b0815621.lc.limacharlie.io
b76093c3662d5b4f.lc.limacharlie.io
aae67d7e76570ec1.lc.limacharlie.io

Your sensors should start resuming connections to the LimaCharlie cloud.

Requesting your assistance

If you have any contacts at Cisco who may be able to get this unblocking expedited, please reach out to us via email or on our community Slack

Additionally, please submit a Web Reputation Support Ticket to Cisco Talos. This will help increase the reputation of the domains above.

Impact

Customers who utilize both the Cisco Umbrella network security product may have found they have suddenly lost connectivity with the LimaCharlie cloud. This includes access to the LimaCharlie:
- web application,
- APIs,
- endpoint telemetry servers,
- output destination servers

Mitigation of Risk

The LimaCharlie sensor continues to collect telemetry while offline, and transmits it once the connection has been re-established. Further details can be found in our article "What happens when the host is offline?".

Timeline

2023-01-13

8:18 a.m. ET
User reported issue via community Slack

1:04 p.m. ET
User reported issue via private Slack channel

1:08 p.m. ET
Opened Cisco Umbrella support ticket #1448986, per reporting procedure

1:31 p.m. ET
Added notice to #status in community Slack

1:39 p.m. ET
Submitted false positive report to Talos Intelligence, submitted the following domains as false positives:

limacharlie.io
9157798c50af372c.lc.limacharlie.io
70182cf634c346bd.lc.limacharlie.io
4d897015b0815621.lc.limacharlie.io
b76093c3662d5b4f.lc.limacharlie.io
aae67d7e76570ec1.lc.limacharlie.io

1:45 p.m. ET
Signed up for Cisco Umbrella trial to verify steps to add limacharlie.io domains to the allow list.
Submitted a categorization dispute

2:46 p.m. ET
Called Cisco TAC via telephone to try to escalate issue; they emailed Umbrella support on our behalf and provided my direct telephone number.

3:30 p.m. ET
Opened another support ticket via Cisco Umbrella support request #1449194; case ID #1449194

7:29 p.m. ET
Submitted escalation request via email on Talos ticket

2023-01-14

1:09 a.m. ET
Validated issue still persists via Talos reputation center

Submitted Web Reputation Support Ticket

Included the following in the ticket:
Cisco Umbrella is incorrectly reporting the LimaCharlie domains listed as Phishing. LimaCharlie provides a legitimate security infrastructure-as-a-service offering, which can be validated at these external sites:

GitHub
Shows our long standing history, first as an open source project and then as a commercial offering. See link to our public repositories which include links to our domain with authenticated date/time stamps.

Crunchbase
Shows funding for our company and the legitimate backers behind us.

Further questions can be directed to Amrik Randhawa, Chief Operating Officer via email at {answers@limacharlie.io -- telephone number provided in the submitted ticket}.

4:11 a.m. ET
Outreach to Cisco Talos Intelligence Group via connection

6:47 a.m. ET
User reported incident resolution.

7:18 a.m. ET
Status of submitted Talos Tickets still appears in "New" state.

Cisco Talos Reputation Center still shows domains in Untrusted state.

1:53 p.m. ET
Cisco Talos Reputation Center now shows domains in Neutral state. Users have started reporting systems returning back to normal operation.

4:22 p.m. ET
We continue to work to get the LimaCharlie domains in a Trusted state with Cisco Talos. We have received reports that other Cisco security products such as IronPort are still negatively impacted.

2023-01-16

3:51 a.m. ET
Received report of Palo Alto XDR flagging limacharlie.io domains as malicious.

7:51 a.m. ET
Cisco Talos Support tickets marked as "RESOLVED - Please allow 24h for your system to update" with comment:

"This domain was penalized by SDR as it used to have a Untrusted Web Reputation score due to malicious or suspicious behavior. The issue has been rectified and the domain reputation has been adjusted accordingly."

10:44 a.m. ET
Posted message to Palo Alto LIVE community to report false positive.