VoIP.ms and other VoIP providers under DDoS attack
At the time of writing, a major VoIP provider called VoIP.ms is under a distributed denial of service (DDoS) attack since over a week. As a result, they are unable to serve their customers with everyone and their dog complaining that they cannot connect to VoIP.ms’s SIP servers as well as other resources. At the same time, someone claiming to be part of the REvil ransomware group is blackmailing the provider.
This is not an isolated case, as earlier this month some UK providers were also under attack and the reports also mentioned REvil as the source of the attack. Various people from the security community have suggested that it is unlikely to be the real REvil but that is not what I’d like to write about here.
We did reach out to someone from one of the UK providers that were hit and he explained that in their sampling of the attack traffic, they did not see any SIP packets. Instead, what they did see were reflected DNS, SNMP and other traffic typically seen in amplification attacks and botnet DDoS attacks.
Not only about SIP denial of service
From reading public reports about these DDoS attacks, it is clear that the victim companies are not only suffering at their SIP endpoints. There are mentions of large amounts of traffic indicating that the attacks may be volumetric in nature rather than application-specific.
In the case of VoIP.ms, they initially had a DNS outage then moved to using Cloudflare for their DNS. They also put their website behind Cloudflare’s website DDoS protection and started asking for CAPTCHA etc.
They were also moving the IP addresses of their POP (point of presence) SIP servers and on the forums one can see people complaining about failing SIP connectivity. And lately, it has been noted that they are routing their SIP traffic through Cloudflare’s Magic Transit which provides UDP DDoS protection.
Yet they still seem to have intermittent connectivity, or appear to be offline at times.
Here’s a log from SIPVicious PRO’s SIP ping:
sipvicious sip utils ping udp://dallas2.voip.ms:5060 INFO[2021-09-24 12:41:47] pinging target (udp://dallas2.voip.ms:5060) with OPTIONS INFO[2021-09-24 12:41:48] 200 OK from target: udp://dallas2.voip.ms:5060 cseq=1 time=268.91614ms INFO[2021-09-24 12:41:49] 200 OK from target: udp://dallas2.voip.ms:5060 cseq=2 time=294.067084ms WARN[2021-09-24 12:41:51] timeout detected INFO[2021-09-24 12:41:52] 200 OK from target: udp://dallas2.voip.ms:5060 cseq=4 time=240.199604ms WARN[2021-09-24 12:41:54] timeout detected INFO[2021-09-24 12:41:55] 200 OK from target: udp://dallas2.voip.ms:5060 cseq=6 time=296.814325ms WARN[2021-09-24 12:41:57] timeout detected WARN[2021-09-24 12:41:59] timeout detected INFO[2021-09-24 12:42:00] 200 OK from target: udp://dallas2.voip.ms:5060 cseq=9 time=251.79393ms INFO[2021-09-24 12:42:01] 200 OK from target: udp://dallas2.voip.ms:5060 cseq=10 time=289.38613ms INFO[2021-09-24 12:42:02] 200 OK from target: udp://dallas2.voip.ms:5060 cseq=11 time=307.496133ms INFO[2021-09-24 12:42:03] 200 OK from target: udp://dallas2.voip.ms:5060 cseq=12 time=228.050179ms INFO[2021-09-24 12:42:04] 200 OK from target: udp://dallas2.voip.ms:5060 cseq=13 time=207.497167ms ^C Interrupt signal received, shutting down. Hit interrupt again for hard shutdown. ERRO[2021-09-24 12:42:06] network error INFO[2021-09-24 12:42:06] test complete - target: udp://dallas2.voip.ms:5060 - status: network error, network error results: - pingstats: 14 SIP messages transmitted, 9 received, 4 error, 35.71% packet loss, time 18s issues: N/A
Not the first time that SIP servers have been attacked
Back in 2011, a botnet infected with malware called Sality started distributing sipvicious. Symantec wrote about this and we had blogged about this at the time on this very blog). Essentially, the target SIP providers were unable to handle SIP cracking attacks that were being distributed. This had the same effect as a DDoS attack on the SIP servers, even if that might have been unintentional.
Attacking SIP is easy
Which brings us to the topic that is most interesting to me personally. What we discover when we do DoS and DDoS simulation exercises is that attacking SIP servers rarely requires volumetric attacks. Instead, SIP flood attacks will often overwhelm most SIP servers that have never been previously tested and therefore hardened against such attacks. With many of our clients, they already would have had protection against SIP DoS, yet we often find that their systems fail anyway as soon as we distribute our attacks. By distribute, I mean a handful of servers rather than the massive botnets that you see in actual attacks.
The main reason for such failures seems to be that protection mechanisms are often put in place without any realistic testing. Thus they tend to have gaps that we, during our DoS simulations end up abusing, causing the target services to stop responding to legitimate users.
That is why we are often successful in our tests. But there is another good question to ask: why do SIP servers fail so easily? The answer is: for various reasons - because SIP servers are complex.
Here are some of the things that often break:
- Authentication mechanisms, due to bottlenecks in the database storing the credentials
- Media servers running out of ports due to massive amount of SIP calls
- CPU and/or memory usage needed to handle SIP sessions and dialog exceeds the resources available
- Specific malformed SIP messages known to cause errors, filling up log files or logging servers
- On SIP over TCP or TLS, we encounter file descriptors resource exhaustion especially during a SIP INVITE flood attack
Most of these failure points are triggered without massive amounts of network traffic. And unfortunately, these attacks work rather well even when rate limiting is in place.
How do we test for SIP DDoS vulnerabilities?
As pentesters, we eat our own dog food, that is, make use of SIPVicious PRO’s SIP DoS Flood tool during our tests. During reconnaissance stage we do various tests to discover the most obvious weaknesses, by testing various different SIP message types for example. In the case of SIP INVITE floods, we also specify how the call is handled, whether to hangup the call at some point or not. The number of concurrent connections or which transport protocol to use for SIP is another important consideration. And finally, we choose a sending rate that runs below any protection mechanisms in place but higher than what the SIP server might withstand.
Then, we can distribute the attack.
For quick example of this, one can watch our unscripted Dangerous Demo for Kamailio World 2019 where we won the award of most entertaining demo.
Naturally, the person who setup the target machine did not have time to prepare for our denial of service tests. The system, running FreeSWITCH (I believe it was a FusionPBX installation) had no protection whatsoever. So this is not a fair example at all but it does show some of our tooling and, in part, how we do such tests.
Protection and mitigation
An excellent solution to all of this is to avoid exposing your critical infrastructure to Internet-based attackers in the first place. Simwood have a lot to say about that on various posts on their blog.
On the other hand, if your business model involves exposing your SIP servers to the Internet, the mitigation is less clear cut. So, we should first differentiate between volumetric attacks and application-level attacks.
The protection needed for volumetric attacks typically involves DDoS mitigation providers with large enough pipes to handle the massive bandwidth required to withstand such attacks. The attacks on VoIP.ms and the UK providers seem to fall under the category of volumetric attacks.
NoteAs part of our security audit and penetration tests, we simulate DDoS attacks on RTC applications and infrastructure. Talk to us to learn more.
On the other hand, DDoS protection for attacks that are SIP (or application) specific, would involve tweaks to the infrastructure, architecture and underlying software configuration. Typically, one might place very well configured Kamailio / OpenSIPS (or similar) servers at the edge - perhaps configured to block excessive SIP traffic. For these changes to be effective, they need to be informed through the results achieved through DDoS simulation. This feedback loop is critical in hardening against such attacks.
If the attacks on VoIP.ms do in fact consist of SIP traffic in a volumetric attack, then it is likely that both protection/mitigation mechanisms are needed to actually address their problems. Which might explain why even behind Cloudflare’s Magic Transit, VoIP.ms seem to be still having major issues. But this is all just speculation at this point.
If you enjoyed this blog post, check out the next one called How I learned to stop worrying and love the flood.