Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.
Most of these ‘attacks’ are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.
For the other attacks, we are using them to investigate and implement measures like rate limiting etc.
Thank you for the amazing job, as always! Cloudflare is a solid solution :)
Sure but maybe something less centralized/proprietary would be preferable
Well for now we’ll have to stick around with cloudflare. I’d just would like to see something managed by a decentralized network. I don’t know if it exists, it’s more of a sentiment or a general idea.
Nothing. DDoS mitigation is inherently an ISP or someone like cloudflare. You will not have success against anybody who knows what they are doing without their help.
Which viable alternative could work to mitigate ddos?
Out of my head, I think OVH offers such a service (but without free tier).
OVH is cheap but their anti-spam/abuse departments are ineffective. Too often they do not action blatant spam reports so in effect OVH is part of the problem with network abuse on the Internet. I’ve had to blackhole many of their netblocks while the people who run mxroute (solid email providers) have written about doing the same.
OVH needs to clean up their act.
HAProxy has some really good features a server admin can use locally without sending all of our data to Cloudflare or OVH.
https://www.haproxy.com/blog/application-layer-ddos-attack-protection-with-haproxy
There are many protection modules for most reverse proxies that provide basic (limiting) or sophisticated (captcha, calculation challenge, etc) DDoS protection. HAProxy is just a very powerful and easily extensible proxy.
That’s easier said than done, DDoS mitigation requires a large amount of servers that are only really useful to persist an active DDoS attack. It’s why everyone uses Cloudflare, because of the amount of customers they serve there’s pretty much always an active attack to fend off. Decentralization wouldn’t work great for it because you would have to trust every decentralized node not to perform man in the middle attacks. But if you know of any such solution I’d love to hear it.
Yeah I see the issue but on the other side you would get a more robust network which could also be incentivised by some sort of underlying blockchain technology. The man in the middle attack could also be mitigated on a technical level.
There are a couple elements that a DDOS mitigation system needs to have.
It needs to be able to absorb the raw network traffic of the attack. A purely volumetric attack seeks to just overload the network pipes that lead to the servers. This can be with junk packets that don’t even make sense to an OS kernel, but have a valid destination IP address so they get through the routers. If the DDOS mitigation system acts as a filter in front of the servers, it has to not get overloaded in the same way the routers do.
It needs to allow good traffic through to the servers. If the attack causes the pipes to just shut down and reject all traffic, then the attack has succeeded. So the mitigation system has to distinguish attack traffic from good traffic, and keep the pipes open enough to let the good traffic through.
For attacks trying to do expensive stuff on the database, or create spam posts, one useful reflex the system can have is to notice when an endpoint is doing those attacks, and then block it at the network layer.
That is not necessarily easy, and it requires control of the network ingress, which arbitrary hosting providers may not be able to provide.
Well, no. Unlike the blockchain, decentralized platforms aren’t snake oil.
Blockchain can bring trust and thus monetisation to a decentralised network. A good example is the Tor network, which is based on voluntariness, and dVPNs, which can have the same network architecture, but where the nodes are paid for their services.
Thanks to the fediverse we were all able to read and search old posts on other instances and interact freely with communities on other instances. Pretty damn great i think.
@jimmy90 @PropaGandalf And I was able to finally open up and stop being a lurker
Thank you for the update!
Thank you for your efforts, work and results. Those “attackers” only deserve disgust.
Anything we can do as “users” to help, other than donating?
Hmm, best would be if those kids find a real hobby so they stop bothering us. On the other hand, it helps us understand Lemmy better and secure it.
If it’s the same people, they’ll probably get tired of it and move on. But the more we talk about it, the more likely it is that new people want to get in on the “fun”. I’d say to not make memes about the downtime and pretty much act like it doesn’t exist (as users, obviously the admins should take action as necessary to mitigate it and post to be transparent).
I don’t understand why people want to take down websites. Especially sites like Lemmy, which isn’t exactly sticking it to anyone because no one owns it!
Are they just Reddit groupies?
For most hackers or wanna-bes (often called Script Kiddies, that is, people (generally young, even children thus the “Kiddies”) who are not technologically inclined enough to be real hackers and see a tutorial online on how to run pre-written scripts that repeatedly perform various functions), the answer to “Why do you do it?” is often:
-
“Because I was bored.”
-
“Because I can.”
Very rarely are other reasons given.
If I’m bored I find something productive and/or fun to do.
Launching a DDoS attack is neither.
Nah, it’s not the 00s anymore. Hacker gangs are a real thing today.
I’m not actually in the security field so take this with a grain of salt. But I believe that these attacks play a similar role to random attacks in low level gangs. It proves that your criminal group has power and the ability to deface a website.
So if you publish that Lemmy.world will go down next week because your hackers are on it… It’s advertising. Its just business. It proves that your hackers have an ability and that you are up for sale.
Cyberspace. A consensual hallucination experienced daily by billions of legitimate operators, in every nation, by children being taught mathematical concepts… A graphic representation of data abstracted from banks of every computer in the human system. Unthinkable complexity. Lines of light ranged in the nonspace of the mind, clusters and constellations of data. Like city lights, receding…
I was using voip.ms last year when they were DDoS’d for over a week, by a group demanding payment via anonymous crypto. The DDoS ended when they switched to CloudFlare (which was probably pretty difficult because they’re a SIP provider.)
Almost any website with a small number of servers is vulnerable to this attack, which happens to be great business for CloudFlare. I wonder which companies are most effectively competing with CloudFlare?
There are others, but I think the craziest thing about Cloudflare is its basic level of protection is free. Free, unmetered, DDOS protection. It’s so popular because so many hobbyists use it for free, and are familiar with it. Then they convince their workplaces to adopt it when the need arises because they are already familiar with it.
They make money by selling support to companies, and selling access to some more advanced features (that often have a free tier as well). It’s honestly so impressive, it made me wonder how much they actually make because it seems unnecessary for most to pay at all. Turns out they cleared almost a billion dollars in revenue in 2022.
They’re just trolls. Lemmy is popular enough that it’s fun target for them, but still small and infantile enough that you don’t have to be hackerman to ddos it. Reddit, twitter, etc… would be constantly getting ddos’d just for the lulz by people if they didn’t have the infrastructure to make it a challenge.
With my tinfoil hat on, I’d say one concern is that Cloudfare is basically a monopoly and nothing is stopping them from DDoSing sites to force them to use their product.
While it’s good to be suspicious, I don’t think we can call CloudFlare a monopoly quite yet.
Akamai is a big, giant competitor. You also have the big cloud providers like AWS that have their own CDN systems, like CloudFront. (I don’t recall GCP’s or Azure’s product names.) Then you have specialized CDNs like Google’s AMP system.
Now, is it possible that there could be a horizontal trust between these companies? Certainly. There’s few enough players for that to happen, but so far, I haven’t seen signs of it happening.
