Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.
Most of these ‘attacks’ are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.
For the other attacks, we are using them to investigate and implement measures like rate limiting etc.
Thank you for the amazing job, as always! Cloudflare is a solid solution :)
Sure but maybe something less centralized/proprietary would be preferable
Nothing. DDoS mitigation is inherently an ISP or someone like cloudflare. You will not have success against anybody who knows what they are doing without their help.
Well for now we’ll have to stick around with cloudflare. I’d just would like to see something managed by a decentralized network. I don’t know if it exists, it’s more of a sentiment or a general idea.
That’s easier said than done, DDoS mitigation requires a large amount of servers that are only really useful to persist an active DDoS attack. It’s why everyone uses Cloudflare, because of the amount of customers they serve there’s pretty much always an active attack to fend off. Decentralization wouldn’t work great for it because you would have to trust every decentralized node not to perform man in the middle attacks. But if you know of any such solution I’d love to hear it.
Yeah I see the issue but on the other side you would get a more robust network which could also be incentivised by some sort of underlying blockchain technology. The man in the middle attack could also be mitigated on a technical level.
Which viable alternative could work to mitigate ddos?
Out of my head, I think OVH offers such a service (but without free tier).
OVH is cheap but their anti-spam/abuse departments are ineffective. Too often they do not action blatant spam reports so in effect OVH is part of the problem with network abuse on the Internet. I’ve had to blackhole many of their netblocks while the people who run mxroute (solid email providers) have written about doing the same.
OVH needs to clean up their act.
HAProxy has some really good features a server admin can use locally without sending all of our data to Cloudflare or OVH.
https://www.haproxy.com/blog/application-layer-ddos-attack-protection-with-haproxy
There are many protection modules for most reverse proxies that provide basic (limiting) or sophisticated (captcha, calculation challenge, etc) DDoS protection. HAProxy is just a very powerful and easily extensible proxy.
Well, no. Unlike the blockchain, decentralized platforms aren’t snake oil.
Blockchain can bring trust and thus monetisation to a decentralised network. A good example is the Tor network, which is based on voluntariness, and dVPNs, which can have the same network architecture, but where the nodes are paid for their services.
There are a couple elements that a DDOS mitigation system needs to have.
It needs to be able to absorb the raw network traffic of the attack. A purely volumetric attack seeks to just overload the network pipes that lead to the servers. This can be with junk packets that don’t even make sense to an OS kernel, but have a valid destination IP address so they get through the routers. If the DDOS mitigation system acts as a filter in front of the servers, it has to not get overloaded in the same way the routers do.
It needs to allow good traffic through to the servers. If the attack causes the pipes to just shut down and reject all traffic, then the attack has succeeded. So the mitigation system has to distinguish attack traffic from good traffic, and keep the pipes open enough to let the good traffic through.
For attacks trying to do expensive stuff on the database, or create spam posts, one useful reflex the system can have is to notice when an endpoint is doing those attacks, and then block it at the network layer.
That is not necessarily easy, and it requires control of the network ingress, which arbitrary hosting providers may not be able to provide.
Thanks to the fediverse we were all able to read and search old posts on other instances and interact freely with communities on other instances. Pretty damn great i think.
@jimmy90 @PropaGandalf And I was able to finally open up and stop being a lurker
Don’t forget. Donate to them. There are no ads here. So we have to maintain the staff and servers.
Lemmy World
https://www.patreon.com/mastodonworld?utm_campaign=creatorshare_fan
Lemmy Devs
https://www.patreon.com/dessalines?utm_campaign=creatorshare_fan
Didn’t the admins for Lemmy[.]world post their expenses recently-ish? I can’t remember how much it would be for a single user to donate. I’d want to donate, but I’d like to know how much of my contribution would affect operation of the server.
Wow if I’m reading their expenses correctly, the maintenance bill doubled from May to June…
Yes. Here you go. https://opencollective.com/mastodonworld
Is this the combined bill for mastodonworld and lemmyworld?
Imagine hosting a service for anyone else to use it, free of charge, no ads, free & open API, yet some idiots think it’s fair to (D)DOS it.
There are more “interesting” targets, worst case - Reddit, who thinks everyone is just a number/noise.
Just leave Lemmy alone. :(
we will all still be here when their hyperactivity wears off.
with the old Reddit simulator, personally I’m not going anywhere anytime soon. This place has a great user base and it feels so old-school.
The new layout with old.lemmy I came back, and new apps coming out for it. It’s been a good replacement. Was on tildes, but got banned for just discussing difficult topics…the admin there is just ban happy and yea he owns the site but will just ban people for no reason. Not to mention that the users over there, assuming new people are using the malicious tag as a down vote button which probably goes right to the admin. So you step out of line and you get banned. I really liked the place too, but it’s not wanting to be a serious place to discuss topics with an admin like that.
I wonder if the owners of deddit, fb, tweetster, et al, might think it financially worthwhile to cause disruption in the fediverse, and even its ultimate failure.
I wouldn’t be surprised, we didn’t take their whole user base of anything but it’s in their interest to keep viable competitors out of the way.
Every account they lose hits them in the pocketbook. The bigger the fediverse gets, the more adherents, the greater the momentum it will have and the harder it will be to stop.
Nipping it in the bud is the best, easiest, and least expensive place to nip it.
The downvotes suggest their operatives are reading the comments.
Most likely their parasocial fans. The Reddit stans who want to be edgy and follow their meme leader. Who will never acknowledge them no matter how much they do.
It’s sad that they could target the real people making the world worse, yet only prop up the people who are oppressors.
I don’t understand why people want to take down websites. Especially sites like Lemmy, which isn’t exactly sticking it to anyone because no one owns it!
Are they just Reddit groupies?
For most hackers or wanna-bes (often called Script Kiddies, that is, people (generally young, even children thus the “Kiddies”) who are not technologically inclined enough to be real hackers and see a tutorial online on how to run pre-written scripts that repeatedly perform various functions), the answer to “Why do you do it?” is often:
-
“Because I was bored.”
-
“Because I can.”
Very rarely are other reasons given.
They’re just trolls. Lemmy is popular enough that it’s fun target for them, but still small and infantile enough that you don’t have to be hackerman to ddos it. Reddit, twitter, etc… would be constantly getting ddos’d just for the lulz by people if they didn’t have the infrastructure to make it a challenge.
I was using voip.ms last year when they were DDoS’d for over a week, by a group demanding payment via anonymous crypto. The DDoS ended when they switched to CloudFlare (which was probably pretty difficult because they’re a SIP provider.)
Almost any website with a small number of servers is vulnerable to this attack, which happens to be great business for CloudFlare. I wonder which companies are most effectively competing with CloudFlare?
There are others, but I think the craziest thing about Cloudflare is its basic level of protection is free. Free, unmetered, DDOS protection. It’s so popular because so many hobbyists use it for free, and are familiar with it. Then they convince their workplaces to adopt it when the need arises because they are already familiar with it.
They make money by selling support to companies, and selling access to some more advanced features (that often have a free tier as well). It’s honestly so impressive, it made me wonder how much they actually make because it seems unnecessary for most to pay at all. Turns out they cleared almost a billion dollars in revenue in 2022.
Nah, it’s not the 00s anymore. Hacker gangs are a real thing today.
I’m not actually in the security field so take this with a grain of salt. But I believe that these attacks play a similar role to random attacks in low level gangs. It proves that your criminal group has power and the ability to deface a website.
So if you publish that Lemmy.world will go down next week because your hackers are on it… It’s advertising. Its just business. It proves that your hackers have an ability and that you are up for sale.
Cyberspace. A consensual hallucination experienced daily by billions of legitimate operators, in every nation, by children being taught mathematical concepts… A graphic representation of data abstracted from banks of every computer in the human system. Unthinkable complexity. Lines of light ranged in the nonspace of the mind, clusters and constellations of data. Like city lights, receding…
If I’m bored I find something productive and/or fun to do.
Launching a DDoS attack is neither.
With my tinfoil hat on, I’d say one concern is that Cloudfare is basically a monopoly and nothing is stopping them from DDoSing sites to force them to use their product.
While it’s good to be suspicious, I don’t think we can call CloudFlare a monopoly quite yet.
Akamai is a big, giant competitor. You also have the big cloud providers like AWS that have their own CDN systems, like CloudFront. (I don’t recall GCP’s or Azure’s product names.) Then you have specialized CDNs like Google’s AMP system.
Now, is it possible that there could be a horizontal trust between these companies? Certainly. There’s few enough players for that to happen, but so far, I haven’t seen signs of it happening.
In case you haven’t considered this, some helpful advice. To keep them from the lemmy.world door after the CDN installation
- Change the public IP addresses
- rotate your certificates
- block all traffic appart from the CDN and only allow a limited known good IP addresses (like yours and your support team). These steps will make your server harder to find, hopefully they move on.
You might have Cloudflare add a request header to the origin request, like x-cloudflare-key: <somesecret>, and then configure nginx on the server to block everything not containing that header.
