UPDATE: @SDF@lemmy.sdf.org has responded
It is temporary as lemmy.world was cascading duplicates at us and the only way to keep the site up reliably was to temporarily drop them. We’re in the process of adding more hardware to increase RAM, CPU cores and disk space. Once that new hardware is in place we can try turning on the firehose. Until then, please patient.
ORIGINAL POST:
Starting sometime yesterday afternoon it looks like our instance started blocking lemmy.world: https://lemmy.sdf.org/instances
This is kind of a big deal, because 1/3rd of all active users originate there!
Was this decision intentional? If so, could we get some clarification about it? @SDF@lemmy.sdf.org
It is temporary as lemmy.world was cascading duplicates at us and the only way to keep the site up reliably was to temporarily drop them. We’re in the process of adding more hardware to increase RAM, CPU cores and disk space. Once that new hardware is in place we can try turning on the firehose. Until then, please patient.
Heads up: a new backend issue has been opened to track performance degradation: lemmy#3466. Looks like there’s also a private matrix channel that sprouted up for performance tuning & troubleshooting that you might potentially be interested in joining: info.
live updates in progress. Moved to SSDs, added more cores and 128GB and 64GB ram.
Here is where we’re at now.
- increased cores and memory, hopefully we never touch swap again
- dedicated server for pict-rs with its own RAID
- dedicated server for lemmy, postgresql with its own RAID
- lemmy-ui and nginx run on both to handle ui requests
Thank you for everyone who stuck around and helped out, it is appreciated. We’re working on additional suggested tweaks from the Lemmy community and hope to let lemmy.world try to DoS us again soon. Hopefully we’ll do much better this time.
Killer stuff! Sorry for my contributing undue pressure on top of what was probably already a taxing procedure happening in the server room.
Out of curiousity: how do you feel about Lemmy performance so far? I’m actually a little bit surprised that we already managed to outstrip the prior configuration. I suppose that inter-instance ActivityPub traffic just punches really hard regardless of intra-instance activity?
What can we (ordinary users) do to help? I’m in Europe, would it be better if I used the SDFeu server as my home instance?
Yes, the fediverse wants to be decentralized, so it is encouraged to use what works best for you. lemmy.sdfeu.org is located in Düsseldorf, Germany.
Two things that would be great:
- Have a tanoy/horn announce icon at the top like with Mastodon where status information can be posted.
- Change the heart icon to link to a method that supports the local instance
Attempts were made to create a thread for the almost daily upgrades we’re going through with BE and UI changes, but even with pinning it doesn’t have the visibility.
We’re on site in about 1 hour to install a new RAID and once that is completed we’ll finish the transfer of pict-rs data.
You’re all doing an amazing job!
I became an ARPA member this past week and am happy to donate again to chip in for this hardware upgrade.
Edit: donated at the $48 yearly level.
If you’re wondering how to donate, like I was-
Or if you want a button to donate: https://sdf.org/support/
Donated! SDF is what the future of the internet should look like! I love you peeps so much! ♥
