I know that Lemmy is open source and it can only get better from here on out, but I do wonder if any experts can weigh in whether the foundation is well written? Or are we building on top of 4 years worth of tech debt?
There’s no reason that a monolith can’t scale. In fact you scale a monolith the same way you scale micro services.
The real reason to use micro services is because you can have individual teams own a small set of services. Lemmy isn’t built by a huge corporation though so that doesn’t really make sense.
I disagree that it being a monolith is immediately a problem, but also
In fact you scale a monolith the same way you scale micro services.
This is just not true. With microservices, it is easy to scale out individual services to multiple instances as demand requires them. Hosting a fleet of entire Lemmy instances is far more expensive than just small slices of it that may require the additional processing power.
What microservices would you split Lemmy into? The database, image hosting and the UI are already separate.
Well, I’m going to start by repeating that I don’t necessarily agree that it being monolithic is necessarily a problem right now.
The immediate thought in my mind would be all of the federation logic. That’s where all of the instances seem to be lagging behind, and it seems the common fix is “just increase the workers to one billion”. Apparently that does something meaningful, but the developer in me wants to know how a few cores can put so many workers to use.
Spinning federation off into a microservice means you could deploy it on something like Cloud Run or AWS ECS, and have it autoscale as the workload demands it. Seems like a pretty prime candidate to me.
I’d split it out into a few systems
- Signup/Login/Account management
- Posting/Commenting/Voting
- Moderation Controls
- DB Readers and Writers in different services
- Community management (may get lumped into moderation controls, but separation of duties may be needed)
- Edit: Federation is a big one I keep forgetting about
The ultimate goal, and I don’t know how possible it is with rust, would be to have a way to run those as individual services or as one part of a larger monolith. Smaller instances would be able to run it as one binary, while larger instances like Lemmy.world or Lemmy.ml can run each part independently. That would allow easier scaling on large instances while (hopefully) leaving it just as simple to deploy on a small scale too.
There’s no reason to split it into 100 different services, but 4-5 mid sized ones might help.
You can easily scale a monolith. You typically horizontally replicate any web server (monolith or not) to handle whatever traffic you’re getting. It shouldn’t really matter what type of traffic it is. Plenty of the world’s biggest websites run monoliths in production. You know how people used to say “rails doesn’t scale”? Well they were wrong because Rails monoliths are behind some huge companies like GitHub and Shopify.
The lemmy backend is also quite lightweight and parallel so it’s cheap and effective to replicate.
In my professional experience microservices are usually a dumpster fire from both the dev perspective and an ops perspective (I’m a Site Reliability Engineer).
Lemmy’s backend is native code, not run in a virtual machine or interpreted from text. You’re not going to pay much extra for an extra megabyte or 10 of RAM being used per instance for the extra code sitting idle. It certainly shouldn’t use much processing power when not in use.
I’d be less concerned with memory (of which Lemmy seems to use very little), and much more concerned with CPU core count. I touched on it in my other comment, but I don’t understand how a few cores is supposed to handle the ridiculous number of federation workers people are setting their instances to.
You definitely can’t scale a monolith the same way you can scale a micro service
You can easily scale a monolith. You typically horizontally replicate any web server (monolith or not) to handle whatever traffic you’re getting. It shouldn’t really matter what type of traffic it is. Plenty of the world’s biggest websites run monoliths in production. You know how people used to say “rails doesn’t scale”? Well they were wrong because Rails monoliths are behind some huge companies like GitHub and Shopify.
The lemmy backend is also quite lightweight and parallel so it’s cheap and effective to replicate.
In my professional experience microservices are usually a dumpster fire from both the dev perspective and an ops perspective (I’m a Site Reliability Engineer).
I can’t say I disagree… Poorly implemented microservice architecture is the bane of my existence. Well implemented, though, and it makes my job so much easier.
Granted, my SRE team has all public facing production infrastructure built using an IAC process, if something causes too much trouble, it’s easier to quarantine and rebuild the offending node(s), and can be complete in under 10 minutes.
The biggest problem is far too many developers ignore the best practices and just shift existing code into smaller services. That will never give you either performance or stability benefits. Honestly, it will probably make any issues worse. Microservice architecture is a huge shift in thinking. The services need to be fairly independent of each other to really make any gains. To get to that point will always take a whole lot of work. That being said, there is nothing inherently wrong with some monoliths, but the benefits of splitting out as much of the higher traffic and resource intensive work should never be overlooked.
