While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.
As I am told, this was the issue:
- There is an vulnerability which was exploited
- Several people had their JWT cookies leaked, including at least one admin
- Attackers started changing site settings and posting fake announcements etc
Our mitigations:
- We removed the vulnerability
- Deleted all comments and private messages that contained the exploit
- Rotated JWT secret which invalidated all existing cookies
The vulnerability will be fixed by the Lemmy devs.
Details of the vulnerability are here
Many thanks for all that helped, and sorry for any inconvenience caused!
Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).
For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.
Very impressed by how quickly action has been taken by this and other instances to patch the issue.
Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.
I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.
In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)
I’m on iOS with the Memmy app. It’s a work in progress that’s officially unfinished so I’m not surprised but it has also been a bit buggy. Doesn’t seem that I can log out without deleting and reinstalling the app so hopefully this doesn’t happen too often XD
So I was actually just struggling with that myself, also in the Memmy app in case that isn’t clear
What I did was add my account (again)
There was no warning or anything, and it populated the list with two of me.
At that point, a “delete account” option appeared under both of them. So I guess in normal circumstances, it wants you to keep one account around at all times?
I deleted one of them, and the app basically reinitialized. Both were gone and it showed me the welcome screen.
I logged back in, and now everything is back to normal
No you can. You just remove the account from the accounts list. It’s labeled “delete this account” which is scary but it just removes it from Memmy. You can add it right back and that logs you back in. Not a great experience.
I sure hope this doesn’t happen a lot. This kind of barrier hurts site growth. I’ve managed a lot of large sites and seen a lot of bugs and when everyone gets logged out there is a measurable impact, and some folks never return. Just look at all the comments here saying “thank I didn’t know to do that.” For every one of those there are 100 people going “huh… Lemmy is down… oh well… on to something else…”
For Memmy, I went to the accounts page in the settings. Click d on my lemmy.world account then to the page where you can change the password then navigated away. That reactivated the account. Maybe we should add a ticket on Memmy’s GitHub about reactivating cookies when there’s an issue. Or at least place à poput to double check credentials or something.
Negative one upvotes would mean that enough people disliked me/another poster to bring my upvote total to zero. (Upvotes and likes are effectively the same thing, it’s just a naming convention). Reddit totals them up and seemingly Lemmy does as well.
huh that’s weird (yes I meant negative one downvote), I already know that the total can be either positive or negative, but shouldn’t the upvote number and downvote number be either positive or zero? (for now I’ll just accept it as a lemmy bug/ inconsistencies between instances)
I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.
As I pointed out in the thread it was probably a few Lemmy users themselves that did it.
And some people do it for teh lulz.
Not everything has to be some conspiracy theory.
it doesn’t matter who, it’s the “why”. They get nothing from this, the only one who benefits from Lemmy going down is spez
Reddit, Threads, Twitter, Elon, and, Zuck seem to be all people know how to talk about in the more popular communities.
Nah. There’s far too much risk for Reddit to be involved. If even one hacker spilled the beans it’d cause a massive panic for Reddit investors.
“It would be extremely stupid and would inevitably backfire if reddit was responsible. Seriously, if spez has one ounce of foresight he would not be involved.”
Well you have convinced me reddit is behind it.
It’s well established that Rdeadit* doesn’t make foolish moves. /s
I’m not saying they’re behind this. I think they are not. But I’m not ruling them out because of their acumen.
During the event a message flashed on my screen, “This website has been seized by Rdeadit for copyright violation”. That suggested two things to me. Rdeadit didn’t write that. Whoever did write it doesn’t understand how a domain would legally be seized.
Deleting hospital fees/debt is very dangerous… In many HUGE regions in the US there’s only one hospital and if that hospital suddenly can’t pay its bills it could shut down, leaving a whole lot of completely innocent people in a very sad, people-are-dying sort of state.
In fact, something like this already happened:
https://www.cbsnews.com/chicago/news/st-maragrets-health-central-illinois-hospital-closing/
Hospitals are special in that they’re often evil organizations (not all though) that are some of the easiest to hack but also provide critical services to the most vulnerable. One should tread lightly. Political solutions are better (hack some politicians that are against healthcare reform instead).
Clearing credit card debt via hacking is nearly impossible but I agree it would be a much more ethical choice for hackers to target. I used to work for the credit card industry. My unique insider perspective, deep industry knowledge, and personal experience is here to let you know they suck. They are just as evil and unethical and unnecessary as everyone thinks they are! Seriously: If Visa, MasterCard, American Express, and all the lesser players suddenly disappeared the world would be a better place.
Before that can happen though people need a backup payment method that doesn’t go through their systems and no: Cash won’t work (there’s not enough in circulation and it’s dangerous to carry large amounts of it). The credit card companies know this threat exists which is why they lobbied Florida (and probably other states) to outlaw alternative, government-run forms of payment (e.g. central bank currency).
As soon as people have a widely accepted payment option that doesn’t go through Visa and MasterCard’s middlemen (e.g. First Data) then hackers can take their gloves off! Until then though… Let’s keep the payment infrastructure working, OK? Thanks!
There’s no limit to the amount of good deeds hackers can do though. So let’s encourage that! For example, there’s plenty of cartels and evil religious organizations (e.g. Taliban, ISIS, Mormon Church, Prosperity Gospel scam artists) that have plenty of money to spare and enormous attack surfaces 👍
clearing credit card debt, deleting hospital fees, or something else that actually serves the public good,
Inflation does very clearly not serve the public good. That aside, causing havoc in banks and medical institutions would have other unpleasant effects.
How about cleaning the bottom 10%'s debt, with the earings from one week of the top 0.1%?
I already know I’m gonna be downvoted for this, but the top 1%/0.1% spending isn’t gonna change, whereas the bottom 10% will cause inflation… That’s why there’s no magic bullet.
Ah, you mean unauthorized “redistribution”, not unauthorized “vanishing debt”.
Technically should do less harm in terms of inflation, but money lying around is different from money being used, so there’ll still be an increase in inflation.
The part about causing havoc - kinda same, there may not be direct inconsistencies as in the initial variant, but there’ll still be some confusion due to the “top 0.1%” possibly being petty and trying to get their money back.
I frankly prefer changing the rules so that there’d be fewer artificial barriers for competition and economic efficiency to this. Say, patent law and trademark laws and IP laws have basically outgrown their usefulness and are now just a plague. Same with various licenses and practices for medical/pharmaceutical stuff (I know that things should be tested and an average person can’t tell a hoax from a normal thing, just entities doing certification shouldn’t be able to block stuff which would then be used to create oligopolies). Same with telecom. And so on.
Except for air traffic, water traffic, road traffic and radio, of course. Not regulating those would mean, eh, real havoc.
Having a dedicated sub for bad understanding of economics seems stupid, it’s already spread over all subs, it’s normal.
Of course, the extremes of bad economics would be usually found someplace with “soc” in the name.
Thanks Ruud for fixing it! Just a reminder guys that If you are using a third party app you need to login again.
what steps are being taken to ensure it doesn’t happen again? was any personal data compromised for users?
Also I am curious, what’s the easiest way to currently reach the admins in case this happens again somehow? Two of them on their account have been seemingly inactive for a month and as per your own statement you rarely check your notifications and dms. Is there a discord somewhere for it?
Mail: info@lemmy.world Mastodon: @mwadmin@mastodon.world Matrix: https://matrix.to/#/#lemmy-support-general:discuss.online
So all our cookies are negated now with the JWT changed, and we just needed to login again? Can attackers have stolen our cookies in order to use our accounts to post as if it was us? I’m sure they were only interested in admin cookies, so most others were “useless” to them? I see nothing wrong with my posts so I should be safe, right?
If you think they could change your password:
YES, they could.
They could have changed the email => “Forgot PW” and with that you lost ur account.
They defaced it with dicks and changed the federation list to be only threads.net. I don’t think it was a state sponsored chinese hacking group. :)
IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.
