Federation is broken.
It’s been broken for days - and that’s not to say for months, because I charitably include the few hours it worked again lately for some reason.
I don’t know if other instances are that thoroughly broken, because the rest of Lemmy seems to be chugging along just fine. It’s just us poor SDF user suckers.
And no-one gives a rat’s ass.
Truly pathetic…
Woah, a few minutes ago my test message to sh.itjust.works from 8 days ago finally arrived. Also a comment I made here 8 days ago.
I’ve sent myself another one about 2 minutes ago, but it hasn’t federated yet. We’ll see…
Edit: 2 more of my comments federated, latest from 4 days ago. I think we’ll be ready to federate smoothly in about 30 minutes, based on last time this happened.
Edit 2: Test message arrived after 17 minutes.
I see the UI has been updated to version 0.19.2, while the backend is still 0.19.1-rc2. Perhaps the backend update is still in progress. Could be that federation is working again simply because they restarted the backend service.
Hopefully the backend update to 0.19.2 is coming and will be more stable.
I also noticed that the web UI is kind of messed up now. None of the button icons display for me, like for upvote, downvote, reply, or notifications, so it’s a little hard to navigate.
I got a reply from membership@sdf.org:
It looks like the site is now running 0.19.2. Can you confirm you still are unable to use it?
I think it’s working now.
At the bottom of the web UI it says: BE: 0.19.2
So I guess the upgrade is complete?
If I remember rightly, the backend update takes a long time as the database needs to do a particularly slow schema update. https://old.lemmy.sdf.org seems to still be ok.
Oh damn, I didn’t even notice that.
It seems the browser cache is saving me for now:
Edit: Hmmm, interesting. The buttons are there, but they only show up briefly after refreshing.
Edit 2: The page with icons returns
HTTP/2 404
server: nginx
date: Sat, 13 Jan 2024 22:31:19 GMT
content-type: text/html; charset=utf-8
vary: Accept-Encoding
x-powered-by: Express
content-security-policy: default-src 'self'; manifest-src *; connect-src *; img-src * data: blob:; script-src 'self' 'nonce-a11a1859d413a6f8d1a7e96994bb6ccb'; style-src 'self' 'unsafe-inline'; form-action 'self'; base-uri 'self'; frame-src *; media-src * data:
cache-control: public, max-age=60
etag: W/"14aa8-/2gJKxI2YSvDtADj1ANGgw9bOSs"
content-encoding: gzip
X-Firefox-Spdy: h2
https://lemmy.sdf.org/static/9aa19d8b/assets/symbols.svg
for comparison:
https://lemmy.ml/static/9aa19d8b/assets/symbols.svg
It isn’t empty if you check the page source, there’s SVG icons.
Example:
So, still broken? How can we even check the status of the federation ?
Easy: post from the SDF instance, then check directly on the other instance if your post made it. If it hasn’t after 10 minutes, feel like you wasted your time. Rinse, repeat - until you’re tired of wasting your time and open an account on another instance.
That’s a shame. Problems like this is why lemmy won’t go very far imo. Expecting a random person on the internet to go through the signup process which involves applying for an account on some instances is wishful thinking, them understanding what federation is and how it works is very optimistic, but then braking the federation with no warning and no action plan for fixing it is just shooting yourself in the foot.
It’s frustrating and it makes me sad and disappointed to see the SDF so unresponsive to pretty much all issues (even those raised on the system itself, about the system).
That being said, I actually feel like it makes me do the opposite of waste time, because it’s virtually impossible for me to get sucked into an upsetting interaction here that has no broader consequence beyond making me feel bad.
Has anyone contacted sdf about this?
What would be the right way? @SDF here on lemmy has been pinged afaik. I don’t use mastodon, and I don’t know where to e-mail.
In the future, maybe we should ask SDF for an official way to contact them outside lemmy. I’m guessing that the admin isn’t really a lemmy user, or they would have noticed the whole thing going on.
I realize that I can’t demand anything, we’re using a free service run by volunteers and donations. I like the community here on SDF, and am interested in finding out how we, together, can make it a great instance.
I asked in #helpdesk on irc and they said to mail membership@sdf.org, which I did earlier this morning.
If I hear back I’ll update the thread.
Looks like there’s a fix incoming: https://github.com/LemmyNet/lemmy/issues/4288#issuecomment-1878442186
FYI, on lemm.ee I have been testing 0.19.1 patched with the #4330 changes by @phiresky for the past week and have noticed no further issues with outgoing federation, so I think this issue will be resolved with the next release
That patch has been applied (https://github.com/LemmyNet/lemmy/pull/4330) so now I guess we’re waiting for a release to be cut. Fingers crossed.
It is, rebooting a service/system/server or other tinkering by the instance admins has let at least one instance come back. But the bug comes back so it’s not a real solution. The instances that don’t show the problem might still have it but also have a solution watching to reset when needed.
Any idea if it’ll “catch up” and sync all the messages over the past few days, or just start properly federating new stuff going forward?
In the past, there’s been no catch up. Repost anything you want to be seen.
Things caught up the last time. I even got replies to some of my comments
Sad…
