The machines, now inaccessible, are arguably more secure than before.
@sailor_sega_saturn And given enough time and enough scale even the most improbably weird things will eventually happen. Update file corrupted by a storage controller that flips a couple of bits at random after every 720 hours of uptime but only if it’s 23.682 seconds after the hour? Weirder shit has happened.
I once helped one of my company’s customers troubleshoot an issue that had seen the same ridiculous edge case error happen three times over the course of a few years. At one point the actual sustaining developer we worked with was able to narrow down a specific bit that was getting flipped somehow, and pitched that cosmic radiation was a plausible solution given how rarely this kind of thing impacted other customers.
It was at this point that we remembered that the customer was either a university with a nuclear physics lab or a hospital with a nuclear medicine program (can’t remember now, ironically enough) that the server rack lived adjacent to.
some twenty four years ago i managed, amongst others, a company’s samba and print server (that was at the time when all the company’s servers were beige boxes with less memory and disk than the laptop i’m using to type this – and still they served a few hundred employees).
the machine developed a strange custom of hard-resetting itself, which we initially tracked to specific files being sent for printing; the behaviour was fully reproducible.
as it happened, it was a hardware fault somewhere between the mainboard and the integrated SCSI card; installing a separate SCSI card and reconnecting the disks and backup tape device fixed the problem. (i did not have the budget for a new serwer, no.)
establishing the actual cause took me fucking weeks.
@m @sailor_sega_saturn
Builds failing, but only at the new office, and only if you tried to build from scratch.
Funny, the Windows network crew that operated the network and suddenly had to operate NFS over UDP on their network, never really realized that their switches were only capable of half-duplex operation. But announced full-duplex. And these Linux boxes fully used that. And big UDP packages used by NFS under load got corrupted.
@m @sailor_sega_saturn
Took a f%cking nightshift of the CTO (German company, so the CTO had PhD in C.S. and still remembered hacking C++ code) and the resident external IT consultant working on the C++ code getting frustrated with the builds crashing and literally debugging the whole shebang to discover that beside a ton of C++ memory bugs, we also had a network issue.
@m @sailor_sega_saturn And philosophically, I’ve been now for a decade in “automatic data entry from 3rd parties”, ETL (nice phrasing for industry level web scraping and data clean-up).
Literally, what I’ve seen (and sometimes, as I’ve also done website development, one wonders what the f%ck the dear colleague was thinking while (s)he developed THAT. Or I want the drugs they were on, that must have been a great trip.), nothing is unthinkable in IT.
