Please just provide a short, medium or long story
Used to work for a gaming company that had a really big Facebook presence.
When most companies launched a new game they would just launch the game connect to Facebook and call it a day.
But when we launched a new game we would turn on cross-promotion and send millions of users to the new game day one.
The problem is that Facebooks load balancers don’t scale quickly. And if you start sending millions of people to a game that just errors, it’s pretty bad for marketing.
So we had a script that you would run for an hour before you went to launch a game that would start making hundreds then thousands than tens of thousands of connections to Facebook giving their load balancers time to scale.
Well, I didn’t write the tool I was just given it and the documentation on it was kind of sparse at the time. I started with the value of 30,000. Weeeellll, turns out a sane value would have been 3-5 as it multiplied whatever input you gave it by 10K.
My cluster made sweet love with their cluster, and Facebook went down for about 3 hours.
Within a week they had some pretty decent changes to their new API limit subsystem.
Were you able to scale the thing after the changes? Did Facebook send an email to your company?
Two big ones in my younger days:
Alt tabbed one too many times, clicked drop database, clicked ok, realized I’d just deleted the live user database for America’s Army. Thankfully it was the east coast site and west coast was the primary, and it was only one way replication. We shut down east coast auth and rebuilt the secondary.
Someone distracted me while typing in a vlan command on a switch, I hit enter without double checking, took out our fiber between two datacenters in the middle of a move. Took me 15 minutes to run to the DC, plug in a console cable and fix it. Took all of our customers out.
America’s Army is such a bizarre game, at a conceptual level. Do you have any other fun stories from that? I have to imagine a lot of weird things went down behind the scenes.
I’m going to tell a story on behalf of my husband.
He was 13 and in Boy Scouts. Their troop was told that some older scouts went missing and the troop had to look for them. They formed patrols and were searching for over 3 hours when the leaders said that the older scouts were located; one of them was disemboweled and needed a medical helicopter to come from Denver. That was the “oh, crap!” moment…
Turns out the whole thing was staged. No one ever went missing. They just wanted the troop to learn how to do search and rescue. There were younger scouts there who were crying and terrified, definitely scarred by the experience.
And that’s how things were done back in the 80s.
I think this is a common thing with boy scouts. We happened upon an accident in between stations at camp. A kid ran up and tells us his dad crashed his car and he needed help. There was a man who was laying in front of the truck on the ground. His arm was bleeding profusely. We needed to administer first aid. The guy had a bunch of blood all over his arm and he was acting all incoherent. We decided to Jerry rig a tourniquet to stop the bleeding and send someone up the road to find a phone (pre cell phone days) The whole experience was super traumatic. All staged. Fuckers. They even had a pump shooting out fake blood from the guys arm.
LMAO. I didn’t remember that name but I had a feeling I knew who it was. This guy’s arm stayed on though
September 12, 2001. I accidentally shut down an entire production plant. Management didn’t even get mad. They closed the plant for the day, I kicked off the boot cycle (takes hours for the system to be ready for production again) and everyone went home to be with their families. Nobody’s head was on right that day anyway.
EDIT: A few years later I was testing some BigIP configs on a tertiary unit when suddenly the entire e-commerce site went down. Apparently this unit used to be a primary before being demoted and someone (not me) forgot to disable replication, so when I wiped all the rules from my “test unit” I inadvertently wiped all the rules to the production units. Technically it wasn’t my fault but it was still an “oh fuck” moment.
Shit job from 10 years ago. Getting too drunk at the office Christmas party and talking so much shit about my horrible fucking boss to a couple of influential senior managers who are close to the CFO. Freaked out all weekend about it, severe anxiety attacks. That mixed with the hangover I was just vomiting all weekend.
And then: Run into one of those senior managers early in the office on Monday morning, apologised for being inappropriate. Get a response “oh, we all know how bad she is, can’t believe you haven’t quit yet”.
Senior managers were also awful then; part of their job is making sure that the lower level managers don’t suck, and they weren’t.
Absolutely. A few months after this I went directly to the person above my boss and explained the situation- mental health, physical health, turned into dependence on weekly psychologist sessions.
Basically told me to get over it.
This was a non profit organisation dedicated to helping homeless, refugees, victims of domestic violence, and elderly people who are unable to afford aged care. The people on the ground doing the work were amazing people. But the people in charge were all cosplaying as big business shitheads.
I’ve heard that kind of thing about a number of non-profits. Makes me wonder how they manage to attract so many awful people.
Or maybe people in general are just awful.
