Ask Lemmy

I didn't make it, but I let it happen to teach a stupid company a lesson.

Worked as a Change Manager for a manufacturing company. They built stuff destined for landfills and you probably have one of their appliances in your home.

To set the stage, my job prior to this was as a Problem Manager/ Major Incident Manager for a large insurance company. On major incident I worked was the when a Pure Storage technician came in to replace a failed drive. A bug in the Storage O/S allowed an active drive to be shown as deactivated and safe for replacement. This did not happen all the time, only during a certain set of circumstances. I don't remember the details of that though.

My fellow IT professionals know full well what that means. That array supported a Tier 1 app and the whole thing went down hard. Not just hard, but it corrupted the database that required a multi-hour restore to fix. I also ran the Problem record on that one and knew everything about the who, what, and why of that outage.

Fast forward a year and I accepted the contract to be the Change Manager for this manufacturer. In comes a change request to replace a failed drive in a Pure Storage Array, except this time instead of paying for a service contract, this company cheaped out and did all the service in house. Which I didn't know Pure Storage allowed you to do, this was 6 years ago, but this company somehow managed it. The storage guy was a bit of prick and hated the whole Change process. Which I get to be honest, but if you want to play in an enterprise environment you need to follow procedures. So he submitted his change with little to no detail and only one director on the CAB had any familiarity with storage systems. So I started asking technical questions, which generally is inappropriate of a Change Manager as that's not the job. When I asked what the O/S version was, I got heavy push back that it wasn't relevant. When I attempted to state WHY I was asking, I was told the change was approved and to move on. As replacing a failed drive in such a large array is generally regarded as low risk. My boss actually reprimanded me for taking so much time and extending CAB longer than it should have been.

I asked her (my boss) who was on call for Major Incidents the time of the change and... It was HER. LOL

So if you've guessed that the Storage team of this company were idiots and the O/S version their Pure Storage Array was running on still had the bug, you are a winner winner chicken dinner. They also had the fantastic luck to run into the circumstances as well.

Cue another Major Incident that my boss had to manage. It took over a day to resolve and all the while several production lines were sitting silent. Hundreds of thousands of dollars of loss in production. If not millions.

When someone with almost 30 years (at that time, I'm over 30 years now) tells you that something can happen, because they've seen it before... FUCKING LISTEN TO THEM.

I didn't have anything to do with the Problem Process at that company, but I did sit in on it and you're damn straight if I didn't deliver the biggest I fucking told you so in the history of that company. The company also bought a support contract from Pure, I made sure that was a task on the Problem Record.

As to the biggest mess I made personally... I was 2 or 3 and had the bright idea of smothering myself in an entire jar of peanut butter. My oldest brother was baby sitting me at the time and our mother made him clean me up. Apparently the smell of peanut butter and water was so bad that my brother, even 53 years later, cannot stand peanut butter at all.