this post was submitted on 24 Mar 2026
37 points (95.1% liked)

Ask Lemmy

38725 readers
917 users here now

A Fediverse community for open-ended, thought provoking questions


Rules: (interactive)


1) Be nice and; have funDoxxing, trolling, sealioning, racism, toxicity and dog-whistling are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them


2) All posts must end with a '?'This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?


3) No spamPlease do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.


4) NSFW is okay, within reasonJust remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com. NSFW comments should be restricted to posts tagged [NSFW].


5) This is not a support community.
It is not a place for 'how do I?', type questions. If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.


6) No US Politics.
Please don't post about current US Politics. If you need to do this, try !politicaldiscussion@lemmy.world or !askusa@discuss.online


Reminder: The terms of service apply here too.

Partnered Communities:

Tech Support

No Stupid Questions

You Should Know

Reddit

Jokes

Ask Ouija


Logo design credit goes to: tubbadu


founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[โ€“] Chippys_mittens@lemmy.world 23 points 21 hours ago (1 children)

The rare 10-30 seconds when something first goes wrong. The mix of surprise, concern and instant mental troubleshooting. I'm an engineer in a big power plant on the operations side. My job 99% of the time is to take numbers and do some wrenching/maintenance when needed. 1% of the time I earn my money by running at the scary shit and figuring it out.

[โ€“] TehBamski@lemmy.world 7 points 20 hours ago

Hats off to workers like you. The unsung heroes of modern society.

I had a stint in IT many years ago. I remember being at my first IT job, working as a help desk person. Months in and there was a lull in work for me. Things were all working just fine or better. It was at this point that I mentioned it to my boss (IT Manager of the whole multi location business). 'Man, there's nothing really to do. It kinda makes me feel guilty that I'm getting paid to basically sit around.' I'll never forget what he replied with after giving a bit of a laugh. He said, "We're here so when a 'fire' breaks out, we can quickly put it out and assess the cause. Then make a preventive plan for this issues like this." I laughed and smiled. As I then realized that I was in a lesser way, a electronic device 'firefighter.' We standby to quickly and effectively address network, computer and printer problems.

He gave an example he had read online a few years prior, of someone who was working for a $4-5 million dollar valued, up and coming corporation. (This I believe happen in 2012 or so.) While the execs were in a meeting, trying balance the books, I guess the founder realized that they were paying for a small team of four IT people and wondered why they had to have so many of them. He complained about how much they were paying to have them staffed and how expensive it was for the two senior IT people they had. I guess they were paying them each over $110,000 salaries. (Each of them specializing in key systems the corp. used daily.) He pointed out his observations, that he would often see and hear them sitting around in their office space, talking and joking with one another. He wanted to know why they kept them onboard if they hardly ever worked and why they 'deserved' to be paid so much. This went on for two days or so where the execs kept trying to find ways to cut costs and balance things out. And the 'IT cost' would keep coming up. They were planning to cut the team down to one or two people and maybe contract out one or two remote, to save money.

Well, on the third day of the execs going over the books and coming close to a final decision, there was a huge server crash that was caused by a hacker or hacker group who had gotten into the network system and into their email server. The IT team entered Red Alert mode and stayed there working until around 3 in the morning, 'putting out fires,' bring things back online safely, and without losing critical server data during it. Even though they had to pull nearly all of the servers to address the issues one by one offline. The company was losing tens of thousands of dollars every hour things were down. And so, come the next morning when the founder and execs came back into the office the next day and get the final report that about 90% of things were back to normal now, and the company had only lost an estimated $400,000 or so in work value while the IT infrastructure was down.

While the IT seniors reported on what had happen, how bad things had gotten and what course of actions the IT team had done to get things back to normal, all while only being about to get a few hours of sleep before the early morning meeting. The founder and execs realized how big of a bullet they had missed but having an in-house IT team and experts that could physically fix things.