Programming

27678 readers

261 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 3 years ago

MODERATORS

snowe@programming.dev

Ategon@programming.dev

UlrikHD@programming.dev

bugsmith@programming.dev

Spyro@programming.dev

Let's blame the dev who pressed "Deploy" - by Dmitry Kudryavtsev (yieldcode.blog)

submitted 2 years ago* (last edited 2 years ago) by mac@programming.dev to c/programming@programming.dev

9 comments fedilink hide all child comments

top 9 comments

sorted by: hot top controversial new old

[–] quinkin@lemmy.world 2 points 2 years ago

If a single person can make the system fail then the system has already failed.

[–] polle@feddit.org 1 points 2 years ago

Microsoft also started blaming th eu. Its such a shitshow its ridiculous.

https://www.tomshardware.com/software/windows/microsofts-eu-agreement-means-it-will-be-hard-to-avoid-crowdstrike-like-calamities-in-the-future

[–] ByteOnBikes@slrpnk.net 1 points 2 years ago

It's never a single person who caused a failure.

[–] solrize@lemmy.world 1 points 2 years ago

Note: Dmitry Kudryavtsev is the article author and he argues that the real blame should go to the Crowdstrike CEO and other higher-ups.

[–] iAvicenna@lemmy.world 1 points 2 years ago

sure it is the dev who is to blame and not the clueless managers who evaluate devs based on number of commits/reviews per day and CEOs who think such managers are on top of their game.

[–] MTK@lemmy.world 1 points 2 years ago* (last edited 2 years ago)

If only we had terms for environments that were ment for testing, staging, early release and then move over to our servers that are critical...

I know it's crazy, really a new system that only I came up with (or at least I can sell that to CrowdStrike as it seems)

[–] Kissaki@programming.dev 0 points 2 years ago* (last edited 2 years ago) (1 children)

It's a systematic multi-layered problem.

The simplest, least effort thing that could have prevented the scale of issues is not automatically installing updates, but waiting four days and triggering it afterwards if no issues.

Automatically forwarding updates is also forwarding risk. The higher the impact area, the more worth it safe-guards are.

Testing/Staging or partial successive rollouts could have also mitigated a large number of issues, but requires more investment.

[–] wizardbeard@lemmy.dbzer0.com 0 points 2 years ago (1 children)

The update that crashed things was an anti-malware definitions update, Crowdstrike offers no way to delay or stage them (they are downloaded automatically as soon as they are available), and there's good reason for not wanting to delay definition updates as it leaves you vulnerable to known malware longer.

[–] merc@sh.itjust.works 1 points 2 years ago

And there's a better reason for wanting to delay definition updates: this outage.