Technology

0? My energy company says I'm using power equivalent to a family of eight. And it's just wifey , the servers and me. I had cops here asking if I grow weed 😁

So unless you steal power, it surely isn't close to 0 😁

I also self host and I wouldn't say the cost is zero. In the UK, energy costs alone mean that a 40W computer cost £8 per month to run (assuming a 28p/kWh price).

Of course, that's assuming you run it 24/7 at full energy use, but I know my PCs run on more than that.

I have 50.000 dollars of server hardware and zero visitors. 😵😜

the guy who designed it is stuck somewhere 10-20 years in the past.

Well, using containerization for everything is very 2015-ish.

Sounds like my homelab has better redundancy than these guys, and my monthly bill isn't much different than their new one. I only pay for power and networking, since I own my own hardware. I'm colocating in my city, so my latency to home is about 1ms, and I've got a full mirrored server in my house. Certain files are further backed up elsewhere for proper 3-2-1 backup (+ each server running raidz2 with disk encryption). Even if my home Internet goes out, I still have full access to my files at home, and all my public services stay running in the data center. If either server fails, it's all set up with containers so it's easy to spin up each service somewhere else.

One thing that's tricky to get right with disk encryption (especially with encrypted /boot) is having a redundant boot partition. I was able to hack this together by having sofware raid duplicate my boot partition to a second drive. Now if I remove either OS boot drive it falls back to the remaining one. To prevent breaking EFI boot, you need to use the Version 1 RAID format so the metadata is stored at the end of the partition, not the front where EFI reads.

every service is running without any containerization and there is a single database for everything... and it all runs on a single host and I didn't read one word about a backup strategy or disk encryption.

Man, a paragraph that can give someone some serious PTSD flashbacks....

The number of times I've had to clean up a customer's environment after they let little Billy play corporate IT and things went boom.....

They migrated from an OS and MySQL version receiving no updates since at least 2 years to MySQL 8.0 which will stop getting updates in 4 days.

I agree that it was an odd choice, as well as the OS. Going to Alma 9 when Alma 10 had already been out for some time. Would think if they wanted the long term updates they would have gone to 10 to get the most out of it. If they went from 8 to 9, sure, some people like staying in the area where RedHat got bored and won't mess with it anymore, but 7 to 9 suggests they didn't do timely upgrades before.

Also every service is running without any containerization and there is a single database for everything

Well, he said explicitly they have 30 databases, though I suppose you meant a single mysql instance. I will say I won't judge one way or another about containerization, as I've seen about as much amateur hour containerization to not immediately judge one way or another on that.

it all runs on a single host

Yeah, that seems pretty dire given his stated usage scenario, and it seems very explicit that their entire internet facing world is that single host...

backup strategy or disk encryption

It was a post narrowly discussing migration, so I don't expect a full inventory of everything they do, so backup strategy and disk encryption and all sorts of other things may be omitted as having nothing to do with the core thing. I guess the most red flag on this front is he explicitly mentions the old setup having "backups enabled" and new setup having "RAID1", which does make me wonder if they think RAID1 is a credible answer for "backup".

Also not a single word about infrastructure as code

Again, not necessarily in-scope for this document, so not sure if I'm going to judge on this one. I routinely take material expressed in terms of an ansible play and "generic it out" for general consumption when discussing with people outside my organization.

The whole stuff is hosted in Germany for a Turkish software company

I'll confess to not liking it being in a single site, however to the extent they select a single site, Germany might make sense because:

Several live mobile apps serving hundreds of thousands of users

Their userbase may be better connected to Germany than Turkey, and the user latency matters more.

My biggest concerns would be mitigated if they said that the German hosted server is their off-prem solution but it is also hosted on-prem giving them multiple sites, but I think that's a bit much to imagine given the process described. The described migration process wouldn't make sense in that scenario.

Lots of respectable EU hosting companies, and also aparently OVH, if they think there's a chance you're taking the piss will ask for a ID so they can ban you. It's not just anti-spam, it's anti-abuse and for preventing non-payment. They think there was a risk involved in accepting your business (whatever that may be, obviously companies don't dilvulge their criteria here), and if you go elsewhere they're not upset about it for that reason.

When I signed up at Hetzner, I had to go through the same anti-abuse check. However I could choose to not upload my ID and pre-pay 20€ instead. Did that and have been a happy customer since.

I never had that kind of experience with Hetzner or OVH as a European. I suppose there are extra hoops to jump through for US customers for some reason?

And get some replication in there. Even if there's not a single point of failure, if a DB instance ever goes tits-up, you'd better have a standby.

Source: I've cleaned up others' messes where they didn't.

Yes it’s ok, in general. It’s not the most modern or efficient way of managing infrastructure but it’s worked for decades now. It all depends on what you’re hosting, for who, and for how many people.

If you’re hosting internal company infrastructure for a relatively static number of users in a single of set few regions to deliver to, bare metal monolithic stuff is absolutely fine. It’s when you’re an app or service company and your infrastructure is for the back end for a public service that needs to be able to scale dynamically, and you’re worried about high 24/7 uptime, and latency to end users is a global issue that things like microservice architecture, containerization, and iac starts becoming important.

The whole containerization crazy is important for microservices architecture where you split your app into different pieces. This lets you scale different parts of you app as needed, it prevents your entire app from failing just because one part of it failed, it allows for lifecycle management like blue/green deployments with no downtime, allows developers to work on different parts of the app and update at a faster cadence than one big release for the entire thing every time you update one small part of it, things like that.

Totally fine. Containerization comes at a cost too. It's a matter of system design, knowing your risks and complexities, and handling them accordingly.

With such a size, before containerization I'm wondering if these services are not independent enough to split them onto multiple servers.

Having everything together reduce system complexities in some ways, but not in other ways.

For example the issue of MySQL 5 being unavailable would be a non-issue with a container

So people careless enough to "just container it" for old, possibly security-compromised software - you call that a "non-issue"? How about upgrading and configuring for compatibility?

Don't quote me, but as far as I know containers can't fix the issue if the host kernel is too old.

you can just set up containers on your bare metal server. in fact if you're going to install insecure services you definitely want to containerize them, though tbh you need to run really far away from whatever it is you're doing that requires sql5, or at least don't let it be reachable on the internet, that should be network-isolated, which really limits its utility.

Do tell

Interesting, why did it happen?