The best time for an update, while hopped up on candy , coffee, and playing trick or treat on the neighbours, enjoying family, friends or just a relaxing day. I hope wherever you are, you are enjoying the holiday season(s).

Reddthat has been super stable and nothing major to report except for pictrs crashing recently (as my lack of updates may have proven).

This sat in my drafts for way too long! But now's the time to finally update you all as there are actually items to update you on!

Moderation

The other admins haven't been online as much lately and I'm doing most of the moderation. So I may end up asking people if they would like to become administrators of reddthat. Mainly I need some people in US time zones and EU time zones. (Which should nearly be all of you 😅!). The majority of the work results in actioning reports and ensuring we don't let bots/people not even trying to read the application process into Reddthat.

If you would like a position, please PM me with:

• Your timezone/Active Hours
• What makes you a good candidate for an Admin. (Prev experience?)

Storage storage Storage. New Server!?

Edit: we completed it in https://reddthat.com/post/57007804

As I went to purchase the new server in June/July right after posting our last update the company IMMEDIATELY jacked up their prices. It must have been a typo because it's now for $135/m. Which would give us 2x500GB and 32GB of Ram. The processor is a Xeon E-2136 , which only comes with 12 threads! With a single thread rating of 2700, which is about double of what OVH is giving us.
So hopefully our database will be even faster once we make the change.

On OVH, we managed to finally get the storage increase and it's worked without any problems. Which was perfect timing really. We now sit at around 75% usage of our 200GB drive. So I'm hoping to get 2-3 more months out of it and pray to the tech gods there are some black friday dedicated server deals or even Christmas ones which will allow us to get a better deal.

And that was what happened. We managed to get a deal on a dedicated server in OVH which was around the $90/m . So only a little bit more than what we are paying now. Which is great. From a brief benchmark we get at least 1.5x faster CPU which equates to shorter scheduled tasks, faster queries, and more resources available to showing everyone their own subscriptions and memes. That's not inclusive of the 500GB ssds, which will allow for our database to grow to 2-3x our current total. Which means this should last us around 4-6 years (at the current growth of Lemmy as a whole).

I've been getting it working over the past 2 weeks Oz. We're taking this time to also upgrade to Debian Trixie, which is currently not supported by the Lemmy ansible repo. So I will be upstreaming that change once I fixed it. This includes keeping it up to date so that when we do the swap over in a week's time it will be as short as 5 minutes and as long as 15 minutes for the swap over to happen. I'm pretty sure it will be closer to 15mins as even at 1Gb/s our 100GB database will take around 800s to completely migrate at the theoretical maximum. (And a brief 15 minutes, is better than mucking around with postgres' HA).

This server change is probably going to happen on the 27th/28th. I'll add another post outlining the timeline on the day.

Cheers,
Tiff

Note: On Liberapay, donations are paid in advance, but you are more than welcome to make it recurring monthly instead of paying yearly. Don't worry too much about the "fees". It's just the cost of doing business via the credit card monopoly.

💸 "Expenses":

• August Costs: ~A$116
• September Costs: ~A$117
• October Costs: ~A$121 (Increased storage costs and falling AUD/USD)
• Nov Costs: ~A$125
• Dec Costs: ~A$125

⭐ Donation "Statistics":

• New Donators in Aug,Sept,Oct: 2
  • Thank you again! September and October was dedicated to both of you!
• New Donators in Nov, Dec: 0
• Total Weekly: ~$24.54
• ("Monthly": 24.54×52÷12 = ~$106.34)
• Our Public Donators:
  • AppleStrudel
  • ~1890351
  • Matthew Fennell

🥅 Goal: 24.54 / 60.00

Want a month dedicated to you? -> https://liberapay.com/reddthat

PS: don't like fees? Use Crypto (Litecoin/Monero) for even better transaction fees than credit cards for your donation. (See the main sidebar for addresses). And validate them again on liberapay too if you want to ensure I get those dollary doos.

As we were originally trolled by OVH in the original server resources upgrade: https://reddthat.com/post/47931907

We now have confirmation that it will now be actioned.

Starting tomorrow, 28th September 00:00 to 01:00 UTC we will be upgrading.

This will be handled by OVH and is estimated (by them) to take between 30 to 60 minutes as it requires them to copy our storage to a (new?) system.

See you all then!

Countdown Link : https://www.timeanddate.com/countdown/weekend?iso=20250928T00&p0=1440&msg=Maintenance&font=cursive&csz=1

Ps. Any update incase there are issues will be on our status page: https://status.reddthat.com/

Small update as I'm now tired and going off to sleep.

I have finally implemented go-away on our main site and it hasn't broken (fingers crossed) any of our frontends!

I have added the non-js challenge which is called a meta-refresh challenge. This means those of you who view sites without javascript will also be able to continue using reddthat without troubles, but the bots are already failing!

We currently have sent the HTTP 418 (I'm a tea pot) Response code over 1900 times since implementing goaway (within the last 5 minutes)

I'll be looking to add a JS challenge in the coming weeks if the bots start implementing proper browsers.

FYI. If you are affected by this, and cannot seem to complete the challenge, you can get around this by using our Voyager frontend to let me know, or any Lemmy application. As both of those directly interface with the API endpoints.

Cheers,

Tiff

Edit. In the past 12 hours we have had the following total queries by http code:

Count   HTTP Code
   4654 200 <- people who "completed" the "challenge"
    406 206
    122 301
      8 302
    243 307
    204 400
      3 403
    479 404
  34946 418 <- very bad bots
      1 499
    186 500
   1617 502 <- meta, openAI, bytedance, etc 
      4 503
      2 504

That's 2 hours from this post.

At 01:00 UTC we will be having a 15-30 minute maintenance due to a server upgrade. (More storage!)

As a reminder we have a status page on: status.reddthat.com which I use when we have these outages.

(Yes I'm aware of 'old' showing up as down, the uptime checker is blocked by our anti-ai protection for old)

See you on the other side.

So apparently it's not instant and we are in a queue to get upgraded... We are... 65th in the queue. I'll edit this post once we get an update.

Hello again to everyone and a special welcome for the people who ended up coming here from Lemm.ee, as well as to anyone joining the Fediverse for their first time!

It's been a weird couple month(s) for us specifically with our issues with our donation holder. Lemm.ee shut down. We turned 2 years old! How time flies...

Donations

It was a stressful time! For those of you who didn't see; I hadn't been reimbursed from our donation account for over 2 months and hadn't heard anything from them. Eventually they must have seen our emails or word got around and the money was transferred to me.

Choosing to outsource the financials was a choice which I think most of the Lemmy servers have done (especially for OpenCollective), and was (probably still is) the easiest to setup. When I started Reddthat right at the start of the exodus of Reddit, I knew people would want to donate, whether it was out of spite, or for a better future. OpenCollective solve that and was incredibly easy to setup. As you don't have to fill out any paperwork besides from your details and bank account for reimbursements. It was a no-brainer for me especially when it came with complete transparency, which I have tried to provide to Reddthat across it's lifetime. This inherit decoupling of assets also came with risks I never imagined but I should have. Especially when beehaw had similar experiences. The writing was on the wall.

Now I have used my already setup Stripe account and finished Reddthat's liberapay. Which is over --> here if you want to see it. The major benefit just like OpenCollective, is that you can make the donation a Secret donation. So I personally will have no way to know who is donating. Obviously if that happens I wont be able to add your username to our list of heroes, but I think it's time for a change. Maybe let me know what you think the new section could be called in the comments?
So the long and the short of it is that now I control all of our financials so no need to worry about someone else not paying me or doing some exit scam.

Server Resources (tech speak time)

Recently we ended up having some timeouts due to memory issues. Fortunately this was solved temporarily by reducing the amount of connections (pools) we allowed Lemmy to use for it's database connections. But this is not why I'm bringing this up. Our database has grown to... Checks 108GB on our filesystem. This has always been on our radar. Especially when in our 1 Year Old post we reached approx 30GB.

Our current OVH server only has 158GB available. With everything else stored on our server we currently have used 124GB of our 158GB. So we only have ~28GB available. Which isn't a lot of wiggle room. Especially when we need to perform daily backups of the database too! Luckily the database does compress quite well and a lot of the indexes take a huge amount of data but there isn't anything I can do about that. To save data on the biggest culprit of indexes I'd have to remove some filtering options from our codebase. Which maybe might help but probably won't.
We are using approximately 5GB of storage per month at the moment. So in about 4 months time if we don't do anything we will be going offline because our server is out of space! Obviously I won't let that happen and have a nice solution planned in the coming months. A dedicated server.

Start reading here again if you skipped the tech speak

So we are slowly growing over time so we will need to increase our server resources again.

(Another) Migration

I seem to like this word. Migration. I use it a tonne at my $job while also seem to be using it more and more at my lemmy job too! In August/September or sooner depending on when our renewal for our OVH server is, but we will be migrating to a Dedicated Server.

• CPU: E-2136 (6Core/12Thread) @ 3.50GHz
• Ram: 32GB
• HDD: 2x500GB (NVMe)
• BW: 10TB/month
• Cost: ~A$90/month

To put it in perspective we get 8 vCore, 16GB Ram and 160GB SSD for ~A$88/month. So for an extra $2/month we can get "infinite" storage, double our RAM and get a way better cpu as well!
Now let's not get ahead of ourselves. I'm not quite ready to pull the pin on it and have to double check a few things first to make sure this company is all on the up-and-up as when they say it's too good to be true, it probably is!

2 years!

Turning 2 years old has been a privilege. For those of you who have stuck with Reddthat since the early days of server issues, the recent server issues, and the many many migrations; I truely appreciate your interactions and the communities we are fostering on our own little corner of the web. Now whenever I type 'red' in my browser if auto fills to a social site that I run. Which is kinda awesome. But it never would have been possible without everyone. So from the bottom of my heart.

Thankyou!

<3 Tiff

PS. I'm also toying with the idea of setting up a piefed instance for those of you whom want a more picture-style feed? Would people enjoy that? And if so, what should our domain name be?

Starting now I'll try and keep a running total at the bottom of these posts and some ideas of what the next months are going to bring. We are very stable at the moment. I manage to scape by as I bundle a lot of things with my own personal bills / items. Things we don't pay for: Email Sending, Git hosting & automated build server(s). As I use my own infrastructure for these items.

💸 "Expenses":

• Budget starting June: A$518.40
• June Costs: ~A$114
• July Costs: ~A$115
• August Costs: ~A$116

💰 Estimated Budget (At the start of the month before payment of that month)

• September: A$173.40 (+A$176.64) ~A$350.04
• October: A$233.04
• November: A$116.04

⭐ Donation "Statistics":

• New Donators in June & July: 5
  • Thank you again! August is dedicated to you!
• Total Weekly: ~A$22 (Advance payment of A$176.64)
• Our first Public Donator: ⭐ ~1883941 ⭐

Want a month dedicated to you? -> https://liberapay.com/reddthat

Opencollective no more, Hello Liberapay!

• Reference Open Collective Post: https://opencollective.com/reddthat/conversations/we-have-migrated-to-liberapay-4pzb7zgp#comment-1749981430361
• Our Liberapay: https://liberapay.com/reddthat
  • Which goes directly to me
• You can also do ko-fi: https://ko-fi.com/reddthat
  • Which goes via Paypal (I think)
• Or send it directly via Crypto if that is your thing.
  • See the main donation page for the addresses

Summary:

It has now been 8 weeks since our last contact with our fiscal host on Opencollective and I have made the hard decision to migrate away to Liberapay.

"New" Donation Platform

When I started Reddthat I wanted to show everything that we were doing in a transparent way. We have the modlog which shows eveything that happens on Reddthat and the Lemmy-verse and I wanted to bring that to the financial side of things as well.

So I looked around at the payment processors. Patreon/etc had higher fees than what I expected and as they were donations I really didn't want to lose up-to 8% of your good will! Liberapay was the obvious choice originally but it turns out they are a platform where people can donate directly to people. Everyone needs to link their Paypal, Stripe, or EU Bank account to accept transactions on Liberapay.
This also looked a bit hard and I don't really trust Paypal, especially if out of the blue I started getting $50 "donations". Working in marketing I've head of paypal closing down accounts and holding money for ransom while you have to consistently prove you are who you are and jump through their hoops.
As I was already an "active" member of the wider Fediverse, I looked up how these services managed to accept donations and eventually stumbled upon Opencollective. This was exactly what I wanted. A open and inviting platform to handle all the money side of things while providing a truly transparent account of what is happening.

Opencollective solved all of the things I could hope for while also solving a secondary issue. Taxes. Something I have no idea about when it comes to donations, but will surely need to be up to speed on it within the next month!
After reading a few documents, you need to pick a Fiscal Host on opencollective, or prove you are a "business?" (or something like that, I don't remember) so you can become a fiscal host and accept money. This looked to be extra work on top of getting Reddthat up and running so I went looking for a local Australian fiscal host, and found one who also had 0% fees! They would accept all the donations and then transfer it back to me as a reimbursement, or I could even get them to pay invoices! This ticked all my boxes. Even if the host evidently ended charging a small 1-2% I would have been happy with that as our relationship after these last two years have been great. We even donated back to them for a few months in the early days.

~~Fast forward to today. We now have 2 months I have paid for without being reimbursed by our host. For a total of A$285.49. That would have left A$518.40 in our account for our future months~~

Update: We have managed to get all of our donations back!

Unfortunately since the payment in April I haven't heard anything from them and so we are saying Goodbye to OpenCollective and Hello to Liberapay! I'll hold out hope that they will come back online, or they've gone away for a huge holiday without internet, but this is a lesson we can all learn from.
I would like to continue using OpenCollective but the thought of losing more money the same way through another fiscal host would be too much. And the alternative of having to setup more paperwork to become our own fiscal host, and getting hit with a transaction fee for taking your donations and then "paying" myself just doesn't seem worth it. So Liberapay it is.

I've setup Liberapay and I've setup Stripe. Given them a fair amount of my details ( 😥 ). So you should be able to donate directly to me and we'll never have any of these unfortunate issues relating to other entities holding our money.

I feel like I've let you all down with ~$500+ disappearing into the ether. We can only learn from our mistakes and moving to Liberapay, Ko-Fi, or Crypto, where I hold the funds is the best way forward to ensure stability.

I'd like to remind everyone that donations are completely optional but they certainly help with it comes to a "big" instance like ours. Lemmy is growing every year and it's great to see everyone and all the things they do and the communities we have created!

And now back to the regular update.

June 2025

We've rolled out v0.9.12 for Lemmy which contained a few bugfixes as well as Peertube federation support, so now you should be able to see more peertube videos i that is your thing.

We turned 2 this month. The 6th of June was our birthday and it was an absolute banger! It was so big we all blacked out at the party and forgot it was happened! As June is a busy month for me, I made sure to renew our domain to make sure nothing happened. ;)

Since last update Lemmy World has successfully turned on the Parallel Sending which we (Reddthat) instigated way back when we started lagging behind and we all found out that sending internet traffic from one side of the world to the other sequentially might be bad. This means we no longer have need for our proxy system and it has saved us 4Euro/month (and the extra management overhead).
This also has had the same affect on our sister instances in AU/NZ allowing them to not lag behind too.

I hope you all had a great year and here's to another amazing one!

Cheers,

Tiff (& The Reddthat Admin Team)

PS. I believe I have managed to cancel all recurring donations. So if you wish to keep donating please do come over to Liberapay. <3

PPS. Did I mention that I've enabled Secret Donations for Liberapay, so I won't be able to know who you are!

April is here!

So much has happened since the last update, we've migrated to a new server, we've failed to update to a new lemmy version, automated our rollouts, fought with OVH about contracts. It's been a lot.

Strap in for story time about the upgrade, or skip till you see the break for the next section.

So good news is that we are successfully on v0.19.11.

The bad news is that we had an extended downtime.

Recently I had some extra time to completely automate the rollout process so Reddthat didn't rely solely on me being on 1 specific computer which had all the variables that was needed for a deployment.
As some people know I co-manage the lemmy-ansible repository. So it wasn't that hard to end up automating the automation. Now when a new Version is announced, I update a file/files, it performs some checks to make sure everything is okay, and i approve and roll it out. Normally we are back online within 30 seconds as the lemmy "backend" containers do checks on start to make sure everything is fine and we are good to go. Unfortunately it never came back up.

So I reverted the change thinking something was wrong with the containers and the rollout proceeded to happen again. Still not up :'( Not having my morning coffee and being a little groggy after just waking up.

Digging into it our database was in a deadlock. Two connections were attempting to do the same but different which resulted in it being locked up and not processing any queries.

Just like Lemmy World, when you are "scaling" sometimes bad things can happen. re: https://reddthat.com/post/37908617.

We had the same problem. When rolling out the update two containers ended up starting at the same time and both tried to do the migrations instead of realising one was already doing them.

After quickly tearing it all down. We started the process of only having 1 container to perform the migration and then once that had finished starting everything else we were back online.

Going forward we'll probably have to have a brief downtime for every version to ensure we don't get stuck like this again. But we are back-up and everything's working.

Now for the scheduled programming.

OVH

OVH scammed me out of the Tax on our server renewal last month. When our previously 12 month contract was coming to the end we re-evaluated our finances and were found wanting. So we ended up scaling down to a more cost-effective server and ended up being able to pay in AUD instead of USD which will allow us to stay at a single known price and not fluctuate month to month.
Unfortunately I couldn't cancel the contract. The OVH system would not let me click terminate. No matter what I did, what buttons I pressed, or how many times I spun my chair around it wouldn't let me cancel. I didn't want to get billed for another month when we were already paying for the new server. So a week before the contract ended I sent a support ticket to OVH. You can guess how that went. The first 2 responses I got from them after 4 days was "use the terminate feature". They didn't even LOOK at the screenshots clearly outlining the steps I had taken and the generic error... So I get billed for another month... and then have to threaten them with legal proceedings. They then reversed the charge. Except for the Tax. So I had to pay 10% of the fee to cancel our service. Really unhappy with OVH after this ordeal.

Automated rollouts

I spent some time after our migration ensuring that we have another system setup which will be able to rollout updates. So we are not dependant on just me and my one random computer :P All was going very well until an upgrade with database migrations happened. We'll be working on that soon to make sure we don't have unforeseen downtime.

Final Forms

Now that the dust has settled and we've performed the migrations starting next month I'll probably go back to our quarterly updates unless something insane happens. (IE: Lemmy drops v1 👀 )

We also modified our "Reddthat Community and Support" community to be a Local Only community. The original idea for the community was to have a place where only reddthat could chat, but back when we started out that wasn't a thing! So now if you want to voice your opinion to other Reddthat users please feel free too knowing other instances won't come in and derail the conversation.

As a reminder we have many ways to donate if you are able and feel like it! A recurring donation of $5 is worth more to me than a once of $60 donation. Only because it allows me to forecast the year and work out when we need to do donation drives or relax knowing everything in it's current state will be fine.

• Open Collective (preferred)
• Kofi
• Litecoin Direct
• Monero Direct (preferred)

Cheers,

Tiff

We just successfully upgraded to the latest Lemmy version, 0.9.10, probably the last before the v1 release.

This addresses some of the PM spam that everyone has been getting. Now when that user is banned and we remove content it also removes the PMS. So hopefully you won't see them anymore!

Over the next couple days will be planning for our migration to our new server as our current server's contract has ended. I expect the down time to last for about an hour, if not shorter. You'll be able to follow updates for the migration by our status page at https://status.reddthat.com/

Normally this update would be a week in advance and more nicely formatted that turns out the contract ends on the 25th and I don't want to get charged for another month at a higher rate when I just purchased the new server.

See you on the other side,

Tiff

EDIT:

22 Mar 2025 02:42: I'm going to start the migration in 5 mins (@ 3:00)

22 Mar 2025 03:01: that was the fastest migration I've ever done. pre-seeding the server and Infra as Code is amazing!

We've turned off our crypt donation p2pool (as no-one was using it), and two of our frontends, alexandrite and next (for the same reasons)

Time to celebrate with some highly accurate Australian content:

Hello Reddthat! We are back for another update on how we are tracking. It's been a while eh? Probably because it was such smooth sailing!

In the middle of February we updated Lemmy to v0.19.9 which contained some fixes for federating between Mastodon and Lemmy so hopefully we will see less spam and more interaction from the larger mastodon community. While that in of itself is a nice fix, the best fix is the recent thumbnail fix! Thumbnails now have extra logic around generating them and now have a higher chance of actually being created! Let us know if you think there has been a change over the past month-ish.

Budget & Upcoming Migration

Reddthat has been lucky to have such a great community that has helped us stay online for over a year and if you can believe it, in just a few more months it will be 2 years, if we can make it.

Our costs have slowly increased over the years as you can all see by our transactions on OpenCollective (https://opencollective.com/reddthat). We've managed to reduce some costs in our S3 hosting after it balooned out and bring it down to a more manageable level. Unfortunately as well, the current economic issues have resulted in the Australian dollar slipping further and as we pay everything in USD or EUR it has resulted in slightly higher costs on a month-to-month basis..

Our best opportunity to keep online for the foreseeable future is to downsize our big server from a 32GB ram instance to a 16GB ram instance which will still provide enough memory that we will be able to function as we currently do while not affecting us in a meaningful way.

This means we'll need to reassess if running all our different front ends are useful, or do we only choose a few? Currently I am looking to turn off next and alexandrite. If you are a regular user of these frontends and prefer them please let me know as from our logs these are the least used while also take up the most resources. (Next still has bugs regarding caching every single image).

We can get a vps for about ~A$60-70 per month which will allow us to still be as fast as we are now while saving 40% off our monthly costs. This will bring us to nearly 90% funded by the community. We'll still be slowly "losing" money from our open collective backlog but we'll have at least another 6 months under our belt, if not 12 months! (S3 costs and other currency conversion not withstanding).

All of this will happen in late March early April as we will need to make sure we do it before the current contract is up so we don't get billed for the next month. Probably the 29th/30th of I don't fall asleep too early on those days.
It'll probably take around 45mins to 60mins but if I get unlucky maybe 2 hours.

Age Restriction

Effective immediately everyone on Reddthat needs to be 18 years old and futher interaction on the platform confirms you are over the age of 18 and agree with these terms.

If you are under the age of 18 you will need to delete your account under Settings

This has also been outlined in our signup form that has been updated around the start of February.

Australian & UK Policy Changes

It seems the UK has also created their own Online Safety Act that makes it nearly impossible for any non-corporation to host a website with user generated content (USG). This is slightly different to the Australian version where it specifically targets Social Media websites.

Help?

I would also like your help!
To keep Reddthat online, and to help comply with these laws, if you see content or user accounts which are under the age of 18 please report the account/post/content citing that the user might be under the age of 18.
We will then investigate and take action if required.

Thanks everyone

As always keep being awesome and having constructive conversations!

Cheers,

Tiff!

PS. Like what we are doing? Keep us online for another year by donating via our OpenCollective or other methods (crypto, etc) via here

Let's bring in the new year with some maintenance.

We are going to update Lemmy to the latest version 0.19.8

To perform this there will be a downtime of 30 to 45 minutes while our database updates.

Unless everything goes wrong we'll be up within the hour.

See you on the other side.

Tiff

Edit 1:
We are all done!

Edit 2:
Frontends didn't start correctly and should be sorted now

Hello Reddthat-thians!

As always here is our semi-whenever there is news update. I would, as always like to thank you all for being here and for the kind support we received last time I made an update.

We hit a few couple of milestones this last quarter:

• Our first BTC transaction was received! Thank you anonymous!
• Lemmy released two versions (which we have yet to update too, more below)
• I only restarted the services once the whole quarter as I thought we were down/stuck.

Lemmy 0.19.7

The latest update brings some fixes to Lemmy as well as ~~new features such as Private/Invite only communities. I can't wait to see what this does to help people find safe spaces and to self regulate.~~ Edit: that's in 0.20.0. The only new features (worth talking about imo) are parallel sending and allowing people to have 1000 characters in their bio.
This update is not live yet as the update requires a 30-60 minute database update that I want to test on a backup to make sure we can safely update. Other big instances have already done the update and most things went successfully so I'm confident that we can update without much fuss but I'd like to actually enjoy my holidays rather than spending 4 hours debugging db migrations! So, testing first.

Users

Something I think we struggle with is our proposition. As we have an opinionated view on what our community should be which doesn't always resonate when trying to sign up. With "only" 100 users active daily compared to other servers with over 1k it puts into perspective that not everyone agrees with our policies, but there are certainly people who do!

As such I'm open to ideas on growing our userbase or ideas on rephrasing our signup page, and sidebar.

Australian Social Media Laws

Here is the biggest news. Last week the Australian Government passed a new law that requires Social Media to no longer be accessible for kids under the age of 16. As such we will need to design a system & possibly modify Lemmy to be compliant with these laws.

I will not debate whether this is good or bad, because at the end of the day, we only have a few options: comply with these laws, don't comply and turn off reddthat, sell / handoff to another admin.
This has been a fun side project for me and I want to continue it. As such I'm looking into ways to comply with the local laws to ensure that Australian minors are blocked.

For Reddthat, this law requires social media to take "reasonable" steps to ensure children under the age of 16 are not be given an account. They do not define what reasonable means and leave it up to the social media platforms to define it. Really helpful for those indie social media platforms...

Unfortunately, like myself, I was thinking that we might technically not be defined as a social media platform thus skirting around the rule. The law defines a social media platform as any site whose primary or significant purpose is to connect 2 or more users, allows end-users to link to or interact with other end-users, and that allows end-users to post content.
So Lemmy, Mastodon, Blusky et al will be affected.

Researching it more leads to the idea which puts the onus on social media companies to continually verify accounts, so they take a proactive step to verify accounts that are believed to be accessed by minors under 16. Thus you need to prove you are above 16 years old. The cherry on top? You are not allowed to use a government issued ID to do it.
Thinking back to when you were 16 years old, what forms of identification did you have? A driver's licence? Maybe a passport? A school ID card? The last one (school ID) would be the only valid id you could use to verify yourself that isn't government issued in this instance. And good luck if you don't go to school anymore and started in the workforce when your 16.

Needless to say this is going to be a recurring theme over the next year and I will keep you all informed about it with our updates.

Future Features

A general outline of what I'm hoping to achieve is more controls in Lemmy to:

• have accounts in a 'monitored' state
• have a way to customise approval processes
• have accounts use a trust rating based on age, post/comment numbers, etc.
  • this would help with spam as well (obviously this would need a lot of factors)
• have automod built in (or actually setup correctly for us)

Obviously these are things that will need Lemmy development to help facilitate and I'll be creating a few issues over the weeks once I've fleshed out what a solution might look like.
If you have ideas please share! let's start the conversation on what the processes would look like to help solve our issues.

Financials

Thanks to the big donation drive from last time I posted we are still looking healthy enough in the financials to last until May next year! Which is good as our server plan renews in April. At that time we will be downgrading to a smaller instance as we may have over exaggerated when purchasing last April. We'll obviously know more closer to April next year about our financials and what we can afford to ensure we are cost neutral (if possible!).

Our LemmyWorld proxy also is an extra cost that we never budgeted for and has been ticking away successfully. Maybe by April LW will have upgraded to >0.19.5 which will give us parallel sending allowing us to remove the proxy.

Our object storage is humming along but costs are creaping up. I'll be doing an audit and possible look into doing a cost analysis to see if other object storage solutions would be cheaper. But for less than $20/m it's probably not even worth my time...

As a reminder we have many ways to donate if you are able. A recurring donation of $5 is worth more to me than a once of $60 donation. Only because it allows me to forecast the year and work out when we need to do donation drives or relax knowing everything in it's current state will be fine.

• Open Collective (preferred)
• Kofi
• Litecoin Direct
• Monero Direct (preferred)

Note: while you still can transfer me Bitcoin, I have removed it as an option because of the current transaction fees. Monero or Litecoin offer transaction fees as low as $0.005 so they are the preferred options compared to the $5 transaction fee of Bitcoin.

Conclusion

You are awesome. Posting, Commenting, and interacting with communities on Reddthat or through Reddthat makes it enjoyable every time to write these updates. So keep being awesome, even if you are a lurker!

If I were not hosting in Australia I would still be required to conform to laws if I were to allow Australian people to use Reddthat. Blocking all of Aus would be an option. But we need to ask ourselves does that fall under our values or ideals? (No obviously). Neither does requiring people to verify they are over a certain age. But we'll see what's to become.
This is the same as how CPPA or GDPR are still enforcable while we are all the way on the other side of the world.
This doesn't require a knee jerk reaction but requires serious thinking, whiteboarding and well thought out communication.

As always,
Cheers,

Tiff

PS Happy Holidays!

We had a brief outage today due to the server running out of space.

I have been tracking our usage but associated it with extra logging and the extra build caches/etc that we've being doing.

Turns out the problem was the frontend Next-UI which has been caching every image since the container was created! All 75GB of cached data!

Once diagnosed it was a simple solution to fix. I'm yet to notify the project of this error/oversight and I'll edit this once Issues/PRs are created.
I also haven't looked at turning the caching off yet as my priority was recovering the main Reddthat service.

Thanks all for being here!

Tiff

So for those of you who were refreshing the page and looking at our wonderful maintenance page it took way longer than we planned! A full write up I'll do after I've dealt with a couple time out issues.

Here is a bonus meme.

So? How'd it go...

Exactly how we wanted it to go... except with a HUGE timeframe.
As part of the initial testing with object storage I tested using a backup of our files. I validated that the files were synced, and that our image service could retrieve them while on the object store.

What I did not account for was the latency to backblaze from Australia, how our image service handled migrations, and the response times from backblaze.

• au-east -> us-west is about 150 to 160ms.
• the image service was single threaded
• response times to adding files are around 700ms to 1500ms (inclusive of latency)

We had 43000 files totaling ~15GB of data relating to images. If each response time is 1.5 seconds per image, and we are only operating on one image at a time, yep, that is a best case scenario of 43000 seconds or just under 12 of transfer time at an average of 1s per image.

The total migration took around 19 hours as seen by our pretty transfer graph:

So, not good, but we are okay now?

That was the final migration we will need to do for the foreseeable future. We have enough storage to last over 1 year of current database growth, with the option to purchase more storage on a yearly basis.
I would really like to purchase a dedicated server before that happens and if we continue having more and more amazing people join our monthly donations on our Reddthat open collective, I believe that can happen.

Closing thoughts

I would like to take this opportunity to apologise for this miscalculation of downtime as well as not fully understanding the operational requirements on our usage of object storage.
I may have also been quite vocal on the Lemmy Admin matrix channel regarding the lack of a multi-threaded option for our image service. I hope my sleep deprived ramblings were coherent enough to not rub anyone the wrong way.
A big final thank you to everyone who is still here, posting, commenting and enjoying our little community. Seeing our community thrive gives me great hope for our future.

As always. Cheers,
Tiff

Edit:

Instability and occasional timeouts

There seems to be a memory leak with Lemmy v0.18 and v0.18.1 which some other admins have reported as well and has since been plaguing us. Our server would be completely running fine, and then BAM, we'd be using more memory than available and Lemmy would restart. These would have lasted about 5-15 seconds, and if you saw it would have meant super long page loads, or your mobile client saying "network error".

Temporary Solution: Buy more RAM.
We now have double the amount of memory courtesy of our open collective contributors, and our friendly VPS host.

In the time I have been making this edit I have already seen it survive a memory spike, without crashing. So I'd count that as a win!

Picture Issues

This leaves us with the picture issues. It seems the picture migration had an error. A few of the pictures never made it across or the internal database was corrupted! Unfortunately there is no going back and the images... were lost or in limbo.

If you see something like below make sure you let the community/user know:

Also if you have uploaded a profile picture or background you can check to make sure it is still there! <3 Tiff

Hello Reddthat! It is I, the person at the top of your feed.

First off. Welcome all new users! Thank you for signing up and joining our Reddthat community.

Bot Defence

Starting yesterday after going through 90+ registration applications, I couldn't do it anymore. I felt compelled to give people a great experience with instantly getting let in and kept my phone on me for over 24 hours, checking every notification to see if that was another registration application I needed to quickly click accept on.

I want to quickly say thank you to the people who obviously read all the information and for those that didn't I'm keeping a close eye on you... 😛
I found a better solution to our signup problems.

As we use cloudflare for our CDN I have turned on their security system for the signup page. ~~Now when anyone goes to the signup page, they will be given a challenge that needs to be solved.
That means any bots that cannot pass cloudflare's automated challenge cannot signup.
A win until we get our captcha back working. ~~ Well I did not check the signup process correctly. It doesn't act as I thought it would, so I'll disable it after the migration.

Downtime / Migration to object storage

Today in the fediverse we have successfully confirmed that object storage will be an acceptable path forward but will not operate as initially hoped.
I initially hoped to offload everything via our CDN, but the data still needs to go through our app server. The silver lining is that we can still cache it heavily on our CDN to ensure that the pictures will be served fast as possible for you.

So it may be slightly pricier than we initially planned for when moving to object storage, but in the end we still benefit, functionally and monetarily. The reason is we were not going to be billed for egress (fetching/displaying images), where as now we will be. The fees are very low and still should be covered by our wonderful monthly donators.

We will have about 15-20GB of storage that needs to be moved and unfortunately our image service is incapable of running at the same time the migration is done, which means we need to turn it off while the migration happens. To top it all off we have... 43000+ (and counting) small image files. If you haven't worked with large swarms of small images before, the one that I can tell you is that transferring small images, sucks.

So we can do two things:

1. Turn off everything

• Dedicate all CPU and bandwidth to the migration
• Ensuring continuity and reducing the risk of something going wrong

2. Turn off the picture service

We can run Reddthat without the picture service & uploads while we perform the migration, but the migration will have an impact in server performance.

• This will amount to having any picture we host (that isn't cached) return a 404.
• Any uploads will timeout during that period, and return an error popup.
• Pages will be slightly slower to respond.
• Something else might break 🤷

Because of the risks associated with running only half our services, I've decided to continue with our planned downtime and go with option 1, turning off everything while we perform the migration.

Date: 28th June (UTC)

• Start Time: 0:05
• End Time: 6:00 (Expected)

It will probably take the whole 6 hours. In our testing, it did 150 items in 10 minutes... I will put up a maintenance page and will keep you all updated during that time frame especially if it is going to take longer, but unfortunately it will take however long it takes.

This will be the last announcement until we do the migration.
Cheers,
Tiff

PS. Like what we are doing? Become a contributor on our Open Collective to help finance these changes!

The following rules governs reddthat.com and all communities hosted on reddthat

Ground Rules

We are a NSFW enabled instance, which means there may be NSFW content in the Local and All categories if you leave NSFW enabled in your account. If you do not wish to view NSFW content you are still welcome here but make sure you toggle off Show NSFW content under your Settings

Another option is to select "Block Community" from the sidebar, when on the specific community. Once you have done that you will never see it again!

Our Instance Rules

• General Code of Conduct
• Any NSFW post must be tagged as NSFW. Failure to do so will be given one warning only
  • Anything that you wouldn't want your boss or coworkers to see, needs to be tagged NSFW
  • NSFW also acts as "Content Warning" outside of the specific NSFW communities
• Remember the human! (no harassment, threats, etc.)
• No racism or other discrimination
• No endorsement of hate speech
• No self-advertisements or spam
• No link-spamming
  • Recommended 5 links per community per day unless noted in Community Rules
• No content against Australian Law

Any posts or comments that are in breach of these rules will be dealt with, and remediation will occur. Whether that be a warning, temporary ban, or permanent ban.

(TLDR) The crux of it boils down to:

• Remember that we are all humans
• Don’t be overtly aggressive towards anyone
• Try and share ideas, thoughts and criticisms in a constructive way
• Tag any NSFW posts as such

Moderation

We agree with the Code of Conduct for Moderation and will due our best to uphold it.

Lemmy provides the Moderation Log for everyone to see as well. This is a federated log where actions taken by other moderators and other instance admins are shown. As such the posts that exist in the log come with a content warning

You can see the modlog in the bottom of the sidebar. Here is the link if you wish to see it.

References

As both links above reference a domain outside of our control (join-lemmy). Here is the git commit and the wayback link:

• this git commit.
• web.archive.org link

Feedback

If you take issue with these rules, or think a rule should be modified, please add a comment on this post with answers to the following as well as any extra information you can think of:

• What rule would you like to change?
• Why you would like to change it?
• What are the benefits to this rule change?
• Have there been any issues and can you reference them?

Thank you for your continued support.

• Tiff

Transparency log:

2023-08-19 - Changed Maximum Links posted in community from 2 to 5 per day.
2023-06-13 - Added "Ground Rules" section, Added Feedback section, `Instance Rules` -> `Our Instance Rules`.  
2023-06-11 - Post Creation