sunaurus

joined 2 years ago
[–] sunaurus@lemm.ee 3 points 9 months ago (5 children)

Hey, for the past few weeks, we have been tuning different rate limits and other bot prevention mechanisms on lemm.ee. We have had a huge increase in traffic lately, as well as a couple of instances of what was effectively a DDoS.

In other words, the problems likely have nothing to do with your app or different versions, and everything to do with our own measures.

For the mark post as read endpoint, I recently made the rate limit a bit more relaxed, so hopefully users won't see issues there anymore. OTOH, we are also presenting a Cloudflare challenge for some IP ranges currently, which may inadvertently affect legit users. This was necessary to mitigate a recent DDoS, but we will continue tuning and hopefully restore things to normal for legit users soon.

[–] sunaurus@lemm.ee 17 points 9 months ago (1 children)

Definitely not 😄

[–] sunaurus@lemm.ee 5 points 9 months ago

We have automated a lot of this stuff on lemm.ee as well

[–] sunaurus@lemm.ee 12 points 9 months ago* (last edited 9 months ago) (1 children)

I'm afraid .world users will also miss out on your mod actions in all of your communities, which is a particularly unfortunate side-effect.

I don't think it's the end of the world (:P), though - .world is a big instance, but there are still tons of users on other instances. I mean, even in your communities, I don't think .world is not making up the majority of activity.

[–] sunaurus@lemm.ee 23 points 9 months ago

I'm not sure why .world has the 18 age requirement - AFAIK GDPR only requires 16+ if you don't specifically ask for parental consent. Of course, there is the matter of pornography etc, but for example Reddit allows 13+ users, and all they do for pornography is show an NSFW warning, which Lemmy also has (although this is a good point - maybe the NSFW toggle should be improved to explicitly ask users to confirm their age on Lemmy as well, similarly to how it works on Reddit).

But at the end of the day, each instance is free to create whatever rules and processes they want, and to ban people according to those rules. I would say that .world admins are probably just trying to do their best in enforcing their rules, and unfortunately that means that most likely you'll be cut off from .world for the next 5 months 🫤

[–] sunaurus@lemm.ee 13 points 9 months ago (7 children)

In my experience, it's mostly a combination of confusing Lemmy UI + user error:

  1. When using the e-mail verification link, Lemmy just shows a blank page with a tiny "e-mail verified" notification in the bottom left corner, which people miss. They then assume that it didn't work, because all they see is a blank page.
  2. When opening the e-mail verification link multiple times, only the first time registers, and further attempts will generate confusing errors. This also makes people assume it didn't work, but in reality, their e-mail got verified successfully.
  3. On lemm.ee, it's very common for people to sign up with typos in their e-mail addresses (happens few times every week recently).
  4. We have a custom question on our sign-up page asking people to state they agree to our rules, and it's relatively common that people just don't read the question and write something random in there - we generally don't accept such applications to try and weed out bots, but I'm pretty sure we also end up rejecting a bunch of legit users this way who just didn't read the instructions.

Generally I'm happy to provide support to people in such cases if they reach out to me directly, but at the end of the day, the easiest and fastest solution if the account is truly stuck in limbo is just to create another account.

It's a shame that there is so much potential friction on sign-up, but at the same time, I don't see us reducing the friction on lemm.ee any time soon, because this is the lesser evil compared to bot sign-ups etc that we have seen in the past.

[–] sunaurus@lemm.ee 3 points 9 months ago (1 children)

I’ve just restored it

[–] sunaurus@lemm.ee 8 points 10 months ago (1 children)

Hey, there isn’t any default community right now. There are a few different databases that track graphs of such things, for example, fedidb.org.

[–] sunaurus@lemm.ee 40 points 10 months ago

Thank you very much for the support!

Our infrastructure costs are currently quite stable at around 200€ per month, and considering that the instance is right now quite decently supporting nearly 6000 monthly active users, you could say that you are indeed relatively contributing a ton - you are effectively covering server costs for 60 people!

The fact that it’s a monthly amount is particularly great, because with recurring income, we will get advanced warning if there is danger of funds starting to run low.

[–] sunaurus@lemm.ee 28 points 10 months ago* (last edited 10 months ago)

We had a few really huge days in 2023, but other than those, it seems like the growth so far in March is definitely outpacing our initial wave of new users in 2023.

[–] sunaurus@lemm.ee 4 points 10 months ago (3 children)

Hey, could you make a post in that community?

[–] sunaurus@lemm.ee 2 points 10 months ago

The e-mail successfully went out from our side - please check your spam filters etc

 

Hey all!

Upcoming lemm.ee cakeday

Can you believe that lemm.ee is almost 1 year old? In just a couple of weeks (specifically, on the 9th of June), we will be able to celebrate our first instance cakeday.

I am thinking of compiling some stats about how lemm.ee has been used in its first year, if you have any specific stats in particular you would like to see, feel free to comment below. I will try to accommodate any ideas as I start gathering this info!

Infrastructure updates

A few weeks ago, I posted about plans to make some changes to our infrastructure in order to deal with different intermittent networking issues.. It took a bit longer than I hoped (just did not manage to get enough free time between then and now), but I am happy to report that this work has now been completed! Additionally, I have decommissioned our stand-alone pict-rs server.

With the two changes mentioned above, I believe lemm.ee should now be much more resilient going forwad, and I expect a significantly lower rate of infrastructure-related issues for the rest of the year!

I'll leave a tehcnical overview about the problem & solution below for those interested, but if these details don't interest you, then you can safely skip the rest of this post.


For context, lemm.ee has been hosted on Hetzner servers for most of this year (having migrated from DigitalOcean initially), with everything except our database being hosted on the Hetzner Cloud side, and the database itself living on a powerful dedicated Hetzner server. This mix allows a great amount of flexibility for redeploying and horizontally scaling our application servers, while still allowing a really cost-effective way of hosting a quite resource-hungry database.

In order to facilitate networking between the cloud servers and the dedicated database server (which live in different networks), Hetzner provides a service named "vSwitch". This service basically allows you to connect different servers together in a private network. Unfortunately, I discovered quite quickly that this service is very unreliable. During the short few months that we have been using the vSwitch, we have gone through one extended period of downtime (where the service was just completely broken for several hours), as well as dozens (if not hundreds at this point) intermittent disconnects, where servers randomly lose their connections over the vSwitch. After such a disconnect, the connection never recovers without manual intervetion.

For most lemm.ee users, the majority of these vSwitch issues have been mostly invisible, as we have redundancy in our servers - if one server loses its connection to the database, other servers will take over the load. Additionally, I have generally been able to respond quite quickly to issues by redeploying the broken servers (or deploying other temporary workarounds). However, in addition to a huge amount of these issues which lemm.ee users hopefully haven't ever noticed, there have also been a few short periods of downtime this year so far, as well as a few cases of federation delays. These more extreme cases were generally caused by multiple servers losing their vSwitch connections at the same time.

After several attempts to work around these issues, I decided that we need to migrate away from vSwitch.

As of earlier today, lemm.ee is no longer using Hetzner's vSwitch at all!

I finally found enough time earlier today to focus on this migration, and I was able to successfully complete it. None of our networking is relying on the vSwitch anymore.

In the end, I went with quite a simple solution - I configured a host-level firewall (nftables) on our database dedicated server, which will deny all connections by default. Whenever any cloud servers are added/removed, their corresponding public IP addresses are added/removed in the allowlist of our database firewall. It would have been ideal to do this whole logic in Hetzner's own firewall, but that one unfortunately has a limit of only 10 rules per server, which is just not enough for our setup.

Bonus: our pict-rs server has been decommissioned!

Pict-rs is the software which Lemmy uses for everything related to media (image storage mostly). Initially, pict-rs required a local filesystem to store both files as well as metadata about files. Since the beginning, lemm.ee has used a dedicated server just for pict-rs, in order to ensure we could easily redeploy the rest of our servers without losing any images.

Over the past year, pict-rs has gained the ability to store files in object storage, and metadata in a PostgreSQL database. This meant that the server running pict-rs itself no longer contained any of the important data, so it became possible to redeploy without losing any images. Additionally, this meant that it would be possible to run multiple pict-rs servers in parallel.

While we had already migrated our pict-rs server to use object storage and PostgreSQL several months ago, we still had the single dedicated pict-rs server up until today. I have been planning for a while to decommission this server, and start running pict-rs directly on each one of our Lemmy application servers. Earlier today, I was able to complete this plan. This should hopefully mean that our pict-rs server is less likely to get overloaded, and it also means a tiny reduction in our overall monthly infrastructure bill (due to one less server running).

With the above changes, I think our infrastructure has become more robust, and hopefully, we will experience less issues with images, federation, and general downtime going forward.


That's all from me for now. Feel free to leave any thoughts or questions in the comments, and as always, I hope you're having a great day!

 

Hey folks!

This is a quick notice about a change to our moderation policy.

We have had a policy on lemm.ee for administration and federation nearly since the very beginning. This policy has also always included a section about moderator responsibilities. Today, we have made two changes to this policy:

  1. The policy has been renamed to Policy for administration, moderation, federation - this is to make it clear that the policy is also relevant for mods
  2. We have introduced a new responsibility for moderators, they must "Ensure that they only provide accurate and clear reasons for mod actions".

The reason for the addition is that mod log actions federate out to other instances, and are more or less permanent (due to how Lemmy and federation works right now). This means that users do not really currently have any easy way to clarify or defend themselves against inaccurate accusations in the mod log.

As always, I am very grateful to all mods for your efforts in building awesome communities on lemm.ee. I hope you can understand why this new policy is necessary - I do not want to make your lives more difficult, the goal is to just try and reduce any mod log related misunderstandings in the future.

Thank you for reading and have a nice day!

 

Hey folks

This is just a quick heads up that I need to perform some maintenance & upgrades on our database server, which unfortunately will require downtime. I don't expect the downtime to last for longer than 2-3 minutes, but just wanted to give a heads up first so you know not to be concerned.

That's all, hope you have a great week!

Edit: maintenance complete!

1
submitted 2 years ago* (last edited 2 years ago) by sunaurus@lemm.ee to c/test@lemm.ee
 

Hello, world!

Edit: first test edit!

 

Hello, friends!

TL;DR: I am working on a new Lemmy frontend in nextJS. There is still much work to be done, but you can already have an early look at https://next.lemm.ee

First of all, quick note to lemm.ee users: I am making this announcement post in !meta@lemm.ee, as this is also a notice that I will be hosting an alternative frontend (lemmy-ui-next) for the first time on lemm.ee. Going forward, I will post updates about lemmy-ui-next in a separate dedicated community: !lemmy_ui_next@lemm.ee. If you're interested in future updates, please subscribe there!

What is lemmy-ui-next?

Lemmy is generally accessed through some kind of frontend UI. By default, Lemmy provides its own web interface (lemmy-ui), which you can find on the front page of most Lemmy instances (including lemm.ee). There are also several other independent frontends, for both the web and different mobile platforms, which I'm sure many of you are familiar with.

Lemmy-ui-next is a brand new alternative frontend, built from the ground up with modern and popular tooling - a framework known as NextJS. Lemmy-ui-next has (or aims to have) the following high-level features:

  • Open source (AGPL)
  • Drop-in replacement for lemmy-ui - same exact URL structure, so all existing links will continue working
  • Very plain & minimalistic UI, strongly inspired by other link aggregator sites (of course including the original lemmy-ui!)
  • Very basic and "typical" NextJS architecture, to encourage open source contributions
  • Fully functional even when JavaScript is disabled (but works better with JS enabled!)
  • Optimized data transfer between your browser and the server (filtering out only relevant data from the Lemmy API, caching, memoization)
  • Strong focus on privacy and security (all authentication with the Lemmy API is done through secure httpOnly cookies, user IP addresses are not leaked to external image hosts, etc)

What is the current status of lemmy-ui-next?

I have mentally split the initial work I want to complete into 3 milestones:

  1. Lurk - All read-only features of Lemmy
  2. Participate - Voting/posting/commenting/DMs/reports, etc
  3. Moderate - Handling reports, creating & managing communities, etc

I am now nearing completion of the first milestone. It's not 100% there yet, but you can already log in, browse, subscribe to communities and even vote. Some things may still look a bit wonky, and some features are still missing, but the core experience is getting there.

In terms of code contributions, I would ask anybody who is interested in getting involved to contact me first before working on anything. I am not looking for PRs just yet - the code structure is still a bit loose, and I am redefining it as I add more stuff. I would ideally really like to complete the first 3 milestones before opening things up for external contributors.

Who can use lemmy-ui-next?

At the moment, it is only hosted on this instance, at https://next.lemm.ee. I do not yet have any formal instructions for running it on other instances, but generally speaking, it is a simple NextJS app - to deploy it, you just need to do: npm install, npm run build and LEMMY_BACKEND=https://<your lemmy api here> npm run start.

Why not just improve lemmy-ui instead?

Lemmy-ui is an extremely important and valuable project. There has been a significant amount of hard effort put into it so far, and nobody can refute that it is the frontend which has really carried Lemmy to this point.

Unfortunately, there are some architectural problems with lemmy-ui (mostly related to how data is fetched and how sessions are stored in memory), all of which would require quite a significant rewrite to fix. Additionally, I think that the core technical solution used for lemmy-ui is just a bit too obscure, which has been an obstacle to my own contributions, as well as to contributions by others. If a rewrite is on the table anyway, then I believe a different technology is the best way forward.

Why not work on lemmy-ui-leptos instead?

Lemmy-ui-leptos is another rewrite of lemmy-ui, which is being lead by Lemmy maintainers. It is based around a Rust web framework called Leptos. I think this is really cool tech, and will be happy to host lemmy-ui-leptos on lemm.ee in the future as well.

There are a two key reasons why I personally decided to start working on another alternative, though:

  • I have heard from several people on Lemmy that they feel like Leptos is a big barrier to entry in terms of them contributing
  • Even for myself personally, I am very comfortable (and think I can move very fast) when working on something like NextJS, but with Leptos, I think the learning curve would be quite big and I would get much less done with any time I invest into it

My hope is that by providing a very vanilla alternative, I can provide an outlet for potential open source contributors who would like to work on Lemmy, but aren't prepared to do it with Leptos.

Does this mean that lemm.ee will change in ways I don't like?

First, let me be clear: lemm.ee will always host the default Lemmy frontend. This means lemmy-ui for now, and most likely lemmy-ui-leptos in the future.

I am however considering the possibility of switching things around at some point in the future, so that lemmy-ui-next will be hosted directly on lemm.ee, and lemmy-ui will be accessible on a different subdomain (like ui.lemm.ee). This would only happen once I have completed all 3 milestones for lemmy-ui-next. The main reason I am considering this is that I feel like I will always be in the best position to offer technical support to users on the frontend which I am myself maintaining. If you have any thoughts about this potential change, please let me know in the comments below!

That's about it for now!

This is something I've been thinking of doing for a while now, and I'm very excited to finally get the ball rolling! If you have a chance, please feel free to check out what https://next.lemm.ee looks like so far, and please let me know if you have any thoughts or feedback!

 

Hello!

I am sunaurus, the head admin of lemm.ee. Ever since I created my instance, I have been following a lot of public and private discussion channels between different parties involved with Lemmy. As I’m sure many others have also noticed, the discussions in such channels sometimes get heated, and in fact recently, I feel like there has been a constant trend in these discussions towards a lot of demands, hostility, negativity, and a general lack of empathy between different participants in the Lemmy network.

I am writing this post for a few reasons:

  1. I would like add a bit of positivity by expressing my gratitude towards every single person who has helped improve Lemmy.
  2. I want to speak up in defense of different people who have been receiving negativity lately.
  3. There are a few false rumors spreading on Lemmy, which I would like to try and counteract with very simple evidence.
  4. I want to remind everybody that at the end of the day, all of us care about building and improving Lemmy. We all have the same goal, and it’s too easy to lose sight of that.

I will split up what I want to say in this post by different user groups - users, mods, admins and developers. I understand that many people belong to several (or even all) of these groups, but I just want to highlight the value of, and express my gratitude to each group separately.

Users

At the end of the day, Lemmy would not be worth anything without the users. Users bring Lemmy to life by posting great content, getting involved in discussions in comments, helping surface interesting content for others through voting and even keeping the platform clean through reports. I am extremely thankful for all the users who have given me so much enjoyment on this platform.

I believe that users often get treated unfairly on Lemmy based on what instance they are participating from. I’m sure so many of you have noticed comments around Lemmy along the lines of “Oh, another user from , I’m going to completely ignore your stupid takes”. I’ve also many cases of people treating users as second-class citizen if they are not on the same instance - for example, I’ve seen users who are active and valuable participants in communities on another instance receive comments like “why are you participating in our discussions, go back to your own instance”. In my opinion this is completely counterproductive to the whole idea of federation. On a human level, I can understand it - you’re far more likely to notice or remember what instance somebody is posting from if you have a negative experience. As a result, as time goes by, people tend to develop negative views of each instance, despite potentially having had many positive interactions with other users of those same instances. The message I want to put out here is that instances, especially bigger ones, are not monoliths - do not judge users based on what instance they are browsing Lemmy from, judge them by their actual words and actions.

Mods

There are some excellent communities already on Lemmy, and these communities are all continuously being built up and maintained by mods. Mods put in huge amounts of their free time and energy in order to provide spaces for all Lemmy users. They form the first line of defense against bad actors, they keep communities alive and often receive no praise, only criticism. I am very grateful to everybody who has dedicated time to building communities on Lemmy.

Users rarely notice the lengths mods go to in order to keep communities running smoothly - mods more often than not only get noticed when users disagree with some mod actions. I believe mods deserve a lot better than this. Constructive criticism can of course be useful to improve communities, but it must be balanced with empathy and kindness towards people who have been putting in effort to provide something for users. Remember that there is another human being reading your words when you start writing about the mods of any particular community. Users who are not happy with mods of a certain community always have the opportunity to start their own community and run it as they like.

Admins

Admins provide two main key functions for the network:

  1. Taking care of the actual infrastructure of Lemmy
  2. Working as a higher level defense against bad actors, in cases where mods are not enough

I can tell from my own experience that being an admin of a bigger instance requires constant energy and attention. I don’t believe that there is a single medium-to-big instance where the admins have not put in hundreds (if not thousands) of hours of their free time, as well as in many cases, probably their own money. This is a service which admins provide for free, and it is necessary in order to keep the Lemmy network healthy. I have endless respect for anybody who is willing to put themselves in the position of a Lemmy admin.

I have seen awful messages towards admins from all the other groups listed here, including other admins. These messages range from condescending and rude, to downright hateful. I have seen admins treated as useless and their work taken for granted. I have seen people getting frustrated with admins for not spending every waking minute on Lemmy. I have seen some users consistently spreading provably false rumors about particular admins in an effort to tarnish their reputation on Lemmy.

Before you take out frustration on admins, please remember that they are also humans who have been working tirelessly to improve Lemmy in their own way.

Also, a reminder: the absolute best feature of Lemmy is that users are free to pick their instance - and as a result, users are also free to pick their admins. Even more than that, users can always become their own admins by spinning up their own instance. Yes, this requires dedication, effort, and research, but that’s exactly my point. It’s not easy running an instance, and mistreating people who do this as a free service is completely unacceptable.

Developers

Lemmy development has been lead by a few key maintainers, with a massive amount of smaller contributors. The software is constantly being improved at a very good pace, and everybody is able to benefit from this effort at no cost whatsoever. I am extremely grateful to everybody who has participated in the development of the Lemmy software, and other related software, as without you folks, none of us would even be here now.

There seems to be a huge amount of people with very little appreciation of the work that has gone into the software. I’m sure many of you have seen countless messages where people express that the devs should be doing more in one way or another. “They should work faster”, “they should prioritize this obviously most important feature”, “they should be available 24/7 to offer support”, etc. I just want to take a moment here and acknowledge what core maintainers have already done for Lemmy:

  • Years worth of work on the code itself
  • Offering support to the community and other admins
  • Reviewing literally thousands of pull requests on GitHub
  • Acting fast in stressful situations where the Lemmy network has been overloaded
  • Not abandoning the project in the face of constant hateful users
  • Sacrificing literally hundreds of thousands of euros in missed salaries which they could have been getting if they were working for a tech company instead of working on Lemmy

I also want to take this moment to discredit some rumors which I have seen repeated too many times:

  1. Rumor: Lemmy devs do not accept outside code contributions

This is completely false - the maintainers are completely open to (and even constantly asking for) contributions. When somebody starts contributing, they will receive support and code reviews very quickly. I can tell you that I have experienced this myself several times, but that’s anecdotal, so let me also provide evidence:

a. Contributors list for the Lemmy backend: https://github.com/LemmyNet/lemmy/graphs/contributors

b. Contributors list for Lemmy UI: https://github.com/LemmyNet/lemmy-ui/graphs/contributors

Both of these lists include 100 different names, and that’s only because GitHub literally caps these pages to 100 users. Actually, the amount of different contributors is even bigger. If Lemmy devs did not accept and encourage outside contributions, then there would be no way for these lists to be so big.

  1. Rumor: Lemmy devs work too slowly

This is an extremely entitled and frankly stupid claim. I try to keep on top of the changes made in the Lemmy repo, and let me tell you, the pace of improvement is very good.

I very firmly believe that if the network started downgrading to Lemmy versions from ~8 months ago, the whole network would just collapse, as none of the instances could keep up with the current volume. That is to say, we have come an extremely long way since last summer alone.

Let me provide some more evidence. Take a look at the Pulse page for the Lemmy backend on GitHub: https://github.com/LemmyNet/lemmy/pulse. As of writing this, Lemmy devs have merged 18 pull requests in the week leading up to this post - that’s an average of 2.5 merged PRs per day. This is extremely good for a project with a small underfunded team.

  1. Rumor: Lemmy devs do not prioritize the important issues

There are two sides to this. First of all, there are endless users who turn to the Lemmy devs with what they believe is the most important issue and should immediately be prioritized - the problem is that almost none of these endless users have the same view of what the most important issue actually is! In that sense, it’s literally impossible to please everybody, because everybody wants different things.

On the other hand, even when Lemmy devs do prioritize things which some users have been desperately asking for, I have on several occasions seen a dismissive response along the lines of “too little too late”. Basically, the demands made are often unrealistic and impossible to meet.

If you are somebody who feels like Lemmy devs are not doing enough, I would ask you to please take a step back, look at the actual contributions which they have made, and consider how you yourself would feel if after making such a massive contribution, you would still need to listen to countless strangers on the internet tell you how you’re not good enough in their opinion.

Conclusion

Lastly, I am very thankful to anybody who took the time to read to the end of this post. Again, my goal is to try and defuse some of the hostility, as well as to put out a message of gratitude and positivity. I am very interested in the success of Lemmy as a whole, and that is much easier to achieve and maintain if we all work together. Thank you, I hope you're doing well, and have a nice weekend!

 

Hey folks

This is a heads up that I will be performing some maintenance and hardware upgrades on our database this Saturday.

We are currently experiencing several spikes throughout the day which cause our database to become overloaded - this results in degraded performance for many users. The spikes are happening due to a combination of continued growth of the database, some expensive periodic scheduled tasks which Lemmy runs, and fluctuating traffic patterns. Some of this can be optimized on the code level in the future, but it seems that the best way to deal with it right now is to add some additional resources to our database server.

I am intending to switch to slightly different hardware in this upgrade, and will be unable to make this switch without downtime, so unfortunately lemm.ee will be unavailable for the duration.

As our database has grown quite a bit, cloning it will most likely take a few hours, so I expect the downtime to last 2-3 hours. Sorry for the inconvenience, I am hopeful that it will be worth it and that this upgrade will significantly reduce some of our recent long page load times!


Edit: upgrade complete!

I have now migrated the lemm.ee database from the original DigitalOcean managed database service to a dedicated server on Hetzner.

As part of this migration, I have also moved all of our Lemmy servers from the DigitalOcean cloud to Hetzner's Cloud. I always want the servers to be as close as possible to the database, in order to keep latencies low. At the same time, I am very interested in having the ability to dynamically spin up and down servers as needed, so a cloud-type solution is really ideal for that. Fortunately, Hetzner allows connecting cloud servers to their dedicated servers through a private network, so we are able to take advantage of a powerful dedicated server for the database, while retaining the flexibility of the cloud approach for the rest of our servers. I'm really happy with the solution now.

In terms of results, I am already seeing far better page load times and far less resource use on the new hardware, so I think the migration has been a success. I will keep monitoring things and tuning as necessary.

 

Hey folks!

Lemmy 0.19 was released this week! It brings a bunch of awesome new features, so I hope you are all willing to forgive some downtime in order to upgrade to this latest version.

Unfortunately some migration will be necessary as part of this upgrade, so it might take a while, but I will try to keep it as short as possible.

I hope you are all having a great holiday period, and I will see you soon in 0.19!


Edit: Update complete!

Welcome to 0.19! Unfortunately, the upgrade took somewhat longer than usual, but I believe everything is in order now. As always, please let me know if you notice anything strange, and have fun!

1
submitted 2 years ago* (last edited 2 years ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hey folks!

I made a short post last night explaining why image uploads had been disabled. This was in the middle of the night for me, so I did not have time to go into a lot of detail, but I'm writing a more detailed post now to clear up where we are now and where we plan to go.

What's the problem?

As shared by the lemmy.world team, over the past few days, some people have been spamming one of their communities with CSAM images. Lemmy has been attacked in various ways before, but this is clearly on a whole new level of depravity, as it's first and foremost an attack on actual victims of child abuse, in addition to being an attack on the users and admins on Lemmy.

What's the solution?

I am putting together a plan, both for the short term and for the longer term, to combat and prevent such content from ever reaching lemm.ee servers.

For the immediate future, I am taking the following steps:

1) Image uploads are completely disabled for all users

This is a drastic measure, and I am aware that it's the opposite of what many of our users have been hoping, but at the moment, we simply don't have the necessary tools to safely handle uploaded images.

2) All images which have federated in from other instances will be deleted from our servers, without any exception

At this point, we have millions of such images, and I am planning to just indiscriminately purge all of them. Posts from other instances will not be broken after the deletion, the deleted images will simply be loaded directly from other instances.

3) I will apply a small patch to the Lemmy backend running on lemm.ee to prevent images from other instances from being downloaded to our servers

Lemmy has always loaded some images directly from other servers, while saving other images locally to serve directly. I am eliminating the second option for the time being, forcing all images uploaded on external instances to always be loaded from those servers. This will somewhat increase the amount of servers which users will fetch images from when opening lemm.ee, which certainly has downsides, but I believe this is preferable to opening up our servers to potentially illegal content.

For the longer term, I have some further ideas:

4) Invite-based registrations

I believe that one of the best ways to effectively combat spam and malicious users is to implement an invite system on Lemmy. I have wanted to work on such a system ever since I first set up this instance, but real life and other things have been getting in the way, so I haven't had a chance. However, with the current situation, I believe this feature is more important then ever, and I'm very hopeful I will be able to make time to work on it very soon.

My idea would be to grant our users a few invites, which would replenish every month if used. An invite will be required to sign up on lemm.ee after that point. The system will keep track of the invite hierarchy, and in extreme cases (such as spambot sign-ups), inviters may be held responsible for rule breaking users they have invited.

While this will certainly create a barrier of entry to signing up on lemm.ee, we are already one of the biggest instances, and I think at this point, such a barrier will do more good than harm.

5) Account requirements for specific activities

This is something that many admins and mods have been discussing for a while now, and I believe it would be an important feature for lemm.ee as well. Essentially, I would like to limit certain activities to users which meet specific requirements (maybe account age, amount of comments, etc). These activities might include things like image uploads, community creation, perhaps even private messages.

This could in theory limit creation of new accounts just to break rules (or laws).

6) Automated ML based NSFW scanning for all uploaded images

I think it makes sense to apply automatic scanning on all images before we save them on our servers, and if it's flagged as NSFW, then we don't accept the upload. While machine learning is not 100% accurate and will produce false positives, I believe this is a trade-off that we simply need to accept at this point. Not only will this help against any potential CSAM, it will also help us better enforce our "no pornography" rule.

This would potentially also allow us to resume caching images from other instances, which will improve both performance and privacy on lemm.ee.


With all of the above in place, I believe we will be able to re-enable image uploads with a much higher degree of safety. Of course, most of these ideas come with some significant downsides, but please keep in mind that users posting CSAM present an existential threat to Lemmy (in addition to just being absolutely morally disgusting and actively harmful to the victims of the abuse). If the choice is between having a Lemmy instance with some restrictions, or not having a Lemmy instance at all, then I think the restrictions are the better option.

I also would appreciate your patience in this matter, as all of the long term plans require additional development, and while this is currently a high priority issue for all Lemmy admins, we are all still volunteers and do not have the freedom to dedicate huge amounts of hours to working on new features.


As always, your feedback and thoughts are appreciated, so please feel free to leave a comment if you disagree with any of the plans or if you have any suggestions on how to improve them.

 

Sorry for the short post, I'm not able to make it nice with full context at the moment, but I want to quickly get this announcement out to prevent confusion:

Unfortunately, people are uploading child sexual abuse images on some instances (apparently as a form of attack against Lemmy). I am taking some steps to prevent such content from making it onto lemm.ee servers. As one preventative measure, I am disabling all image uploads on lemm.ee until further notice - this is to ensure that lemm.ee can not be used as gateway to spread CSAM into the network.

It will not possible to upload any new avatars or banners while this limit is in effect.

I'm really sorry for the disruption, it's a necessary trade-off for now until we figure out the way forward.

 

Hey folks

I have been receiving a lot of messages every single day about federation with hexbear. Some of our users are vehemently against it, others are in full support. The conversation does not seem to be dying down, rather, the volume of messages I receive about it seems to be increasing, so I am opening this public space where we can openly discuss the topic.

I am going to write a wall of text about my own thoughts on the situation, I’m sorry, but no tl;dr this time, and I ask anybody participating in this thread to first read through this post before commenting.

Before I go any further, I want to be clear that for anybody who participates here, it is required to focus on the quality of your posts. That means:

  • Be kind to each other, even if you disagree
  • Use arguments rather than calling people names
  • Realize that this is a divisive topic, so your comments should be even more thoughtful than usual

With that out of the way, there are a few things I want to cover.

On defederation in general

First of all, I am a firm believer that defederation must be reserved only for cases where all other methods have failed. If defederation is used liberally, then a small group of malicious users can effectively completely shut down the federated network, by simply creating the type of drama between instances which would inevitably result in defederation. In my view, federation is the biggest strength of Lemmy compared to any centralized discussion forum, so naturally I think maintaining federation by default is an important goal in general.

I am also a believer in the value of deplatforming hateful content, but I think defederation is not the best way to do this. Banning individual users, banning communities and establishing a culture of mutual support between mods and admins of different instances should be the first line of defense against such content. There are some further steps that can be taken before defederation as well, but these are not really documented anywhere (in order to prevent circumvention). The point is: for myself, defederation is the absolute last resort, only to be used when it is completely clear that other methods are ineffective.

Finally, I am wary of creating a false expectation among lemm.ee users that lemm.ee admins endorse all users and communities and content on instances we are federated with. Here at lemm.ee, we use a blocklist for federation, which means our default apporach is to federate with all new instances. We do not have the resources (manpower, skills and knowledge) necessary to pass judgement on all instances which exist out there, as a result, users on lemm.ee are expected to curate their own content to quite a high degree. In addition to downvoting and/or reporting as necessary, individual lemm.ee users are also able to block specific users and communities, and the ability to block entire instances is coming very soon as well.

Having said all that, in a situation where all other methods do indeed fail, defederation is not out of the question. Making such a call is up to the discretion of lemm.ee admins, and doing it as a last resort is completely in line with our federation policy.

Regarding hexbear

Hexbear is an established Lemmy instance, focused on many flavors of leftism. They have quite a large userbase who are very active on Lemmy (often so active that they leave the impression brigading all popular Lemmy posts). One important thing to note is that while some forms of bigotry seem to be quite accepted by many hexbear users (but seemingly not by mods - more on that below), they at least are very protective of LGBT rights (and yes, I am quite certain that they are not just pretending to do this, as many users seem to believe). Additionally, while I have noticed quite high quality posts from hexbear users, there are also several users there who seem to really enjoy trolling and baiting (very reminiscent of 4chan-type “for the lulz” posting), and it’s important to note that this kind of posting is in general allowed on hexbear itself.

The reason this whole topic is important to so many people right now (despite hexbear being a relatively old instance), is that hexbear only recently enabled federation. A combination of their volume of posts, their strong convictions, the excitement about federation, and the aforementioned trolling has made them very visible to almost all Lemmy users, and this has sparked discussions about the value of federation with hexbear on a lot of Lemmy instances.

My own experience with hexbear

I want to write down my own experience with interacting with hexbear users, mods, and admins over the past few days. I believe this experience will highlight why I am hesitant to advocate for immediate full defederation from hexbear at this point in time, and am for now still more in favor of taking action on a more individual user basis. Please read and see how you feel about the situation afterwards.

Background

My first real contact with hexbear users was in the comments section of a post in this meta community requesting defederation from hexbear by @glimpythegoblin@lemm.ee. That post is now locked, because several hexbear users very quickly started doing the aforementioned “for the lulz” type spamming of meme images in the comments (these are actually just emojis, but they are rendered as full-size images on all instances other than the source instance, due to a current Lemmy bug).

I did not want to take further actions in that thread in general (for archival purposes), but I did take one action, which in retrospect was a mistake: I removed a comment which contained the hammer and sickle symbol. I ignorantly associated this symbolism with Kremlin propaganda, and the atrocities my own people suffered at the hands of the soviet union during the previous century. Many users (including hexbear users) correctly (and politely) pointed out to me in DMs that the symbol has a much broader use than just as the symbol of the USSR, and people elsewhere in the world may not associate it with the USSR at all. I am grateful for users who pointed this out to me without resorting to personal attacks.

Let me be clear here: while I do not have anything against leftism or communist ideas in general (in fact in today’s world, I think discussion of such ideas is quite necessary), Kremlin propaganda has no place on lemm.ee. Any dehumanizing talking points of the Kremlin on lemm.ee are treated as any other bigotry, and if communist symbolism is used in context of Kremlin propaganda (that is the context in which I have been exposed to it throughout my whole life), then it will still be removed. But there is no blanket ban on communist symbolism in general on lemm.ee, and discussing and advocating for leftist and communist topics (as distinct from the imperialist and dehumanizing policies of the Kremlin) is certainly allowed on lemm.ee.

Hexbear user response

Coming back to the events of the past few days: soon after my removal of the comment containing the symbol from the meta thread, two posts popped up on hexbear. One was focused on insulting and spreading lies about me personally. Another was focused on diminishing the horrors of the soviet occupation in my country. In the comments under both of these posts (and in a few other threads on hexbear), I noticed some seriously disturbing bigotry against my people. There were comments which reflected the anti-Estonian propaganda of the current Russian state, things like:

  • Suggesting that my people has no right to exist
  • Stating that my people (and other Baltic nations) are subhuman
  • Claiming that anybody critical of both nazi and soviet occupations is themselves a nazi and a holocaust denier

I expect to hear such statements from the Russian state - here in Estonia, we are subjected to this and other kinds of bigotry constantly from Russian media - but to see it spread openly in non-Russian channels is extremely disturbing. Such bigotry is completely against lemm.ee rules in general. Additionally, my identity is public information, because I feel it’s important for the integrity of lemm.ee that I don’t hide behind anonymity. Considering this, I’m sure you can understand why I am very worried about my own safety when people leave comments in many unrelated threads (where my original posts are not even visible), baselessly calling me a nazi and a holocaust denier.

Note that the goal of this post is not to start a new debate in the comments about the the repressions of the soviet union in Estonia or other occupied territories, but if the topic interests any users, I can recommend the 2006 documentary The Singing Revolution (imdb). The trailer is a bit cheesy, but the actual film contains lots of historical footage from the soviet occupation, and also many interviews with people who experienced it, who share stories which are deeply familiar to all Estonians. If anybody is interested in further discussion, then I suggest making a post about it in the Estonian community here: !eesti@lemm.ee.

Hexbear admin response

After the above events had played out, I reached out to hexbear admins for clarification on their moderation policies and how they handle such cases. I was actually very happy with their response:

  1. They immediately removed the personal attacks and dehumanizing comments containing Kremlin propaganda from Hexbear, and assured me that such content is always handled by mods
  2. They told me that while there are all kinds of leftists on hexbear, Russian disinformation is generally either refuted in comments or removed by mods
  3. They implemented some additional rules on hexbear to try and reduce the trolling experienced by many other instances, including ours: https://hexbear.net/post/352119
My personal take-aways

Let me play the devil’s advocate here and employ some “self-whataboutism”: among all users that have been banned on lemm.ee for bigotry, the majority were actually not users from other instances, and in fact people with lemm.ee accounts. If we judge any larger instance only by bigoted posts that some of its users make, then we might as well declare all instances as cesspools and close down Lemmy completely. I believe it’s far more useful to judge instances based on moderation in response to such content. Just as we remove bigoted content from lemm.ee, I have also witnessed bigoted content being removed from hexbear.

At the same time, I am aware of some internal conflict between hexbear users over the more strict moderation they are now starting to employ, and I am definitely keeping an eye on that situation and how admins handle it.

I am also still quite worried about the amount of distinct users on hexbear who have posted Kremlin propaganda. I so far don't have reason to believe that these users are employed by the Russian state, but the fact that they are spreading the same hateful content which can be seen on Russian television seems problematic to say the least, and it remains to be seen if moderators can truly keep up with such content.

Where thing stand right now

I am not convinced that we are currently at a point where the “last resort” of defederation is necessary. This is based on the presumption that our moderation workload at lemm.ee will not get out of hand just due to users from that particular instance. My current expectation is that as the excitement of federation calms down (and as new rules on hexbear go into effect), the currently relatively high volume of low effort trolling will be replaced by more thoughtful posts. If this is not the case then we will certainly need to re-evaluate things.

Additionally, nothing is changing about our own rules regarding bigotry. Especially relevant in the context of Kremlin propaganda, I want to say that dehumanizing anybody is not allowed on lemm.ee (hopefully I do not have to spell it out, but this of course includes Ukrainians, LGBT folks, and others that the Kremlin despises), and action will be taken against any users who do this, regardless of what instance they are posting from.

Finally, I am very interested to hear thoughts and responses from our own users. I am super grateful to anybody who actually took the time to read through this massive dump of my own thoughts, and I am very interested to get a proper understanding of how our users feel about what I’ve written here. Please share any thoughts in the comments.

view more: ‹ prev next ›