"Moderation tools are nonexistent on here. It also eats up storage like crazy [...] The software is downright frustrating to work with" - Can any other instance admins relate to this?
"Moderation tools are nonexistent on here. It also eats up storage like crazy [...] The software is downright frustrating to work with" - Can any other instance admins relate to this?
After a year online the free speech-focused instance 'Burggit' is shutting down. Among other motivations, the admins point to grievances with the Lemmy software as one of the main reasons for shutting down the instance. In a first post asking about migrating to Sharkey, one of the admins states:
This Lemmy instance is much harder to maintain due to the fact that I can’t tell what images get uploaded here, which means anyone can use this as a free image host for illegal shit, and the fact that there’s no user list that I can easily see. Moderation tools are nonexistent on here. It also eats up storage like crazy due to the fact that it rapidly caches images from scraped URLs and the few remaining instances that we still federate with. The software is downright frustrating to work with, and It feels less rewarding overall putting effort into this instance because it feels like we’re so isolated.
A few weeks later, in the post announcing that Burggit was shutting down, another admin says the same:
The amount of hoops that burger has to go to in order to bring you this site is ridiculous. To give you an idea of how bad this software is, there’s no easy way to check all the images uploaded to the site (such as through private messages). When the obvious concern of potential illegal imagery is brought up to lemmy devs, they shrug and say to plug in an expensive AI image checker to scan for illegal imagery. That response genuinely has me thinking that this is by design, and they want it to be like this. We can’t even easily look at the list of registered users without looking through the DB, absolute insanity.
The other thing is there’s no real way to manage storage properly in Lemmy, the storage caches every image ever uploaded to any instance forever.
Also the software is constantly breaking.
They also say that Kbin has many of the same problems, so I'm just curious to know if the admins of bigger Lemmy & Kbin instances feel the same way about these software.
You're viewing a single thread.
(instance admin here, but for a small one) woof well, for me, I agree, but I wouldn't use that wording.
Lemmy for sure isn't a plug and play site. Setting it up took leaps and bounds, learning way more about nginx than I ever really cared to, and figuring out documentation that was very clearly out of date. Very little logging or error messaging exists to help with that problem.
Very little errors exist at all, it's very much a "happy path" project. That's why we get constant spinners everywhere, because when an HTTP error occurs there's no actual error message. (Come on guys, just add it to your standard HTTP messages, if statusCode < 200 || >= 300 then show a toast message).
But yeah, the moderation tools have to be the worst. Lemmy has an amazing development group that's separate from the main developers who have patched together a good set of tools, from automods to CSAM and illegal scanning, huge props to them - but these issues are routinely ignored by the main devs. I was shocked, honestly shocked that when we were under CSAM attacks that there was not an immediate roundtable of the head devs to try to solve the problem officially. Here was a problem that 99% of countries would immediately and gladly throw us, the instance admins, in jail over and they just handwaved it away. In fact, I don't know that there was ever an official post about it, or even that there are things coming to help with it.
I love Lemmy and being here, and the devs have done a great job at building this platform for us, but we're at a critical point right now. It's no longer software that is just fun side projects and building stuff that looks cool, it has some real issues now that it has a real userbase. I'm definitely one to say "But it's FOSS, and other people can pick up and submit a PR" - but it also says something when the head devs just completely ignore a massively huge issue with it.
Bugs and caches and that sort of thing I can overlook. Those I can wait on and see them get smoothed out over time. Actual issues that could land me in jail or get the feds to beat down my door? Those I kind of expect a fast response.
So, I'll say I'm extremely conflicted. I want to host lemmy long term, and I'm happy to bring the fediverse to a few more people, but the csam attacks really altered my view of the devs.
Edit - because my favorite manager said "Bring me solutions, not problems" a few things that would really help immediately -
- Integrate db0's CSAM checker natively, more or less a plug and play option, or a checkbox. His checker sits at an endpoint. The admin page of lemmy could easily have you plop in the endpoint and it would start checking
- Have an image management portal, with capabilities to:
- Auto remove images after X time (to help with ballooning storage costs)
- Perma-delete images and users (maybe blurred too if the CSAM checker flagged it, so I don't need eye bleach) (Edit again, 0.19.4 might have fixed this, I need to upgrade so I'll see)
- Federating image purges, so one purge on one server will force purge it on everyone else's
- ~~Disabling of caching other server's images ~~ (Edit again, I see 0.19.4 just dropped which has this, so this is good). This way I'm only responsible for my own users.
- View images that are not related to a post (DM'd messages that I'm hosting, or people just uploading images to my site)
- Bring in a logging system into the UI itself, so I can keep tabs on the error logs. I can pipe them somewhere, but this would be a major plus as an admin
Dont forget the complete ignorance of gpdr. That shit will get you fucked over, and its not as simple to follow as it seems.
In fact, Im not 100% sure that federation works with gpdr, since you cant garantee all data will be deleted.
I'm not sure it's that difficult to follow. If you offer a service in the EU, you are responsible for your server deleting personal data (or, even better, not even hosting it in the fist place!); you are not responsible for other people not deleting their copy of personal data.
But I'm not that well-informed in the actual legalese so my best understanding is the big issue is the EU's definition of "provide service to the EU" more than anything else. They seem to think that just because your users might upload a local copy of a picture of someone from the EU, even if you yourself are not allowing connections from the EU, then you are serving to the EU. And with how nazi the EU has been going lately with stuff like ChatControl, the last thing I'd want as an instance owner is to be upheld to arbitrary boomers' (lack of) understanding of technology.
but we’re at a critical point right now. It’s no longer software that is just fun side projects and building stuff that looks cool, it has some real issues now that it has a real userbase. I’m definitely one to say “But it’s FOSS, and other people can pick up and submit a PR” - but it also says something when the head devs just completely ignore a massively huge issue with it.
This is a general issue I think, not just for lemmy but the whole fediverse (whatever one's opinions might be on particular priorities).
It's all non-profit and being run and built at a much smaller scale than many users would appreciate (I think). Sure there are plenty of people here, but not that many. Combined with no obvious revenue streams, such as ads or subscription fees, there really is only so much that can be done. Some time last year even the Mastodon team (by far the most successful fediverse platform) admitted that they didn't have the capacity to work on new things for a while ... they were just busy keeping things running. And they are (apparently) notorious at being slow to ship new features. Meanwhile platforms like firefish just straight up died last year.
So yea, it might be a critical point, for sure. But putting more on the core dev teams may not be the answer for the simple reason that it's just not viable in the long run.
If we enjoy the bigger community focus and open and non-profit organisations that makeup the fediverse, the "answer" at this critical point might be to find a way to give back somehow ... to organise, build communities, run fund-raising campaigns, think of ideas for more sustainable funding, find devs who can help etc etc. It's perhaps onerous and annoying, even to read perhaps ... but this is likely the tradeoff we have to make for a place like this.
I fully understand that, and I explained it in my reply to nutomic. It's not that I wanted to just pile on, but rather at an emergency like that, an all-stop would have been justified in my opinion, to stop all work and go into emergency "What can we all do to stop this and prevent it". All other issues were secondary in that moment compared to stopping CSAM, and it didn't feel that way. They added it to the pile of issues, and I'm glad to see changes came out, but in that moment I didn't see much if any support from them
I hear you for sure on this.
I personally appreciate the push the core devs make for a "less demanding" relationship between users and open source devs. I think they have a point and it's good for long term sustainability and I'll probably find myself defending it.
But I like you're framing, and in retrospect it seems (as is usually the case) that some people organising was what was missing just to get everyone helping each out as much and efficiently as possible. While any member of the community can (and should) do that, at some point it makes sense for the core devs to take on a task like that, I agree.
But yeah, the moderation tools have to be the worst. Lemmy has an amazing development group that’s separate from the main developers who have patched together a good set of tools, from automods to CSAM and illegal scanning, huge props to them - but these issues are routinely ignored by the main devs. I was shocked, honestly shocked that when we were under CSAM attacks that there was not an immediate roundtable of the head devs to try to solve the problem officially. Here was a problem that 99% of countries would immediately and gladly throw us, the instance admins, in jail over and they just handwaved it away. In fact, I don’t know that there was ever an official post about it, or even that there are things coming to help with it.
My impression at the time was that admins were handling the CSAM wave just fine with existing mod tools and through Matrix chats. A roundtable wouldnt have solved anything except make people feel good. Besides we still were extremely busy at the time to scale up Lemmy and resolve problems revealed by the huge amount of new users. Keep in mind that Lemmy is still at version 0.x which means that its not feature complete. So if something is missing that you find important, consider waiting a year or two and checking back then. Or get it implemented yourself, thats what open source is all about.
That said most of the features you mentioned have already been implemented, including a list of all locally uploaded images.
I was not handling it fine, it was generally chaotic, and in the Matrix chats I remember it being chaotic, for both attacks. Luckily by the second one we had db0's tooling to help a bit more, but there still many of us who were exposed to the images. We lost a lot of instances during those two attacks from admins who justifiably didn't want to take on the risk.
I completely understand how crazy it was, but the lack of response from you guys was disheartening, it really did make me wonder if I should continue hosting or if I should bail out. Ultimately, I decided to stay obviously, but had to do some hard extra steps, like reducing privacy and registering with the feds for CSAM.
So like I said, I'm torn. I respect you guys for everything you do, but that was a moment where all other development should have stopped to immediately address a real problem, and while you think a roundtable would have just been feel good, I think we could have kept a lot of instances online if it had been done. Assurances that yes, new changes are coming, and official suggestions like "Here are the endpoints to delete the images", or nominating db0 or someone as the person in charge of the outbreak. It was honestly a scary time, and for us owners who accept a lot of risk, for many of them it was too much.
Anyway, I have a habitual case of foot in mouth disease, so it was immediately after posting that comment that I heard about 0.19.4, and immediately felt stupid. I tried it last night but I kept getting timeout errors and something about "Could not get user's /inbox" or something, I'll try 0.19.5 today. Thank you for bringing additional mod tools, they've been hugely needed. I know they're not glamorous to make, but they keep the communities healthy and strong.
Edit: 0.19.5 also failed. I wrote up a github bug on it, until then I unfortunately have to stay on 0.19.3 https://github.com/LemmyNet/lemmy/issues/4850
The entire time after the Reddit migration was extremely chaotic. I dont remember when exactly the CSAM attacks happened, but around that time we were already very exhausted from all the urgent work we had to do on scaling, patching security vulnerabilities and fixing countless bugs. I also dont remember receiving any requests from admins to help out with this. So if you notice something similar in the future, feel free to message me directly. Anyway we are only two people working full-time on Lemmy, and have lots of different tasks to take care of. So it gets very difficult to give everything the attention it deserves, and to prioritize things correctly.
I honestly think you peeps need to somehow invest in your communication strategy somehow. Such communication breakdowns is/was causing schisms in the lemmy community which is an extreme shame as that's in turn driving away the same potential contributors that would help the software improve faster to cover these same points. I would argue that saying things like "we're still in beta, come back in 2 years if you can't handle the heat" is not doing you any favours. I know you are technically correct, but there's no reason to phrase it like that, yanno? Not everyone interprets such statements the same way and for non-ASD/ADHD people, this can parse very hostile and confrontational, even if you honestly didn't mean it to be read like that.
Apologies in advance for the unsolicited advice, but have you considered reaching our for some community outreach person to join your team? Such positions won't necessarily fill themselves and you need to ask for it. But at this point I think it might significantly help the lemmy project avoid such drama.
Nice suggestion
How else would you say this? And who do you suggest reaching out to? Keep in mind that it would have to be a volunteer position as we dont have the funds to pay for it.
You could have volunteers working on the communication aspect. That doesn't have to be a paid position.
I'll let db0 suggest a better phrasing for the "we're still in beta" part
Sure but its not so easy to find volunteers. Would you or db0 be willing to do this?
I feel like people already see me around a lot, I would avoid being "overexposed" if that makes sense.
Maybe someone with less activity on Lemmy would be better for that job?
At the end of the day, it's mostly to discuss with you guys how to present things to the audience, and being more present when issues like the CSAM issues arise, potentially even switching priorities to convey the message to the community that this issue was serious and that you were aware of it.
Again, I'm not saying you guys didn't do anything, but as always, it's mostly on how to communicate in a situation rather than having a look at what was done or not.
If I had the skills and the time, I would. But I'm an AuDHD tech-nerd, and vastly unsuited and unsuitable for such a role. The point is not to ask the specific people making this suggestion as a gotcha, it's to recognize there's a problem area which might be solved with another volunteer with the right skillset and take the steps to find one. To put it another way, you are already painfully cognizant that lemmy needs some DB attention. There's no point in asking people who bring up performance issues if they can do the necessary DB work, but there is a point in making an open call for people with DB optimization expertise. Does this make sense?
Having another volunteer also means more work for us, as we need to communicate with this person regularly. It also means that we maintainers get more removed from the users, and wont be able to talk with them directly anymore. And in my experience, volunteers are very motivated in the beginning, but most of them get bored or busy after a while and then you need to find someone new again. Not really worth the hassle in this case.
Also the database issues mentioned in this thread may simply be from lack of ram.
Yes I understand that. I meant making an open call for volunteers on that aspect. Note that I'm in the same boat with my own FOSS project where it's not always easy to find the volunteers, but at least I don't have the same pressures as you face on this area. Not saying this is going to be an easy fulfillment, but it can be perhaps something to pursue.
How else would you say this?
Note that I'm AuDHD, so I'm not the best person to actually do these things. It's because I know my limitations that I have compensated by learning to notice these pitfalls in communication. I would also need someone to help me in the same situation as you.
Nevertheless, If I were to speculate, I wouldn't even go in that direction. From what I've seen, most people who know what they're doing in this aspect will just say something like "we acknowledge the issue and we'll do what we can to handle it asap" or whatever, just so people don't feel left in the cold, you know? Again, don't take what examples I write as the exact practice as it's not my area of expertise. I'm just (badly) parroting what I saw work better.
Its easy to say this now, more than half a year later. But youre ignoring that we were completely overworked and exhausted back then. That said Im taking your feedback into account and will hopefuly to handle it better in the future.
I was around when a user when the CSAM attacks happened. That was crazy stuff, sorry you admins had to go through it.
The introduction of a plugin system seems interesting here, though it's still alpha and basically looking for feedback from would-be plugin developers
AFAICT, there's no established way for a plugin to surface affect the UI, which would be a somewhat unwieldy problem anyway due to the apps and frontends ecosystem. Probably the best path for any plugin that provides a UI would be to have a system for aggregating links to plugin UIs.
With something like that in place, plugins and other services that just use the DB/API, could really go a long way to filling these holes, if they haven't already.
So it seems that the work needed here is perhaps "distribution" work ... where there's a more "plug and play" Lemmy distribution with plugins etc bundled?
This is what excites me the most. There are huge potentials for plugins, and I think it'll ease some of the strain from the core engineers. Most of the "ideas" I see posted really could be plugins. Things like badges on posts, verification of links, etc etc could all be plugins that individuals could make. The problem with developing against the core repo is that you have to learn and understand the core repo, so you don't fuck up something else in some other place accidentally. Plugins are a neat way where we can say "I'm a function that does one thing, just do the thing here, and then do what you need to with that data"
Absolutely!
The bit I'm conceptually stuck on (not know much at all about how a good plugin architecture would work) is how a plugin can surface or affect the UI, especially in an ecosystem with multiple UIs/Apps/Frontends, and, a federated ecosystem at that.
Given the apps, I figure it's not possible without a convention of plugins providing APIs which apps can then implement against when available, which adds a good amount of complexity but should be viable for popular/useful plugins. Though, tangentially, this does affirm for me that the whole native mobile app expectation is a bit of a trap for a social system like the fedi (as webUIs are naturally more universal and maleable).
So, for immediate results, I can see only two options:
- a plugin operates on the backend directly manipulating or creating content not unlike a bot
- a plugin provides its own webUI which is made available through a simple and dedicated location in the UI
Is there something I'm missing about how a plugin system could work?