Posts to your instances communities seems sometimes to take multiple hours before federating and sometimes not at all.

Any ideas what's the cause? I also wonder if that "Parallel federation sending" feature introduced in Lemmy v0.19.6 could help with the issue?

Do something with it !

I get this error message whenever I try to

{"message":"{\"msg\":\"No space left on device (os error 28)\",\"files\":null}"}

Hey all! I've done a lot of database maintenance work lately and other things to make lemdro.id better. Wanted to give a quick update on what's up and ask for feedback.

For awhile, we were quite a ways behind lemmy.world federation (along with many other instances) due to a technical limitation in lemmy itself that is being worked on. I ended up writing a custom federation buffer that allowed us to process activities more consistently and am happy to say that we are fully caught up with LW and will not have that problem again!

Additionally, on the database side of things, I've setup barman in the cluster to allow for point of time backups. Basically, we can now restore the database to any arbitrary point in time. This is on top of periodic automatic backups which also gets pulled to storage both on my personal NAS as well as a Backblaze bucket (both encrypted of course).

Today, I deployed a new frontend at https://next.lemdro.id. This one is very early stages and experimental but is being developed by https://lemm.ee and seems promising!

If you live outside of the US and experience consistently long load times I want to hear from you! I am deploying the first read replica node to Europe soon, so if you live in that region you'll soon notice near-instaneous loading of content. Very exciting!

Finally, looking for feedback. Is there anything you want to see changed? Please let me know!

I am rolling out the Photon UI as a replacement to the default lemmy UI right now. Initially, only about 50% of requests will be routed to Photon, determined by a hash of your IP address and user agent (sorry for any inconsistencies...). As I determine that this configuration is stable I will be slowly increasing the percentage until Photon is the new default frontend for lemdro.id.

If you have any difficulties, please reach out. Additionally, the "old" lemmy frontend will remain available at https://l.lemdro.id

Edit: I am aware of some problems with l.lemdro.id. It wasn't designed to run on a subdomain so I'll need to add a proxy layer to it to redirect requests. A task for tomorrow!

FINAL EDIT: https://l.lemdro.id is now fully operational, if you choose to use the old lemmy UI it is available there

Over the course of the last couple weeks, I managed to root cause and solve the problem causing stale sorting on lemdro.id. My apologies!

Edit: Upgrade went off without much of a hitch! Some little tasks left to do but we are now running 0.19.3. Seems like a lot of the federation issues have been solved by this too.

You will have to re-login and 2FA has been reset.

This update is scheduled to take place this weekend. No specific day or time because I am infamously bad at following those. I will try to minimize impact by keeping downtime to lower-traffic periods.

Ideally, there will be no downtime, but if there is it is likely to only last an hour maximum. During this time I will add an "under maintenance" page so you can understand what we are up to.

Feel free to join our Matrix space for more information and ongoing updates! My apologies for how long this took - I was in the middle of a big move and a new job.

Additionally, there may be small periods of increased latency or pictures not loading as I perform maintenance on both the backend database and pictrs server in preparation for this upgrade.

Hello everyone,

Small question for you: do you have any idea on when you will update to 19.3? There is an issue with upvotes federation between 18.5 and 19.3 at the moment

There was a brief (~5 minute) period of downtime on lemdro.id recently. This occurred during a routine database upgrade when some strange issue caused a member of the cluster to become inconsistent and refuse to form quorum.

This locked the entire cluster into an invalid state which took some troubleshooting to fix. My apologies.

I will be rolling out read replicas for folks on the East coast of the US as well as those in Europe sometime in the next week, you should notice a pretty dramatic reduction in latency if you are from those areas. Additionally, other recent changes have increased reliability and decreased latency which may or may not be noticeable.

I will post another update before I start rolling out the read replicas since it is kind of a big change (and I will schedule a time for it)

I recently rolled out an infrastructure upgrade this last Monday (Dec 4) with the intent of reducing peak response times and removing occasional scaling errors.

Unfortunately, my metrics system showed slightly elevated error rates, so I've decided to rollback these changes for now. I will make another announcement before I roll them back out in the future. Thanks for your support!

lemmy.studio and sopuli.xyz are not populating for me. Probably soft blocked? Also, it may be worth opening and pinning a megathread for this topic so the community doesn't get spammed.

edit 10/15: sopuli has started working, studio is still not showing

Hey there everyone. I think the Photon project has matured enough to the point where I feel ready to replace the default lemmy frontend with it. Since this instance serves roughly 1000 people now, I figured this was worth holding a vote on!

Please check out Photon as currently hosted at https://p.lemdro.id.

If you support changing the default frontend to Photon, upvote my comment on this post. If you don't support it, downvote that same comment.

Thanks!

Has anyone else been getting frequent error messages on lemdro.id the last couple days? I am using the standard web interface and keep seeing something to the effect of:

FetchError: request to http://lemdroid-lemmy.flycast/api/v3/post/list?type_=Subscribed&page=1&limit=20&sort=Hot

Basically title, can't get any communities from the instance to load. It is in the list of federated instances in the footer.

Start your journey into the Fediverse by subscribing to our starter communities. We're actively working with subreddit communities and moderators on their transition over.

Our Mission

Lemdro.id strives to be a fully open source instance with incredible transparency. Visit our GitHub for the nuts and bolts that go into making this instance soar and our Matrix Space to chat with our team and access the read-only backroom admin chat.

Interfaces

• lemdro.id powered by Photon
• l.lemdro.id powered by Lemmy-UI
• m.lemdro.id powered by Voyager
• old.lemdro.id powered by mlmym
• a.lemdro.id powered by Alexandrite

Resources

• List of Lemmy apps
• Fediverse Community Search

Our Communities

• !android@lemdro.id
• !askandroid@lemdro.id
• !androidmemes@lemdro.id
• !techkit@lemdro.id
• !aistuff@lemdro.id
• !google@lemdro.id
• !googlepixel@lemdro.id
• !xiaomi@lemdro.id
• !sony@lemdro.id
• !samsung@lemdro.id
• !galaxywatch@lemdro.id
• !nothing@lemdro.id
• !oneplus@lemdro.id
• !motorola@lemdro.id
• !meta@lemdro.id
• !apple@lemdro.id
• !microsoft@lemdro.id
• !chatgpt@lemdro.id
• !bing@lemdro.id
• !reddit@lemdro.id
• !materialdesign@lemdro.id

Other Neat Communities

• !cybersecurity@sh.itjust.works
• !ps5@lemmy.world
• !pcgaming@lemmy.ca
• !steamdeck@lemmy.ml
• !selfhost@lemmy.ml
• !firefox@fedia.io
• !opensource@lemmy.ml
• !technology@beehaw.org
• !science@lemmy.ml

Seeking Experienced Mods

Are you interested in exploring options to migrate your tech subreddit to the Fediverse in a way that supports decentralization or are you an experienced moderator who is interested in joining one of our mod teams? Get in touch!

A Fediverse home for developers

Are you developing a Lemmy app and looking for a home community for your project? Get in touch!

Earlier today, I identified the root cause of an issue causing annoying 502 errors intermittently. If you've ever had an action infinitely load until you refreshed the page, that is this issue. I deployed a fix, and am slowly scaling it down to stress test it. If you encounter an infinite loading occurrence or an HTTP 502 error please let me know!

UPDATE: Stress testing complete. Theoretically we should be equipped to handle another 5k users without any intervention from me

Hello folks! I am migrating the image backend to an S3-compatible provider for cost and reliability reasons. During this time, thumbnails and other images hosted here will be borked, but the rest of Lemdro.id will remain online. Thank you for your patience!

UPDATE: Image migration is gonna take a hot minute. Should be done in around 6 hours, I'll get it fully fixed up in around 7-8 hours when I wake up (~08:30 PDT)

UPDATE 2: It failed, yay! Alright, fine. I turned the image proxying back on. I am migrating to S3 in the background and will switch over when it is done. Any images uploaded in the next 8 hours or so may end up being lost.

UPDATE 3: Migration complete. Will be rolling out the update to S3-backed image storage in around 6 hours (~6pm PDT)

UPDATE 4: Object storage backend deployed! Thanks for your patience folks.

I'm sure you all have noticed the latency problems on this instance. Stage 1 of my 4 stage scaling roadmap is taking place tonight as I migrate the database to physically run closer to the machines running lemmy.

I will do a more detailed write-up on this later, but the gist is that each db operation required a new connection from lemmy, and that means a brand new SSL handshake since the db is managed elsewhere. Pooling would solve this, but lemmy does not handle a properly configured pg bouncer correctly in my testing. So the solution is move the database closer and within the private network to avoid SSL handshakes altogether.

TL;DR instance gonna go brrrr, downtime starting at 10:30pm pacific time tonight, should be done by 11:30pm

cross-posted from: https://lemdro.id/post/4976

> I shared bits and pieces of this before, but it's officially up and running now: https://www.search-lemmy.com/ > > This is an enhanced search engine for Lemmy. With a few primary goals: > * You can choose a preferred instance. After choosing what your primary instance is, and performing a search ALL links will open in that instance. > * This aims to be a replacement for using site:reddit.com in Google, but just for the fediverse. > * You can filter the search results by: > * Instance -- This will filter the results to only show communities that belong to a particular instance. Just type something like instance:lemmy.wrold or instance:https://lemmy.world/. This is separate from your preferred instance, such that you can search for posts on lemmy.world while still opening them on lemmy.ml. > * Community -- You can refine the search by a specific community. You use the same syntax that you'd use here community:!fediverse@lemmy.world. > * Author -- Similar to the above you can also filter by a specific author such as: author:@marsara9@lemmy.world. > * The entire thing is open-source. You can view the code and even host your own instance... See more details here: https://github.com/marsara9/lemmy-search. > > NOTE: This only supports Lemmy instances for now. Other fediverse type instances may be in the future depending on how this works out. > > I've been working on this over just the last few weeks, so it hasn't had a chance to crawl much of the fediverse yet. For now it only supports lemmy.world and lemmy.ml but other preferred-instances will come online as time goes by. > > If anyone finds any bugs, and I'm sure you will, or if anyone has any suggestions PLEASE raise an issue on GitHub for me to track. Lastly, if anyone wants to help contribute please feel free to reach out. > > NOTE TO SERVER ADMINS: You can prevent your site from being crawled by adding lemmy-search to your robots.txt for the user-agent.

cross-posted from: https://lemdro.id/post/4376

> I got tired of hitting ⌘Enter and not having my post automatically go through, so I wrote a little userscript that enables exactly that.

Lemdro.id App List

This is a curated list of various apps for Lemmy, an open-source federated alternative to Reddit. The list includes web apps, Android and iOS clients, as well as Linux and browser extensions.

The Lemdro.id Lemmy App List will be kept updated and evergreen. Feel free to share or pin across your respective communities and instances. Comment below if you have a suggestion.

Don't forget that you can also add the main https://lemdro.id page to your home screen. The mobile version is surprisingly good!

Notes:

• Originally adapted from rist097@lemmy.world.
• Lemmy apps haven't been audited for security vulnerabilities. Use at your own discretion.

Web Tools

! Lemmy Instance Assistant

A browser extension to make Lemmy and Kbin easier to use (GitHub).

Web Apps

! Voyager

Formerly wefwef. An Apollo-like web client for Lemmy. Works great on mobile and desktop (Github, Play Store, Web App).

! mlmym

A familiar Reddit-like desktop experience for lemmy (Github, Web App).

! Nemmy

The Neat Lemmy App for the Web (Github, Web App).

! Photon

An alternative web client for Lemmy with the UI of Xylo (Github, Web App, !photon@lemmy.xylight.dev).

! Alexandrite

A desktop-first alternative web UI for Lemmy (Github, Web App, !alexandrite@lemmy.world).

! Createlab

A Lemmy UI rewrite in SvelteKit. (Github, Web App Preview).

! Leanish

A lean-ish web client for Lemmy (Github, Web App).

Android

! Jerboa

The Official Android Lemmy client (Github, Play Store, !jerboa@lemmy.ml).

! Thunder

A Lemmy client for iOS and Android built with Flutter (Github, IzzyDroid, !thunder_app@lemmy.world).

! Liftoff

A mobile client for Lemmy (Github, Play Store, !liftoff@lemmy.world).

! Connect for Lemmy

Aimed at being easy to understand, fast, and clean (Play Store, !lemmyconnect@lemmy.ca).

! Summit

A mobile Lemmy client aimed at being fast, simple, and effective (Play Store, !summit@lemmy.world).

! Lemming

An Android client for Lemmy (Github, !lemming_app@lemmy.world).

! Arctius for Lemmy

A simple and accessible mobile client for Lemmy built on React Native (Github, !arctius@lemmy.world).

! Beyond

An Android and iOS Lemmy built heavily inspired by Infinity for Reddit (Announcement, Play Store)

! Lemmotif for Lemmy

A simple app for Lemmy (Play Store, !lemmotif@lemmy.world).

! Infinity for Everything

Alpha available! Fork of the full-featured Reddit app for Lemmy (Github)

! Infinity for Lemmy

New project! A clean UI and smooth browsing Lemmy client written in Java, fork of the full-featured Reddit app (Codeberg, Announcement).

! Slide for Lemmy

Builds are available! Developer is working on bringing this full-featured Reddit app to Lemmy (Github).

! Sync for Lemmy

Initial release of their Reddit app brought to Lemmy (Play Store, !syncforlemmy@lemmy.world).

! Boost for Lemmy

Coming soon! Developer is working on bringing their full-featured Reddit app to Lemmy (Play Store, !boostforlemmy@lemmy.world).

! Artemis

Coming soon! React Native + Typescript app for Android and iOS that uses its own API to scrape kbin (Dev Diary, Discord, !ArtemisApp@kbin.social).

! Warami

Coming soon! A mobile app in early development (Github).

iOS

! Memmy for iOS

An Apollo-inspired iOS client for Lemmy (Github, Testflight, !memmy@lemmy.ml).

! Thunder App

A Lemmy client for iOS and Android built with Flutter (Github, Testflight, !thunder_app@lemmy.world).

! Mlemapp

An open source Lemmy client focused on ease of use, intuitiveness, and optimization (Github, Testflight, !mlemapp@lemmy.ml).

! Liftoff

A mobile client for Lemmy (Github, Testflight, !liftoff@lemmy.world).

! Olympus

An Apollo-inspired iOS client for Lemmy (Testflight, !olympusapp@sh.itjust.works).

! Lemmios

An Apollo-inspired iOS client for Lemmy (GitHub, Testflight, !lemmiosapp@lemmy.world).

! Artemis

Coming soon! React Native + Typescript app for Android and iOS that uses its own API to scrape kbin (Dev Diary, Discord, !ArtemisApp@kbin.social).

! Ares

Coming soon! A mobile app in early development (Github).

! Beyond

Coming soon! An Android and iOS Lemmy built heavily inspired by Infinity for Reddit (Announcement)

! Morpha

Coming soon! An Android and iOS Lemmy built heavily inspired by Infinity for Reddit (Announcement, !morpha@vlemmy.net)

For all curious, Lemdroid now runs via fly.io and neon.tech.

The database is hosted by neon tech while the frontend and backend are horizontally scalable and deployed to fly.io. This should be massively scalable both up and down, so we can handle nearly any amount of users. Pretty neat, right?