This isn’t the only answer but it’s a big one. Having both the communities where people can authoritatively answer niche questions and the ability for new people to find those communities/questions is absolutely critical.
It doesn't help that the thread URLs are some old school "post/4268567".
I also noticed that the markdown format is included (e.g. the hashmarks for headings, asterisks for bold/italics) in search results while every other site doesn't look like that.
What do you mean you cant see the comments until you click a further link? What do you see when you click on this https://lemmy.world/post/2383782 i see the post and all the comments?
I wonder how that'd work though. Like imagine you made a backpack focused Lemmy instance, say backpack.social. How would you get the SEO show up posts under that instance if they add "Lemmy" at the end of the search? It's probably possible, but I dunno how it works or if that would cause problems. Also would we go with Lemmy? Because should kbin posts also show up, or should they just use Lemmy as an SEO tag?
Maybe “fediverse” might be better. All fediverse instances would have the context of fediverse in the scraped data. When someone searches “best budget backpack fediverse” the search engine would show the fediverse instances with the best seo score. Higher quality posts get a letter seo score just as they do today on Reddit. It does not matter which instance the post would be on.
The bigger problem is that search engines can’t even really scrape some parts of the fediverse (like Lemmy) because the default UI does not show any post or comment data.
It's not so much the instances but the communities that are important on Lemmy, unlike most of the fediverse. If your community's instance is federated with the big instances, it helps get people to your community either way if the post shows as a link on the bigger instance or the host instance. Hopefully crawlers will eventually add some smarts so we link the host instances eventually too.
I've been adding "Lemmy" to some of my Google searches lately for things I'm sure there are answers to here (been looking into buying my first NAS), but I think it's only returning results from instances that have the word Lemmy in them, like lemmy.world and lemmy.ml. As far as I can tell, if there's an answer on an instance like aussie.zone, using "Lemmy" in the query will not return it in the results.