‘Websites have control over whether crawlers can access their content.’
Google can’t display tweets and pages from Twitter in search results as effectively as it usually does because of changes at Twitter, according to a statement given to The Verge. “We’re aware that our ability to crawl Twitter.com has been limited, affecting our ability to display tweets and pages from the site in search results,” spokesperson Lara Levin said. “Websites have control over whether crawlers can access their content.”
“Could”, more like “already is”. It’s a darn shame, and honestly I kind of wish the Fediverse had taken off long ago so we wouldn’t be stuck with these repositories of information being walled off by centralized tech companies
Brave Search has a beta tool called Goggles with which you can limit/sort results. There is one called 1k Short for example that omits first 1000 results.