Found this post super informative as it relates to Mastodon, and thought Lemmy might also benefit from this perspective. I’m not sure I share his optimism, but his points seem sound to dampen some of the alarm bells over Meta joining the Fediverse.
Why use a crawler if you could spin up some camoflaged small instances and get the info right via the regular api?
Or create accounts and get the info from the client api like apps?