Personally, it happens to me quite frequently that I encounter a niche problem, Google it, and find the solution in an obscure blog from 2007.

However, for more recent content, I’ve found it increasingly difficult. For example, with older brute-force chess engines, I can easily search online and find abundant documentation, forum posts, and personal experiences. In contrast, for modern chess engines, like those based on neural networks, I’ve found it significantly harder to locate what I’m looking for through Google, because much of the technical discussion and support takes place on private Discord servers.

Moreover, I often limit my searches to specific blogs or forums, like Fedora forums or my favorite personal blogs, but this approach doesn’t work when the information is confined to Discord.

Funnily enough, I’m not blaming AI by any means. Closed walled gardens are a modern problem, stemming from the decline of forums and independent blogs. AI didn’t cause it; the issue already existed. In fact, you might argue that closed gardens like Discord are pissing off everyone: AI companies dislike them because they make access to training data more difficult, and members of this community dislike them because they make it harder to find human-written content online through search engines like Google.

It’s not technically that hard to scrape content from the most popular Discord servers. The real issue is that you can’t legally share those findings publicly on the open web because of copyright concerns.

It’s a problem when we can’t find things just by searching online, because all the information is spread across thousands of different Discord servers that Google can’t index.

“Just join the server” isn’t a real solution either for several reasons:

1. You don’t know which server actually has the info you’re looking for.
2. You might not even know where to find the link to join the right server.
3. Some servers hit their member limit and won’t let new people join.
4. There’s no way to search across multiple servers at once. You have to check them one by one.
5. Even within a single server, Discord’s search function doesn’t work very well.
6. It’s also bad for long-term archiving. These servers can’t be properly archived by services like the Internet Archive or Wayback Machine. If a server shuts down due to drama, mismanagement, or malicious actions, all the information in it can vanish overnight with no backup.

So telling someone to “just join” doesn’t solve the core issues of discoverability, accessibility, or preservation.

You can already use proprietary cloud-based LLMs like OpenAI’s ChatGPT or xAI’s Grok. If you explain to them in the prompt what “niche, enthusiast, passionate websites” are and how to find them, they can definitely help you and give you much better results than Google even in their current state. “Hallucinations” are a complete non-issue. If the LLM gives you two non-relevant links out of ten, with the rest being correct, that is still better than Google, where you might only get one relevant link out of fifty.

Now, thankfully, you do not have to rely on the cloud. If you have some DIY skills and a fair amount of computing power at home, you can run a setup locally that rivals cloud-based LLM searches in performance.

Unfortunately, it is somewhat of an arms race as you said. Advertisers and marketers aim to target people who stick to defaults: the ones who search for “top 5 password managers” on Google and click the first result. That is their audience. LLMs are not a complete solution. There are clever ways to use them with well-crafted prompts, and there are simpler, less effective approaches. Those who remain with default behaviors will be absorbed by the system; those who make the effort to resist stand a better chance of avoiding marketing influence.

As an example, some people began adding “site:reddit.com” to their searches in an attempt to get real opinions from real users. I can assure you, marketing firms have caught up with this tactic. Due to widespread astroturfing, I no longer consider Reddit a reliable source.

Ads aren’t only about the blatant banners on the side. There is also SEO blogspam, aggressive affiliate links and marketing, commercial websites trying desperately to sell you their services, and recently, AI-slop.

The only reliable way to filter all of this would be to use an intelligent LLM (ideally run locally) with your criteria in the prompt, filtering out websites and trying to find the “small and/or clean guys.” If you can’t beat AI, join them!

Otherwise, I like to use alternative search engines like Yandex, Qwant, Mojeek, Marginalia, and Wiby. If you’re willing to pay a bit: Kagi is really cool, check it out. I really like old-school webrings too: they are places where you can find a list of websites curated by other people.

But friend, you gotta learn to research smarter. Learn to use search operators, read about blogs that share search tricks such as this one: https://searchresearch1.blogspot.com/

AI will only automate and accelerate fact-checking, at best. If you have no clue about which sources to use, then a less capable AI is even more clueless.

Of course, if people rely on black-box systems in the cloud, I wish them good luck assuming their AI will remain unbiased on controversial topics.

The only saving grace is that there is no single, well-known, authoritative human source with reliably accurate facts on controversial subjects, so at least we’re “no better than AI” in that regard.

Use your brain, your principles, and your own judgment too, for goodness sake.