Changing search on the web is less daunting than it sounds (getting rid of entrenched incumbents aside). There is no law of technology stating that search has the work the way it does today. The architecture we have is a mix of historical accidents, laziness from technologists who are slow to spot problems, and deliberate decisions by a small handful of powerful companies. We can make different decisions, and the alternatives are feasible.
Berjon has a lot of great ideas here, and the article is worth reading if for no other reason than the excellent explanation of how search currently functions.
That said, I’m not sure how you spend this much time thinking and writing about how to fix search on the web without addressing the question of how search gets paid for. I’m not going to defend the practices of the tech giants that currently own the search business—they’re awful!—but it seems to me that a core problem here is that the code and infrastructure that is required to support a search engine is substantial, and building and running it costs money. On top of that, our stupid economy expects companies to not only make more money than they spend, but to make ever-increasing amounts of money in perpetuity, and it actively punishes companies that do not show never-ending growth.
If you suggest that people ought to pay for search, you get laughed out of the room, and probably rightly so. I generally believe that if a product or service is valuable, then you ou should pay for it, but my guess is that any company that tried this strategy would last about a month. Consumers expect this stuff to be free.
Even if you could get people to subscribe to your web search service, the market would look for your subscriber numbers to always be growing, which makes for all sorts of weird incentives. You would probably still end up with a broken product.
So the tech companies that build search attract users by not charging them for using it, and then make money by selling advertising against the service and mining the users’ data. And then they still have to show the market perpetual growth, so they turn to monopolistic practices like browser lock-in to keep growing their user-base.
As long as all of these dynamics remain, none of these great ideas ideas for technical fixes to the problem of search are going to make a lick of difference, because the underlying economics driving the whole thing have not changed.