Finally, I believe that any site with an a large number of pages — such as a corporate site with products, information and support, or an offroad vehicle parts store — should provide a search option. I find that some people just don’t process links on pages and prefer to search, other people want to skip multiple levels of navigation and go straight to a known item, and yet other people are trying to understand the scope of the site and what content it covers.

Improving search is great. Insulting search is not necessary.

From The Anatomy of a Search Engine section 4.5.1

In order to rank a document with a single word query, Google looks at that document’s hit list for that word. Google considers each hit to be one of several different types (title, anchor, URL, plain text large font, plain text small font, …), each of which has its own type-weight. The type-weights make up a vector indexed by type. Google counts the number of hits of each type in the hit list. Then every count is converted into a count-weight. Count-weights increase linearly with counts at first but quickly taper off so that more than a certain count will not help. We take the dot product of the vector of count-weights with the vector of type-weights to compute an IR score for the document. Finally, the IR score is combined with PageRank to give a final rank to the document.

Our current thinking hasn’t changed much since 1997. Local Search is only necessary if you can’t make the investment in ensuring the right links are on the right pages. Some local search can be very inexpensive (such as tying google.com search into your site as we’ve done on uie.com), so it may be the more cost-effective investment. (Warning: Google works because of some parlor tricks that only succeed because the Internet has billion of links. Networks removed from the Internet, such as an intranet, don’t work so well with Google.)

We still recommend our clients solve findability issues with better links, not better Search. Better Search will always be fixing the symptoms, not the problem, and is unlikely to ever reach desired goals of success.

The best site in the world would not need a in-site search engine cause everyone would find what they need right away.

When you look at the userbility view of things, a search engine should in theory only be used when you dont want your users to find stuff throu the navigation.
And when looking appart fron VERY BIG sites, when do you do that? Besides.. even on very big sites.. what people search for is the “not-deep-down” things, but still basics like contact, adress, products etc.

The point is … In-site search engine usage ofte indicates the lack of userfriendlyness of the site, why people tend to “run home to mama” and use the search engine.
Improving the in-site search engine is solving the symptoms, not the illnes.

At least that’s my 2 cents….

But yea, you are right, the in site search engines are pretty bad. For many reasons, mostly because Google or MSN can be improved in one single point, and can do ROI.
An in-site search engine can not prove its ROI that easy. You can’t sell banner space on your in-site search engine … if you could would you? So what is the point in making it good? It doesn’t sell your products – alt least not directly – and it doesn’t make more ppl come to the site? At least that’s what ppl tend to think.
This in terms means that you are not motivated to improve it. Unless you have a high standard for your site or other independent reasons come in to play.

Some times search engines are so bad, I use Google to search on some sites. The syntax is btw, “site:www.yoursite.com Keyword1 keyword2 etc.” in Google….

But again.. It should not be necessary.