Bing-Windows Mobile “conspiracy”, with screenshots

October 9, 2009

Yesterday in our Directions editorial meeting, another analyst noticed how Microsoft’s search engine, Bing, offers a “Reference” search option (on the top toolbar, click “More” then “Reference”) that returns “enhanced” Wikipedia pages, right on the Bing site. (See the example entry for “Microsoft.”)  You can still click through to the original Wikipedia entry, but he thought it was odd that there was this extra page. Following Microsoft and being subject to their bizarre efforts at spin is enough to give anybody a conspiratorial mindset, so he checked the “enhanced” entry for “Microsoft” against the original entry on Wikipedia. There was no evidence of foul play–the Microsoft “enhanced” version was identical to the original, including negative references about the company.

But that got me wondering. What about regular search results? Might Microsoft be tempted to tweak them a bit to put its other products in a more positive light?

To test it, I entered “Windows Mobile 6.5 reviews” on Bing. The latest version of Microsoft’s OS for smartphones launched on Tuesday, and if you follow the mobile space at all, you’ll know that it was almost universally panned. (The Gizmodo and CrunchGear reviewers were particularly brutal.)

Search results change frequently for the same queries as indexes are updated, so I conducted the search this morning and took a screenshot, which is here.

Bing results for the query "Windows Mobile 6.5 reviews," as of about 11 a.m. on 10/9

Bing results for the query "Windows Mobile 6.5 reviews," as of about 11 a.m. on 10/9

Lo and behold–the first page of search results is surprisingly neutral, with neither of these brutally negative reviews anywhere on the page, and very neutral headlines. While clicking through to some of these headlines leads in fact to a fairly negative review, the headlines and exerpts themselves reveal nothing about the widespread negative sentiment being expressed about Windows Mobile 6.5.

Now compare that with the equivalent Google results:

Google's results for the query "Windows Mobile 6.5 reviews" as of about 11 a.m. on 10/9.

Google's results for the query "Windows Mobile 6.5 reviews" as of about 11 a.m. on 10/9.

Ouch! The top news headline notes how the product has been “stung with bad reviews.” That’s followed by the Gizmodo headline (“There’s No Excuse For This”), TechCrunch headline (“It Still Sucks”), and, toward the bottom of the page, a PC world headline (“Mostly Disappoints“).

One possible conclusion you could draw: Google is purposely over-weighting negative reviews because it’s a Microsoft competitor. So I did the same test with two products that have gotten mostly positive reviews, Windows 7 and the new Zune HD.

Here are Bing results for “Windows 7 reviews”:

Bing results for "Windows 7 reviews" as of 11 a.m. on 10/9.

Bing results for "Windows 7 reviews" as of 11 a.m. on 10/9.

The top news result is striking for its complete irrelevance–it’s a story about a tapas restaurant in Nebraska–but the other results on the page include at least one positive headline from Computerworld, although the story’s very old (“This time Microsoft gets it right“), alongside various other more recent reviews with neutral-to-positive headlines and summaries. (For what it’s worth, when I did this search yesterday, Walt Mossberg’s mostly positive review came up tops in the news results.)

What about the Google results for the same query?

Google results for "Windows 7 reviews" as of  about 11 a.m. on 10/9.

Google results for "Windows 7 reviews" as of about 11 a.m. on 10/9.

Also mostly positive, although the top news story is actually criticizing Mossberg. But overall, lots of positive headlines like “Mossberg Says Windows 7 is the Best Version…” and “You Can Quit Complaining Now.”

Last test. Here are Bing’s results for “Zune HD reviews”:

Bing results for "Zune HD reviews" as of about 11 a.m. on 10/9.

Bing results for "Zune HD reviews" as of about 11 a.m. on 10/9.

Oddly enough, here the top Bing news result is…talking about poor reviews for Windows Mobile 6.5! But below that, the first link is a positive headline from a Zune fan site (“Sets Zune Apart“), and the other headlines are mostly neutral.

Here are the Google results for the same query:

Google results for "Zune HD reviews" as of about 11 a.m. on 10/9.

Google results for "Zune HD reviews" as of about 11 a.m. on 10/9.

Very positive headlines, including Donald Bell’s “tears of joy” (from a preview glimpse; here’s his full review, which also shows up on page 1 of results) and Ars Technica’s “Third time’s the charm.”

So here’s a quick recap of this very unscientific test. For a Microsoft product that has received almost universally negative reviews, Microsoft’s search engine doesn’t show the worst reviews and gives no first-glimpse impression of the widespred negative sentiment about the product. For two Microsoft products that received mostly positive reviews, Microsoft’s search engine showed at least one headline that clearly conveys a positive first-glimpse impression of the product.

Bias or innocent? Let’s look at how this could be happening.

The first and most likely scenario is that Bing indexes and ranks reviews differently than Google. If you look at all three sets of search results, Bing seems to offer on a higher proportion of what I’d call relatively obscure sites: Ubergizmo? WMExperts? Asktechman? I’d never heard of any of these sites before finding them on the Bing results for Windows Mobile 6.5 reviews. Same with Windows7Review.com for the Windows 7 query, and Blogoncherry/boncherry.com (which turns out to be an invalid result) for the Zune HD review. Bing results do include some higher profile sources like Engadget, CNET, and Computerworld, but overall there’s a high proportion of obscure sites. Google, in contrast, sticks primarily with the tried and true: Gizmodo, CNET, Engadget, PC World.

This might be a purposeful choice on the part of Bing’s engineers–perhaps they feel that consumers want more variety of opinion, so purposely pull in less-popular sites. Or it could be a flaw with Bing’s relevance-ranking algorithms, which may not be as good as Google’s. (That Omaha tapas restaurant review and the boncherry.com page suggest the latter.)

The second possible reason–less likely–is that the Windows Mobile team did some serious search engine optimization as part of its marketing for the launch of Windows Mobile 6.5 this week.  SEO is a dark art that I don’t understand very well, but it might be possible to influence which results appear near the top of particular search queries by judicious use of cross-linking. For example, if Microsoft wanted certain positive or neutral results to appear more highly in results, it might create its own Web pages that link to these reviews. (I’m not alleging they do this, nor that it works–I actually have no idea.) In this scenario, the conclusion would be that Google is more resistant to efforts to game the system than Bing is.

The third possible reason, which I’m almost sure did not happen, is that somebody in the Windows Mobile team called up Bing engineering and told them “hey guys, can you please tweak your algorithms to suppress those really negative reviews of Windows Mobile 6.5?” Equally unlikely is that somebody on the Bing team did this of his/her own accord, reasoning that Windows Mobile needed all the help it could get. Why am I almost sure this didn’t happen? Mainly because Bing and Windows Mobile are in completely different parts of the company. Microsoft’s product teams are notoriously averse to cross-company coordination–they’re not evaluated on it, and therefore have no incentive to help out like this at the expense of hurting their own product. If the Windows Mobile team were craven enough to make such a call, the Bing team would almost certainly tell them to pound sand.

Whatever the reason, this simple experiment shows me that Google, for whatever reason, does a better job of delivering an accurate snap-impression of queries related to Microsoft products.


Follow

Get every new post delivered to your Inbox.