Is Marissa Mayer Restarting Yahoo Search?

In July, Yahoo announced that they had hired Google famed Marissa Mayer as their new CEO. Unlike everyone else in the tech sector, I was disappointed. My disappointment stemmed from the fact that for the last several years Yahoo has been the surefire place where things go to die. This made me sad because Mayer has always been at the center of innovation as Google’s Vice President of Search Products and User Experience during the company’s most radical growth period. Yahoo seemed like a step down. But at the same time I have been hopeful that she could turn Yahoo around and build something amazing. Then this weekend something weird happened…

Over the weekend I was checking the traffic stats for one of my less serious projects with Statcounter and saw something odd in the visitor report:

When I saw “Yahoo! Slurp 3.0″ I was shocked and confused. Slurp is the name of Yahoo’s web crawler. Version 3.0 launched in April of 2008. However, almost a year later Yahoo quit crawling the web and replaced their results with Bing.  Because of this I was almost certain that someone was spoofing their user agent to hide their identity. In fact many of the commercial scrappers on the market allow you the ability to change your user agent to what ever you want. There is even a handy little browser plugin that does this as well.

I posted a status on Facebook and soon learned that my buddy Meg Geddes has been seeing the same thing in Statcounter.

A buttload?? This got me thinking….so I went back to my Statcounter login and drilled down a bit further and this is what I found:

From the above we can see that the request is absolutely coming from Yahoo, that the client has Javascript enabled, and that a screen resolution is recorded. At first glance this looks like a typical request from a browser such as Chrome or Firefox. But why wouldn’t they just disclose that in the user agent? Because maybe this is a browser/crawler hybrid?

So this morning I wake up and can’t stop thinking about Meg’s buttload…Its one thing to see this type of behavior in random intervals, its another to have a buttload. (Also, I like repeating the term “buttload”, I am so sorry.) So, to get further to the bottum of this, I sent Meg an email with some questions:

JH: When did you first start seeing Slurp in your reports this year?

MEG: I don’t know when they started, but I probably first noticed around late spring.

JH: Is there ever any Referring URLs?

MEG: No there are never any referring URLs.

JH: How many pages per domain does it typically visit?

MEG: Probably around 50-100 per day across many sites. Interestingly, I don’t think they’re showing up in Google Analytics, so it may well just be a Statcounter thing.

JH: Do your server logs show the same type of data that Statcounter does?

MEG: Yep.

JH: Peanut butter: crunchy or creamy?

MEG: Dude. Extra Crunchy Jif. Sometimes I add MORE peanuts. I’m hardcore like that.

Meg also shared with us this forum posting where Statcounter moderators point out that these might be image crawling bots which are indexing the 1 pixel image Statcounter uses to send visitor data back to their servers. However, last time I checked (just now), Yahoo’s image search still says, “Powered by Bing™” at the bottom.

Yahoo’s alliance with Bing is supposed to continue til 2019, but that doesn’t impede on Yahoo to continue developing their own internal search products. In fact, they could be developing something that integrates with a future product set that is not search related at all.

Either way its fun to think that maybe somewhere on the horizon there is the possibility of a more competitive search landscape with Mayer commanding a serious competitor. If that is ever the case, then shes already got a new fanboy!

Comments

  1. What do you think is on the horizon then? Are yahoo going to rebrand. You’ve gotten me interested when you’ve said “they could be developing something that integrates with a future product set that is not search related at all.” Are you thinking social network?

    • Honestly, I don’t know. I know they have put a lot of effort into monetizing properties with exclusive content, so maybe something the combines content and search… Your guess is as good as mine.

  2. Now, this is interesting. Have we noticed anything different in results? Yahoo and Bing have been pretty consistent up to now.

    Marjory

    • No, I haven’t. And I doubt that Yahoo would ever back out of their deal with Bing…or at least integrate their own results with out Bing’s knowledge….they both have way to much at stake to not be transparent with each other.

      • It’s actually not outside the realm of possibility that Yahoo might back out of its search deal with Microsoft.

        Danny Sullivan published a great piece on the deal back in July: http://searchengineland.com/yahoo-microsoft-search-alliance-google-127843

        In the article, Danny covers the RPS Gap, which is one of the biggest out clauses in the search deal. He also discusses various hypothetical situations (e.g., Yahoo buying Bing, Yahoo pursuing a search deal with Google, etc.).

        I’m not sure how things have evolved since July, but it’s always fun to speculate ;-)

  3. Imagine a world where Yahoo Site Explorer is running again. Good old days. Ahhh I digress. Good catch Joe.

  4. “From the above we can see that the request is absolutely coming from Yahoo, that the client has Javascript enabled, and that a screen resolution is recorded. At first glance this looks like a typical request from a browser such as Chrome or Firefox. But why wouldn’t they just disclose that in the user agent?”

    BTW, It’s possible to make crawler executing Javascript. I even know how to do that with open source frameworks ) The only big problem is performance.

  5. Whatever it is, we all need yahoo, bing is going pretty good, google is a search engine for ads mainly, totally useless now, so sad. Yahoo has a lot of potential and I didnt use them for years, then after the google search failure this year I started to use yahoo, and guess what, I actually find stuff, bing is nice also but I personally like yahoo more, I just wish they clean up that home page

  6. I’m sure I’m wrong on this, but I thought the deal with Bing was to use their algorithms, not necessarily their index? If so, Yahoo will still have to do their own crawling.

    Failing that, assuming the cost of running that crawler now its developed isn’t too high, it might well be nothing more than a safety net:- Any company that relies completely on one separate third party to function is in a risky position (Hi SEO’s!) , it could well be an arse-covering exercise in case Bing ever goes belly-up.

  7. Nice catch, Joe! Some interesting possibilities are certainly conjured up.
    In Danny’s piece last July, he said:
    ‘It won’t be Yahoo. Yahoo has lost too much key search talent and hasn’t kept its core search technology up-to-date. When it gave itself over to Microsoft, Yahoo really left itself without a “Plan B.”’

    At the time, I would have agreed with him. but Marissa is the wild card that is known for doing things that popular wisdom says can’t be done. Her Plan Bs are sometimes better than our Plan As. Personally, I think that if anyone can restore Yahoo to a strong business position, she’s it.

    Whether that position will be in search or not, of course, is totally up for grabs. My bet is that we’ll all be caught off-guard by an unexpected new direction.

    Do me a favor… save me a front row seat at the next MM Fanboy gathering. And BTW, congrats on the new gig!

Trackbacks

  1. [...] Is Marissa Mayer Restarting Yahoo Search?, joehall.me [...]

Speak Your Mind

*