Twitter Search is Failing Us: Is There a Solution?

One of the crucial parts of using Twitter and social media for business intelligence is is search. The problem is that when using Twitter search for information is that Twitter holds all the cards, and I think Twitter is dealing from the bottom of the deck.

Here are a couple of examples. First my Twitter stream (it opens in a new window), then a search on Twitter for “from:trishussey”. Notice anything, um, missing?

Yep, none of my tweets are indexed by Twitter. This isn’t new, of course, Corvida Raven wrote about this very issue last October. The problem is that Twitter’s “solution” to his is rather pathetic.

You can contact support to let them know that you’re being filtered out, and now (from an April 28th update) that “resource constraints” are causing a part of the problem:

Update 4/28/10: We’ve updated this help page with additional information on current resource constraints. We have recently seen an increase in users missing from search, and our engineers are working on this issue. We will post further updates here.
link: Twitter Support : My Tweets or Hashtags Are Missing from Search!

This is just the tip of the iceberg.

Okay, some unknown number of people are being filtered out of Twitter search results (this would also include their hashtags). Google can take up some of the slack, right? Sure, that should work. If you can use Google blog search to save as an RSS feed.

There is another problem: the depth of Twitter search. We know that although Twitter has said that they are keeping all tweets (somewhere), but try searching for something more than a few weeks back. Yep, nothing.

The remaining question is then: what now?

Part of the solution is up to Twitter. I suspect Twitter is going to have to build a partnership with Google or (gasp) Microsoft for the resources to bulk up their resources. The other part is learning how to attack search in a way that will try to minimize the potential bias and gap in your search results.

Me? I use Yahoo Pipes. I published a Pipe that pulls in several separate RSS feeds, filters them, and republishes them. That’s just a sample. I have other tricks up my sleeve, of course, and believe it or not, I’ll be sharing them as part of the Social Media Club Vancouver event on Thrusday—Social Media 101: Strategy, Business Intelligence & Metrics.

There is an open question if companies could be provided with special API keys to tap into deeper search results. The flaw in that is even another company would need tremendous resources to store and serve all the thousands, millions, billions of tweets that would need to be indexed.

Until then, search, search broadly, search well.

Oh, and Twitter, add me back into the index please.