Wowd Blog

Wowd whitepaper on real-time search

Boris (our founder and CTO) has packaged-up his blog postings on real-time search, and we're releasing the result as a Wowd whitepaper. 

The whitepaper is available here: http://wowd.com/publications/rts.pdf.

We're very interested in these topics (as you'll have guessed by now), so if you have thoughts, comments, or an opposing view... please get in touch.

A "sliding window"? What kind of search is that?

Our founder and CTO, Boris Agapiev, has written his third blog post on the topic of real-time search. 

In this installment, he essentially asks the question: if a real-time search system is just filtering a stream of information against a given query, is it really search in any useful sense? 

And more generally, he looks at the relationship between the very limited notion of matching offered by today's "real-time search engines", and true indexing-based search. 

You know you need the answers: http://distributedsearch.blogspot.com/2009/08/sliding-window.html .

Real-time search: RAM or Disk?

In his second installment of a four-part whitepaper on real-time search, Boris Agapiev explains the potential role of Random-Access Memory in building a fast and adaptive index, and contrasts this with the traditional use of hard disk storage.  (By the way, this is way cooler than you might first think... ;) )

What the heck is "real-time search"? (Historical Foundations)

We've been seeing ever-more discussion of "real-time search", and have decided that it's time for us to enter the fray.

Fundamentally, all the soi-disant real-time search engines that we've seen really aren't combining recent information and historical information using any meaningful ranking mechanism.

So Boris Agapiev, our founder, CTO, and VP of Search, is writing a series of blog posts intended to lay out some of the history behind search and indexing, and to then use this historical foundation to explain what "real time search" is to most people, and what it should be (and indeed, what it is to us here at Wowd).

Boris' first post is available now.  He'll be posting more on this topic over the next few days, and we'll roll-up all his blog posts into a Wowd whitepaper, for general distribution.

You can follow us on Twitter if you want to be notified of these blog posts as they become available (we'll also send a Tweet when the full paper itself is ready).

Toolbar data, local computation, and privacy control

Doug Sherrets (VentureBeat) wrote a comprehensive analysis on the nature and value of "toolbar data".  It's a great article.  In particular, I like the comments he's collected from a number of different people in the industry with respect to their perspective on the nature and value of toolbar data.

It's clear that toolbar data (as he defines it) is indeed valuable.

By understanding the pages that real people like, and by knowing where real people actually spend their time, one can implement a fantastic analysis to deliver better search ranking (ordering results when someone knows what they're looking for) and discovery & recommendation (figuring out what someone might like, when they're not sure what to look for).

The sort of data needed to support such analysis can be naturally captured in a browser toolbar.  But algorithms must have access to the data from all users who participate in the ranking or recommendation process.

So, provided that you're prepared to ship all that critical data from your personal computer to some centralized processing system, then the algorithms can work their magic.

But sending toolbar data from each user's personal computer is a process fraught with privacy concerns.

So, at Wowd we take a different approach... no toolbar data comes off the local machine, but the user still benefits from web search ranking, recommendations, etc., that use anonymous site "vote" data.

Roughly, it works like this.

I'm a member of the Wowd network, and I visit a new web page. My local Wowd client first checks to see if that web page is publically available -- meaning, can other people out there on the web see the same page that I’m seeing.

If I’m visiting a public page, then the site is nominated for inclusion in the Wowd index. No personally identifiable information leaves my machine!

By visiting the site I'm simply and implicitly voting for it. That's all. The indexing of the publically available site is done from another machine in the Wowd network, not mine.

So Wowd users get all the benefits of a system that understands toolbar / click-stream data, without having to actually share a single scrap of personally identifiable information. The results are better for our users, and the privacy is better too.