Wowd Blog

Wowd 2.0 Released!

I am happy to announce the release of Wowd 2.0. 

There are lots of great new features in the release, including a re-vamped UI that we think everyone will find more effective and easier to use. Wowd now lets you control how important the "real-time" factor is in ranking your search results. The default setting, "Best now", emphasizes what people are viewing and talking about right now, showing results that are minutes old, while "Best all time" shows you the definitive long-term results as reflected by our users' collective judgment. We also include a "Hot Topics" tag cloud that provides a window into what the hottest pages in the web are really about.

But things get really exciting in the full version of Wowd. New capabilities include:

Topic Tag Cloud

 Wowd now generates a personalized Topic Tag Cloud on every search. Wowd will analyze the contents of the pages that match any search and show you a list of the topics that are prevalent in those pages. Clicking on any of the tags performs a search on that topic, and a fresh topic cloud is generated for the new results. This provides a great way to refine your search, to drill into "nearby topics" that might more closely match what you are looking for, or to just head off on some interesting tangent.

 This image shows a Topic Tag Clould for a search I did on "privacy" this morning. You can go in a lot of interesting directions by clicking on one of those tags. Note also how fresh the topic cloud is, emphasizing topics that are currently getting a lot of attention. That is because the topics are generated from scratch each time you do a search. Try a search on "privacy" in Wowd yourself and you will get different results because the conversation will have moved on.

SearchStream(TM)

Another innovation is the Wowd SearchStream. When you do a search in Wowd you see the latest results from our continuously-updated index. But new items are entering the index all the time, so the Wowd SearchStream monitors this "stream" of new information and will show you new items matching your search as they come in.  Think about that for a bit: when a member of the Wowd community finds new content that matches your search anywhere on the web -- not just on Twitter or Facebook, but on any web page -- Wowd will bring it to you in real time! This demonstrates one of the benefits of having a local browser application running on your computer: we can execute continuing long-duration tasks on your behalf.

And More

Wowd 2.0 has some other interesting features, such as a full tagging and date filtering in the My Pages web history tab. I'll be writing more about those later. Until then, give Wowd 2.0 a try at www.wowd.com.

Forbes on Wowd

What a great title for an article: "Wowd: Searching The Darkness".  Taylor Buley and Quentin Hardy explore the origins and idea of Wowd in a Forbes article with that title, available today.

Only one comment... The article says that "... [people will] lend [Wowd] their computers and privacy for the purpose of searching the Web".  

In fact, I would argue that the approach that Wowd uses is much, much better for personal privacy than centralized approaches.

Let's say that you use a toolbar issued by a major search company.  Chances are, that toolbar is shipping back to some central location all the URLs that pass through your browser. 

After being captured in this manner, the centralized search provider has a decision to make for any given URL: should they have seen the URL, or should they not have seen it?  That is, your browser toolbar will be sending back URLs that are, in fact, quite private.  You're counting on the centralized authority to recognize its mistake, after the fact, and you're counting on them to keep the private URLs to themselves!  (Will the URLs be deleted?  Or just kept but not used, for ever?  If they're used, then how will they be used?  Who knows!)

With Wowd, a private URL is never, ever shared off your computer.  WIth Wowd, what happens is that when you visit a public web site, the URL for that site gets an anonymous vote, and then from some other point in the Wowd cloud of computers, the page pointed-to by that URL is retrieved and indexed.  The page is not retrieved from your computer.  There's no way to trace back from the retrieval of the page to the person who issued the nomination for the page.

No personal information is shared, no "fingerprints" are left in the system, and there's no way to trace back to any given user the pages that that user votes for in the system.  This is actually a much better approach than what most people put up with today.

So in no way can Wowd be considered "spyware".  Wowd protects individual identity much more effectively than centralized approaches ever could.  At Wowd, we take privacy seriously, and have designed our architecture from the ground up to put you in control of the information that describes what you do and where you go on the web.

Whither the real-time web in 2010?

I had a conversation about real-time search recently, with a journalist, and here's the question that I was asked: Does real-time search solve a problem?

And my answer was: No, not really.  I think that real-time search solves half a problem.

I think that the real-time web is primarily about discovery.  The "reference web" is about stuff that I already know exists.  In the reference web, I search because I already know it's out there, I just don't know all the details that I need to know. 

This is true whether I'm searching for a historical place or event (which often ends at Wikipedia), or if I'm searching for a web site (so-called navigational search), or if I'm searching for details on some product that I'm thinking about buying.  All of these searches involve me knowing something, and wanting to find out more about that something. 

In traditional reference search, when I first arrive at the web page that contains the search box, I already know what I'm going to type into it.

With real-time, my experience largely starts with discovery.  I don't know what's new, noteworthy, or popular, right now, out there on the web.  So when I first engage with a real-time web site, I need to be told what's "out there" on the web.  Whether the new information is about a celebrity, an earthquake, or a product recall, I first need a summary of what's going on -- and that summary informs me with respect to any particular search I might then do.

When I first arrive at the web page that contains a real-time search box, I typically have no idea what I'm going to type into it.

So that's why I said, in my conversation with this journalist, that real-time search solves half a problem.  It's an important half, to be sure, but focusing just on real-time search leaves out the key part, the starting point, which is real-time discovery.

So let me use this observation to springboard into a couple of brief predictions about what will happen to the entire concept of real-time, in 2010.

First, the entire web is real-time.  It's not just about search.  There's real-time discovery, real-time recommendation, and real-time contextually appropriate push.  All the old-school activities that used to happen on a timescale of weeks and months will (or already are) happening on a timescale of seconds and minutes.  We need tools for engaging with this accelerated world of information, and of course, that's exactly what Wowd is doing.

Another thing that we'll see in 2010 is that searching the space of tweets will take on its own special character.  It's a form of vertical search, like automotive search, where you want to be able to search on some specific structured attribute of a vehicle, like a VIN, or body type, or transmission type.  Essentially, there's structure to the stuff you're searching through, and systems that exploit that structure can deliver a better user experience. 

In Twitter, there's obvious structure that can be used in search.  There are people who issue tweets, there's the time of a tweet, there's possibly the location from which the tweet was issued, and there are people who are following the twitterer.  And so on. 

Searching effectively through this sea of information has special requirements, where a user might want to search for something by specifying some combination of "tweet must contain this text", and "twitterer must be followed by this person" or "tweet must have been issued in this time frame".  And so on.

2010 will be an interesting year for real-time.  We'll see increasing diversity in the ways that the real-time web can be embraced, and we'll see increasing power and flexibility to the way that Twitter-specific search can be performed.  It'll be an interesting year.  See you there.

Looking forward to 2010!

Happy New Year, and all the best for 2010 to all Wowd's friends, users, and followers.

Speaking of 2010, Forbes has a great article about some companies to watch and Wowd is number one on their watch list.

We've got a lot of amazing new features coming soon for digging more effectively into the real-time web, and we're looking forward to getting them into your hands.  So please watch this space!

Wowd: A great idea of 2009!

Wowd is mentioned in the NY Times Sunday Magazine today:

http://www.nytimes.com/projects/magazine/ideas/2009/#technology-17

One of the great ideas of 2009!

Crawling the entire web, in real-time?

There's been a lot of news this week about real-time search.  Google has made a number of announcements related to a new real-time search product, and these announcements really have people talking.

Unfortunately, a lot of what's being said about Google's real-time search is confusing and misleading.

Here's our opinion, at Wowd, of recent developments -- in particular, the Google announcements.

Google has said that they will offer a section called "latest results" in a regular Google search results page.  This section will automatically refresh content from sources like Twitter.

Ah, well that's a key phrase:  "... sources like Twitter".  What this phrase really means is, Twitter, and Facebook, and a few hand-picked blogs and news sites.  That's a great list of information resources, but it does not mean that Google is now indexing the entire web in real-time.

While I haven't seen anyone from Google actually make this claim directly (that Google now indexes the entire web in real time) there have been others who have explicitly said this on their behalf.

For example, the Technology Review article "Google Takes Search Real-Time" (Mon Dec 7 2009, by David Talbot) says this: "Gradually, over the past decade, Google has compressed the gap between fresh indexing of the Web from months to mere minutes."

Sorry, but that's technically just not possible.

Google clearly has a crawler-based architecture that's able to make a sweep of the web in some amount of time.  How much time is required for Google to index the entire web?  It's a period of time that's best measured in weeks or months, not seconds or minutes.

Clearly, Google deals with some sites specially.  Google gets page content updates from its AdSense program, from its Urchin-based analytics, from its toolbar, and also from special-purpose information feeds like the one that it now receives from Twitter.

It's clear that Twitter is one source of "real-time" information.  And, depending on what sorts of pages are made public through Facebook, it too can provide good real-time information.  But Twitter and Facebook and a handful of specially selected blogs and news sites do not make up the entire real-time web.

Google is clearly able to refresh some parts of its index in minutes, and other parts in a much longer time frame.

Another article that's quite misleading is "Google launches real-time search" (Mon Dec 7 2009, by Tom Krazit).  In this article, we get the following claim:

"Real-time search at Google involves more than just social-networking and microblogging services.  While Google will get information pushed to it through deals with those companies, it also has improved its crawlers to index and display virtually any Web page as it is generated."

And is supposed to work... how?  The physics of computation preclude an instantaneous crawl of all pages as they're updated or first created.

A more reasonable account of the real-time web appears in "Startups Mine the Real-Time Web" (Wed Dec 9 2009, by Erica Naone).  In fact, the sub-title of this article is "There's more to it than microblog posts and social network updates."

We have two predictions involving Google's incorporation of Twitter results in traditional search results pages:

1. Many people will realize that they don't want Twitter results in their normal search experience.  There's a lot of chatter in tweets, and in general, if people want to search inside the content of tweets, then they know that they can do that, in real-time, at Twitter.com itself.
2. Many people will realize that there's more to real-time search than just having access to Twitter and Facebook and a few news sites.  A lot more. 

On this second point, it's clear now that the entire web is moving to a real-time foundation.  The revolution that started with Web 2.0, where those people formerly known as the audience became active participants in an open, public, high-velocity conversation, has become a global torrent of new information.  That information is not localized to just Twitter and Facebook.  Those two sites represent just a tiny fraction of what's new and interesting and changing in real time on the entire web.

A analysis in this vein is provided in the article "Facebook Will Be Google-able (If Your Profile is Set to Public)" (Mon Dec 7 2009, by Marshall Kirkpatrick).  Marshall makes an excellent point:

"Ultimately, the real-time web remains larger than both of these sources and the newly-included MySpace partnership (also announced today). There is so much implicit real-time data online that few real-time search startups use only explicit data, like shared links, from social networks."

At Wowd, we believe that the challenges of the real-time web are two-fold:

1. A credible real-time service must help people discover what's out there on the web, in real-time, before those people decide to search for something specific.
2. A credible real-time service must implement ranking, in real-time, across all possible web pages, not just Tweets and Facebook pages.

To this end, Wowd offers a user experience in the real-time web that starts with real-time discovery.  Using the Wowd Hot List, you can see what's popular, from pages across the entire web, including the so-called "deep web".

The Wowd Hot List changes in real-time, to reflect what real humans think about different web pages.  Once you see something that's interesting to you, you can use Wowd's real-time search features to dive more deeply into this real-time, web-wide information, finding pages (again, from the entire web) that are just seconds old.

Wowd also incorporates Twitter information, but Wowd uses Twitter information as "meta-data", to help figure out what real people think are good web pages.  Wowd takes tweets, determines which ones contain URLs, and then filters that set according to some re-tweet and follower criteria.  (This helps to ensure that only quality tweets are considered.)

Wowd then de-references URLs in the quality tweets and indexes the pages that the tweets mention, essentially using such information as a 'vote' for the mentioned page.  This small stream of human attention data is folded in to Wowd's own, proprietary, anonymous attention stream, and this combined data is used to rank pages from all over the web.

Wowd uses an incremental link-based distributed algorithm to calculate ranking.  The real-time performance of this algorithm is only possible because of the distributed nature of the Wowd cloud computation engine.

The key point as far as the current discussion is concerned is that Wowd is not a search engine for looking inside the contents of tweets... if you want to do that, we suggest using Twitter's own rather good search system.

Wowd is instead a tool for discovering and then searching the entire web, in real-time, where search results are ranked according to what real human beings think is good.  Twitter is just one (small) input to that overall calculation.

However one ranks search results, either according to popularity or according to freshness, one must take into account relevance.  It's not enough to simply match a search term against a page or a tweet or some other information source, and to return the page or tweet if that term is found.  That approach breaks down quickly.

What's needed is a method of doing ranking in a way that takes into account what a user wants (the most popular information for a given search term, or the freshest), but that also pays attention to how relevant any given source is for the given term.  If you'd like to read more about the Wowd approach to this problem, please take a look at Boris Agapiev's recent blog posting.

If you'd like to see what a real-time web discovery and search engine looks like, please visit us at www.wowd.com.  See what's gaining in popularity on the Wowd Hot List, and try a search and sort your results by Freshness.

Not to pick on Tiger Woods in his hour (or many months) of personal difficulty, but you might try a real-time, web-wide freshness search for web pages that mention him.

Clearly, there are many approaches to real-time search.  At Wowd, we plan to offer you a service that starts with real-time discovery, to allow you to see what's new, and popular, with your fellow human beings.  Once you find things that are interesting to you personally, you can use Wowd's real-time search feature to dig more deeply into that material.

We've got a lot of new features coming soon, to help you do real-time discovery even more effectively.  So, watch this space, as they say.

We'd love to know what you think.  Please feel free to get in touch.

Attention Frontier whitepaper

What's a crawler?  It's a 15-year-old idea about how to build an index, and is part of the foundation of determining what's good on the web today, for pretty well all search-related services you use.

Wowd takes a different approach, based on an analysis of what we call the "Attention Frontier".  This approach has significant benefits in terms of being able to keep up with the volume and velocity of new material entering the web.

You can get the Attention Frontier whitepaper here: http://wowd.com/publications/AttentionFrontier.pdf

The Buzz on Wowd

Wowd has launched its public Beta.  Here's some of the buzz:

InventorSpot, by Ron Callari

Google and Wowd Putting Humans Back Into the Search Equation

ReadWriteWeb, by Jolie O'Dell

Real-time Search Start-ups on Google/Bing/Twitter News:  Who Will Prevail, Who Will Profit?

Daily Finance, Alex Salkever

Meet Wowd, a peer-to-peer search engine that could challenge Google

TechCrunch, by Erick Schonfeld

Wowd Takes a Stab at Realtime Search with a Peer-to-Peer Approach

For more articles, see Wowd's press page

Wowd version 1.3.1 released

Wowd version 1.3.1 has been released. This version contains several bug fixes and performance improvements.

It was brought to our attention that the Wowd web application contained a bug that created a "non-persistent" cross-site scripting security vulnerability. We fixed this bug on our web site on October 28, the day after it was discovered, and have fixed it in version 1.3.1. To our knowledge, no exploit for this bug was actually attempted and none of our users were affected.

As of tonight, the 1.3.1 update has been applied to approximately 98% of running Wowd instances through the auto-update mechanism.

Please contact me or email us at feedback@wowd.com if you have any questions.

Wowd launches today

Today, Wowd will launch its public Beta at this year's Web 2.0 Summit at San Francisco's Westin Hotel.

We invite you to visit Wowd.com to see our new look and feel.  Use the "feedback" button to tell us what you think.

Now that Wowd is open to the public, please tell your friends to visit Wowd.com so that they can also enjoy fresh, real-time search results from sources across the entire web as well as Wowd's Hot List of popular pages.