Research Sources for the Deep Web

Posted on February 4, 2009
Filed Under online search, online tools, search engine, web access, web evolution | View Comments

deep-web
Photo credit: ANDI2..

The Invisible or Deep Web is represented by all kind of content not being indexed by broad search engines, even if it’s actually online. In this sense is a complement of the Surface Web.

Just think of the Internet as an ocean to fish in for information. You can either drag your Net on the surface (and still catch a vast amount of information) or dive deeper for the most rare, most useful items for your niche of interest.

Estimates of the New York Times set the Surface Web over 8 billion pages; search engines are said to crawl only 16-20% of the Internet.

The Deep Web is mainly about informations stored in databases, like directories, dictionary definitions, items for sale in a Web store or on Web-based auctions, patents, laws, and anything that can be located in subject directories.

So how to search deeper?

Sometimes you just have to go through a “split search“: first, you search for a site on a common search engine (say, you google “United States newspapers“). By doing so, you retrieve the site of NewsDirectory, a database of links to newspapers sites.

To get further in this topic you can take a look at Deep Web Technologies, a company that has developed some databases, mainly for science and business research.

You can find a vast list of Deep Web resources on the Online Education Database, or on DeepWebResearch.

For scholarly content, look for Questia, a really large online library with journal articles not available elsewhere.

Or visit Turbo10, to search more than 800 deep search engines at the same time; and also Direct Search.

This list is definetely not meant to be complete…
so if you have anything to add or report, your feedbacks are welcome.

Comments

  • Hi Victor,

    Many thanks for the mention of Deep Web Technologies. We kicked off our Notably Accelerated Research (NAR) campaign for the Deep Web with Biznar.com and Mednar.com last year. This year you'll see Lawnar, PharmaNar, GreeNar, EnerNar and others come to the surface. The deep web is about to become significantly more accessible!

    Also, if you have suggestions for a Deep Web vertical search engine, please feel free to email me. darcy at deepwebtech dot com
  • Hello Darcy

    thanks to you for the info about your company project,
    which i find really interesting.

    looking forward for the developments you mentioned.
    i'll stay tuned.

    victor
blog comments powered by Disqus