Defense Advanced Research Projects Agency (DARPA) is developing a new search engine that is designed to provide search for the hidden content in the so-called “shadow” of the Internet. The project, called Memex, now employs 17 independent development teams.
The head of the project Dr. Chris White main problem of traditional search engine that tries to solve the developers called generics approach to the Internet, in which search results are based on rankings and consumer advertising.
As is known, conventional search engines like Google and Bing, able to index only 5% of all Internet.
Memex should cover the content of millions of resources, ignored by commercial search engines, as well as thousands of undocumented websites operating in Tor-networks, which allow to establish anonymous connections and access blocked Internet censorship information.
Applying latent indexing sites, experts from DARPA trying vyvit regularities and patterns of space “shadow” Internet segment. Ultimately, this information should help law enforcement agencies to track illegal activity.
Another goal of the project is a more accurate estimate of the number of hidden resources, operating in a network TOR. According to Dr. Chris White, the agency’s preliminary estimate of their number amounted to 10 thousand. Web resources. Now the project specialists believe that a more accurate number of such sites – about 40 thousands.
Memex project developed over the last year. To date, in tests involving 8 patterns and debug prototypes, but the details of the project were not disclosed.
We only know that the final cost of the development of such a search engine can be up to several tens of millions of dollars. If the search engine on the “shadow” Internet all the same runs, he will be able to track not only find and index the hidden web resources, but also to keep track of available connections between them.
All this should provide invaluable assistance to law enforcement and intelligence agencies to combat those Internet sites, with which there is trade in drugs, arms, obscenity, stolen data, etc.
Through a traditional search engine to find similar sites can only be accidental, and it is likely to come across a direct link in any forum or blog. Websites mentioned specifically block indexing of your pages by search engines in order to get to them could only be for a direct link.
Now, if the project is all the same will be possible, a special search engine will still index such resources and “pull them out of the shadows.”