What is the Deep Web?

The internet iceberg

Believe it or not, the majority of the World Wide Web exists beyond the reach of Google’s search bar. Familiar websites you know and love (like Facebook, Youtube, Wikipedia, and The Beacon) make up less than 1% of the entire internet. Everything else is part of the Deep Web—pages, domains and files that can’t be accessed by standard search engines. Before we delve any deeper, let’s take a look at how the surface web actually works.

Crawl. Index. Serve. Repeat.

Search engines like Google, Bing and Yahoo aim to connect their users with relevant webpages. Their software crawls billions of pages, traversing the web with the help of sitemaps and hyperlinks. After getting a feel for the structure of each site, the engines index (organize) every page based on the content it contains. Finally, they display the indexed sites that best match your search terms.

The pages beyond

There’s just one caveat: search engines can only index static pages. A static page (like the one you’re on now) stays the same once it’s published. Most search engines can’t process dynamic pages, which display different content each time they’re accessed. Dynamic pages can range from the search results page of a database to the inbox of your password-protected Gmail account.

Into the deep

So what lies beneath the surface? Well, there’s a lot of junk. Broken links. Inaccessible intranet sites. Private Facebook messages. Loads of seemingly-useless data dating back decades. But there’s also important information—scholarly articles, public records, research studies—just out of Google’s grasp. Unfortunately, the Deep Web is also home to the internet’s more sinister side.

It gets darker

The Tor network has built a reputation as the deepest, darkest corner of the web. But, its intentions aren’t all bad. Tor (short for The Onion Router) runs on a network of servers that allow for maximum online anonymity. Reporters use the network to communicate with whistleblowers or political dissidents in countries where censorship reigns supreme. Corporations take advantage of Tor to protect sensitive business communications. Even the U.S. Navy uses the network for open source intelligence gathering.

The vast majority of Tor users are drawn to the network solely for the ability to speak and search anonymously. But there are also a few seedy sites that facilitate black market transactions using virtual bitcoin currency. These sites offer everything from illegal weapons to narcotics and human organs. Pretty spooky stuff.

The future of invisible internet

Though it’s often misrepresented as a slimy underworld for illegal activity, most of the Deep Web is more helpful than harmful. It hosts a deluge of scholarly resources and databases loaded with everything from artwork to literature anthologies. It’s a platform for people who can’t safely speak out against injustice. And it’s rife with ways to protect personal data and prevent invasive web traffic monitoring. As interest in big data grows and law enforcement agencies crack down on the darker side of the Deep Web, we’ll likely discover a whole new world of possibility just beneath the surface.

Sources:

• http://money.cnn.com/2014/03/10/technology/deep-web/index.html
• https://moz.com/beginners-guide-to-seo/how-search-engines-operate
• https://support.google.com/webmasters/answer/70897?hl=en
• https://www.torproject.org/about/overview.html.en
• https://www.torproject.org/about/overview.html.en