The Deepweb

[Guile]

The Deep Web (also called Deepnet, the invisible Web, DarkNet, Undernet, or the hidden Web) refers to World Wide Web content that is not part of the Surface Web, which is indexed by standard search engines.
Mike Bergman, credited with coining the phrase,[1] has said that searching on the Internet today can be compared to dragging a net across the surface of the ocean: a great deal may be caught in the net, but there is a wealth of information that is deep and therefore missed. Most of the Web's information is buried far down on dynamically generated sites, and standard search engines do not find it. Traditional search engines cannot "see" or retrieve content in the deep Web – those pages do not exist until they are created dynamically as the result of a specific search. The deep Web is several orders of magnitude larger than the surface Web.

In 2000, it was estimated that the deep Web contained approximately 7,500 terabytes of data and 550 billion individual documents.Estimates based on extrapolations from a study done at University of California, Berkeley in the year 2000, speculate that the deep Web consists of about 91,000 terabytes. By contrast, the surface Web (which is easily reached by search engines) is about 167 terabytes; the Library of Congress, in 1997, was estimated to have 3,000 terabytes. More accurate estimates are available for the number of resources in the deep Web: He et al. detected around 300,000 deep web sites in the entire Web in 2004, and, according to Shestakov, around 14,000 deep web sites existed in the Russian part of the Web in 2006

Deep Web resources may be classified into one or more of the following categories:
Dynamic content: dynamic pages which are returned in response to a submitted query or accessed only through a form, especially if open-domain input elements (such as text fields) are used; such fields are hard to navigate without domain knowledge.
Unlinked content: pages which are not linked to by other pages, which may prevent Web crawling programs from accessing the content. This content is referred to as pages without backlinks (or inlinks).
Private Web: sites that require registration and login (password-protected resources).
Contextual Web: pages with content varying for different access contexts (e.g., ranges of client IP addresses or previous navigation sequence).
Limited access content: sites that limit access to their pages in a technical way (e.g., using the Robots Exclusion Standard, CAPTCHAs, or no-cache Pragma HTTP headers which prohibit search engines from browsing them and creating cached copies).
Scripted content: pages that are only accessible through links produced by JavaScript as well as content dynamically downloaded from Web servers via Flash or Ajax solutions.
Non-HTML/text content: textual content encoded in multimedia (image or video) files or specific file formats not handled by search engines.
Text content using the Gopher protocol and files hosted on FTP that are not indexed by most search engines. Engines such as Google do not index pages outside of the HTTP protocol.

To discover content on the Web, search engines use web crawlers that follow hyperlinks. This technique is ideal for discovering resources on the surface Web but is often ineffective at finding deep Web resources. For example, these crawlers do not attempt to find dynamic pages that are the result of database queries due to the infinite number of queries that are possible.It has been noted that this can be (partially) overcome by providing links to query results, but this could unintentionally inflate the popularity for a member of the deep Web.

-Copied from Wikipedia.

In plain English, and "For those who don't know, the deep web represents a gargantuan part of the internet which is not accessible through regular searches via google or other search engines.
Searching on the Internet today can be compared to dragging a net across the surface of the ocean. While a great deal may be caught in the net, there is still a wealth of information that is deep, and therefore, missed. The reason is simple: Most of the Web's information is buried far down on dynamically generated sites, and standard search engines never find it.

Here are some facts on The Deep Web:
· Public information on the deep Web is currently 400 to 550 times larger than the commonly defined World Wide Web.
· The deep Web contains 7,500 terabytes of information compared to 19 terabytes of information in the surface Web.
· The deep Web contains nearly 550 billion individual documents compared to the 1 billion of the surface Web.
· More than 200,000 deep Web sites presently exist.
· Sixty of the largest deep-Web sites collectively contain about 750 terabytes of information — sufficient by themselves to exceed the size of the surface Web forty times.
· The deep Web is the largest growing category of new information on the Internet.
· Deep Web sites tend to be narrower, with deeper content, than conventional surface sites.
· Total quality content of the deep Web is 1,000 to 2,000 times greater than that of the surface Web.
· Deep Web content is highly relevant to every information need, market, and domain.
· More than half of the deep Web content resides in topic-specific databases.
· A full ninety-five per cent of the deep Web is publicly accessible information — not subject to fees or subscriptions.
What lies beneath the surface is a who's who of hackers, scientists, drug dealers, astronomers, assassins, physicists, revolutionaries, Government officials, Police, Feds, terrorists, perverts, data miners, kidnappers, sociologists, etc. "


What do you guys think, that there is actually an entire layer of internet below the visible internet? Sure, a large portion of it is just unqueried databases, encrypted files, and unpublished pages, but what about things such as Silkroad, where you can essentially buy any drug imaginable? Or the fact that the deepweb is essentially where the major viruses are born, anonymous operates (4Chan is just scratching the surface), and the vast stores of hidden files/data.

I'm sure we all know about Wikileaks, that came out what, six months, a year ago? That was big on the deepweb far before it even popped up on the surfaceweb. The deepweb can be hard to access, and you often need to the actual address, most pages are rarely linked together. And sometimes you might see a little vein of the deepweb popping up (such as Wikileaks), but just imagine, what else exists down there?


The internet is an iceberg, you can see that bit at the top, Youtube, Facebook, Teenhelp, but that's maybe 5% of what really exists...

[TigerTank77]

I have seen things, man.

[BDF]

I've known this for a while. I'm surprised google with it's team of geniuses haven't come up with a search engine yet that can somehow inflitrate this "deep web" more efficiently than doing it in person.

Came into contact with it all the way back in school.

[dr2005]

Doesn't surprise me in the slightest, to be honest. It's no different to traditional data archiving systems in that respect - if you think of a central library, for example, the vast majority of the stuff they own probably isn't on display because it's called upon so rarely. The problem comes when, as is becoming increasingly obvious, it's also being used to cover for the less salubrious parts of the Internet. But no, the actual notion of there being another, quite sizeable section of information not readily accessible is not something I find particularly shocking. What's actually stored in there, on the other hand...

[Guile]

Originally Posted by dr2005

Doesn't surprise me in the slightest, to be honest. It's no different to traditional data archiving systems in that respect - if you think of a central library, for example, the vast majority of the stuff they own probably isn't on display because it's called upon so rarely. The problem comes when, as is becoming increasingly obvious, it's also being used to cover for the less salubrious parts of the Internet. But no, the actual notion of there being another, quite sizeable section of information not readily accessible is not something I find particularly shocking. What's actually stored in there, on the other hand...

Yes, a good portion is honest archives, outdated systems, and such. Unfortunately, it is also where the pedophiles, hackers, and criminals also hide.

[Cosmo]

Oh man I love exploring the deep web. The wealth of information on there that people don't want you to know about is incredible.

[TheNumber42]

The seedier part of the deep web is fun to browse sometimes, as long as your careful not to actually visit the illegal parts.

Especially if you get into one of the really fun parts like TOR sites that aren't even accessible conventionally.

[Cabron]

I've heard of this, though never gone on it myself. Maybe I will one day.

[TheNumber42]

Originally Posted by Cabron

I've heard of this, though never gone on it myself. Maybe I will one day.

It's pretty cool to see. Don't have it set up on this computer though. You have to go through TOR or a similar service to access a lot of it. Forget the exact procedure, someone posted it on /b/ awhile back.

EDIT: Just looked it up, and basically all you gotta do is get Firefox+TOR and you can access .onion sites.

[Guile]

Here it is...

[dr2005]

In the interests of protecting the site and its members, I feel obliged to point out that the above post should not be taken as meaning TeenHelp condones or promotes engaging in such activity. I would also advise that step-by-step instructions should not really be posted on the forums in such a manner - remember who our target audience is, after all...

[TheNumber42]

Originally Posted by dr2005

In the interests of protecting the site and its members, I feel obliged to point out that the above post should not be taken as meaning TeenHelp condones or promotes engaging in such activity. I would also advise that step-by-step instructions should not really be posted on the forums in such a manner - remember who our target audience is, after all...

Well, you could just think of it as helping them find porn depicting people their own age, huh? (notserious)


I do kind of agree though, even if it wasn't full of CP, there is a lot of shit that most people under 18 (or over, I suppose) don't need to be seeing. Thanks for the guide nonetheless, always interesting to browse.


EDIT: Also, just so people know, the Hidden Wiki is currently down. You'll need to find a mirror of it if you want a list of .onion sites

[Guile]

Woh woh! Rwemember who posted it

I'd never support CP, I just posted it because I'd rather people know how to explore the Deepweb safely and not get hacked.