Visiting those domains revealed that the reason was that those pages were usually a login page. While examining the number of pages we found per domain, we found that many domains have only one page available. The next issue was that all the “good stuff” (and by good I mean really bad), was on sites that require login. It is an ongoing process of constant discovery as sites don’t live long on the dark web.
The solution was to group them together by the site’s name.
The next problem we now faced was that many domains were down, went down, or quickly changed mirrors.
We created a discovery service that crawled external links found in anchors and domains that were mentioned in the text. The solution wasn’t that complicated since we already had a seed list of sites. Illicit services are hard to find and usually are on a need to know basis. The question raised was, are we doing something wrong? Or maybe the TOR network is nothing but a fad? Well, we were doing something wrong, and down the rabbit hole we went. We were able to crawl the TOR network and download some data, and we even got a lot of traction, however when push comes to shove, our clients didn’t find a lot of value in the content we’ve found. Did it work? Well kinda… actually not really. I thought we will be able to shed light on the criminal activity going on in the darkest corners of the web. The idea was to leverage our current crawling technology that downloads and automatically structures content (extract the content, author name, comments, dates etc.) from the open web, and just use a proxy to connect to the TOR network. The following short story portrays the surprising technological and logical challenges we faced while developing our dark web monitoring technology.īack in 2017 when I initially had the idea of adding content from the TOR network to our data repository, I thought it should be quite straight forward.
0 kommentar(er)
