Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

domain_history: fix false negative results #221

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

KhasMek
Copy link

@KhasMek KhasMek commented Dec 30, 2017

Netcraft is heavily reliant upon javascript to check for automated scraping and rendering of the page. Switching over to selinium from requests and rendering the page with either PhantomJS, Chrome or Firefox circumvents this issue and actually returns results if present.

You can see the proof of this issue by printing out the BeautfiulSoup response in the unpatched module

diff --git a/domain/domain_history.py b/domain/domain_history.py
index ff8fe69..fb6d0f9 100755
--- a/domain/domain_history.py
+++ b/domain/domain_history.py
@@ -23,6 +23,7 @@ def netcraft_domain_history(domain):
     req = requests.get(endpoint)
 
     soup = BeautifulSoup(req.content, 'html.parser')
+    print(soup)
     urls_parsed = soup.findAll('a', href=re.compile(r'.*netblock\?q.*'))
     for url in urls_parsed:
         if urls_parsed.index(url) != 0:

If the user ignores the instructions and somehow PhantomJS, Chrome or Firefox aren't installed on the system, this module will spit out a selenium error that I can't catch and redirect. However, the error will not cause datasploit (or anything else) to hard fail/exit. So, I figure this rare edge case issue is mostly acceptable and far more acceptable/expected than 100% false negative results.

  Netcraft's is heavily reliant upon javascript to check
  for automated scraping and rendering of the page.
  Switching over to selinium from requests and rendering the
  page with either PhantomJS, Chrome or Firefox circumvents
  this issue and actually returns results if present.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant