Open Source Intelligence Automation: Spiderfoot


SpiderFoot is an open source footprinting tool designed to be easy to use, fast and extensible

SpiderFoot is an open source footprinting tool, available for Windows and Linux. It is written in Python and provides an easy-to-use GUI. SpiderFoot obtains a wide range of information about a target, such as web servers, netblocks, e-mail addresses and more. SpiderFoot’s simple web-based interface enables you to kick off a scan immediately after install – just give your scan a name, the domain name of your target and select which modules to enable.

SpiderFoot 2.4.0 released

Open Source Intelligence Automation: Spiderfoot Open Source Intelligence Automation: Spiderfoot Open Source Intelligence Automation: Spiderfoot

The main objective of SpiderFoot is to automate this process to the greatest extent possible, freeing up a penetration tester’s time to focus their efforts on the security testing itself.

  • Start with a target of more than just domains (Hostnames, IPs, Netblocks, etc.)
  • Clean-up back-end data model to be more flexible
  • Simultaneous scans
  • More threading for faster performance
  • Search/Filtering
  • Bunch of bug fixes

 

SpiderFoot is an open source intelligence automation tool Its goal is to automate the process of gathering intelligence about a given target

 

Purpose

There are three main areas where SpiderFoot can be useful:

  1. If you are a pen-tester, SpiderFoot will automate the reconnaisance stage of the test, giving you a rich set of data to help you pin-point areas of focus for the test.
  2. Understand what your network/organisation is openly exposing to the outside world. Such information in the wrong hands could be a significant risk.
  3. SpiderFoot can also be used to gather threat intelligence about suspected malicious IPs you might be seeing in your logs or have obtained via threat intelligence data feeds.

 

Features

  • Utilises a shedload of data sources; over 40 so far and counting, including SHODAN, RIPE, Whois, PasteBin, Google, SANS and more.
  • Designed for maximum data extraction; every piece of data is passed on to modules that may be interested, so that they can extract valuable information. No piece of discovered data is saved from analysis.
  • Runs on Linux and Windows. And fully open-source so you can fork it on GitHub and do whatever you want with it.
  • Visualisations. Built-in JavaScript-based visualisations or export to GEXF/CSV for use in other tools, like Gephi for instance.
  • Web-based UI. No cumbersome CLI or Java to mess with. Easy to use, easy to navigate. Take a look through the gallery for screenshots.
  • Highly configurable. Almost every module is configurable so you can define the level of intrusiveness and functionality.
  • Modular. Each major piece of functionality is a module, written in Python. Feel free to write your own and submit them to be incorporated!
  • SQLite back-end. All scan results are stored in a local SQLite database, so you can play with your data to your heart’s content.
  • Simultaneous scans. Each footprint scan runs as its own thread, so you can perform footprinting of many different targets simultaneously.
  • So much more.. check out the documentation for more information.

 

Data Sources

This is an ever-growing list of data sources SpiderFoot uses to gather intelligence about your target. A few require API keys but they are freely available.

SourceLocationNotes
abuse.chhttp://www.abuse.chVarious malware trackers.
AdBlockhttps://easylist-downloads.adblockplus.org/easylist.txtAdBlock pattern matches
AlienVaulthttps://reputation.alienvault.comAlienVault’s IP reputation database.
Autoshun.orghttp://www.autoshun.orgBlacklists.
AVG Site Safety Reporthttp://www.avgthreatlabas.comSite safety checker.
Binghttp://www.bing.comScraping but future version to also use API.
Blocklist.dehttp://lists.blocklist.deBlacklists.
Checkusernames.comhttp://www.checkusernames.comLook up username availability on popular sites.
DNSYour configured DNS server.Defaults to your local DNS but can be configured to whatever IP address you supply SpiderFoot.
DomainToolshttp://www.domaintools.com
DroneBLhttp://www.dronebl.org
Facebookhttp://www.facebook.comScraping but future version to also use API.
FreeGeoIPhttp://freegeoip.net
Googlehttp://www.google.comScraping but future version to also use API.
Google+http://plus.google.comScraping but future version to also use API.
Google Safe Browsinghttp://www.google.com/safebrowsingSite safety checker.
LinkedInhttp://www.linkedin.comScraping but future version to also use API.
malc0de.comhttp://malc0de.comBlacklists.
malwaredomainlist.comhttp://www.malwaredomainlist.comBlacklists.
malwaredomains.comhttp://www.malwaredomains.comBlacklists.
McAfee SiteAdvisorhttp://www.siteadvisor.comSite safety checker.
NameDroppershttp://www.namedroppers.org
Nothink.orghttp://www.nothink.orgBlacklists.
OpenBLhttp://www.openbl.orgBlacklists.
PasteBinhttp://www.pastebin.comAchieved through Google scraping.
PGP Servershttp://pgp.mit.edu/pks/PGP public keys.
PhishTankhttp://www.phishtank.orgIdentified phishing sites.
Project Honeypothttp://www.projecthoneypot.orgBlacklists. API key needed.
RIPE/ARINhttp://stat.ripe.net/
Robtexhttp://www.robtex.com
SANS ISChttp://isc.sans.eduInternet Storm Center IP reputation database.
SHODANhttp://www.shodanhq.comAPI key needed.
SORBShttp://www.sorbs.netBlacklists.
SpamHaushttp://www.spamhaus.orgBlacklists.
ThreatExperthttp://www.threatexpert.comBlacklists.
TOR Node Listhttp://torstatus.blutmagie.de
TotalHash.comhttp://www.totalhash.comDomains/IPs used by malware.
UCEPROTECThttp://www.uceprotect.netBlacklists.
VirusTotalhttp://www.virustotal.comDomains/IPs used by malware. API key needed.
WhoisVariousWhois servers for different TLDs.
Yahoohttp://www.yahoo.comScraping but future version to also use API.
Zone-Hhttp://www.zone-h.orgEasy to get black-listed. Log onto the site in a browser from the IP you’re scanning from first and enter the CAPTCHA, then it should be fine.

 

Open Source Intelligence Automation: Spiderfoot

Open Source Intelligence Automation: Spiderfoot

SpiderFoot is designed from the ground-up to be modular. This means you can easily add your own modules that consume data from other modules to perform whatever task you desire. As a simple example, you could create a module that automatically attempts to brute-force usernames and passwords any time a password-handling webpage is identified by the spidering module.

Open Source Intelligence Automation: Spiderfoot

Pre-Requisites

Linux/BSD/Solaris

SpiderFoot is written in Python (2.7), so to run on Linux/Solaris/FreeBSD/etc. you need Python 2.7 installed, in addition to the lxml, netaddr, M2Crypto, CherryPy and Mako modules.

To install the dependencies using PIP, run the following:

~$ pip install lxml netaddr M2Crypto cherrypy mako

Other modules such as MetaPDF, SOCKS and more are included in the SpiderFoot package, so you don’t need to install them separately.

 

 

Source && Download

spiderfoot download


Visitors Rating
Rate Here
Ease Of Use
73%
Features
69%
Value
77%
Overall Rating
75%
73%
Visitors Rating
4 ratings
You have rated this
What's your reaction?
OWND
0%
Cool
67%
Nice
33%
WHAT ?
0%
MEH
0%
zzzZZzz
0%
Rage
0%

You must log in to post a comment