Category: Tools

Jupyter Notebook for Queries

Hey All,

Long time no blog. During a recent OSINT investigation, I found that I needed to pull all domains found from my query on The problem I had, however, was that the results weren’t all that usable without a lot of copying, pasting, and cleaning.

To address this problem, and to save time in the future, I created a Jupyter Notebook to programmatically query the website, dump the results into a pandas data frame (thinking that I’ll want to further enrich the data at a later date), and then printing out the unique list of results to the screen.

The code is written in Python 3, and relies on BeautifulSoup4, Pandas, and NumPy.

I’m calling it CrtShcrape (pronounced cert-shcrape) and you can download it from my GitHub here:

Hopefully, you can get some use from it. Until next time!

New Git Repositories That I’m Following

github-8-xxlEvery now and then I star a Git repo that looks interesting, has a tool I want to try later, or is something immediately useful. Most times, however, I tend to star them and forget about them. In reviewing some of my more recent ‘stars’, I thought it might be useful to share them with my readers.

[list icon=”chevron-sign-right”]harelba/q[/list]

q is a command line tool that allows direct execution of SQL-like queries on CSVs/TSVs (and any other tabular text files). q treats ordinary files as database tables, and supports all SQL constructs, such as WHERE, GROUP BY, JOINs etc. It supports automatic column name and column type detection, and provides full support for multiple encodings.

[list icon=”chevron-sign-right”]wmetcalf/buildcuckoo-trusty[/list]

A dumb set of scripts for building a cuckoo rig

[list icon=”chevron-sign-right”]ChrisTruncer/EyeWitness[/list]

EyeWitness is designed to take screenshots of websites, provide some server header info, and identify default credentials if possible.

Inspiration came from Tim Tomes’s PeepingTom Script. I just wanted to change some things, and then it became a thought exercise to write it myself.

EyeWitness is designed to run on Kali Linux. It will auto detect the file you give it with the -f flag as either being a text file with URLs on each new line, nmap xml output, or nessus xml output. The -t (timeout) flag is completely optional, and lets you provide the max time to wait when trying to render and screenshot a web page. The –open flag, which is optional, will open the URL in a new tab within iceweasel.

[list icon=”chevron-sign-right”]packetloop/packetpig[/list]

An Open Source Big Data Security Analytics tool that analyses pcap files using Apache Pig.

[list icon=”chevron-sign-right”]cure53/Flashbang[/list]

This tool is an open-source Flash-security helper with a very specific purpose: Find the flashVars of a naked SWF and display them, so a security tester can start hacking away without decompiling the code.

Flashbang is built upon Mozilla’s Shumway project. It runs in the browser but has a bunch of requirements to work properly.

[list icon=”chevron-sign-right”]technoskald/maltrieve[/list]

A tool to retrieve malware directly from the source for security researchers.

[list icon=”chevron-sign-right”]guelfoweb/peframe[/list]

PEframe is a open source tool to perform static analysis on (Portable Executable) malware. It’s released under GPL v2. JSON output and SQlite database support are been introduced since version 4.0.

[list icon=”chevron-sign-right”]holman/spark[/list]

Shell script to create spark lines in your shell – e.g. ▁▂▃▅▇

[list icon=”chevron-sign-right”]mlsecproject/combine[/list]

Combine gathers OSINT Threat Intelligence Feeds

[list icon=”chevron-sign-right”]mlsecproject/tiq-test[/list]

Threat Intelligence Quotient Test – Code and data repository for the statistical analysis of TI feeds

[list icon=”chevron-sign-right”]CIRCL/AIL-framework[/list]

AIL is a modular framework to analyze potential information leak from unstructured data source like pastes from Pastebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sensitive information.

New Tool: web2intel


Script to fetch malicious domain and URL lists from sites that publish RSS feeds or raw HTML pages.


To obtain the tool, please visit and download the associated files or simply run the following command at your command prompt:


Supported Lists


./web2intel.rb <option> <extras> 

For command syntax, please visit the GitHub repository.

Example 1 – Domains only

$ ./web2intel.rb --sucuri_iframe
#Title: Sucuri Research Labs Hidden iframes list
#2014-07-20 15:08:14 -0700
....list of domains....

Example 2 – Full URLs

$ ./web2intel.rb --sucuri_iframe --urls
#Title: Sucuri Research Labs Hidden iframes list
#2014-07-20 15:08:42 -0700
....list of URLs....


For any questions, bugs, or concerns, please use the GitHub issue submission system and/or reach out to @andrewsmhay on Twitter.

© Andrew Hay, 2014

Scroll to top