Andrew Hay

the man, the myth, the blog

August 11, 2014
by Andrew Hay

New Git Repositories That I’m Following

github-8-xxlEvery now and then I star a Git repo that looks interesting, has a tool I want to try later, or is something immediately useful. Most times, however, I tend to star them and forget about them. In reviewing some of my more recent ‘stars’, I thought it might be useful to share them with my readers.

  • harelba/q
    • q is a command line tool that allows direct execution of SQL-like queries on CSVs/TSVs (and any other tabular text files). q treats ordinary files as database tables, and supports all SQL constructs, such as WHERE, GROUP BY, JOINs etc. It supports automatic column name and column type detection, and provides full support for multiple encodings.
  • wmetcalf/buildcuckoo-trusty
    • A dumb set of scripts for building a cuckoo rig
  • ChrisTruncer/EyeWitness
    • EyeWitness is designed to take screenshots of websites, provide some server header info, and identify default credentials if possible.
    • Inspiration came from Tim Tomes’s PeepingTom Script. I just wanted to change some things, and then it became a thought exercise to write it myself.
    • EyeWitness is designed to run on Kali Linux. It will auto detect the file you give it with the -f flag as either being a text file with URLs on each new line, nmap xml output, or nessus xml output. The -t (timeout) flag is completely optional, and lets you provide the max time to wait when trying to render and screenshot a web page. The –open flag, which is optional, will open the URL in a new tab within iceweasel.
  •  packetloop/packetpig
    • An Open Source Big Data Security Analytics tool that analyses pcap files using Apache Pig.
  • cure53/Flashbang
    • This tool is an open-source Flash-security helper with a very specific purpose: Find the flashVars of a naked SWF and display them, so a security tester can start hacking away without decompiling the code.
    • Flashbang is built upon Mozilla’s Shumway project. It runs in the browser but has a bunch of requirements to work properly.
  • technoskald/maltrieve
    •  A tool to retrieve malware directly from the source for security researchers.
  • guelfoweb/peframe
    • PEframe is a open source tool to perform static analysis on (Portable Executable) malware. It’s released under GPL v2. JSON output and SQlite database support are been introduced since version 4.0.
  • holman/spark
    • Shell script to create spark lines in your shell – e.g. ▁▂▃▅▇
  • mlsecproject/combine
    • Combine gathers OSINT Threat Intelligence Feeds
  • mlsecproject/tiq-test
    • Threat Intelligence Quotient Test – Code and data repository for the statistical analysis of TI feeds
  • CIRCL/AIL-framework
    • AIL is a modular framework to analyze potential information leak from unstructured data source like pastes from Pastebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sensitive information.

July 27, 2014
by Andrew Hay

Unveiling The Open Source Visualization Engine For Busy Hackers at Black Hat 2014

This year marks the first year in my security career that I get to speak at the Black Hat security conference – and I couldn’t be more excited. On Tuesday, August 6th at 2:15pm local time, I’ll be co-presenting Unveiling The Open Source Visualization Engine For Busy Hackers with Thibault Reuille. Here is the abstract for the talk:

The way a human efficiently digests information varies from person-to-person. Scientific studies have shown that some individuals learn better through the presentation of visual/spatial information compared to simply reading text. Why then do vendors expect customers to consume presented data following only the written word method as opposed to advanced graphical representations of the data? We believe this approach is dated.

To help the neglected visually inclined masses, we decided to create a free and Open Source engine to remove the complexity of creating advanced data visualizations. The ultimate goal of the project was to allow for the visualization of any loosely related data without having to endlessly reformat that data. For the visual/spatial learners, the engine will interpret their own data, whether it be a simple or complex system, and present the results in a way that their brains can understand.

Learning, for visual-spatial learners, takes place all at once, with large chunks of information grasped in intuitive leaps, rather than in the gradual accretion of isolated facts or small steps. For example, a visual-spatial learner can grasp all of the multiplication facts as a related set in a chart much easier and faster than memorizing each fact independently. We believe that some security practitioners might be able to better utilize their respective data sets if provided with an investigative model that their brains can understand.

During this presentation, we will show you how you can take any relational data set, quickly massage the format, and visualize the results. We will also share some observations and conclusions drawn from the results of the visualization that may not have appeared in simple text form. We have used this engine within OpenDNS to track CryptoLocker and CryptoDefense ransomware, Red October malware, and the Kelihos botnet. Additionally, specific Syrian Electronic Army (SEA) campaigns, carding sites, and even a map of the Internet via Autonomous Systems have been visualized using the engine.

Interesting data can also be isolated through the use of Python and JavaScript-based plugins that can be easily added to the engine’s framework. These plugins affect the way the data is visualized and allow analysts to make sense of their data as it relates to the question they’re trying to answer. The “big picture” model will help visually inclined incident responders, security analysts, and malware researchers visually stitch together complex data sets without needing a PhD in math or particle physics.

OpenGraphiti, what we’ve named the tool, will be made available the day of the presentation. Having used it at work (and for play) I can tell you that it’s going to blow your mind. See you in Vegas and I hope to see some of my readers at the talk :)

July 21, 2014
by Andrew Hay

New Tool: web2intel


Script to fetch malicious domain and URL lists from sites that publish RSS feeds or raw HTML pages.


To obtain the tool, please visit and download the associated files or simply run the following command at your command prompt:


Supported Lists


./web2intel.rb <option> <extras> 

For command syntax, please visit the GitHub repository.

Example 1 – Domains only

$ ./web2intel.rb --sucuri_iframe
#Title: Sucuri Research Labs Hidden iframes list
#2014-07-20 15:08:14 -0700
....list of domains....

Example 2 – Full URLs

$ ./web2intel.rb --sucuri_iframe --urls
#Title: Sucuri Research Labs Hidden iframes list
#2014-07-20 15:08:42 -0700
....list of URLs....


For any questions, bugs, or concerns, please use the GitHub issue submission system and/or reach out to @andrewsmhay on Twitter.

© Andrew Hay, 2014