Long time no blog. During a recent OSINT investigation, I found that I needed to pull all domains found from my query on crt.sh. The problem I had, however, was that the results weren’t all that usable without a lot of copying, pasting, and cleaning.
To address this problem, and to save time in the future, I created a Jupyter Notebook to programmatically query the crt.sh website, dump the results into a pandas data frame (thinking that I’ll want to further enrich the data at a later date), and then printing out the unique list of results to the screen.
The code is written in Python 3, and relies on BeautifulSoup4, Pandas, and NumPy.
I’m calling it CrtShcrape (pronounced cert-shcrape) and you can download it from my GitHub here: https://github.com/andrewsmhay/CrtShcrape.
Hopefully, you can get some use from it. Until next time!
Every now and then I star a Git repo that looks interesting, has a tool I want to try later, or is something immediately useful. Most times, however, I tend to star them and forget about them. In reviewing some of my more recent ‘stars’, I thought it might be useful to share them with my readers.
q is a command line tool that allows direct execution of SQL-like queries on CSVs/TSVs (and any other tabular text files). q treats ordinary files as database tables, and supports all SQL constructs, such as WHERE, GROUP BY, JOINs etc. It supports automatic column name and column type detection, and provides full support for multiple encodings.
A dumb set of scripts for building a cuckoo rig
EyeWitness is designed to take screenshots of websites, provide some server header info, and identify default credentials if possible.
Inspiration came from Tim Tomes’s PeepingTom Script. I just wanted to change some things, and then it became a thought exercise to write it myself.
EyeWitness is designed to run on Kali Linux. It will auto detect the file you give it with the -f flag as either being a text file with URLs on each new line, nmap xml output, or nessus xml output. The -t (timeout) flag is completely optional, and lets you provide the max time to wait when trying to render and screenshot a web page. The –open flag, which is optional, will open the URL in a new tab within iceweasel.
An Open Source Big Data Security Analytics tool that analyses pcap files using Apache Pig.
This tool is an open-source Flash-security helper with a very specific purpose: Find the flashVars of a naked SWF and display them, so a security tester can start hacking away without decompiling the code.
Flashbang is built upon Mozilla’s Shumway project. It runs in the browser but has a bunch of requirements to work properly.
A tool to retrieve malware directly from the source for security researchers.
PEframe is a open source tool to perform static analysis on (Portable Executable) malware. It’s released under GPL v2. JSON output and SQlite database support are been introduced since version 4.0.
Shell script to create spark lines in your shell – e.g. ▁▂▃▅▇
Combine gathers OSINT Threat Intelligence Feeds
Threat Intelligence Quotient Test – Code and data repository for the statistical analysis of TI feeds
AIL is a modular framework to analyze potential information leak from unstructured data source like pastes from Pastebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sensitive information.