Tag: tool

Jupyter Notebook for crt.sh Queries

Hey All,

Long time no blog. During a recent OSINT investigation, I found that I needed to pull all domains found from my query on crt.sh. The problem I had, however, was that the results weren’t all that usable without a lot of copying, pasting, and cleaning.

To address this problem, and to save time in the future, I created a Jupyter Notebook to programmatically query the crt.sh website, dump the results into a pandas data frame (thinking that I’ll want to further enrich the data at a later date), and then printing out the unique list of results to the screen.

The code is written in Python 3, and relies on BeautifulSoup4, Pandas, and NumPy.

I’m calling it CrtShcrape (pronounced cert-shcrape) and you can download it from my GitHub here: https://github.com/andrewsmhay/CrtShcrape.

Hopefully, you can get some use from it. Until next time!

Suggested Blog Reading – Monday February 18th, 2008

ReadUgh….I haven’t had a case of the flu like this for years. I’m finally over it (I think) and hopefully things will be getting back to normal soon.

Here is the list:

PHPIDS – Security Layer & Intrusion Detection for PHP Based Web Applications – This is an interesting tool that I haven’t heard about until today.

PHPIDS (PHP-Intrusion Detection System) is a simple to use, well structured, fast and state-of-the-art security layer for your PHP based web application. The IDS neither strips, sanitizes nor filters any malicious input, it simply recognizes when an attacker tries to break your site and reacts in exactly the way you want it to. Based on a set of approved and heavily tested filter rules any attack is given a numerical impact rating which makes it easy to decide what kind of action should follow the hacking attempt.

From the SANS Information Security Reading Room:

Enterprise Security 2008 Learning Guide – Good collection of articles to check out.

2008 won’t just be a year of the same old network, application and compliance issues. New malware has hit the scene, cyberterrorist attacks have become more common, and virtualization technology has presented different enterprise network security challenges. Mike Chapple, Michael Cobb, Joel Dubin, Mike Rothman and Ed Skoudis explore various information security areas and point out the new threats that every organization needs to be ready for.

More on Hating Agents – Everyone hates them but they are required – no, not lawyers….I’m talking about log agents. Anton lists some good pros and cons for leveraging an agent to get you your logs.

I responded to a question about using agents for log collection on a mailing list (semi-public); I think this content also begs to be blogged.

Password Cracking Wordlists and Tools for Brute Forcing – Ever want to find a good word list for your audits?

I quite often get people asking me where to get Wordlists, after all brute forcing and password cracking often relies on the quality of your word list.

Do note there are also various tools to generate wordlists for brute forcing based on information gathered such as documents and web pages (such as Wyd – password profiling tool) These are useful resources that can add unique words that you might not have if your generic lists.

Also add all the company related words you can and if possible use industry specific word lists (chemical names for a lab, medical terms for a hospital etc).

Is the mobile malware threat overblown? – Overblown…maybe. Under-exploited…possibly. Not receiving the amount of attention it deserves…definitely!

The trouble for some IT pros is that security experts have been warning of growing mobile phone attacks for more than three years and the big event has yet to materialize.

Does this mean the mobile phone threat has been overblown all this time, over-hyped by security vendors generating FUD to sell new products? Not exactly.

True, enterprises continue to experience little by way of mobile phone attacks. But that’s only because companies are still limiting the functionality of such devices among employees. Just about everyone uses cell phones with Internet capabilities these days. But in the working world, use of the devices are still limited to making phone calls and checking email.

New Docs at SWGDE – Some new docs on forensics. Thanks Harlan.

The Scientific Working Group on Digital Evidence (SWGDE) has released some new documents, the most notable of which are the Vista Technical Notes, and the document on “Live Capture”.

Could computer forensics help your organisation? – Umm…ya?

Forensics is not yet a mainstream field and descriptions and definitions vary. Yet how do organisations integrate incident response, breach handling and forensic examination into a security strategy? That security strategy should be defined by policies and procedures to minimise security risk at the lowest cost and least disruption. It is a major challenge facing many CIOs…

Scary concept: Friendly worms – If this ever became a reality, which I doubt it will, how long would you expect it would take before someone exploited the updating and transport mechanism to “do evil”?

This isn’t a new idea, the concept of creating worms that patch your computer when you catch them. There are even some malware out there now that patches vulnerabilities on systems to make sure other worms can’t exploit the same vulnerabilities. But the problem is, if both beneficial and malign software show the same basic behavior patterns, how do you differentiate between the two? And what’s to stop the worm from being mutated once it’s started, since bad guys will be able to capture the worms and possibly subverting their programs.

SQL Injection Tutorial Now Available! – Very cool. Good for Oracle in taking a step to help people secure their product and applications.

By taking this self-study tutorial, you can arm yourself with techniques and tools to strengthen your code and applications against these attacks. This tutorial employs text and diagrams to present concepts, design issues, coding standards, processes, and tools. Flash-based demos and simulations allow you to visualize what you have learned, and assessment quizzes help you gauge your learning progress.

Suggested Blog Reading – Sunday February 10th, 2008

ReadI’m a little confused why more snow has fallen over the past 3 months than has fallen over the past 2 years. I’m getting sick of clearing it!

Here is the list:

Birth of IPv6 – Is your organization pushing towards IPv6? I didn’t think so 🙂

Well tonight’s the night. For the first time, IPv6 domain resolution will be possible from a root server. Just a few addresses mind you, according to this article. You may ask “what took so long?”. The answer is that we did not really need it. IPv6 bakes in some security that was addressed by SSL in IPv4 so that driver did not help. The other issue, a rapidly depleting address space, was managed by NAT(Network Address Translation). But now depletion is really staring us in the face. It is getting hard to get address space. Soon you will see the first bidding wars for owners of large blocks of free IP addresses. Technically you are not allowed to sell IP addresses so don’t expect a market for them. But do expect high valuations for shells that control IP address blocks.

(IN)SECURE Magazine Issue 15 – Looks like Issue 15 is finally out.

Articles in this issue include: Proactive analysis of malware genes holds the key to network security, Advanced social engineering and human exploitation, part 1, Free visualization tools for security analysis and network monitoring, Hiding inside a rainbow, Internet terrorist: does such a thing really exist?, Weaknesses and protection of your wireless network, Fraud mitigation and biometrics following Sarbanes-Oxley, QualysGuard visual walkthrough, Application security matters: deploying enterprise software securely, Web application vulnerabilities and insecure software root causes: solving the software security problem from an information security perspective, A dozen demons profiting at your (jn)convenience, The insider threat: hype vs. reality, Interview with Andre Muscat, Director of Engineering at GFI Software, How B2B gateways affect corporate information security, Reputation attacks, a little known Internet threat, Italian bank’s XSS opportunity seized by fraudsters, The good, the bad and the ugly of protecting data in a retail environment, Interview with Mikko Hypponen is the Chief Research Officer for F-Secure, Interview with Richard Jacobs, Technical Director of Sophos and Interview with Raimund Genes, CTO Anti-Malware at Trend Micro.

A funny thing happened on the way to reviewing my logs – Interesting article from Andy Willingham on his journey implementing a SIEM solution.

At work we’re in the process of implementing a SIEM (Security Information Event Management) system. I’ll leave the vendor nameless for the moment but they have a reputation of making most everything harder than it needs to be. Until that time all logs have to be reviewed manually and obviously that means that they are not reviewed in real time. I have others that monitor most of the logs but I monitor our IPS logs from the UTM device. Usually I review them each morning when I come in but last week I didn’t get a change to so yesterday I was playing catchup.

Interesting tool – pdump.exe – I’ll have to give this a shot.

Toni at Teamfurry.com has a new tool that has some interesting functionality, it dumps process memory, but it also saves each allocated memory region to a separate file.

I’ve played with it a little bit and it seems like it has potential.

Rebecca Herold’s 2008 speaking dates – If you’re in the area I strongly suggest you drop by and check out one of Rebecca’s presentations.

January 18: The Importance of Verifying Third Party Security Programs
Learning event at the Grand Rapids, Michigan ISSA chapter meeting
Web Site: http://www.gr-issa.org/

February 21: Anatomy of a Privacy Breach
Learning event at the University of California, Berkeley
Web Site: http://www.truststc.org/seminar.htm

March 18: Anatomy of a Privacy Breach
Learning event at the Iowa ISACA chapter meeting

April 27: The 30 Second Security Pitch
Learning event at the CSI SX conference
Web Site: http://www.csisx.com/conference/view-by-day.php

April 30 & May 1: Executive Summit: Security and Privacy Collaboration
2-day learning workshop at the CSI SX conference
Web Site: http://www.csisx.com/conference/workshops.php

July 23 & 24: Executive Summit: Security and Privacy Collaboration
2-day learning workshop hosted by the Charlotte, North Carolina ISACA chapter.

Getting over the hump with vulnerability counts – What is more important? The total number of vulnerabilities or the number of highly exploitable vulnerabilities?

Should our vulnerability counts be going up or going down? That is an important question every security professional should be considering when laying out a security program.

If you believe vulnerability counts should be increasing, then presumably you believe that we are only covering the tip of the iceberg with respect to the total number of vulnerabilities in production. In this case, you are taking a short-term view of what is happening in security – it is okay to be hoping the counts increase in the short term, but eventually you want them to decrease (right?).

Give yourself a little time with SQL Injection – Interesting article on blind SQL injection.

I was recently involved in web application assessment and discovered something that I wanted to pass along. Keep in mind that this has probably been utilized before, but it is something that I just noticed so … I wanted to throw it out for your amusement.
To set the stage, I had been looking at this application for quite some time and had an idea that SQL Injection might exist, but I was having much difficulty determining if the injection was actually present. The application was catching errors, displaying 404’s, (etc) and really not displaying any good data to make a decision. So …. the question was … if the application is catching our errors and really not giving us anything to work with … how could we ask the question to the database to indicate if we were actually getting our requests processed by the database server?
Answer? Time.

Security Metrics – How Often Should We Scan? – Personally, I think your frequency of scans should be dictated by the criticality of the systems, the type of systems, the data stored on the systems, and of course…your documented security policy.

I get this question from Nessus users and Tenable customers very often. They want to know if they are scanning too often, not often enough and they also want to know what other organizations are doing as well. In this blog entry, we will discuss the many different reasons why people perform scans and what factors can contribute to their scanning schedule.

German Police Creating LE Trojan – “Law Enforcement Trojan”? I’m not sure if this will fly.

German cops are pushing ahead with controversial plans, yet to be legally approved, to develop “remote forensic software” – in other words, a law enforcement Trojan. Leaked documents outline proposals by German firm Digitask to develop software to intercept Skype VoIP communications and SSL transmissions. A second leaked document from the Bavarian Ministry of Justice outlines costing and licensing proposals for the software. Both scanned documents (in German, natch) have found their way onto the net after being submitted to Wikileaks…

From the SANS Information Security Reading Room:

Spending for IT security gains ground in 09 budget – I can’t remember a time when security/IT/(random item) spending wasn’t “gaining ground”.

New details on federal IT spending plans, made available by the Office of Management and Budget today, show that $103 out of every $1,000 requested for IT spending next fiscal year — or about $7.3 billion in total — will be devoted to improving IT security. That is 9.8 percent more than what was slated for fiscal 2008, and 73 percent more than the $4.2 billion budgeted for cybersecurity in fiscal 2004.

DFRWS 2008 Announcement – I need to come up with a paper for this 🙂

The DFRWS 2008 CfP and Challenge have been posted!

The CfP invites contributions on a wide range of subjects, including:

  • Incident response and live analysis
  • File system and memory analysis
  • Small scale and mobile devices
  • Data hiding and recovery
  • File extraction from data blocks (“file carving”)

And here’s a couple that should be interesting:

  • Anti-forensics and anti-anti-forensics
  • Non-traditional approaches to forensic analysis

Submission deadline is 17 Mar, with author notification about 6 wks later.

Python for Bash scripters: A well-kept secret – Good post for all of us who know Bash scripting but want to break into Python.

Python is easy to learn, and more powerful than Bash. I wasn’t supposed to tell you this–it’s supposed to be a secret. Anything more than a few lines of Bash could be done better in Python. Python is often just as portable as Bash too. Off the top of my head, I can’t think of any *NIX operating systems, that don’t include Python. Even IRIX has Python installed.

The Flow of MBR Rootkit Trojan Resumes – Why…won’t…this…die?

Back in final weeks of 2007 the GMER team discovered the emergence of a new rootkit that hooked into the Windows master boot record (MBR) in order to take control of a compromised computer. The people responsible for this threat kept busy cranking out newly compiled versions of this Trojan in the weeks following its discovery. However, near the beginning of January the output of new variants mysteriously halted. Taking a quick look at the following table of Trojan.Mebroot sample data it appears as though a massive QA plan was performed by the gang, starting back in November 2007.

A Practical Approach to Managing Information System Risk – Another paper to check out.

The mantra spinning around in the heads of most security managers affirms that managing security is about managing risk. Although they know this is the right approach, and they understand the importance of balance in designing and implementing security controls, many of them—including me—came up through the ranks of network engineering, programming, or some other technical discipline. While this prepared us for the technology side of our jobs, the skills necessary to assess and understand business risk arising from the use of information systems were not sufficiently developed.

Scroll to top