I’m in Salt Lake City, Utah this week teaching what is probably going to be my last Q1 Labs QRadar 5.x course. Since I’ve been promoted, the training aspect of my job is something that is going to be taken off my plate due to the time required for both training and leading the Integration Team at Q1.

I must say that the team that I’m teaching is quite sharp, experienced, and obviously well funded based on their infrastructure and ability to track down issues. Another reason I say “well funded” is because I’m in an actual classroom with about 30 workstations for students, a good quality white board, a mobile podium/desk for the instructor computer, and a perfectly placed overhead projector. Typically, while on training assignments, I’m stuffed into a small board room or server room which isn’t always the best environment for training.

Not only is the training facilities nice but the location is amazing. If you’ve never been to Salt Lake City I strongly suggest you go for a vacation. Here is a sample image of the city which is surrounded by mountains.

As the “Handler on Duty”, Ed Skoudis has written an excellent diary entry showing how to utilize netstat in a couple of common scenarios.

From the article:

I’ve often lamented the fact that Windows does not have a built-in lsof-like tool. On Linux and UNIX, lsof gives all kinds of details about what various processes are up to. Sure, we’ve got the Microsoft Sysinternals Process Monitor tool, which is really cool, but is not built in. And, of course, Windows doesn’t include a built-in sniffer…

One technique that I’ve been using a lot in incident handling, vulnerability assessment, malware analysis, and other sysadmin work over the last few months involves the traditional, humble netstat tool. Although netstat is limited, I’ve found a specific use of it to be tremendously helpful. Here are some scenarios.

Fellow handler Mike Poor and I were at a client site, and Mike was doing a network scan. I had one of the client’s laptops, on which we could install no additional software. I wanted to see when Mikey’s wide-ranging scan reached my box, which did have an open port. Here’s what I ran:

C:> netstat –na 1 | find "[Scan_Host_IP_Addr]"

The netstat command, used this way, shows TCP and UDP port activity. The –n means to list numbers. The –a indicates that we want all connections and listening ports. In Windows netstat, the 1 means we want to run every second, repeatedly dumping the output on standard out. And, we are looking through our output with the find command to see an indication of when Mike’s box had accessed ours. Note that I’m using find here, but another alternative would be the findstr command. The find command can locate strings nicely, but findstr can process regular expressions. I believe in using the appropriate tool for the job, and these simple searches work just fine with find. If you want regexp stuff, use the more powerful findstr command. Anyway, because the 3-way handshake or an actual connection will likely last more than 1 second, this technique will work. Sadly, the technique does not work to capture sub-1-second events. As Mikey continued the scan… Bingo! We could see with 1-second accuracy when it reached my box.

I’ve used this technique elsewhere as well. A gentleman taking the SANS Security 504 class had a dilemma. He was seeing a weird ICMP Host Unreachable message in his network. When he looked at the destination address, it was going from his router back to his Domain Controller. So, his DC was pushing out a packet to a machine that his router couldn’t reach. But, what process on his DC was sending this packet? On the Domain Controller, we ran:

C:> netstat –nao 1 | find "[Dest_IP_Addr]"

Here, I’ve added the –o flag, which makes Windows netstat print the PID. You can then look up that PID using “wmic process list brief”, “tasklist”, or, if you insist, Task Manager (yuck!). Then, you can see what process is emitting that packet, provided that it is using the TCP or UDP stack of Windows to send it, and that it takes at least a second. Note that netstat also offers the –b flag, which makes it show the EXE and its associated DLLs that are using TCP and UDP ports. However, I didn’t use –b here, because it seriously hurts performance. For whatever reason, it takes netstat a lot of CPU cycles to get the EXE and DLL info, cycles that we cannot spare on a Domain Controller. And, running “netstat –naob” every second would be a serious drain on processor resources.

Here’s another one. We were working on an investigation where an evil process would start up, and eventually (not instantly) listen on TCP port 2222. We wanted to know when it started listening, so we ran:

C:> netstat –na 1 | find "2222"

And, here’s one final one for you. I was working on an investigation, and we had a process listening on a given TCP port (let’s say, for example, it was TCP port 4444). We wanted to know when the bad guy connected to it. We ran:

C:> netstat –na 1 | find "4444" | find "ESTABLISHED"

This will print nothing until the output of netstat includes an established connection on port 4444. So, with approximately 1-second accuracy, we were able to see when someone connected to the port, knowing that our bad guy had come calling. Also, this output includes the source IP address connected to the port, a helpful thing in an investigation.

Now, obviously you could do all of this with a sniffer, with more accuracy and detail. But, netstat is built-in, and these command are easy and quick to type.