On Jun 5, 2012, at 9:52 AM, Peter Kristolaitis wrote: > > As far as horror stories... yeah. My most memorable experience was a guy > (with a CISSP designation, working for a company who came highly recommended) > who: > - Spent a day trying to get his Backtrack CD to "work properly". When I > looked at it, it was just a color depth issue in X that took about 45 seconds > from "why is this broken?" to "hey look, I fixed it!". > - Completely missed the honeypot machine I set up for the test. I had > logs from the machine showing that his scanning had hit the machine and had > found several of the vulnerabilities, but the entire machine was absent from > the report. > - Called us complaining that a certain behavior that "he'd never seen > before" was happening when he tried to nmap our network. The "certain > behavior" was a firewall with some IPS functionality, along with him not > knowing how to read nmap output. > - Completely messed up the report -- three times. His report had the > wrong ports & vulnerabilities listed on the wrong IPs, so according to the > report, we apparently had FreeBSD boxes running IOS or MS SQL... > - Stopped taking our calls when we asked why the honeypot machine was > completely missing from the report. > > In general, my experience with most "pen testers" is a severe disappointment, > and isn't anything that couldn't be done in-house by taking the person in > your department who has the most ingrained hacker/geek personality, giving > them Nessus/Metasploit/nmap/etc, pizza and a big ass pot of coffee, and > saying "Find stuff we don't know about. Go.". There is the occasional pen > tester who is absolutely phenomenal and does the job properly (i.e. the guys > who actually write their own shellcode, etc), but the vast majority of "pen > testers" just use automated tools and call it a day. Like everything else in > IT, security has been "commercialized" to the point where finding really good > vendors/people is hard, because everyone and their mom has CEH, CISSP, and > whatever other alphabet soup certifications you can imagine.
I agree with a lot of what you've said, but there are absolutely good security guys (pen tester, vulnerability assessors, etc) that use both open source and commercial automated tools, but still do a fantastic job because they understand the underlying technologies and protocols. I used to do a lot of this in the past, had lots of automated tools, and only occasionally wrote some assessment modules or exploit code if necessary. But again, a person in that position has to understand technology holistically (network, systems, software, protocols, etc). -b
