=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
             ║ Some Dark Corners of Pentesting ║
               =-=-=-=-=-=- s1m0n -=-=-=-=-=-=


  > Intro
  > Sins of the pentesters
  > Finding and hiring pentesters
  > Retaining pentesters
  > Engaging a pentest(er| provider)
  > Pentest planning
  > Pentest delivery
  > Missed vulnerabilities
  > Credits
  > References


--[ Intro
To be totally honest with you dear reader, I wasn't sure if I want to
write this text at all but at some point there was this one straw that
broke the camel's back. It's a non-technical text revolving around random
issues in pentesting that are often annoying, rarely interesting and
sometimes remain unspoken. Even though I'm mainly focusing in it on
pentesting and pentesters, then I believe that some of the points made
herein are generic enough so they can be also applied to other infosec
areas. I wrote it without any expectations but it would be cool if it
would contribute to some changes for better. And let's be clear about one
thing, the text was not meant to attack anyone. If you feel offended by
it, then there's a big chance that perhaps you're doing something in the
wrong way.

--[ Sins of the pentesters
Howdy fellow pentesters. From time to time we can be perceived by others
as a special breed with better than average computer skills. However,
talking like one tester to another now. We're not free of annoying habits
and... let's call it logic flaws.

We all know it's great to break stuff, isn't it? Similarly, ranting about
things being broken is so common and easy. Everyone can do that. However,
building/fixing/improving/... it is equally hard if not harder. Some
people say defence is sexy. No it's not. It's a constant hard work,
underpaid, quite often frustrating and not appreciated enough by so many.
All it takes to be blamed and finger pointed is one fail, one oversight,
one weak spot. Keep it in mind next time you will rant about devs and
blue-team's incompetence. Sharing is caring. If you can, then contribute
ideas, submit code and suggest solutions to problems after you dropped the
bomb. Surely you can do better than that.

If your pentester's work is limited to running tools written by others and
reading security news, then congrats, you just earned a pro-skiddie award.
It doesn't matter if you're an experienced pentester/ethical
hacker/whitehat/cyber security professional or whatever the name is in use
today. If you're not inquisitive anymore, you became lazy, you burned out,
you look for excuses instead of solutions, you mainly rant about
everything and everyone, you're known nowadays only because you know
everyone else on the security band wagon, you spend more time drinking
booze and attending conferences instead of reading and writing code (don't
get me wrong on this one though, most of us know how to party properly;>),
then know you can do much better than that. Continuously learn and adapt
or... else.

It's not uncommon to encounter pentesters who eagerly express their
opinions on basically anything security related. There's nothing wrong
about it until they have at least some experience on the matter
(preferably not based on factoids). Unfortunately, quite often pentesters
talk without actually knowing or checking facts first. Talking about
memory corruption when they didn't write a single rop chain in their life,
about malware when they never actually analysed a sample, one form of
disclosure being better over another if they published no advisory,
talking about the source code when all they wrote was pascal decades ago
etc. Such approach doesn't give them too much credibility, does it? The
general advise is to stick to what you know or keep it quiet until you do
your homework. If unsure (RTFM|POC)||GTFO.

Oh, and if your work is publicity driven, then it might be a good time to
re-think your approach. It's never too late.

--[ Finding and hiring pentesters
For some time now the "skills gap in security" term is omnipresent and
people are generally getting very excited about this phenomenon. It was
already the case during my uni times circa 2004. In fact, it was the same
long before that. Sure, the problem of skills shortage was called
differently and it wasn't of the same scale as it is presently but
unquestionably the problem was already there. Educated workers, engineers,
programmers, analytics, mathematicians, scientists and other bright minds
were always difficult to find and hire. After they were found, they could
then work on/with "new technologies" and whenever we deal with new
technologies, there's practically always a security factor involved. Think
for a moment of cryptography & intelligence during and after IIWW,
mainframes, ALGOL/COBOL/..., financial institutions, industrial PLCs,
telcos, NASA and more. I believe it means the underlying problem is at
least 60-70 years old and only the names and technologies change. It seems
there will still be a deficit of capable people in a foreseeable future
because nothing indicates otherwise.

Why finding people with IT security capabilities became such a big deal?
If you consider recent years and state-sponsored attacks/groups,
high-profile hacks & data leaks, GDPR/Patriot/Cyber Privacy
Fortification/... acts, national cyber security strategies, exploit
markets, IoT, transportation and generally the way technologies are used
to gather and process information about everyone and everything, the
answer is obvious.

Getting capable people to join the team is really hard[1][2]. The
following observations are from the perspective of being recruited,
occasionally being involved in recruiting others and from numerous
discussions with friends.

If you even think about becoming a respectable and prospering company
providing security services, then take a good care of your employees in
the first place. It is no one else but them who will take you there.

The important question that you should ask yourself before you even start
looking for pentesters is who are you looking for exactly? Knowing your
own expectations regarding the position you're hiring for, you should
consider that security is a specific field comprising of voodoo people
[3]. Let those people know beforehand if you're looking for someone in a
shiny shoes, tie and suit, or perhaps a wool picker who will run tools all
day long, or perhaps you need a security rockstar travelling from one conf
to another etc. You can save yourself and others lot of time/effort/money
by simply being honest and transparent.

Remember that interviews do work both ways. Candidates will know if you
actually spend time reading their CV and any other information they
possibly provided to you. It also doesn't hurt to do a quick web search
about your candidate on your own. If you show at this stage that you don't
care about them they will more likely choose employer that will.

During technical interviews avoid asking blatant questions. Common
examples includes, but are not limited to:
 * what service is listening on port A?
 * how B works/is implemented in tool C?
 * what parameter/argument X will do?
 * would you be able to write a script in Y?
 * can you run a scan using Z?

If you ask similar questions then it means you're doing it wrong. This is
definitely not the way to find a capable pentester. Instead you could ask
candidate to solve a practical problem or confront him with a hands-on
challenge. Next time you can ask candidate to review a source code, solve
a technical challenge, document her findings and walk you through it.
That would be for a good start.

Please, save candidates time and money. If possible provide them with a
toll free numbers or call them, and don't make them wait for you on the
line for minutes because you had another meeting etc. Reimbursing and/or
participation in candidate's costs (e.g. travel tickets) is an extra mile
that some companies are willing to walk and believe me, they benefit from
it in the long term. Remember that pentesters and voodoo people in general
meet and discuss from time to time despite general opinions about them.
This means no more no less that if you're failing on one candidate it's
very likely that at the same time you're failing on many.

Are you asking your candidates about salary expectations? No one told you
it's millions of billions? Seriously, it's optimal to inform candidates
how much is the offer (even range will do) before anything else. Otherwise
it's again, only waste of everyone's time.

Yes, do not provide candidates with updates, give them no feedback and you
will surely not burn any bridges. Such bad experiences travel via word of
mouth and people will be less eager to apply to/via such companies.

--[ Retaining pentesters
If you hired people only to tell them what to do, then I believe any
person who knows how to turn on the computer will be able to follow your
instructions. On the other hand, if you hired smart people/talents/top
performers/A-players/..., then it's probably because they can
invent/improve/create/deliver/... amazing things. Listen to those people
then and let them tell you what can be done and how. It would be also wise
not to kill their ideas. Even if you have to (for whatever the reason),
then don't do that using weak or worse, dumb arguments.

Probably all of us are familiar with cases where pentesters are kept
artificially on their positions for way too long (think of
tenures/grads/juniors/seniors/...). If the main reason for doing so is to
keep their salaries low(er), then you should be aware they will eventually
be picked up by companies which see value in them, or they'll become a
contractors, or they'll decide to start their own business or... They will
simply fly away somewhere else as soon as they'll see a better opportunity
on the horizon.

If you don't want to make your pentesters feel bad, then appreciate their
work. Believe me, you're not doing it by informing your pentester that the
4 week engagement they're just delivering is an equivalent of their annual
salary. Furthermore, stories told by executives about their yachts,
horses, golf games, new rides and similar don't build morale within the
team too well. Better keep those for your fellow execs.

Another great example of how to loose people quickly is, to give them
unreal deadlines, force them to work after hours and on the weekends only
to tell them afterwards, that you wanted to see how they behave under
stress. Nope, it's not gonna fly.

Offering budget to your pentesters so they can spend it on trainings and
conferences is awesome and a necessary thing. However, even better is to
give them time so they can do their own research. Through research they
learn, they can share with results at conferences, give trainings, gather
experience and also promote your company in many other ways. Allow them
that time so they can focus on what hacking is really about.

--[ Engaging a pentest(er| provider)
You want someone to check security of your assets? Even if you used in the
past security services from different providers, I think you may still
find a useful pointer or two in the following section about how to find
the right one.

First and foremost, remember it's a pentester who will be doing the job
for you and not the company/brand/sales/managers/ads/... Therefore, focus
on pentesters when choosing your next provider or hire those pentesters
directly as freelancers/contractors/self-employed/... Keep also in mind
that pentest is a technical exercise. You don't want people who can talk
about it, you want people who can actually do it.

Some providers are a pure disasters. The have all the know how about
pentesting and they will use it to sell you as many services they can. In
practice however all the technical skills, spirit of hacking, ethics and
quality is long time gone along with people who left these companies.

I'm going to skip the part about where to look for a good
pentesters/companies. If you would need help with that, feel free to
contact me directly and I'll be happy to advise something.

Naturally, first thing to ask about are testers CV's/bio's. Next step
would be to verify by yourself information contained within. Some good
ways of verifying pentester's capabilities would be to see if she:
 * Published security advisories.
 * Has any CVE numbers for vulnerabilities found.
 * Did security research.
 * Wrote custom code/tool.
 * Has some kind of a certificate(s).
 * Presented at conferences.
 * Published technical content in e-zine/book/magazine/blog/...

You should consider in the first place pentesters who demonstrated their
capabilities based on any of the above. However, there's also a tricky
part about it. Some pentesters can probably tick off some or even most of
those boxes but they still won't be technically skilled enough to do a
good job. To find the bests in the field further verification is required.

You should be aware that too many individuals publish for various reasons
advisories that are of a very low impact/value/quality. If you don't
understand contents of the advisory, then don't blindly assume it must be
good.

More and more people is interested in security nowadays. Some really great
tools are available for the masses like
fuzzers/compilers/analysers/scanners/... Unsurprisingly it is now much
easier to use them and find defects in code and potential vulnerabilities
in anything that electricity flows through. Quite often however finders
don't bother or don't know how to verify if the given finding has security
implications or not. They announce that this is a security issue and
that's it. Just so you know, there's not enough people (including vendors)
who have time/willingness/resources to verify/debunk all of those
findings. Many findings ultimately end up being classified as security
vulnerabilities with the CVE number being assigned and/or fix being
released.

I'm pretty sure that if you can read phrack/poc||gtfo/projectzero with
understanding, then you know how to filter out crap and distinguish good
security research/presentation/code/publication. If in doubt, find someone
who can do that and ask for help.

Some companies/markets have culture of interviewing and even training
pentesters before they are allowed to participate in the engagement.
Pentesters might not necessarily like it but it's your money in the end
and you want to make sure that you have the best people for the job. That
should be your next step on the way to choosing the right pentester(s).

If you can afford, then hire different providers for testing different
targets/assets. After some time you'll be able to stick to those who
proved themselves and avoid botchers. Some obvious stuff but good to
remind it at this point:
 * Competition is healthy.
 * Size does matter, sometimes in a positive sometimes in a negative way.
 * Low(est) price usually means that you sacrifice time and/or quality
   (there are exceptions like e.g. freelancer vs. company with
   execs/sales/HR/PR/... you're paying for).
 * One provider will be better in testing A and another will be in testing
   B.

Try to sond a pentester/provider using different (unofficial) channels.
Sadly, there's too many unethical people and companies in this business.
Would you choose a pentesting company for your job where (yes, this
actually happened):
 * Sales ask over the phone customer and not recognising who he is - "Who
   the hell are you"?
 * Chief Officer says after finishing a call - "Cunt will wait for a week
   now" only because customer was late with things on her end?
 * Business strategist admits with disarming honesty - "If i were them I
   would have changed our services long time ago"?

Sometimes customers are after someone who could test
uncommon/peculiar/exotic/complex... target of theirs. In most cases the
only question asked is if that someone has experience in testing such
target? There are chances that they eventually will find someone who
tested something like that before. But be aware of one thing. There are
people and companies who will say literally anything you want to hear only
to get your money. It should be fairly easy to verify their claims though.
There aren't too many companies in the world which can say they have
pentesters with experience in testing almost anything you'll throw at
them. There are however companies with capable people, a real hackers, who
can dissect/adapt/learn/harness/mangle/test/improve/... your target. It is
them and their knowledge you're after and not company's portfolio and its
marketing bullshit. You are reading how to recognise them in the crowd in
this very moment.

Ultimately, what you get after paying for a pentest is a report. Request
sample to avoid any unpleasant surprises [4][5]. It is good to see a
sample but remember that majority of reports are very similar in terms of
their structure and contents. Yes, some might be prettier than others but
that's it. I know of one company who charges its customers big bucks and
justifies it by claiming it's because their reporting is elite and
consistent. But what's the point if they keep missing serious stuff during
the tests? So keep calm and choose wisely.

--[ Pentest planning
From time to time people are asking me how much time do I think is needed
to test Q? That usually triggers a series of questions from my end to
better understand context/expectations/target/limitations/... Scoping is a
tricky beast and without a little of ping-pong it can hardly be done
right. Based on my experience the scoping process is non or at best
semi-transparent to customers. Projects occasionally are under or over
scoped, that happens, everywhere. It is however unacceptable if that
happens intentionally. Some people just want to rip you off and it doesn't
matter to them what you need from a pentest. They will attempt to sell you
as many days as possible or, regardless the target, constant number of
days without any questions asked about the target(s) in scope. In that
case you know you ended up with a money factory and do not expect a
profound testing and customised approach. If something doesn't seem sound
request for the assessment breakdown/plan.

Pentest engagements are delivered within a pre-defined time frames. In
order to avoid confusion and frustration everyone should be on the same
page and know if that time means effort or perhaps the overall
engagement's duration. The difference between one and another can be
substantial and will determine if the total engagement cost will be lower
or higher.

Practically every pentest requires some sort of pre-requisites.  If all
parties involved won't be able to sort that bit before the start date,
then it can have serious impact on the overall delivery. Skipping all
possible scenarios when that can happen, it is clear that it's a customer
in the end who will be mostly impacted by this. Hence, don't wait to the
last minute to sort out what needs to be. Every hour pentester spends her
time on twiddling thumbnails means customer is wasting money. My personal
best was being for seven days on-site waiting. Customer during that time
was sorting out access issues on their end.

Only on rare occasions people who are more security aware request
additional and very specific information from pentesters/providers. For
instance, when discussing pentest details they ask:
 * What tools/techniques/procedures/scenarios will be used during the
   assessment?
 * If any third party assets will be involved in testing, e.g. cloud
   providers/VPSs/VPNs/proxies/... ?
 * How data will flow, how and where it will be stored?
 * What are the risks of something going wrong during testing and
   suggested precautions?

Way too often there are companies contacting pentest providers only to
tick off a compliance box. All they care about is to get it done quickly,
possibly cheap and the report to be all roses. Those guys can be a real
pain in the arse. They will question your every finding, attempt to modify
your report to make it look better, won't tell you everything or will be
trying to bend the reality, they'll give you only access to targets
prepared especially for that occasion while everything else is running on
aid-bands and using sticky tape. If your consultancy skills failed and you
weren't able to show customer there are other and better ways to address
security, then honestly, they deserve to get pwned. History shows that
many companies changed their thinking and started taking things seriously
only after such incidents. Anyway, avoid if possible, not worth it.

Pentesting won't go away anytime soon but at the same time, it might not
be the best option to assess/implement security of the given target
anymore. Depending on the circumstances more appropriate might be to use
devsecops or purple/black/red teaming for that purpose. This is where
consultancy skills comes into play.

--[ Pentest delivery
Probably every pentesting company will communicate to you at some point
that it uses some kind of methodology for testing. The truth is that
pentesters rarely follow methodologies, fully. That can be the case
because:
 * They are not familiar with it.
 * There is no time to follow it within the test's time frame.
 * It is incomplete/wrong/outdated/...
 * Pentester intentionally deviates from it as it seems to be in the best
   interest of the customer.

Sometimes this will lead to various problems and sometimes it's even
better that methodology wasn't followed throughly. It's a gray area I
would say. After all common reason supported by experience is what should
guide a pentester I suppose.

Next issue affects to some extent all pentest providers. Pentesters
performance is measured mainly by their utilisation. The minimal
utilisation varies from one company to another but it starts from around
65% and sky is the limit. There are wool factories where 110% utilisation
is a norm. To meet management/board/stakeholders expectations in this area
pentesters are squeezed like lemons. That means they are frequently forced
to do multitasking. They jump from one engagement on another without time
required to properly prepare to the job. They handle two, three and more
engagements concurrently. This leads to a number of issues like
errors/miscommunication/decrease of quality/burn out/... That's exactly
why finding a right pentester/provider and pentest planning is so
important, to avoid situations like this.

Some people just love meetings. I wouldn't be surprised if someone already
wrote a book or two on the subject. I'll keep it brief then. Daily catch
ups/mails/calls/SCRUM sessions/... make sense only if they serve a
purpose. Otherwise it's waste of time (and money). I remember engagements
where there were three meetings per day with different people involved but
working on the same project. True, it's rare and extreme example but
proves my point. The more and longer the meetings the less time lefts for
testing itself. It's simple as that.

Penetest finished and you got yourself a lengthy report. Depending on its
contents (how it looks is a secondary issue) it will be either good or
not. Some indicators the report is good includes, but are not limited to:
 * It has been customised, e.g. references & recommendations were adjusted
   to technologies and configuration you use, severity rating was calculated
   including any worsening and/or mitigating factors etc.
 * Based on the finding's description you're able to understand the
   problem's root cause and reproduce/verify/remediate the issue.
 * It doesn't contain copypasta from the net, tools or worse, other
   customers reports.
 * It doesn't contain false positives.
 * If not provided by you, it used a reasonable risk matrix (the one from
   OWASP is a good example).
 * If not specified by you, it used a reasonable risk scoring system (e.g.
   CVSS/low/med/hig/Bugbar/custom/...).
 * It was written using correct grammar.

It is only my personal opinion that any medium and higher severity
vulnerability should come with a working proof of concept demonstrating
the problem. The POC||GTFO principle applies here more than anywhere else.
Otherwise customer is flooded with bunch of maybe's/could's/if's and left
with a lot of FUD.

Pentesters should be able to provide customers at any time with the
evidences of work they performed. It is however a good idea to ask them
beforehand if you will need anything extra to avoid disappointments. On
the flip side, pentesters have supporting evidences in case customer would
accuse them for wrongdoing or not doing something. Examples of such
evidences includes, but are not limited to:
 * The times during which testing was conducted.
 * Any customers' data acquired during testing.
 * Logs and dumps from tools that were used.
 * PCAPs containing generated network traffic.
 * Screen captures.

--[ Missed vulnerabilities
As a pentester you can be challenged and asked why the frag didn't you
report some serious vulnerability that was out there? Obviously,
consequences if that would happen can be severe, e.g. lost customer/trust,
reputational/brand damage, lawsuits, target was pwned but not by yourself
etc. Nevertheless, shit happens, and then what? I discussed the issue with
some fellas and we all agreed on one thing - this can occur for various
reasons and there's no one good answer. I know, shocker.

Before the engagement it should be communicated to customer in one way or
another that missed vulnerabilities should be a calculated risk. It's a
separate issue how to minimise such risk. The risk management however is
not my cup of tea. The clause stating that "pentest is a snapshot in
time", or something between those lines, is not enough on its own.

You may have no opportunity to respond to missed vuln situation whatsoever
and then it's most likely game over at this point.

If you failed as a pentester, then of course admitting it would be the
first step. Don't play dumb and look for fault somewhere else.

Assuming that customer wants to discuss with you what went wrong, try to
get from them as much information as possible about the vulnerability they
know about but was overlooked during the pentest. Based on that
information you should be able to determine the root cause for the fail,
e.g.:
 * Target was not in scope.
 * Pentest pre-requisites were not delivered.
 * Some tests/vectors were not covered by the methodology.
 * The vulnerability/technique was not publicly known when testing was
   done.
 * Vulnerability presents itself only when specific conditions occur.
 * Target was in scope but due to limitations testing didn't cover it.
 * Testing tools were not configured properly.
 * Pentesters are only humans and do mistakes too.

Remember that pentesters/pentests are limited by
methodology/scope/time/costs/... and attackers usually aren't tied to any
of these.

During some types of assessments it is more probable to overlook vulns. I
would say that pentesting of web/native/mobile apps is less prone to such
situations as it's very repeatable process. If that happens it's most
likely due to limitations mentioned above.
On the other hand, during pentests involving vulnerability assessment,
networks testing, reverse engineering and most tricky of them all, source
code reviewing, pentesters are more likely to miss things. That's simply
nature of those tests in the great scheme of things. There are too many
random factors for which pentesters could account them all for.

Explain to all interested parties what went wrong and advise what can be
done to improve things to minimise chances of occurring similar situations
in the future. You should also describe methodology that was used in every
detail, how it was followed and support everything with evidences that you
have. Not perfect but it's still better than "sorry, we'll try better next
time".

--[ Credits
Thanks guys for all the discussions over the beers. You know who you are.
Keep doing the amazing things that you do.

--[ References
[1] http://carnal0wnage.attackresearch.com/2012/11/the-biggest-problem-in-computer-security.html
[2] http://blog.silentsignal.eu/2015/04/03/the-story-of-a-pentester-recruitment/
[3] http://www.unicri.it/special_topics/securing_cyberspace/current_activities/hackers_profiling/
[4] http://it.toolbox.com/blogs/securitymonkey/the-worlds-worst-penetration-test-report-by-scumbagpentester-58747
[5] http://ipsec.pl/penetration-testing/2014/writing-meaningful-and-professional-penetration-testing-reports.html