Guest User


a guest
Aug 4th, 2017
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 28.35 KB | None | 0 0
  4. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
  5. ║ Some Dark Corners of Pentesting ║
  6. =-=-=-=-=-=- s1m0n -=-=-=-=-=-=
  9. > Intro
  10. > Sins of the pentesters
  11. > Finding and hiring pentesters
  12. > Retaining pentesters
  13. > Engaging a pentest(er| provider)
  14. > Pentest planning
  15. > Pentest delivery
  16. > Missed vulnerabilities
  17. > Credits
  18. > References
  21. --[ Intro
  22. To be totally honest with you dear reader, I wasn't sure if I want to
  23. write this text at all but at some point there was this one straw that
  24. broke the camel's back. It's a non-technical text revolving around random
  25. issues in pentesting that are often annoying, rarely interesting and
  26. sometimes remain unspoken. Even though I'm mainly focusing in it on
  27. pentesting and pentesters, then I believe that some of the points made
  28. herein are generic enough so they can be also applied to other infosec
  29. areas. I wrote it without any expectations but it would be cool if it
  30. would contribute to some changes for better. And let's be clear about one
  31. thing, the text was not meant to attack anyone. If you feel offended by
  32. it, then there's a big chance that perhaps you're doing something in the
  33. wrong way.
  35. --[ Sins of the pentesters
  36. Howdy fellow pentesters. From time to time we can be perceived by others
  37. as a special breed with better than average computer skills. However,
  38. talking like one tester to another now. We're not free of annoying habits
  39. and... let's call it logic flaws.
  41. We all know it's great to break stuff, isn't it? Similarly, ranting about
  42. things being broken is so common and easy. Everyone can do that. However,
  43. building/fixing/improving/... it is equally hard if not harder. Some
  44. people say defence is sexy. No it's not. It's a constant hard work,
  45. underpaid, quite often frustrating and not appreciated enough by so many.
  46. All it takes to be blamed and finger pointed is one fail, one oversight,
  47. one weak spot. Keep it in mind next time you will rant about devs and
  48. blue-team's incompetence. Sharing is caring. If you can, then contribute
  49. ideas, submit code and suggest solutions to problems after you dropped the
  50. bomb. Surely you can do better than that.
  52. If your pentester's work is limited to running tools written by others and
  53. reading security news, then congrats, you just earned a pro-skiddie award.
  54. It doesn't matter if you're an experienced pentester/ethical
  55. hacker/whitehat/cyber security professional or whatever the name is in use
  56. today. If you're not inquisitive anymore, you became lazy, you burned out,
  57. you look for excuses instead of solutions, you mainly rant about
  58. everything and everyone, you're known nowadays only because you know
  59. everyone else on the security band wagon, you spend more time drinking
  60. booze and attending conferences instead of reading and writing code (don't
  61. get me wrong on this one though, most of us know how to party properly;>),
  62. then know you can do much better than that. Continuously learn and adapt
  63. or... else.
  65. It's not uncommon to encounter pentesters who eagerly express their
  66. opinions on basically anything security related. There's nothing wrong
  67. about it until they have at least some experience on the matter
  68. (preferably not based on factoids). Unfortunately, quite often pentesters
  69. talk without actually knowing or checking facts first. Talking about
  70. memory corruption when they didn't write a single rop chain in their life,
  71. about malware when they never actually analysed a sample, one form of
  72. disclosure being better over another if they published no advisory,
  73. talking about the source code when all they wrote was pascal decades ago
  74. etc. Such approach doesn't give them too much credibility, does it? The
  75. general advise is to stick to what you know or keep it quiet until you do
  76. your homework. If unsure (RTFM|POC)||GTFO.
  78. Oh, and if your work is publicity driven, then it might be a good time to
  79. re-think your approach. It's never too late.
  81. --[ Finding and hiring pentesters
  82. For some time now the "skills gap in security" term is omnipresent and
  83. people are generally getting very excited about this phenomenon. It was
  84. already the case during my uni times circa 2004. In fact, it was the same
  85. long before that. Sure, the problem of skills shortage was called
  86. differently and it wasn't of the same scale as it is presently but
  87. unquestionably the problem was already there. Educated workers, engineers,
  88. programmers, analytics, mathematicians, scientists and other bright minds
  89. were always difficult to find and hire. After they were found, they could
  90. then work on/with "new technologies" and whenever we deal with new
  91. technologies, there's practically always a security factor involved. Think
  92. for a moment of cryptography & intelligence during and after IIWW,
  93. mainframes, ALGOL/COBOL/..., financial institutions, industrial PLCs,
  94. telcos, NASA and more. I believe it means the underlying problem is at
  95. least 60-70 years old and only the names and technologies change. It seems
  96. there will still be a deficit of capable people in a foreseeable future
  97. because nothing indicates otherwise.
  99. Why finding people with IT security capabilities became such a big deal?
  100. If you consider recent years and state-sponsored attacks/groups,
  101. high-profile hacks & data leaks, GDPR/Patriot/Cyber Privacy
  102. Fortification/... acts, national cyber security strategies, exploit
  103. markets, IoT, transportation and generally the way technologies are used
  104. to gather and process information about everyone and everything, the
  105. answer is obvious.
  107. Getting capable people to join the team is really hard[1][2]. The
  108. following observations are from the perspective of being recruited,
  109. occasionally being involved in recruiting others and from numerous
  110. discussions with friends.
  112. If you even think about becoming a respectable and prospering company
  113. providing security services, then take a good care of your employees in
  114. the first place. It is no one else but them who will take you there.
  116. The important question that you should ask yourself before you even start
  117. looking for pentesters is who are you looking for exactly? Knowing your
  118. own expectations regarding the position you're hiring for, you should
  119. consider that security is a specific field comprising of voodoo people
  120. [3]. Let those people know beforehand if you're looking for someone in a
  121. shiny shoes, tie and suit, or perhaps a wool picker who will run tools all
  122. day long, or perhaps you need a security rockstar travelling from one conf
  123. to another etc. You can save yourself and others lot of time/effort/money
  124. by simply being honest and transparent.
  126. Remember that interviews do work both ways. Candidates will know if you
  127. actually spend time reading their CV and any other information they
  128. possibly provided to you. It also doesn't hurt to do a quick web search
  129. about your candidate on your own. If you show at this stage that you don't
  130. care about them they will more likely choose employer that will.
  132. During technical interviews avoid asking blatant questions. Common
  133. examples includes, but are not limited to:
  134. * what service is listening on port A?
  135. * how B works/is implemented in tool C?
  136. * what parameter/argument X will do?
  137. * would you be able to write a script in Y?
  138. * can you run a scan using Z?
  140. If you ask similar questions then it means you're doing it wrong. This is
  141. definitely not the way to find a capable pentester. Instead you could ask
  142. candidate to solve a practical problem or confront him with a hands-on
  143. challenge. Next time you can ask candidate to review a source code, solve
  144. a technical challenge, document her findings and walk you through it.
  145. That would be for a good start.
  147. Please, save candidates time and money. If possible provide them with a
  148. toll free numbers or call them, and don't make them wait for you on the
  149. line for minutes because you had another meeting etc. Reimbursing and/or
  150. participation in candidate's costs (e.g. travel tickets) is an extra mile
  151. that some companies are willing to walk and believe me, they benefit from
  152. it in the long term. Remember that pentesters and voodoo people in general
  153. meet and discuss from time to time despite general opinions about them.
  154. This means no more no less that if you're failing on one candidate it's
  155. very likely that at the same time you're failing on many.
  157. Are you asking your candidates about salary expectations? No one told you
  158. it's millions of billions? Seriously, it's optimal to inform candidates
  159. how much is the offer (even range will do) before anything else. Otherwise
  160. it's again, only waste of everyone's time.
  162. Yes, do not provide candidates with updates, give them no feedback and you
  163. will surely not burn any bridges. Such bad experiences travel via word of
  164. mouth and people will be less eager to apply to/via such companies.
  166. --[ Retaining pentesters
  167. If you hired people only to tell them what to do, then I believe any
  168. person who knows how to turn on the computer will be able to follow your
  169. instructions. On the other hand, if you hired smart people/talents/top
  170. performers/A-players/..., then it's probably because they can
  171. invent/improve/create/deliver/... amazing things. Listen to those people
  172. then and let them tell you what can be done and how. It would be also wise
  173. not to kill their ideas. Even if you have to (for whatever the reason),
  174. then don't do that using weak or worse, dumb arguments.
  176. Probably all of us are familiar with cases where pentesters are kept
  177. artificially on their positions for way too long (think of
  178. tenures/grads/juniors/seniors/...). If the main reason for doing so is to
  179. keep their salaries low(er), then you should be aware they will eventually
  180. be picked up by companies which see value in them, or they'll become a
  181. contractors, or they'll decide to start their own business or... They will
  182. simply fly away somewhere else as soon as they'll see a better opportunity
  183. on the horizon.
  185. If you don't want to make your pentesters feel bad, then appreciate their
  186. work. Believe me, you're not doing it by informing your pentester that the
  187. 4 week engagement they're just delivering is an equivalent of their annual
  188. salary. Furthermore, stories told by executives about their yachts,
  189. horses, golf games, new rides and similar don't build morale within the
  190. team too well. Better keep those for your fellow execs.
  192. Another great example of how to loose people quickly is, to give them
  193. unreal deadlines, force them to work after hours and on the weekends only
  194. to tell them afterwards, that you wanted to see how they behave under
  195. stress. Nope, it's not gonna fly.
  197. Offering budget to your pentesters so they can spend it on trainings and
  198. conferences is awesome and a necessary thing. However, even better is to
  199. give them time so they can do their own research. Through research they
  200. learn, they can share with results at conferences, give trainings, gather
  201. experience and also promote your company in many other ways. Allow them
  202. that time so they can focus on what hacking is really about.
  204. --[ Engaging a pentest(er| provider)
  205. You want someone to check security of your assets? Even if you used in the
  206. past security services from different providers, I think you may still
  207. find a useful pointer or two in the following section about how to find
  208. the right one.
  210. First and foremost, remember it's a pentester who will be doing the job
  211. for you and not the company/brand/sales/managers/ads/... Therefore, focus
  212. on pentesters when choosing your next provider or hire those pentesters
  213. directly as freelancers/contractors/self-employed/... Keep also in mind
  214. that pentest is a technical exercise. You don't want people who can talk
  215. about it, you want people who can actually do it.
  217. Some providers are a pure disasters. The have all the know how about
  218. pentesting and they will use it to sell you as many services they can. In
  219. practice however all the technical skills, spirit of hacking, ethics and
  220. quality is long time gone along with people who left these companies.
  222. I'm going to skip the part about where to look for a good
  223. pentesters/companies. If you would need help with that, feel free to
  224. contact me directly and I'll be happy to advise something.
  226. Naturally, first thing to ask about are testers CV's/bio's. Next step
  227. would be to verify by yourself information contained within. Some good
  228. ways of verifying pentester's capabilities would be to see if she:
  229. * Published security advisories.
  230. * Has any CVE numbers for vulnerabilities found.
  231. * Did security research.
  232. * Wrote custom code/tool.
  233. * Has some kind of a certificate(s).
  234. * Presented at conferences.
  235. * Published technical content in e-zine/book/magazine/blog/...
  237. You should consider in the first place pentesters who demonstrated their
  238. capabilities based on any of the above. However, there's also a tricky
  239. part about it. Some pentesters can probably tick off some or even most of
  240. those boxes but they still won't be technically skilled enough to do a
  241. good job. To find the bests in the field further verification is required.
  243. You should be aware that too many individuals publish for various reasons
  244. advisories that are of a very low impact/value/quality. If you don't
  245. understand contents of the advisory, then don't blindly assume it must be
  246. good.
  248. More and more people is interested in security nowadays. Some really great
  249. tools are available for the masses like
  250. fuzzers/compilers/analysers/scanners/... Unsurprisingly it is now much
  251. easier to use them and find defects in code and potential vulnerabilities
  252. in anything that electricity flows through. Quite often however finders
  253. don't bother or don't know how to verify if the given finding has security
  254. implications or not. They announce that this is a security issue and
  255. that's it. Just so you know, there's not enough people (including vendors)
  256. who have time/willingness/resources to verify/debunk all of those
  257. findings. Many findings ultimately end up being classified as security
  258. vulnerabilities with the CVE number being assigned and/or fix being
  259. released.
  261. I'm pretty sure that if you can read phrack/poc||gtfo/projectzero with
  262. understanding, then you know how to filter out crap and distinguish good
  263. security research/presentation/code/publication. If in doubt, find someone
  264. who can do that and ask for help.
  266. Some companies/markets have culture of interviewing and even training
  267. pentesters before they are allowed to participate in the engagement.
  268. Pentesters might not necessarily like it but it's your money in the end
  269. and you want to make sure that you have the best people for the job. That
  270. should be your next step on the way to choosing the right pentester(s).
  272. If you can afford, then hire different providers for testing different
  273. targets/assets. After some time you'll be able to stick to those who
  274. proved themselves and avoid botchers. Some obvious stuff but good to
  275. remind it at this point:
  276. * Competition is healthy.
  277. * Size does matter, sometimes in a positive sometimes in a negative way.
  278. * Low(est) price usually means that you sacrifice time and/or quality
  279. (there are exceptions like e.g. freelancer vs. company with
  280. execs/sales/HR/PR/... you're paying for).
  281. * One provider will be better in testing A and another will be in testing
  282. B.
  284. Try to sond a pentester/provider using different (unofficial) channels.
  285. Sadly, there's too many unethical people and companies in this business.
  286. Would you choose a pentesting company for your job where (yes, this
  287. actually happened):
  288. * Sales ask over the phone customer and not recognising who he is - "Who
  289. the hell are you"?
  290. * Chief Officer says after finishing a call - "Cunt will wait for a week
  291. now" only because customer was late with things on her end?
  292. * Business strategist admits with disarming honesty - "If i were them I
  293. would have changed our services long time ago"?
  295. Sometimes customers are after someone who could test
  296. uncommon/peculiar/exotic/complex... target of theirs. In most cases the
  297. only question asked is if that someone has experience in testing such
  298. target? There are chances that they eventually will find someone who
  299. tested something like that before. But be aware of one thing. There are
  300. people and companies who will say literally anything you want to hear only
  301. to get your money. It should be fairly easy to verify their claims though.
  302. There aren't too many companies in the world which can say they have
  303. pentesters with experience in testing almost anything you'll throw at
  304. them. There are however companies with capable people, a real hackers, who
  305. can dissect/adapt/learn/harness/mangle/test/improve/... your target. It is
  306. them and their knowledge you're after and not company's portfolio and its
  307. marketing bullshit. You are reading how to recognise them in the crowd in
  308. this very moment.
  310. Ultimately, what you get after paying for a pentest is a report. Request
  311. sample to avoid any unpleasant surprises [4][5]. It is good to see a
  312. sample but remember that majority of reports are very similar in terms of
  313. their structure and contents. Yes, some might be prettier than others but
  314. that's it. I know of one company who charges its customers big bucks and
  315. justifies it by claiming it's because their reporting is elite and
  316. consistent. But what's the point if they keep missing serious stuff during
  317. the tests? So keep calm and choose wisely.
  319. --[ Pentest planning
  320. From time to time people are asking me how much time do I think is needed
  321. to test Q? That usually triggers a series of questions from my end to
  322. better understand context/expectations/target/limitations/... Scoping is a
  323. tricky beast and without a little of ping-pong it can hardly be done
  324. right. Based on my experience the scoping process is non or at best
  325. semi-transparent to customers. Projects occasionally are under or over
  326. scoped, that happens, everywhere. It is however unacceptable if that
  327. happens intentionally. Some people just want to rip you off and it doesn't
  328. matter to them what you need from a pentest. They will attempt to sell you
  329. as many days as possible or, regardless the target, constant number of
  330. days without any questions asked about the target(s) in scope. In that
  331. case you know you ended up with a money factory and do not expect a
  332. profound testing and customised approach. If something doesn't seem sound
  333. request for the assessment breakdown/plan.
  335. Pentest engagements are delivered within a pre-defined time frames. In
  336. order to avoid confusion and frustration everyone should be on the same
  337. page and know if that time means effort or perhaps the overall
  338. engagement's duration. The difference between one and another can be
  339. substantial and will determine if the total engagement cost will be lower
  340. or higher.
  342. Practically every pentest requires some sort of pre-requisites. If all
  343. parties involved won't be able to sort that bit before the start date,
  344. then it can have serious impact on the overall delivery. Skipping all
  345. possible scenarios when that can happen, it is clear that it's a customer
  346. in the end who will be mostly impacted by this. Hence, don't wait to the
  347. last minute to sort out what needs to be. Every hour pentester spends her
  348. time on twiddling thumbnails means customer is wasting money. My personal
  349. best was being for seven days on-site waiting. Customer during that time
  350. was sorting out access issues on their end.
  352. Only on rare occasions people who are more security aware request
  353. additional and very specific information from pentesters/providers. For
  354. instance, when discussing pentest details they ask:
  355. * What tools/techniques/procedures/scenarios will be used during the
  356. assessment?
  357. * If any third party assets will be involved in testing, e.g. cloud
  358. providers/VPSs/VPNs/proxies/... ?
  359. * How data will flow, how and where it will be stored?
  360. * What are the risks of something going wrong during testing and
  361. suggested precautions?
  363. Way too often there are companies contacting pentest providers only to
  364. tick off a compliance box. All they care about is to get it done quickly,
  365. possibly cheap and the report to be all roses. Those guys can be a real
  366. pain in the arse. They will question your every finding, attempt to modify
  367. your report to make it look better, won't tell you everything or will be
  368. trying to bend the reality, they'll give you only access to targets
  369. prepared especially for that occasion while everything else is running on
  370. aid-bands and using sticky tape. If your consultancy skills failed and you
  371. weren't able to show customer there are other and better ways to address
  372. security, then honestly, they deserve to get pwned. History shows that
  373. many companies changed their thinking and started taking things seriously
  374. only after such incidents. Anyway, avoid if possible, not worth it.
  376. Pentesting won't go away anytime soon but at the same time, it might not
  377. be the best option to assess/implement security of the given target
  378. anymore. Depending on the circumstances more appropriate might be to use
  379. devsecops or purple/black/red teaming for that purpose. This is where
  380. consultancy skills comes into play.
  382. --[ Pentest delivery
  383. Probably every pentesting company will communicate to you at some point
  384. that it uses some kind of methodology for testing. The truth is that
  385. pentesters rarely follow methodologies, fully. That can be the case
  386. because:
  387. * They are not familiar with it.
  388. * There is no time to follow it within the test's time frame.
  389. * It is incomplete/wrong/outdated/...
  390. * Pentester intentionally deviates from it as it seems to be in the best
  391. interest of the customer.
  393. Sometimes this will lead to various problems and sometimes it's even
  394. better that methodology wasn't followed throughly. It's a gray area I
  395. would say. After all common reason supported by experience is what should
  396. guide a pentester I suppose.
  398. Next issue affects to some extent all pentest providers. Pentesters
  399. performance is measured mainly by their utilisation. The minimal
  400. utilisation varies from one company to another but it starts from around
  401. 65% and sky is the limit. There are wool factories where 110% utilisation
  402. is a norm. To meet management/board/stakeholders expectations in this area
  403. pentesters are squeezed like lemons. That means they are frequently forced
  404. to do multitasking. They jump from one engagement on another without time
  405. required to properly prepare to the job. They handle two, three and more
  406. engagements concurrently. This leads to a number of issues like
  407. errors/miscommunication/decrease of quality/burn out/... That's exactly
  408. why finding a right pentester/provider and pentest planning is so
  409. important, to avoid situations like this.
  411. Some people just love meetings. I wouldn't be surprised if someone already
  412. wrote a book or two on the subject. I'll keep it brief then. Daily catch
  413. ups/mails/calls/SCRUM sessions/... make sense only if they serve a
  414. purpose. Otherwise it's waste of time (and money). I remember engagements
  415. where there were three meetings per day with different people involved but
  416. working on the same project. True, it's rare and extreme example but
  417. proves my point. The more and longer the meetings the less time lefts for
  418. testing itself. It's simple as that.
  420. Penetest finished and you got yourself a lengthy report. Depending on its
  421. contents (how it looks is a secondary issue) it will be either good or
  422. not. Some indicators the report is good includes, but are not limited to:
  423. * It has been customised, e.g. references & recommendations were adjusted
  424. to technologies and configuration you use, severity rating was calculated
  425. including any worsening and/or mitigating factors etc.
  426. * Based on the finding's description you're able to understand the
  427. problem's root cause and reproduce/verify/remediate the issue.
  428. * It doesn't contain copypasta from the net, tools or worse, other
  429. customers reports.
  430. * It doesn't contain false positives.
  431. * If not provided by you, it used a reasonable risk matrix (the one from
  432. OWASP is a good example).
  433. * If not specified by you, it used a reasonable risk scoring system (e.g.
  434. CVSS/low/med/hig/Bugbar/custom/...).
  435. * It was written using correct grammar.
  437. It is only my personal opinion that any medium and higher severity
  438. vulnerability should come with a working proof of concept demonstrating
  439. the problem. The POC||GTFO principle applies here more than anywhere else.
  440. Otherwise customer is flooded with bunch of maybe's/could's/if's and left
  441. with a lot of FUD.
  443. Pentesters should be able to provide customers at any time with the
  444. evidences of work they performed. It is however a good idea to ask them
  445. beforehand if you will need anything extra to avoid disappointments. On
  446. the flip side, pentesters have supporting evidences in case customer would
  447. accuse them for wrongdoing or not doing something. Examples of such
  448. evidences includes, but are not limited to:
  449. * The times during which testing was conducted.
  450. * Any customers' data acquired during testing.
  451. * Logs and dumps from tools that were used.
  452. * PCAPs containing generated network traffic.
  453. * Screen captures.
  455. --[ Missed vulnerabilities
  456. As a pentester you can be challenged and asked why the frag didn't you
  457. report some serious vulnerability that was out there? Obviously,
  458. consequences if that would happen can be severe, e.g. lost customer/trust,
  459. reputational/brand damage, lawsuits, target was pwned but not by yourself
  460. etc. Nevertheless, shit happens, and then what? I discussed the issue with
  461. some fellas and we all agreed on one thing - this can occur for various
  462. reasons and there's no one good answer. I know, shocker.
  464. Before the engagement it should be communicated to customer in one way or
  465. another that missed vulnerabilities should be a calculated risk. It's a
  466. separate issue how to minimise such risk. The risk management however is
  467. not my cup of tea. The clause stating that "pentest is a snapshot in
  468. time", or something between those lines, is not enough on its own.
  470. You may have no opportunity to respond to missed vuln situation whatsoever
  471. and then it's most likely game over at this point.
  473. If you failed as a pentester, then of course admitting it would be the
  474. first step. Don't play dumb and look for fault somewhere else.
  476. Assuming that customer wants to discuss with you what went wrong, try to
  477. get from them as much information as possible about the vulnerability they
  478. know about but was overlooked during the pentest. Based on that
  479. information you should be able to determine the root cause for the fail,
  480. e.g.:
  481. * Target was not in scope.
  482. * Pentest pre-requisites were not delivered.
  483. * Some tests/vectors were not covered by the methodology.
  484. * The vulnerability/technique was not publicly known when testing was
  485. done.
  486. * Vulnerability presents itself only when specific conditions occur.
  487. * Target was in scope but due to limitations testing didn't cover it.
  488. * Testing tools were not configured properly.
  489. * Pentesters are only humans and do mistakes too.
  491. Remember that pentesters/pentests are limited by
  492. methodology/scope/time/costs/... and attackers usually aren't tied to any
  493. of these.
  495. During some types of assessments it is more probable to overlook vulns. I
  496. would say that pentesting of web/native/mobile apps is less prone to such
  497. situations as it's very repeatable process. If that happens it's most
  498. likely due to limitations mentioned above.
  499. On the other hand, during pentests involving vulnerability assessment,
  500. networks testing, reverse engineering and most tricky of them all, source
  501. code reviewing, pentesters are more likely to miss things. That's simply
  502. nature of those tests in the great scheme of things. There are too many
  503. random factors for which pentesters could account them all for.
  505. Explain to all interested parties what went wrong and advise what can be
  506. done to improve things to minimise chances of occurring similar situations
  507. in the future. You should also describe methodology that was used in every
  508. detail, how it was followed and support everything with evidences that you
  509. have. Not perfect but it's still better than "sorry, we'll try better next
  510. time".
  512. --[ Credits
  513. Thanks guys for all the discussions over the beers. You know who you are.
  514. Keep doing the amazing things that you do.
  516. --[ References
  517. [1]
  518. [2]
  519. [3]
  520. [4]
  521. [5]
Add Comment
Please, Sign In to add comment