joemccray

Command-Line Log Analysis

Feb 20th, 2016
10,631
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. ##################################
  2. # Pentester Academy Log Analysis #
  3. ##################################
  4.  
  5. I'm doing this set of videos for my good friend Vivek Ramachandran at SecurityTube.net/PentesterAcademy.com
  6.  
  7.  
  8.  
  9.  
  10.  
  11. ##########
  12. # VMWare #
  13. ##########
  14. - For this workshop you'll need the latest version of VMWare Workstation (Windows), Fusion (Mac), or Player.
  15.  
  16. - Although you can get the VM to run in VirtualBox, I will not be supporting this configuration for this class.
  17.  
  18. VM for these labs
  19. -----------------
  20. https://s3.amazonaws.com/infosecaddictsvirtualmachines/Win7x64.zip
  21. username: workshop
  22. password: password
  23.  
  24.  
  25.  
  26.  
  27. ##############################################
  28. # Log Analysis with Linux command-line tools #
  29. ##############################################
  30. The following command line executables are found in the Mac as well as most Linux Distributions.
  31.  
  32. cat – prints the content of a file in the terminal window
  33. grep – searches and filters based on patterns
  34. awk – can sort each row into fields and display only what is needed
  35. sed – performs find and replace functions
  36. sort – arranges output in an order
  37. uniq – compares adjacent lines and can report, filter or provide a count of duplicates
  38.  
  39.  
  40.  
  41. ###############
  42. # Apache Logs #
  43. ###############
  44.  
  45. Reference:
  46. http://www.the-art-of-web.com/system/logs/
  47.  
  48. wget https://s3.amazonaws.com/SecureNinja/Python/access_log
  49.  
  50.  
  51. You want to list all user agents ordered by the number of times they appear (descending order):
  52.  
  53. awk -F\" '{print $6}' access_log | sort | uniq -c | sort -fr
  54.  
  55.  
  56.  
  57. Using the default separator which is any white-space (spaces or tabs) we get the following:
  58.  
  59. awk '{print $1}' access_log # ip address (%h)
  60. awk '{print $2}' access_log # RFC 1413 identity (%l)
  61. awk '{print $3}' access_log # userid (%u)
  62. awk '{print $4,5}' access_log # date/time (%t)
  63. awk '{print $9}' access_log # status code (%>s)
  64. awk '{print $10}' access_log # size (%b)
  65.  
  66. You might notice that we've missed out some items. To get to them we need to set the delimiter to the " character which changes the way the lines are 'exploded' and allows the following:
  67.  
  68. awk -F\" '{print $2}' access_log # request line (%r)
  69. awk -F\" '{print $4}' access_log # referer
  70. awk -F\" '{print $6}' access_log # user agent
  71.  
  72.  
  73. awk -F\" '{print $6}' access_log \
  74. | sed 's/(\([^;]\+; [^;]\+\)[^)]*)/(\1)/' \
  75. | sort | uniq -c | sort -fr
  76.  
  77.  
  78. The next step is to start filtering the output so you can narrow down on a certain page or referer. Would you like to know which pages Google has been requesting from your site?
  79.  
  80. awk -F\" '($6 ~ /Googlebot/){print $2}' access_log | awk '{print $2}'
  81. Or who's been looking at your guestbook?
  82.  
  83. awk -F\" '($2 ~ /guestbook\.html/){print $6}' access_log
  84.  
  85.  
  86. Reference:
  87. https://blog.nexcess.net/2011/01/21/one-liners-for-apache-log-files/
  88.  
  89. # top 20 URLs from the last 5000 hits
  90. tail -5000 ./access_log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
  91. tail -5000 ./access_log | awk '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
  92.  
  93. # top 20 URLS excluding POST data from the last 5000 hits
  94. tail -5000 ./access_log | awk -F"[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
  95. tail -5000 ./access_log | awk -F"[ ?]" '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
  96.  
  97. # top 20 IPs from the last 5000 hits
  98. tail -5000 ./access_log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
  99. tail -5000 ./access_log | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
  100.  
  101. # top 20 URLs requested from a certain ip from the last 5000 hits
  102. IP=1.2.3.4; tail -5000 ./access_log | grep $IP | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
  103. IP=1.2.3.4; tail -5000 ./access_log | awk -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
  104.  
  105. # top 20 URLS requested from a certain ip excluding, excluding POST data, from the last 5000 hits
  106. IP=1.2.3.4; tail -5000 ./access_log | fgrep $IP | awk -F "[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
  107. IP=1.2.3.4; tail -5000 ./access_log | awk -F"[ ?]" -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
  108.  
  109. # top 20 referrers from the last 5000 hits
  110. tail -5000 ./access_log | awk '{print $11}' | tr -d '"' | sort | uniq -c | sort -rn | head -20
  111. tail -5000 ./access_log | awk '{freq[$11]++} END {for (x in freq) {print freq[x], x}}' | tr -d '"' | sort -rn | head -20
  112.  
  113. # top 20 user agents from the last 5000 hits
  114. tail -5000 ./access_log | cut -d\ -f12- | sort | uniq -c | sort -rn | head -20
  115.  
  116. # sum of data (in MB) transferred in the last 5000 hits
  117. tail -5000 ./access_log | awk '{sum+=$10} END {print sum/1048576}'
  118.  
  119.  
  120. ##############
  121. # Cisco Logs #
  122. ##############
  123.  
  124. wget https://s3.amazonaws.com/StrategicSec-Files/LogAnalysis/cisco.log
  125.  
  126.  
  127. AWK Basics
  128. ----------
  129. To quickly demonstrate the print feature in awk, we can instruct it to show only the 5th word of each line. Here we will print $5. Only the last 4 lines are being shown for brevity.
  130.  
  131. cat cisco.log | awk '{print $5}' | tail -n 4
  132.  
  133.  
  134.  
  135.  
  136. Looking at a large file would still produce a large amount of output. A more useful thing to do might be to output every entry found in “$5”, group them together, count them, then sort them from the greatest to least number of occurrences. This can be done by piping the output through “sort“, using “uniq -c” to count the like entries, then using “sort -rn” to sort it in reverse order.
  137.  
  138. cat cisco.log | awk '{print $5}'| sort | uniq -c | sort -rn
  139.  
  140.  
  141.  
  142.  
  143. While that’s sort of cool, it is obvious that we have some garbage in our output. Evidently we have a few lines that aren’t conforming to the output we expect to see in $5. We can insert grep to filter the file prior to feeding it to awk. This insures that we are at least looking at lines of text that contain “facility-level-mnemonic”.
  144.  
  145. cat cisco.log | grep %[a-zA-Z]*-[0-9]-[a-zA-Z]* | awk '{print $5}' | sort | uniq -c | sort -rn
  146.  
  147.  
  148.  
  149.  
  150.  
  151. Now that the output is cleaned up a bit, it is a good time to investigate some of the entries that appear most often. One way to see all occurrences is to use grep.
  152.  
  153. cat cisco.log | grep %LINEPROTO-5-UPDOWN:
  154.  
  155. cat cisco.log | grep %LINEPROTO-5-UPDOWN:| awk '{print $10}' | sort | uniq -c | sort -rn
  156.  
  157. cat cisco.log | grep %LINEPROTO-5-UPDOWN:| sed 's/,//g' | awk '{print $10}' | sort | uniq -c | sort -rn
  158.  
  159. cat cisco.log | grep %LINEPROTO-5-UPDOWN:| sed 's/,//g' | awk '{print $10 " changed to " $14}' | sort | uniq -c | sort -rn
  160.  
  161.  
  162.  
  163.  
  164. #################################
  165. # Using Python for log analysis #
  166. #################################
  167.  
  168.  
  169.  
  170.  
  171. ###########################################
  172. # Python Basics Lesson 1: Simple Printing #
  173. ###########################################
  174.  
  175. >>> print 1
  176.  
  177. >>> print hello
  178.  
  179. >>> print "hello"
  180.  
  181. >>> print "Today we are learning Python."
  182.  
  183.  
  184.  
  185. ###################################################
  186. # Python Basics Lesson 2: Simple Numbers and Math #
  187. ###################################################
  188.  
  189. >>> 2+2
  190.  
  191. >>> 6-3
  192.  
  193. >>> 18/7
  194.  
  195. >>> 18.0/7
  196.  
  197. >>> 18.0/7.0
  198.  
  199. >>> 18/7
  200.  
  201. >>> 9%4
  202.  
  203. >>> 8%4
  204.  
  205. >>> 8.75%.5
  206.  
  207. >>> 6.*7
  208.  
  209. >>> 6*6*6
  210.  
  211. >>> 6**3
  212.  
  213. >>> 5**12
  214.  
  215. >>> -5**4
  216.  
  217.  
  218.  
  219.  
  220.  
  221.  
  222. #####################################
  223. # Python Basics Lesson 3: Variables #
  224. #####################################
  225.  
  226. >>> x=18
  227.  
  228. >>> x+15
  229.  
  230. >>> x**3
  231.  
  232. >>> y=54
  233.  
  234. >>> x+y
  235.  
  236. >>> age=input("Enter number here: ")
  237. 43
  238.  
  239. >>> age+32
  240.  
  241. >>> age**3
  242.  
  243. >>> fname = raw_input("Enter your first name: ")
  244.  
  245. >>> lname = raw_input("Enter your first name: ")
  246.  
  247. >>> fname = raw_input("Enter your name: ")
  248. Enter your name: Joe
  249.  
  250. >>> lname = raw_input("Enter your name: ")
  251. Enter your name: McCray
  252.  
  253. >>> print fname
  254. Joe
  255.  
  256. >>> print lname
  257. McCray
  258.  
  259. >>> print fname lname
  260.  
  261. >>> print fname+lname
  262. JoeMcCray
  263.  
  264.  
  265.  
  266. NOTE:
  267. Use "input() for integers and expressions, and use raw_input() when you are dealing with strings.
  268.  
  269.  
  270.  
  271.  
  272.  
  273. #################################################
  274. # Python Basics Lesson 4: Modules and Functions #
  275. #################################################
  276.  
  277. >>> 5**4
  278.  
  279. >>> pow(5,4)
  280.  
  281. >>> abs(-18)
  282.  
  283. >>> abs(5)
  284.  
  285. >>> floor(18.7)
  286.  
  287. >>> import math
  288.  
  289. >>> math.floor(18.7)
  290.  
  291. >>> math.sqrt(81)
  292.  
  293. >>> joe = math.sqrt
  294.  
  295. >>> joe(9)
  296.  
  297. >>> joe=math.floor
  298.  
  299. >>> joe(19.8)
  300.  
  301.  
  302.  
  303.  
  304.  
  305.  
  306.  
  307.  
  308.  
  309. ###################################
  310. # Python Basics Lesson 5: Strings #
  311. ###################################
  312.  
  313. >>> "XSS"
  314.  
  315. >>> 'SQLi'
  316.  
  317. >>> "Joe's a python lover"
  318.  
  319. >>> 'Joe\'s a python lover'
  320.  
  321. >>> "Joe said \"InfoSec is fun\" to me"
  322.  
  323. >>> a = "Joe"
  324.  
  325. >>> b = "McCray"
  326.  
  327. >>> a, b
  328.  
  329. >>> a+b
  330.  
  331.  
  332.  
  333.  
  334.  
  335.  
  336.  
  337.  
  338. ########################################
  339. # Python Basics Lesson 6: More Strings #
  340. ########################################
  341.  
  342. >>> num = 10
  343.  
  344. >>> num + 2
  345.  
  346. >>> "The number of open ports found on this system is " + num
  347.  
  348. >>> num = str(18)
  349.  
  350. >>> "There are " + num + " vulnerabilities found in this environment."
  351.  
  352. >>> num2 = 46
  353.  
  354. >>> "As of 08/20/2012, the number of states that enacted the Security Breach Notification Law is " + `num2`
  355.  
  356.  
  357.  
  358. NOTE:
  359. Use "input() for integers and expressions, and use raw_input() when you are dealing with strings.
  360.  
  361.  
  362.  
  363.  
  364.  
  365.  
  366.  
  367. ###############################################
  368. # Python Basics Lesson 7: Sequences and Lists #
  369. ###############################################
  370.  
  371. >>> attacks = ['Stack Overflow', 'Heap Overflow', 'Integer Overflow', 'SQL Injection', 'Cross-Site Scripting', 'Remote File Include']
  372.  
  373. >>> attacks
  374. ['Stack Overflow', 'Heap Overflow', 'Integer Overflow', 'SQL Injection', 'Cross-Site Scripting', 'Remote File Include']
  375.  
  376. >>> attacks[3]
  377. 'SQL Injection'
  378.  
  379. >>> attacks[-2]
  380. 'Cross-Site Scripting'
  381.  
  382.  
  383.  
  384.  
  385.  
  386.  
  387. ########################################
  388. # Python Basics Level 8: If Statement #
  389. ########################################
  390. >>> attack="SQLI"
  391. >>> if attack=="SQLI":
  392. print 'The attacker is using SQLI'
  393.  
  394. >>> attack="XSS"
  395. >>> if attack=="SQLI":
  396. print 'The attacker is using SQLI'
  397.  
  398.  
  399. #############################
  400. # Reference Videos To Watch #
  401. #############################
  402. Here is your first set of youtube videos that I'd like for you to watch:
  403. https://www.youtube.com/playlist?list=PLEA1FEF17E1E5C0DA (watch videos 1-10)
  404.  
  405.  
  406.  
  407.  
  408.  
  409. #####################################
  410. # Lesson 9: Intro to Log Analysis #
  411. #####################################
  412.  
  413. Login to your StrategicSec Ubuntu machine. You can download the VM from the following link:
  414.  
  415. https://s3.amazonaws.com/StrategicSec-VMs/Strategicsec-Ubuntu-VPN-163.zip
  416. username: strategicsec
  417. password: strategicsec
  418.  
  419. Then execute the following commands:
  420. ---------------------------------------------------------------------------------------------------------
  421.  
  422.  
  423. wget https://s3.amazonaws.com/SecureNinja/Python/access_log
  424.  
  425.  
  426. cat access_log | grep 141.101.80.188
  427.  
  428. cat access_log | grep 141.101.80.187
  429.  
  430. cat access_log | grep 108.162.216.204
  431.  
  432. cat access_log | grep 173.245.53.160
  433.  
  434. ---------------------------------------------------------
  435.  
  436. Google the following terms:
  437. - Python read file
  438. - Python read line
  439. - Python read from file
  440.  
  441.  
  442.  
  443.  
  444. ########################################################
  445. # Lesson 10: Use Python to read in a file line by line #
  446. ########################################################
  447.  
  448.  
  449. Reference:
  450. http://cmdlinetips.com/2011/08/three-ways-to-read-a-text-file-line-by-line-in-python/
  451.  
  452.  
  453.  
  454.  
  455.  
  456.  
  457. Let's have some fun.....
  458.  
  459.  
  460. >>> f = open('access_log', "r")
  461.  
  462. >>> lines = f.readlines()
  463.  
  464. >>> print lines
  465.  
  466. >>> lines[0]
  467.  
  468. >>> lines[10]
  469.  
  470. >>> lines[50]
  471.  
  472. >>> lines[1000]
  473.  
  474. >>> lines[5000]
  475.  
  476. >>> lines[10000]
  477.  
  478. >>> print len(lines)
  479.  
  480.  
  481.  
  482.  
  483.  
  484.  
  485.  
  486.  
  487.  
  488. ---------------------------------------------------------
  489. vi logread1.py
  490.  
  491.  
  492. ## Open the file with read only permit
  493. f = open('access_log', "r")
  494.  
  495. ## use readlines to read all lines in the file
  496. ## The variable "lines" is a list containing all lines
  497. lines = f.readlines()
  498.  
  499. print lines
  500.  
  501.  
  502. ## close the file after reading the lines.
  503. f.close()
  504.  
  505. ---------------------------------------------------------
  506.  
  507.  
  508. Google the following:
  509. - python difference between readlines and readline
  510. - python readlines and readline
  511.  
  512.  
  513.  
  514.  
  515.  
  516. #################################
  517. # Lesson 11: A quick challenge #
  518. #################################
  519.  
  520. Can you write an if/then statement that looks for this IP and print "Found it"?
  521.  
  522.  
  523. 141.101.81.187
  524.  
  525.  
  526.  
  527.  
  528.  
  529.  
  530. ---------------------------------------------------------
  531. Hint 1: Use Python to look for a value in a list
  532.  
  533. Reference:
  534. http://www.wellho.net/mouth/1789_Looking-for-a-value-in-a-list-Python.html
  535.  
  536.  
  537.  
  538.  
  539. ---------------------------------------------------------
  540. Hint 2: Use Python to prompt for user input
  541.  
  542. Reference:
  543. http://www.cyberciti.biz/faq/python-raw_input-examples/
  544.  
  545.  
  546.  
  547.  
  548. ---------------------------------------------------------
  549. Hint 3: Use Python to search for a string in a list
  550.  
  551. Reference:
  552. http://stackoverflow.com/questions/4843158/check-if-a-python-list-item-contains-a-string-inside-another-string
  553.  
  554.  
  555.  
  556.  
  557.  
  558. Here is my solution:
  559. -------------------
  560. $ python
  561. >>> f = open('access_log', "r")
  562. >>> lines = f.readlines()
  563. >>> ip = '141.101.81.187'
  564. >>> for string in lines:
  565. ... if ip in string:
  566. ... print(string)
  567.  
  568.  
  569.  
  570.  
  571. Here is one student's solution - can you please explain each line of this code to me?
  572. -------------------------------------------------------------------------------------
  573. #!/usr/bin/python
  574.  
  575. f = open('access_log')
  576.  
  577. strUsrinput = raw_input("Enter IP Address: ")
  578.  
  579. for line in iter(f):
  580. ip = line.split(" - ")[0]
  581. if ip == strUsrinput:
  582. print line
  583.  
  584. f.close()
  585.  
  586.  
  587.  
  588.  
  589. -------------------------------
  590.  
  591. Working with another student after class we came up with another solution:
  592.  
  593. #!/usr/bin/env python
  594.  
  595.  
  596. # This line opens the log file
  597. f=open('access_log',"r")
  598.  
  599. # This line takes each line in the log file and stores it as an element in the list
  600. lines = f.readlines()
  601.  
  602.  
  603. # This lines stores the IP that the user types as a var called userinput
  604. userinput = raw_input("Enter the IP you want to search for: ")
  605.  
  606.  
  607.  
  608. # This combination for loop and nested if statement looks for the IP in the list called lines and prints the entire line if found.
  609. for ip in lines:
  610. if ip.find(userinput) != -1:
  611. print ip
  612.  
  613.  
  614.  
  615. ##################################################
  616. # Lesson 12: Look for web attacks in a log file #
  617. ##################################################
  618.  
  619. In this lab we will be looking at the scan_log.py script and it will scan the server log to find out common hack attempts within your web server log.
  620. Supported attacks:
  621. 1. SQL Injection
  622. 2. Local File Inclusion
  623. 3. Remote File Inclusion
  624. 4. Cross-Site Scripting
  625.  
  626.  
  627.  
  628. wget https://s3.amazonaws.com/SecureNinja/Python/scan_log.py
  629.  
  630. The usage for scan_log.py is simple. You feed it an apache log file.
  631.  
  632. cat scan_log.py | less (use your up/down arrow keys to look through the file)
  633.  
  634.  
  635.  
  636.  
  637.  
  638. ################################
  639. # Log Analysis with Powershell #
  640. ################################
  641.  
  642. VM for these labs
  643. -----------------
  644. https://s3.amazonaws.com/infosecaddictsvirtualmachines/Win7x64.zip
  645. username: workshop
  646. password: password
  647.  
  648.  
  649. You can do the updates in the Win7 VM (yes, it is a lot of updates).
  650.  
  651. You'll need to create directory in the Win7 VM called "c:\ps"
  652.  
  653. #####################
  654. # Powershell Basics #
  655. #####################
  656.  
  657. PowerShell is Microsoft’s new scripting language that has been built in since the release Vista.
  658.  
  659. PowerShell file extension end in .ps1 .
  660.  
  661. An important note is that you cannot double click on a PowerShell script to execute it.
  662.  
  663. To open a PowerShell command prompt either hit Windows Key + R and type in PowerShell or Start -> All Programs -> Accessories -> Windows PowerShell -> Windows PowerShell.
  664.  
  665. dir
  666. cd
  667. ls
  668. cd c:\
  669.  
  670.  
  671. To obtain a list of cmdlets, use the Get-Command cmdlet
  672.  
  673. Get-Command
  674.  
  675.  
  676.  
  677. You can use the Get-Alias cmdlet to see a full list of aliased commands.
  678.  
  679. Get-Alias
  680.  
  681.  
  682.  
  683. Don't worry you won't blow up your machine with Powershell
  684. Get-Process | stop-process What will this command do?
  685. Get-Process | stop-process -whatif
  686.  
  687.  
  688. To get help with a cmdlet, use the Get-Help cmdlet along with the cmdlet you want information about.
  689.  
  690. Get-Help Get-Command
  691.  
  692. Get-Help Get-Service –online
  693.  
  694. Get-Service -Name TermService, Spooler
  695.  
  696. Get-Service –N BITS
  697.  
  698. Start-Transcript
  699.  
  700. PowerShell variables begin with the $ symbol. First lets create a variable
  701.  
  702. $serv = Get-Service –N Spooler
  703.  
  704. To see the value of a variable you can just call it in the terminal.
  705.  
  706. $serv
  707.  
  708. $serv.gettype().fullname
  709.  
  710.  
  711. Get-Member is another extremely useful cmdlet that will enumerate the available methods and properties of an object. You can pipe the object to Get-Member or pass it in
  712.  
  713. $serv | Get-Member
  714.  
  715. Get-Member -InputObject $serv
  716.  
  717.  
  718.  
  719.  
  720.  
  721. Let’s use a method and a property with our object.
  722.  
  723. $serv.Status
  724. $serv.Stop()
  725. $serv.Refresh()
  726. $serv.Status
  727. $serv.Start()
  728. $serv.Refresh()
  729. $serv.Status
  730.  
  731.  
  732.  
  733.  
  734. Methods can return properties and properties can have sub properties. You can chain them together by appending them to the first call.
  735.  
  736.  
  737.  
  738. #############################
  739. # Simple Event Log Analysis #
  740. #############################
  741.  
  742. Step 1: Dump the event logs
  743. ---------------------------
  744. The first thing to do is to dump them into a format that facilitates later processing with Windows PowerShell.
  745.  
  746. To dump the event log, you can use the Get-EventLog and the Exportto-Clixml cmdlets if you are working with a traditional event log such as the Security, Application, or System event logs.
  747. If you need to work with one of the trace logs, use the Get-WinEvent and the ExportTo-Clixml cmdlets.
  748.  
  749. Get-EventLog -LogName application | Export-Clixml Applog.xml
  750.  
  751. type .\Applog.xml
  752.  
  753. $logs = "system","application","security"
  754.  
  755. The % symbol is an alias for the Foreach-Object cmdlet. It is often used when working interactively from the Windows PowerShell console
  756.  
  757. $logs | % { get-eventlog -LogName $_ | Export-Clixml "$_.xml" }
  758.  
  759.  
  760.  
  761. Step 2: Import the event log of interest
  762. ----------------------------------------
  763. To parse the event logs, use the Import-Clixml cmdlet to read the stored XML files.
  764. Store the results in a variable.
  765. Let's take a look at the commandlets Where-Object, Group-Object, and Select-Object.
  766.  
  767. The following two commands first read the exported security log contents into a variable named $seclog, and then the five oldest entries are obtained.
  768.  
  769. $seclog = Import-Clixml security.xml
  770.  
  771. $seclog | select -Last 5
  772.  
  773.  
  774. Cool trick from one of our students named Adam. This command allows you to look at the logs for the last 24 hours:
  775.  
  776. Get-EventLog Application -After (Get-Date).AddDays(-1)
  777.  
  778. You can use '-after' and '-before' to filter date ranges
  779.  
  780. One thing you must keep in mind is that once you export the security log to XML, it is no longer protected by anything more than the NFTS and share permissions that are assigned to the location where you store everything.
  781. By default, an ordinary user does not have permission to read the security log.
  782.  
  783.  
  784. Step 3: Drill into a specific entry
  785. -----------------------------------
  786. To view the entire contents of a specific event log entry, choose that entry, send the results to the Format-List cmdlet, and choose all of the properties.
  787.  
  788.  
  789. $seclog | select -first 1 | fl *
  790.  
  791. The message property contains the SID, account name, user domain, and privileges that are assigned for the new login.
  792.  
  793.  
  794. ($seclog | select -first 1).message
  795.  
  796. (($seclog | select -first 1).message).gettype()
  797.  
  798.  
  799.  
  800. In the *nix world you often want a count of something (wc -l).
  801. How often is the SeSecurityPrivilege privilege mentioned in the message property?
  802. To obtain this information, pipe the contents of the security log to a Where-Object to filter the events, and then send the results to the Measure-Object cmdlet to determine the number of events:
  803.  
  804. $seclog | ? { $_.message -match 'SeSecurityPrivilege'} | measure
  805.  
  806. If you want to ensure that only event log entries return that contain SeSecurityPrivilege in their text, use Group-Object to gather the matches by the EventID property.
  807.  
  808.  
  809. $seclog | ? { $_.message -match 'SeSecurityPrivilege'} | group eventid
  810.  
  811. Because importing the event log into a variable from the stored XML results in a collection of event log entries, it means that the count property is also present.
  812. Use the count property to determine the total number of entries in the event log.
  813.  
  814. $seclog.Count
  815.  
  816.  
  817.  
  818.  
  819.  
  820.  
  821. ############################
  822. # Simple Log File Analysis #
  823. ############################
  824.  
  825.  
  826. You'll need to create the directory c:\ps and download sample iss log http://pastebin.com/raw.php?i=LBn64cyA
  827.  
  828.  
  829. mkdir c:\ps
  830. cd c:\ps
  831. (new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=LBn64cyA", "c:\ps\u_ex1104.log")
  832.  
  833.  
  834.  
  835.  
  836.  
  837.  
  838.  
  839.  
  840. ###############################################
  841. # Intrusion Analysis Using Windows PowerShell #
  842. ###############################################
  843.  
  844. Download sample file http://pastebin.com/raw.php?i=ysnhXxTV into the c:\ps directory
  845.  
  846.  
  847.  
  848.  
  849.  
  850. (new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=ysnhXxTV", "c:\ps\CiscoLogFileExamples.txt")
  851.  
  852. Select-String 192.168.208.63 .\CiscoLogFileExamples.txt
  853.  
  854.  
  855.  
  856.  
  857. The Select-String cmdlet searches for text and text patterns in input strings and files. You can use it like Grep in UNIX and Findstr in Windows.
  858.  
  859. Select-String 192.168.208.63 .\CiscoLogFileExamples.txt | select line
  860.  
  861.  
  862.  
  863.  
  864. To see how many connections are made when analyzing a single host, the output from that can be piped to another command: Measure-Object.
  865.  
  866. Select-String 192.168.208.63 .\CiscoLogFileExamples.txt | select line | Measure-Object
  867.  
  868.  
  869.  
  870. To select all IP addresses in the file expand the matches property, select the value, get unique values and measure the output.
  871.  
  872. Select-String “\b(?:\d{1,3}\.){3}\d{1,3}\b” .\CiscoLogFileExamples.txt | select -ExpandProperty matches | select -ExpandProperty value | Sort-Object -Unique | Measure-Object
  873.  
  874.  
  875.  
  876. Removing Measure-Object shows all the individual IPs instead of just the count of the IP addresses. The Measure-Object command counts the IP addresses.
  877.  
  878. Select-String “\b(?:\d{1,3}\.){3}\d{1,3}\b” .\CiscoLogFileExamples.txt | select -ExpandProperty matches | select -ExpandProperty value | Sort-Object -Unique
  879.  
  880.  
  881. In order to determine which IP addresses have the most communication the last commands are removed to determine the value of the matches. Then the group command is issued on the piped output to group all the IP addresses (value), and then sort the objects by using the alias for Sort-Object: sort count –des.
  882. This sorts the IP addresses in a descending pattern as well as count and deliver the output to the shell.
  883.  
  884. Select-String “\b(?:\d{1,3}\.){3}\d{1,3}\b” .\CiscoLogFileExamples.txt | select -ExpandProperty matches | select value | group value | sort count -des
  885.  
  886.  
  887.  
  888.  
  889. This will get the setting for logs in the windows firewall which should be enabled in GPO policy for analysis.
  890. The command shows that the Firewall log is at:
  891. %systemroot%\system32\LogFiles\Firewall\pfirewall.log, in order to open the file PowerShell will need to be run with administrative privileges.
  892.  
  893.  
  894. First step is to get the above command into a variable using script logic.
  895. Thankfully PowerShell has a built-in integrated scripting environment, PowerShell.ise.
  896.  
  897. netsh advfirewall show allprofiles | Select-String FileName | select -ExpandProperty line | Select-String “%systemroot%.+\.log" | select -ExpandProperty matches | select -ExpandProperty value | sort –uniq
  898.  
  899.  
  900. ##############################################
  901. # Parsing Log files using windows PowerShell #
  902. ##############################################
  903.  
  904. Download the sample IIS log http://pastebin.com/LBn64cyA
  905.  
  906.  
  907. (new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=LBn64cyA", "c:\ps\u_ex1104.log")
  908.  
  909. Get-Content ".\*log" | ? { ($_ | Select-String "WebDAV")}
  910.  
  911.  
  912.  
  913. The above command would give us all the WebDAV requests.
  914.  
  915. To filter this to a particular user name, use the below command:
  916.  
  917. Get-Content ".\*log" | ? { ($_ | Select-String "WebDAV") -and ($_ | Select-String "OPTIONS")}
  918.  
  919.  
  920.  
  921. Some more options that will be more commonly required :
  922.  
  923. For Outlook Web Access : Replace WebDAV with OWA
  924.  
  925. For EAS : Replace WebDAV with Microsoft-server-activesync
  926.  
  927. For ECP : Replace WebDAV with ECP
  928.  
  929.  
  930.  
  931. To find out the count of the EWS request we can go ahead and run the below command
  932.  
  933. (Get-Content ".\*log" | ? { ($_ | Select-String "WebDAV") -and ($_ | Select-String "Useralias")}).count
  934.  
  935.  
  936.  
  937.  
  938.  
  939.  
  940.  
  941.  
  942.  
  943. Explain to me how this script works.
Add Comment
Please, Sign In to add comment