View difference between Paste ID: 15Zee2cP and AKiC3EhQ
SHOW: | | - or go back to the newest paste.
1
#################
2
#  Log Analysis #
3
#################
4
5
6
##########
7
# VMWare #
8
##########
9
- For this workshop you'll need the latest version of VMWare Workstation (Windows), Fusion (Mac), or Player.
10
 
11
- Although you can get the VM to run in VirtualBox, I will not be supporting this configuration for this class.
12
13
14
VM for these labs
15
-----------------
16
https://s3.amazonaws.com/StrategicSec-VMs/StrategicsecUbuntu-v3.zip
17
user: strategicsec
18
pass: strategicsec
19
20
https://s3.amazonaws.com/StrategicSec-VMs/Win7x64.zip
21
username: workshop
22
password: password
23
24
25
26
27
##############################################
28
# Log Analysis with Linux command-line tools #
29
##############################################
30
The following command line executables are found in the Mac as well as most Linux Distributions.
31
32
cat –  prints the content of a file in the terminal window
33
grep – searches and filters based on patterns
34
awk –  can sort each row into fields and display only what is needed
35
sed –  performs find and replace functions
36
sort – arranges output in an order
37
uniq – compares adjacent lines and can report, filter or provide a count of duplicates
38
39
40
41
###############
42
# Apache Logs #
43
###############
44-
-----------------------------Type this-----------------------------------------
44+
45-
wget https://s3.amazonaws.com/infosecaddictsfiles/cisco.log
45+
46-
-------------------------------------------------------------------------------
46+
47
48
wget https://s3.amazonaws.com/SecureNinja/Python/access_log
49
50
51
You want to list all user agents ordered by the number of times they appear (descending order):
52-
-----------------------------Type this-----------------------------------------
52+
53
awk -F\" '{print $6}' access_log | sort | uniq -c | sort -fr
54-
-------------------------------------------------------------------------------
54+
55
56
57
Using the default separator which is any white-space (spaces or tabs) we get the following:
58
59
awk '{print $1}' access_log         # ip address (%h)
60-
-----------------------------Type this-----------------------------------------
60+
61
awk '{print $3}' access_log         # userid (%u)
62-
-------------------------------------------------------------------------------
62+
63
awk '{print $9}' access_log         # status code (%>s)
64
awk '{print $10}' access_log        # size (%b)
65
66
You might notice that we've missed out some items. To get to them we need to set the delimiter to the " character which changes the way the lines are 'exploded' and allows the following:
67
68-
-----------------------------Type this-----------------------------------------
68+
69
awk -F\" '{print $4}' access_log    # referer
70-
-------------------------------------------------------------------------------
70+
71
72
73
awk -F\" '{print $6}' access_log \
74
  | sed 's/(\([^;]\+; [^;]\+\)[^)]*)/(\1)/' \
75
  | sort | uniq -c | sort -fr
76
77-
-----------------------------Type this-----------------------------------------
77+
78
The next step is to start filtering the output so you can narrow down on a certain page or referer. Would you like to know which pages Google has been requesting from your site?
79
80
awk -F\" '($6 ~ /Googlebot/){print $2}' access_log | awk '{print $2}'
81
Or who's been looking at your guestbook?
82
83
awk -F\" '($2 ~ /guestbook\.html/){print $6}' access_log
84
85-
--------------------------------------------------------------------------------
85+
86
Reference:
87
https://blog.nexcess.net/2011/01/21/one-liners-for-apache-log-files/
88
89
# top 20 URLs from the last 5000 hits
90
tail -5000 ./access_log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
91
tail -5000 ./access_log | awk '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
92
 
93
# top 20 URLS excluding POST data from the last 5000 hits
94
tail -5000 ./access_log | awk -F"[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
95
tail -5000 ./access_log | awk -F"[ ?]" '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
96
 
97
# top 20 IPs from the last 5000 hits
98
tail -5000 ./access_log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
99
tail -5000 ./access_log | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
100
 
101
# top 20 URLs requested from a certain ip from the last 5000 hits
102
IP=1.2.3.4; tail -5000 ./access_log | grep $IP | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
103
IP=1.2.3.4; tail -5000 ./access_log | awk -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
104
 
105
# top 20 URLS requested from a certain ip excluding, excluding POST data, from the last 5000 hits
106
IP=1.2.3.4; tail -5000 ./access_log | fgrep $IP | awk -F "[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
107
IP=1.2.3.4; tail -5000 ./access_log | awk -F"[ ?]" -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
108
 
109
# top 20 referrers from the last 5000 hits
110
tail -5000 ./access_log | awk '{print $11}' | tr -d '"' | sort | uniq -c | sort -rn | head -20
111
tail -5000 ./access_log | awk '{freq[$11]++} END {for (x in freq) {print freq[x], x}}' | tr -d '"' | sort -rn | head -20
112
 
113
# top 20 user agents from the last 5000 hits
114
tail -5000 ./access_log | cut -d\  -f12- | sort | uniq -c | sort -rn | head -20
115
 
116
# sum of data (in MB) transferred in the last 5000 hits
117
tail -5000 ./access_log | awk '{sum+=$10} END {print sum/1048576}'
118
119
120
##############
121
# Cisco Logs #
122
##############
123
124
wget https://s3.amazonaws.com/StrategicSec-Files/LogAnalysis/cisco.log
125
126
127
AWK Basics
128
----------
129
To quickly demonstrate the print feature in awk, we can instruct it to show only the 5th word of each line. Here we will print $5. Only the last 4 lines are being shown for brevity.
130
131
cat cisco.log | awk '{print $5}' | tail -n 4
132
133
134
135
136
Looking at a large file would still produce a large amount of output. A more useful thing to do might be to output every entry found in “$5”, group them together, count them, then sort them from the greatest to least number of occurrences. This can be done by piping the output through “sort“, using “uniq -c” to count the like entries, then using “sort -rn” to sort it in reverse order.
137
138
cat cisco.log | awk '{print $5}'| sort | uniq -c | sort -rn
139
140
141
142
143
While that’s sort of cool, it is obvious that we have some garbage in our output. Evidently we have a few lines that aren’t conforming to the output we expect to see in $5. We can insert grep to filter the file prior to feeding it to awk. This insures that we are at least looking at lines of text that contain “facility-level-mnemonic”.
144
145
cat cisco.log | grep %[a-zA-Z]*-[0-9]-[a-zA-Z]* | awk '{print $5}' | sort | uniq -c | sort -rn
146
147
148
149
150
151
Now that the output is cleaned up a bit, it is a good time to investigate some of the entries that appear most often. One way to see all occurrences is to use grep.
152
153
cat cisco.log | grep %LINEPROTO-5-UPDOWN:
154
155
cat cisco.log | grep %LINEPROTO-5-UPDOWN:| awk '{print $10}' | sort | uniq -c | sort -rn
156
157
cat cisco.log | grep %LINEPROTO-5-UPDOWN:| sed 's/,//g' | awk '{print $10}' | sort | uniq -c | sort -rn
158
159
cat cisco.log | grep %LINEPROTO-5-UPDOWN:| sed 's/,//g' | awk '{print $10 " changed to " $14}' | sort | uniq -c | sort -rn
160
161
162
163
164
#################################
165
# Using Python for log analysis #
166
#################################
167
168
169
170
171
###########################################
172
# Python Basics Lesson 1: Simple Printing #
173
###########################################
174
175-
python
175+
176
177-
>>>
177+
178
179
>>> print "hello"
180
 
181
>>> print "Today we are learning Python."
182
 
183
 
184
 
185
###################################################
186
# Python Basics Lesson 2: Simple Numbers and Math #
187
###################################################
188
 
189
>>> 2+2
190
 
191
>>> 6-3
192
 
193
>>> 18/7
194
 
195
>>> 18.0/7
196
 
197
>>> 18.0/7.0
198
 
199
>>> 18/7
200
 
201
>>> 9%4
202
 
203
>>> 8%4
204
 
205
>>> 8.75%.5
206
 
207
>>> 6.*7
208
 
209
>>> 6*6*6
210
 
211
>>> 6**3
212
 
213
>>> 5**12
214
 
215
>>> -5**4
216
 
217
 
218
 
219
 
220
 
221
 
222
#####################################
223
# Python Basics Lesson 3: Variables #
224
#####################################
225
 
226
>>> x=18
227
 
228
>>> x+15
229
 
230
>>> x**3
231
 
232
>>> y=54
233
 
234
>>> x+y
235
 
236
>>> age=input("Enter number here: ")
237
        43
238
 
239
>>> age+32
240
 
241
>>> age**3
242
243
>>> fname = raw_input("Enter your first name: ")
244
245
>>> lname = raw_input("Enter your first name: ")
246
247
>>> fname = raw_input("Enter your name: ")
248
Enter your name: Joe
249
250
>>> lname = raw_input("Enter your name: ")
251
Enter your name: McCray
252
253
>>> print fname
254
Joe
255
256
>>> print lname
257
McCray
258
259
>>> print fname lname
260
261
>>> print fname+lname
262
JoeMcCray
263
264
 
265
 
266
NOTE:
267
Use "input() for integers and expressions, and use raw_input() when you are dealing with strings. 
268
 
269
 
270
 
271
 
272
 
273
#################################################
274
# Python Basics Lesson 4: Modules and Functions #
275
#################################################
276
 
277
>>> 5**4
278
 
279
>>> pow(5,4)
280
 
281
>>> abs(-18)
282
 
283
>>> abs(5)
284
 
285
>>> floor(18.7)
286
 
287
>>> import math
288
 
289
>>> math.floor(18.7)
290
 
291
>>> math.sqrt(81)
292
 
293
>>> joe = math.sqrt
294
 
295
>>> joe(9)
296
 
297
>>> joe=math.floor
298
 
299
>>> joe(19.8)
300
 
301
 
302
 
303
 
304
 
305
 
306
 
307
 
308
 
309
###################################
310
# Python Basics Lesson 5: Strings #
311
###################################
312
 
313
>>> "XSS"
314
 
315
>>> 'SQLi'
316
 
317
>>> "Joe's a python lover"
318
 
319
>>> 'Joe\'s a python lover'
320
 
321
>>> "Joe said \"InfoSec is fun\" to me"
322
 
323
>>> a = "Joe"
324
 
325
>>> b = "McCray"
326
 
327
>>> a, b
328
 
329
>>> a+b
330
 
331
 
332
 
333
 
334
 
335
 
336
 
337
 
338
########################################
339
# Python Basics Lesson 6: More Strings #
340
########################################
341
 
342
>>> num = 10
343
 
344
>>> num + 2
345
 
346
>>> "The number of open ports found on this system is " + num
347
 
348
>>> num = str(18)
349
 
350
>>> "There are " + num + " vulnerabilities found in this environment."
351
 
352
>>> num2 = 46
353
 
354
>>> "As of 08/20/2012, the number of states that enacted the Security Breach Notification Law is " + `num2`
355
 
356
357
 
358
NOTE:
359
Use "input() for integers and expressions, and use raw_input() when you are dealing with strings.
360
 
361
 
362
 
363
 
364
 
365
 
366
 
367
###############################################
368
# Python Basics Lesson 7: Sequences and Lists #
369
###############################################
370
 
371
>>> attacks = ['Stack Overflow', 'Heap Overflow', 'Integer Overflow', 'SQL Injection', 'Cross-Site Scripting', 'Remote File Include']
372
 
373
>>> attacks
374
['Stack Overflow', 'Heap Overflow', 'Integer Overflow', 'SQL Injection', 'Cross-Site Scripting', 'Remote File Include']
375
 
376
>>> attacks[3]
377
'SQL Injection'
378
 
379
>>> attacks[-2]
380
'Cross-Site Scripting'
381
 
382
 
383
 
384
 
385
 
386
 
387
########################################
388
# Python Basics Level 8: If Statement #
389
########################################
390
>>> attack="SQLI"
391
>>> if attack=="SQLI":
392
        print 'The attacker is using SQLI'
393
 
394
>>> attack="XSS"
395
>>> if attack=="SQLI":
396
        print 'The attacker is using SQLI'
397
398
 
399
#############################
400
# Reference Videos To Watch #
401
#############################
402
Here is your first set of youtube videos that I'd like for you to watch:
403
https://www.youtube.com/playlist?list=PLEA1FEF17E1E5C0DA (watch videos 1-10)
404
 
405
 
406
 
407
 
408
 
409
#####################################
410
# Lesson 9: Intro to Log Analysis #
411-
>>> exit()
411+
412
 
413
Login to your StrategicSec Ubuntu machine. You can download the VM from the following link:
414
 
415
https://s3.amazonaws.com/StrategicSec-VMs/Strategicsec-Ubuntu-VPN-163.zip
416
        username: strategicsec
417
        password: strategicsec
418
 
419
Then execute the following commands:
420
---------------------------------------------------------------------------------------------------------
421
 
422
 
423
wget https://s3.amazonaws.com/SecureNinja/Python/access_log
424
 
425
 
426
cat access_log | grep 141.101.80.188
427
 
428
cat access_log | grep 141.101.80.187
429
 
430
cat access_log | grep 108.162.216.204
431
 
432
cat access_log | grep 173.245.53.160
433
 
434
---------------------------------------------------------
435
 
436
Google the following terms:
437
        - Python read file
438
        - Python read line
439
        - Python read from file
440
 
441
 
442
 
443
 
444
########################################################
445
# Lesson 10: Use Python to read in a file line by line #
446
########################################################
447
 
448
 
449
Reference:
450
http://cmdlinetips.com/2011/08/three-ways-to-read-a-text-file-line-by-line-in-python/
451
452
453
454
455
456
457
Let's have some fun.....
458
459
460
>>> f = open('access_log', "r")
461
462
>>> lines = f.readlines()
463
464
>>> print lines
465
466
>>> lines[0]
467
468
>>> lines[10]
469
470
>>> lines[50]
471
472
>>> lines[1000]
473
474
>>> lines[5000]
475
476
>>> lines[10000]
477
478
>>> print len(lines)
479
480
481
482
483
484
485
 
486
 
487
 
488
---------------------------------------------------------
489
vi logread1.py
490
 
491
 
492
## Open the file with read only permit
493
f = open('access_log', "r")
494
 
495
## use readlines to read all lines in the file
496
## The variable "lines" is a list containing all lines
497
lines = f.readlines()
498
 
499
print lines
500
 
501
 
502
## close the file after reading the lines.
503
f.close()
504
 
505
---------------------------------------------------------
506
 
507
 
508
Google the following:
509
        - python difference between readlines and readline
510
        - python readlines and readline
511
 
512
 
513
 
514
 
515
 
516
#################################
517
# Lesson 11: A quick challenge #
518
#################################
519
 
520
Can you write an if/then statement that looks for this IP and print "Found it"?
521
 
522
 
523
141.101.81.187
524
 
525
 
526
 
527
 
528
 
529
 
530
---------------------------------------------------------
531
Hint 1: Use Python to look for a value in a list
532
 
533
Reference:
534
http://www.wellho.net/mouth/1789_Looking-for-a-value-in-a-list-Python.html
535
 
536
 
537
 
538
 
539
---------------------------------------------------------
540
Hint 2: Use Python to prompt for user input
541
 
542
Reference:
543
http://www.cyberciti.biz/faq/python-raw_input-examples/
544
 
545
 
546
 
547
 
548
---------------------------------------------------------
549
Hint 3: Use Python to search for a string in a list
550
 
551
Reference:
552
http://stackoverflow.com/questions/4843158/check-if-a-python-list-item-contains-a-string-inside-another-string
553
 
554
 
555
 
556
 
557
 
558
Here is my solution:
559
-------------------
560
$ python
561
>>> f = open('access_log', "r")
562
>>> lines = f.readlines()
563
>>> ip = '141.101.81.187'
564
>>> for string in lines:
565
...     if ip in string:
566
...             print(string)
567
 
568
 
569
 
570
 
571
Here is one student's solution - can you please explain each line of this code to me?
572
-------------------------------------------------------------------------------------
573
#!/usr/bin/python
574
 
575
f = open('access_log')
576
 
577
strUsrinput = raw_input("Enter IP Address: ")
578
 
579
for line in iter(f):
580
    ip = line.split(" - ")[0]
581
    if ip == strUsrinput:
582
        print line
583
 
584
f.close()
585
 
586
 
587
 
588
 
589
-------------------------------
590
 
591
Working with another student after class we came up with another solution:
592
 
593
#!/usr/bin/env python
594
 
595
 
596
# This line opens the log file
597
f=open('access_log',"r")
598
 
599
# This line takes each line in the log file and stores it as an element in the list
600
lines = f.readlines()
601
 
602
 
603
# This lines stores the IP that the user types as a var called userinput
604
userinput = raw_input("Enter the IP you want to search for: ")
605
 
606
 
607
 
608
# This combination for loop and nested if statement looks for the IP in the list called lines and prints the entire line if found.
609
for ip in lines:
610
    if ip.find(userinput) != -1:
611
        print ip
612
 
613
 
614
 
615
##################################################
616
# Lesson 12: Look for web attacks in a log file #
617
##################################################
618
 
619
In this lab we will be looking at the scan_log.py script and it will scan the server log to find out common hack attempts within your web server log.
620
Supported attacks:
621
1.          SQL Injection
622
2.          Local File Inclusion
623
3.          Remote File Inclusion
624
4.          Cross-Site Scripting
625
 
626
 
627
 
628
wget https://s3.amazonaws.com/SecureNinja/Python/scan_log.py
629
 
630
The usage for scan_log.py is simple.  You feed it an apache log file.
631
 
632
cat scan_log.py | less                  (use your up/down arrow keys to look through the file)
633
634
635
636
637
638
################################
639
# Log Analysis with Powershell #
640
################################
641
642
VM for these labs
643
-----------------
644
https://s3.amazonaws.com/StrategicSec-VMs/Win7x64.zip
645
        username: workshop
646
        password: password
647
648
 
649
You can do the updates in the Win7 VM (yes, it is a lot of updates).
650
 
651
You'll need to create directory in the Win7 VM called "c:\ps"
652
 
653
#####################
654
# Powershell Basics #
655
#####################
656
 
657
PowerShell is Microsoft’s new scripting language that has been built in since the release Vista.
658
 
659
PowerShell file extension end in .ps1 .
660
 
661
An important note is that you cannot double click on a PowerShell script to execute it.
662
 
663
To open a PowerShell command prompt either hit Windows Key + R and type in PowerShell or Start -> All Programs -> Accessories -> Windows PowerShell -> Windows PowerShell.
664
 
665
dir
666
cd
667
ls
668
cd c:\
669
 
670
 
671
To obtain a list of cmdlets, use the Get-Command cmdlet
672
 
673
Get-Command
674
 
675
 
676
 
677
You can use the Get-Alias cmdlet to see a full list of aliased commands.
678
 
679
Get-Alias
680
 
681
 
682
 
683
Don't worry you won't blow up your machine with Powershell
684
Get-Process | stop-process                              What will this command do?
685
Get-Process | stop-process -whatif
686
 
687
 
688
To get help with a cmdlet, use the Get-Help cmdlet along with the cmdlet you want information about.
689
 
690
Get-Help Get-Command
691
 
692
Get-Help Get-Service –online
693
 
694
Get-Service -Name TermService, Spooler
695
 
696
Get-Service –N BITS
697
 
698
Start-Transcript
699
 
700
PowerShell variables begin with the $ symbol. First lets create a variable
701
 
702
$serv = Get-Service –N Spooler
703
 
704
To see the value of a variable you can just call it in the terminal.
705
 
706
$serv
707
 
708
$serv.gettype().fullname
709
 
710
 
711
Get-Member is another extremely useful cmdlet that will enumerate the available methods and properties of an object. You can pipe the object to Get-Member or pass it in
712
 
713
$serv | Get-Member
714
 
715
Get-Member -InputObject $serv
716
 
717
 
718
 
719
 
720
 
721
Let’s use a method and a property with our object.
722
 
723
$serv.Status
724
$serv.Stop()
725
$serv.Refresh()
726
$serv.Status
727
$serv.Start()
728
$serv.Refresh()
729
$serv.Status
730
 
731
 
732
 
733
 
734
Methods can return properties and properties can have sub properties. You can chain them together by appending them to the first call.
735
 
736
 
737
 
738
#############################
739
# Simple Event Log Analysis #
740
#############################
741
 
742
Step 1: Dump the event logs
743
---------------------------
744
The first thing to do is to dump them into a format that facilitates later processing with Windows PowerShell.
745
 
746
To dump the event log, you can use the Get-EventLog and the Exportto-Clixml cmdlets if you are working with a traditional event log such as the Security, Application, or System event logs.
747
If you need to work with one of the trace logs, use the Get-WinEvent and the ExportTo-Clixml cmdlets.
748
 
749
Get-EventLog -LogName application | Export-Clixml Applog.xml
750
 
751
type .\Applog.xml
752
 
753
$logs = "system","application","security"
754
 
755
The % symbol is an alias for the Foreach-Object cmdlet. It is often used when working interactively from the Windows PowerShell console
756
 
757
$logs | % { get-eventlog -LogName $_ | Export-Clixml "$_.xml" }
758
 
759
 
760
 
761
Step 2: Import the event log of interest
762
----------------------------------------
763
To parse the event logs, use the Import-Clixml cmdlet to read the stored XML files.
764
Store the results in a variable.
765
Let's take a look at the commandlets Where-Object, Group-Object, and Select-Object.
766
 
767
The following two commands first read the exported security log contents into a variable named $seclog, and then the five oldest entries are obtained.
768
 
769
$seclog = Import-Clixml security.xml
770
 
771
$seclog | select -Last 5
772
 
773
 
774
Cool trick from one of our students named Adam. This command allows you to look at the logs for the last 24 hours:
775
 
776
Get-EventLog Application -After (Get-Date).AddDays(-1)
777
 
778
You can use '-after' and '-before' to filter date ranges
779
 
780
One thing you must keep in mind is that once you export the security log to XML, it is no longer protected by anything more than the NFTS and share permissions that are assigned to the location where you store everything.
781
By default, an ordinary user does not have permission to read the security log.
782
 
783
 
784
Step 3: Drill into a specific entry
785
-----------------------------------
786
To view the entire contents of a specific event log entry, choose that entry, send the results to the Format-List cmdlet, and choose all of the properties.
787
 
788
 
789
$seclog | select -first 1 | fl *
790
 
791
The message property contains the SID, account name, user domain, and privileges that are assigned for the new login.
792
 
793
 
794
($seclog | select -first 1).message
795
 
796
(($seclog | select -first 1).message).gettype()
797
 
798
 
799
 
800
In the *nix world you often want a count of something (wc -l).
801
How often is the SeSecurityPrivilege privilege mentioned in the message property?
802
To obtain this information, pipe the contents of the security log to a Where-Object to filter the events, and then send the results to the Measure-Object cmdlet to determine the number of events:
803
 
804
$seclog | ? { $_.message -match 'SeSecurityPrivilege'} | measure
805
 
806
If you want to ensure that only event log entries return that contain SeSecurityPrivilege in their text, use Group-Object to gather the matches by the EventID property.
807
 
808
 
809
$seclog | ? { $_.message -match 'SeSecurityPrivilege'} | group eventid
810
 
811
Because importing the event log into a variable from the stored XML results in a collection of event log entries, it means that the count property is also present.
812
Use the count property to determine the total number of entries in the event log.
813
 
814
$seclog.Count
815
 
816
 
817
 
818
 
819
 
820
 
821
############################
822
# Simple Log File Analysis #
823
############################
824
 
825
 
826
You'll need to create the directory c:\ps and download sample iss log http://pastebin.com/raw.php?i=LBn64cyA
827
 
828
 
829
mkdir c:\ps
830
cd c:\ps
831
(new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=LBn64cyA", "c:\ps\u_ex1104.log")
832
 
833
 
834
 
835
 
836
       
837
       
838
       
839
 
840
###############################################
841
# Intrusion Analysis Using Windows PowerShell #
842
###############################################
843
 
844
Download sample file http://pastebin.com/raw.php?i=ysnhXxTV into the c:\ps directory
845
 
846
 
847
 
848
 
849
 
850
(new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=ysnhXxTV", "c:\ps\CiscoLogFileExamples.txt")
851
 
852
Select-String 192.168.208.63 .\CiscoLogFileExamples.txt
853
 
854
 
855
 
856
 
857
The Select-String cmdlet searches for text and text patterns in input strings and files. You can use it like Grep in UNIX and Findstr in Windows.
858
 
859
Select-String 192.168.208.63 .\CiscoLogFileExamples.txt | select line
860
 
861
 
862
 
863
 
864
To see how many connections are made when analyzing a single host, the output from that can be piped to another command: Measure-Object.
865
 
866
Select-String 192.168.208.63 .\CiscoLogFileExamples.txt | select line | Measure-Object
867
 
868
 
869
 
870
To select all IP addresses in the file expand the matches property, select the value, get unique values and measure the output.
871
 
872
Select-String “\b(?:\d{1,3}\.){3}\d{1,3}\b” .\CiscoLogFileExamples.txt | select -ExpandProperty matches | select -ExpandProperty value | Sort-Object -Unique | Measure-Object
873
 
874
 
875
 
876
Removing Measure-Object shows all the individual IPs instead of just the count of the IP addresses. The Measure-Object command counts the IP addresses.
877
 
878
Select-String “\b(?:\d{1,3}\.){3}\d{1,3}\b” .\CiscoLogFileExamples.txt | select -ExpandProperty matches | select -ExpandProperty value | Sort-Object -Unique
879
 
880
 
881
In order to determine which IP addresses have the most communication the last commands are removed to determine the value of the matches. Then the group command is issued on the piped output to group all the IP addresses (value), and then sort the objects by using the alias for Sort-Object: sort count –des.
882
This sorts the IP addresses in a descending pattern as well as count and deliver the output to the shell.
883
 
884
Select-String “\b(?:\d{1,3}\.){3}\d{1,3}\b” .\CiscoLogFileExamples.txt | select -ExpandProperty matches | select value | group value | sort count -des
885
 
886
 
887
 
888
 
889
This will get the setting for logs in the windows firewall which should be enabled in GPO policy for analysis.
890
The command shows that the Firewall log is at:
891
%systemroot%\system32\LogFiles\Firewall\pfirewall.log, in order to open the file PowerShell will need to be run with administrative privileges.
892
 
893
 
894
First step is to get the above command into a variable using script logic.
895
Thankfully PowerShell has a built-in integrated scripting environment, PowerShell.ise.
896
 
897
netsh advfirewall show allprofiles | Select-String FileName | select -ExpandProperty line | Select-String “%systemroot%.+\.log" | select -ExpandProperty matches | select -ExpandProperty value | sort –uniq
898
 
899
 
900
##############################################
901
# Parsing Log files using windows PowerShell #
902
##############################################
903
 
904
Download the sample IIS log http://pastebin.com/LBn64cyA
905
 
906
 
907
(new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=LBn64cyA", "c:\ps\u_ex1104.log")
908
 
909
Get-Content ".\*log" | ? { ($_ | Select-String "WebDAV")}  
910
 
911
 
912
 
913
The above command would give us all the WebDAV requests.
914
 
915
To filter this to a particular user name, use the below command:
916
 
917
Get-Content ".\*log" | ? { ($_ | Select-String "WebDAV") -and ($_ | Select-String "OPTIONS")}  
918
 
919
 
920
 
921
Some more options that will be more commonly required :
922
 
923
For Outlook Web Access : Replace WebDAV with OWA
924
 
925
For EAS : Replace WebDAV with Microsoft-server-activesync
926
 
927
For ECP : Replace WebDAV with ECP
928
 
929
 
930
 
931
To find out the count of the EWS request we can go ahead and run the below command
932
 
933
(Get-Content ".\*log" | ? { ($_ | Select-String "WebDAV") -and ($_ | Select-String "Useralias")}).count
934
935
936
937
938
939
940
941
942
 
943
Explain to me how this script works.