awk
Created Saturday 11 May 2013

------------------------------------------------------------------------------------------------
Output Separators
------------------------------------------------------------------------------------------------
As mentioned previously, a print statement contains a list of items separated by commas. In the output, the items are normally separated by single spaces. However, this doesn’t need to be the case; a single space is simply the default. Any string of characters may be used as the output field separator by setting the predefined variable OFS. The initial value of this variable is the string " " (i.e., a single space).
The output from an entire print statement is called an output record. Each print statement outputs one output record, and then outputs a string called the output record separator (or ORS). The initial value of ORS is the string "\n" (i.e., a newline character). Thus, each print statement normally makes a separate line.
In order to change how output fields and records are separated, assign new values to the variables OFS and ORS. The usual place to do this is in the BEGIN rule (see BEGIN/END), so that it happens before any input is processed. It can also be done with assignments on the command line, before the names of the input files, or using the -v command-line option (see Options). The following example prints the first and second fields of each input record, separated by a semicolon, with a blank line added after each newline:
------------------------------------------------------------------------------------------------
EXAMPLE
$ awk 'BEGIN { OFS = ";"; ORS = "\n\n" }
> { print $1, $2 }' mail-list
-| Amelia;555-5553
-|
-| Anthony;555-3412
-|
-| Becky;555-7685
------------------------------------------------------------------------------------------------
USING GSUB TO SEARCH/REPLACE STRINGS
IN THIS EXAMPLE I AM REMOVING "double quote" and commas
curl -s http://hgfix.net/paste/view/raw/708cd7c5 | awk '{ gsub ("\"|,",""); print }'
------------------------------------------------------------------------------------------------
for i in 123456 admin123 info123 test password ; do awk -F : -v i="$i" '{OFS="\t";split($0, f, "$");p="openssl passwd -1 -salt "f[3]" "i;if(p|getline o);split(o, q, "$");m=match($0,q[4]);}m!=0{print$1,i}' shadow ; done

for i is passing serveral variables in a loop
awk sets a delimiter : and then defines the variable i=$i then we set the output field seperator to \t(tab) and perform a split on the entire line $0 where f is the array and "$" is the delimiter
the "p" variable is performing an openssl passwd -1 -salt on the array[3] (here is an example line admin:$1$GK3i7sYW$bH1.QzR9izDa3eRqboRcL/:15916:::::: )
since our split is of the entire line "$0" with an additional delimiter set as "$" which would provide the string GK3i7sYW from the example above then " " providing a space and finally i which would be (123456 admin123 info123 test password)
to better understand I will provide the output being ran in this command below:
openssl passwd -1 -salt GK3i7sYW admin123
$1$GK3i7sYW$bH1.QzR9izDa3eRqboRcL/
---------------------------------
now we perform an if $p getline which would get the output above and set it to variable "o"
we do a split on variable "o" where o is the entire line ( $1$GK3i7sYW$bH1.QzR9izDa3eRqboRcL/ ) q is the array and "$" is the delimiter
afterwords we define "m" to perform a match on ($0,q[4]) where $0 is the entire line ( admin:$1$GK3i7sYW$bH1.QzR9izDa3eRqboRcL/:15916:::::: ) and q is "QzR9izDa3eRqboRcL/"
if m does not equal 0 ( meaning a match is found or true ) we print $1 of the entire line ( admin:$1$GK3i7sYW$bH1.QzR9izDa3eRqboRcL/:15916:::::: ) which would be "admin
and $i which is one of our variables set at the very beginning of our command ( 123456 admin123 info123 test password ) in this case a match was found on admin123
------------------------------------------------------------------------------------------------
CONVERT UNIX TIMESTAMP TO HUMAN READABLE
awk '{$2=strftime("%Y-%m-%d %H:%M:%S", $2); print $0}'
------------------------------------------------------------------------------------------------
CONVERT UNIX TIME STAMP FROM BASH HISTORY FILE
awk -F- '{x=$1;sub(/^\#/,"",$1);sub(/#.*/,strftime("#%Y-%m-%d %H:%M:%S",$1),x);$1=x;}1' ~/.bash_history
------------------------------------------------------------------------------------------------
EXAMPLE STRING MATCHING AND USING SUB
awk -F, '{x=$1;sub(/^\#/,"",$1);sub(/#.*/,strftime("#%Y-%m-%d %H:%M:%S",$1),x);$1=x;}1' time_log
------------------------------------------------------------------------------------------------
time_log
#1394663100
cd public_html/
#1394663104
nano wp-config.php
#1394663115
ll
#1394663162
exit
------------------------------------------------------------------------------------------------
#2014-03-12 17:25:00
cd public_html/
#2014-03-12 17:25:04
nano wp-config.php
#2014-03-12 17:25:15
ll
#2014-03-12 17:26:02
exit
---------------------------------
Example:
grep bhabin /var/log/mysql_queries.log | awk '{$2=strftime("%Y-%m-%d %H:%M:%S", $2); print $0}'
[ 2017-03-14 01:51:01 ] [ bhabin_wrdp31 ] [ 3570009 ] [ 134 ] [ bhabin_wrdp31 ] [ Sleep ] [ KILLED ] [ ] [ ]
[ 2017-03-25 02:00:01 ] [ bhabin_wrdp31 ] [ 7173763 ] [ 103 ] [ bhabin_wrdp31 ] [ Sleep ] [ KILLED ] [ ] [ ]
[ 2017-03-26 04:36:01 ] [ bhabin_wrdp26 ] [ 7530669 ] [ 109 ] [ bhabin_wrdp26 ] [ Sleep ] [ KILLED ] [ ] [ ]
[ 2017-03-26 04:51:01 ] [ bhabin_wrdp26 ] [ 7533318 ] [ 68 ] [ bhabin_wrdp26 ] [ Sleep ] [ KILLED ] [ ] [ ]
[ 2017-03-27 11:03:01 ] [ bhabin_wrdp38 ] [ 7987001 ] [ 81 ] [ bhabin_wrdp38 ] [ Sleep ] [ KILLED ] [ ] [ ]
[ 2017-03-28 02:27:01 ] [ bhabin_wrdp26 ] [ 8252842 ] [ 96 ] [ bhabin_wrdp26 ] [ Sleep ] [ KILLED ] [ 'cloud1028.hostgator.com' ] [ ]
[ 2017-03-28 07:30:01 ] [ bhabin_wrdp26 ] [ 8311080 ] [ 92 ] [ bhabin_wrdp26 ] [ Sleep ] [ KILLED ] [ ] [ ]

------------------------------------------------------------------------------------------------
jcook:~/Documents/QA/AuditsInProgress/2017-01-19$ curl -s http://hgfix.net/paste/view/raw/9e9b7145 |grep -oP '\d+\s.*(?=01\/\d{2}\/2017)' | sort -k1,1 -u | awk 'BEGIN { OFS = ","}$1 ~ /[0-9]{8}/{tid=$1}{if (match($0, /([0-9]{2}:){2}[0-9]{2}/,tit)) print tid,tit[0]}' > restore_contact_drivers.csv
------------------------------------------------------------------------------------------------
MATCH FIELD ONE "docroot" If the NR variable is equal to 1, then it sends the first line as the argument to printf; else it sends a comma concatenated with the current line.
(so if is the first part with ? as the delimiter to identify the then statement, and the : is the delimiter to identify the else statement)
When the first line is seen (NR == 1), only it is printed; otherwise a comma and the line are sent as arguments to printf.
This solution uses AWK's ternary operator ?:, that is:
NR == 1 ? $0 : ","$0
curl -s http://hgfix.net/paste/view/raw/10673f0a | awk '$1=="docroot" {printf("%s", NR == 1 ? $2 : ","$2);} END {printf("\n");}'
------------------------------------------------
EXAMPLE TEXT
docroot /home4/b0y4o3s3/public_html/basic.wwus.net
domain basic.wwus.net
docroot /home4/b0y4o3s3/public_html/basic.wwus.net
domain basicw.fun2fart.com
docroot /home4/b0y4o3s3/public_html/biz.wwus.net
------------------------------------------------
EXAMPLE OUTPUT
/home4/b0y4o3s3/public_html/basic.wwus.net,/home4/b0y4o3s3/public_html/basic.wwus.net,/home4/b0y4o3s3/public_html/biz.wwus.net,/home4/b0y4o3s3/public_html/biz.wwus.net,/home4/b0y4o3s3/public_html/hg.wwus.net,/home4/b0y4o3s3/public_html/hg1.wwus.net,/home4/b0y4o3s3/public_html/hg1.wwus.net,/home4/b0y4o3s3/public_html/hg.wwus.net,/home4/b0y4o3s3/public_html/pro.wwus.net,/home4/b0y4o3s3/public_html/pro.wwus.net,/home4/b0y4o3s3/public_html/test
------------------------------------------------------------------------------------------------
WILL MATCH FIELD ONE "ADDON_ID" AND IF ANYTHING ON EACH RECORD MATCHES "Could\ not\ add\ zone\ record|CSR\ validation|No\ CSR\ defined" ASSIGN IT TO err THEN PRINT id,err [ARRAY]
------------------------------------------------------------------------------------------------
curl -s http://hgfix.net/paste/view/raw/b3327a0f | awk '$1=="ADDON_ID"{id=$2}{ if (match($0,/Could\ not\ add\ zone\ record|CSR\ validation|No\ CSR\ defined/,err)) print id,err[0] }'
------------------------------------------------------------------------------------------------
FIND ALL LOGS WITH Dec-2016.gz, USE ZCAT PIPED TO AWK TO READ THE FILE CONTENT ALSO DEFINING FILE AS F TO PRINT WHERE COLUMN 4 MATCH DATE AND COLUMN 7 MATCHES xmlrpc THEN GET A COUNT AND PIPE BACK TO AWK TO ONLY PRINT COUNT GREATER THAN 1000
------------------------------------------------------------------------------------------------
find /home/*/logs -type f -name "*Dec-2016.gz" -print | while read FILE ; do zcat "$FILE" | awk -v F="$FILE" '$4 ~ /12\/Dec\/2016/ && $7 ~/xmlrpc/{ print F }' | uniq -c |awk '$1 > 1000 {print}';done
------------------------------------------------
EXAMPLE OUTPUT:
1449 /home/madredel/logs/madredellachiesa-settimo.it-Dec-2016.gz
6018 /home/webstari/logs/web-star.info-Dec-2016.gz
------------------------------------------------------------------------------------------------
LOCATING ALL EMAIL ACCOUNTS. USING DELIMITER ":" AND @ AS THE OFS THEN SPLIT FILENAME INTO AN ARRAY WITH DELIMITER "/" OUTPUTING FIELD "2"
------------------------------------------------------------------------------------------------
awk -F : '{OFS="@";split(FILENAME, d, "/"); print$1,d[2]}' etc/*/shadow
info@propheticcenter.net
sandralugo@propheticcenter.net
sandra@propheticcenter.net
events@propheticcenter.net
prayer@propheticcenter.net
orderdept@propheticcenter.net
insync@propheticcenter.net
membership@propheticcenter.net
kingdomconnections@propheticcenter.net
------------------------------------------------------------------------------------------------
TRYING TO FIGURE OUT HELP HOURS FOR AGENT, SO COPIED CHART RESULTS OF HH PAGE OUTPUT TO TEXT FILE AND RAN FOLLOWING
https://inet.houston.hostgator.com:8443/?gnet_pid=29&p=1479708000&e=10700&m=0&uf=0&ns=0
SNIPPET : http://hgfix.net/paste/view/raw/498c9b6b
sum_min += $4 this means add up all values in column 4 and assign the output to sum_min
sum_hour=int(hours+(sum_min/60) i had to use int here because, by default, awk numbers are always floating point
awk -F "[\t:]" '{OFS=":"}$1!~/[A-Za-z]+/{sum_min += $4;min=sum_min%60;hours+=$3;sum_hour=int(hours+(sum_min/60))}END{print sum_hour,min}' hh_checker
2:37
------------------------------------------------------------------------------------------------
STRING MATCH 0-9 WITH 3-5 CHARACTERS FOR COLUMN ( FILE OWNERSHIP ) IN MAIL
ls -lhc /home*/*/mail/ | awk '$3~/[0-9]{3,5}/{print$0}'
------------------------------------------------------------------------------------------------
AWK MATCH A STRING
for i in `find / -type f -name '*.c' -exec grep -l mysql {} \; | awk '/\/include\// {print}'`; do dirname $i; done
------------------------------------------------------------------------------------------------
DEFINE FIELDS USING MATCH STRING ON TWO SEPERATE LINES ONLY PRINTING IF BOTH RETURN A VALUE THEN UNSETTING THEM THEN PRINTING THE OUTPUT ON A SINGLE LINE
curl -s http://hgfix.net/paste/view/raw/75dfc19d | awk '$1=="docroot"{dr=$2}$1=="domain"{dom=$2}dr&&dom{print dr,dom;dr=dom=0}'|while read docroot domain; do echo "Docroot: $docroot :: Domain: $domain";done
------------------------------------------------------------------------------------------------
ADDING UP BANDWIDTH TOTAL FOR USER FROM MAIL LOGS
grep vcshaeffer@kiskipby.org /var/log/maillog | grep -oP '(?<=bytes=)\S*' | awk 'BEGIN {FS = "/"} ; {sum+=$1; sum2+=$2} END {print "IN",sum/1024/1024,"MB","\n""OUT",sum2/1024/1024,"MB"}'
------------------------------------------------------------------------------------------------
USING PRINTF WITH ARGUMENTS
grep "Created\ Ticket" vps_dedi_empowerment_escalations.csv | awk -F, '{printf("%s,%s,%s\n",$1,$2,$3)}' | sort -k1
------------------------------------------------------------------------------------------------
FILTER OUTPUT USING GREATER THAN/LESS THAN
------------------------------------------------------------------------------------------------
FIND COLUMN 5 GREATER THAN 141 AND PRINT COLUMN 7
EXAMPLE: awk '$5 <= 141 { print $7}' /opt/hgmods/kill_imap.log | grep ttchildren.org | sort | uniq -c | sort -rn
------------------------------------------------
FIND WHERE COLUMN 9 IS LESS THAN COLUMN 10 PRINT 1,2 AND 9
EXAMPLE: awk '$9 < 10 {print $1,$2,$9}'
------------------------------------------------------------------------------------------------
STRING MATCH A FIELD
------------------------------------------------------------------------------------------------
MATCH FIELD 7 PRINT COLUMN 5
EXAMPLE: awk 'match($7, /200/){print $5}'
------------------------------------------------------------------------------------------------
STRING MATCH SEARCHING THE ENTIRE LINE, THEN PRINT OUT COLUMN 1 AS WELL THE MATCHING STRING AND EVERYTHING FOLLOWING
------------------------------------------------------------------------------------------------
awk 'match($0, /Mozilla.*$/){print $1,substr($0,RSTART,RLENGTH)}'
------------------------------------------------------------------------------------------------
PRINT OUT A COLUMN AND EVERYTHING AFTER IT WHICH IN THE EXAMPLE PROVIDED IS COLUMN 12
awk '{ s = ""; for (i = 12; i <= NF; i++) s = s $i " "; print s }'
------------------------------------------------------------------------------------------------
PRINT THE LAST FIELD
tail -f file | grep A1 | awk '{print $NF}'
------------------------------------------------------------------------------------------------
USING DELIMTER TO PRINT OUT EVERYTHING WITHIN IT
tail -100 access-logs/pennystocktweets.com | awk -F \" '{print$(NF-1)}'
------------------------------------------------------------------------------------------------
REMOVE DUPLICATE ENTIRES FOR COLUMN 2 BASED ON MATCHES IN COLUMN 1
------------------------------------------------------------------------------------------------
awk '$1~/regextomatch/&&!_[$2]++{print$2}' filename
------------------------------------------------------------------------------------------------
SET MULTIPLE DELIMITERS :. THEN MATCH /malek/ AND WHERE DELIMITER : THEN . NOT EQUAL STRING 'coderhall' GSUB 'malek' with nothing and Print
------------------------------------------------------------------------------------------------
awk -F "[:.]" '$NF~/malek/&&$(NF-2)!="coderhall"{ gsub (": malek",""); print }' /etc/userdomains
akaind.com
siavash.rocks
hamidlighting.com
iranianeyeclinic.com
akagroup.org
isecho.org
akhgarelectric.com
smartgym.akafitness.co
scsir.org
akafitness.co
faradidafzar.com
akafitness.net
----------------------------------------------------
FULL OUTPUT OF DOMAINS MATCHING malek
------------------------------------------------------------------------------------------------
grep malek /etc/userdomains
akaind.com: malek
akhgarelectric.coderhall.com: malek
akagroup.coderhall.com: malek
coderhall.com: malek
siavash.coderhall.com: malek
siavash.rocks: malek
iranianeyeclinic.coderhall.com: malek
persiangallery.coderhall.com: malek
hamidlighting.com: malek
isecho.coderhall.com: malek
tornado.coderhall.com: malek
------------------------------------------------------------------------------------------------
PULLING ADDON DOMAINS AND PATH
------------------------------------------------------------------------------------------------
addon_domains () { awk -v user="$1" -F "[:=]" '$6 ~ /addon/ && $2 ~ " "user"$" {print $10,$1}' /etc/userdatadomains;}; addon_domains sami
/home3/sami/public_html/berraquerapaisa.com berraquerapaisa.com
/home3/sami/public_html/criaderoaristogatos.com criaderoaristogatos.com
/home3/sami/public_html/esperanzagomez.online esperanzagomez.online
/home3/sami/public_html/fincalacascada.com fincalacascada.com
/home3/sami/public_html/limpiezadelser.com limpiezadelser.com
/home3/sami/public_html/lonchiseda.com lonchiseda.com
/home3/sami/public_html/marketingypromociones.com marketingypromociones.com
/home3/sami/public_html/orgullosamentepaisa.com orgullosamentepaisa.com
/home3/sami/public_html/orgullosamentepaisas.com orgullosamentepaisas.com
/home3/sami/public_html/transportesdya.com transportesdya.com
------------------------------------------------------------------------------------------------
To pull Addon domains and output only the docroot domain on a single line
------------------------------------------------------------------------------------------------
hal cpanel_api server_id 163484 username hgdesign function listaddondomains module AddonDomain | awk '$1=="dir"{dr=$2}$1=="domain"{dom=$2}dr&&dom{print dr,dom;dr=dom=0}' | while read docroot domain; do echo "$docroot $domain";done
/home1/hgdesign/public_html/iheartmontrose.com iheartmontrose.com
/home1/hgdesign/public_html/sandwichboard.net sandwichboard.net
------------------------------------------------------------------------------------------------
for ip in 216.172.184.9{0..9} ; do echo $ip ; whois $ip | grep -oP "(?<=network:Organization;I:)[a-z]\S+" | while read dom ; do dig A $dom | awk '{OFS="\t"}BEGIN{nores=1;}{gsub(".\t","\t");if ($1~/^'$dom'/){print$1,$NF; nores=0}}END{if (nores) print "'$dom'";}'; done ; done
------------------------------------------------------------------------------------------------
awk -F "[ :]" '$9=="teaevent"&&$8=="CREATE"&&$7=="2016"&&!_[$NF]++{system("if [ -e /var/cpanel/users/"$NF" ] ; then echo -e \""$NF"\t"$(NF-2)"\" ; fi")}' /var/cpanel/accounting.log
------------------------------------------------------------------------------------------------
PULL ALL USERS CREATED IN 2016 THAT ARE STILL PRESENT ON THE SERVER
------------------------------------------------------------------------------------------------
awk -F "[ :]" '$8=="CREATE"&&$7=="2016"&&!_[$NF]++{system("if [ -e /var/cpanel/users/"$NF" ] ; then echo -e \""$NF"\t"$(NF-2)"\" ; fi")}' /var/cpanel/accounting.log
------------------------------------------------------------------------------------------------
ALTERNATE METHOD W/O SYSTEM CALL
------------------------------------------------------------------------------------------------
awk -F "[ :]" '{OFS="\t"}$8=="CREATE"&&$7=="2016"&&!_[$NF]++{ xxx = " ls /var/cpanel/users/"$NF " 2>/dev/null";if (xxx | getline yyy);else yyy=0 ;close (xxx); if (yyy) print$NF,$(NF-2);close (yyy)}' /var/cpanel/accounting.log
------------------------------------------------------------------------------------------------
AWK USING GETLINE
------------------------------------------------------------------------------------------------
[root@gator3314 /home2/ibmperu]# awk -F : '{OFS=":"}$(NF-1)=="ibmperu"{ cmd = "date -d @"$1 ; if (cmd | getline t) $1=t; print$0; close (CMD)}' /var/log/abusetool.log
Fri Jul 15 05:05:52 CDT 2016:http:disable:ibmperu:NBA-45676225
Fri Jul 15 15:38:33 CDT 2016:http:enable:ibmperu:NBA-45676225
Thu Aug 25 02:09:09 CDT 2016:http:disable:ibmperu:GKM-508-47251
Thu Aug 25 09:54:39 CDT 2016:http:enable:ibmperu:GKM-508-47251
Mon Sep 5 08:53:43 CDT 2016:http:disable:ibmperu:HZW-968-32594
Mon Sep 5 11:23:38 CDT 2016:http:enable:ibmperu:HZW-968-32594
Mon Sep 5 15:58:00 CDT 2016:http:disable:ibmperu:FBO-312-70888
Mon Sep 5 23:23:25 CDT 2016:http:enable:ibmperu:FBO-312-70888
Sat Sep 10 02:54:20 CDT 2016:http:disable:ibmperu:CDT-46695619
Sat Sep 10 16:27:44 CDT 2016:http:enable:ibmperu:CDT-46695619
Sat Sep 10 16:34:56 CDT 2016:http:disable:ibmperu:CDT-46695619
Sat Sep 10 16:35:08 CDT 2016:http:enable:ibmperu:CDT-46695619
Sun Oct 23 09:09:51 CDT 2016:http:disable:ibmperu:DLK-445-69830
normally, that file has the epoch time at the beginning. i just used awk and system to replace the epoch time with the output of date -d
that close(cmd) should be lowercase. apparently it doesn't matter though.
------------------------------------------------------------------------------------------------
File Output
------------------------------------------------------------------------------------------------
1456606799:http:enable:adesanya:BHU-42368795
1457721176:http:enable:dinhdoan:LES-42371168
1457938626:http:disable:altopode:FKN-42911813
1458287053:http:disable:delta12:CFK-43040585
1458345475:http:enable:delta12:CFK-43040585
1458345566:http:disable:delta12:CFK-43040585
1458505315:http:enable:delta12:CFK-43040585
1461223348:http:disable:medwards1965:IAX-800-64427
1462354690:http:disable:mija:TLG-957-30790
1462505493:http:disable:onebrady:QLA-521-54355
1464183707:http:disable:booker10:ZGU-376-21306
1464310054:http:enable:booker10:ZGU-376-21306
1464398105:http:disable:imrand:BTD-628-48445
1464793551:http:enable:imrand:BTD-628-48445
1464954518:http:disable:imrand:YYF-698-47816
1465337702:http:enable:imrand:YYF-698-47816
1468697876:http:enable:mija:TLG-957-30790
1469009543:http:disable:imrand:XRA-635-30977
1469220697:http:disable:jmakhoul:IJS-476-61292
1469640446:http:enable:imrand:XRA-635-30977
1469640740:http:disable:imrand:XRA-635-30977
1469640819:http:enable:imrand:XRA-635-30977
1471273992:http:enable:jmakhoul:IJS-476-61292
1472649407:http:disable:abuzuluf:UIS-685-57379
1472691443:http:disable:kiddcuzzclo:BEU-256-20074
1473148395:http:disable:resttemp:SFP-893-85816
1473149742:http:disable:alliance:SNH-766-33920
1473168119:http:enable:resttemp:SFP-893-85816
1474496232:http:disable:jmakhoul:OSK-128-23639
1474665094:http:enable:alliance:SNH-766-33920
1475611369:http:enable:jmakhoul:OSK-128-23639
1476295452:http:disable:jmakhoul:FMW-723-46783
1476299671:http:enable:jmakhoul:FMW-723-46783
1476383235:http:disable:jmakhoul:WHQ-843-26992
1476469890:http:disable:khosrov:DXE-903-74816
1476530736:http:disable:james948:ZJM-394-63813
1476554214:http:enable:james948:ZJM-394-63813
1476752907:http:enable:jmakhoul:WHQ-843-26992
1477207923:http:disable:jolib:IET-300-45720
------------------------------------------------------------------------------------------------
to figure out help hours for agents, so i copied the chart results of the hh page output into a text file and ran the following.
sum_min += $4 this means add up all values in column 4 and assign the output to sum_min
sum_hour=int(hours+(sum_min/60) i had to use int here because, by default, awk numbers are always floating point.
awk -F "[\t:]" '{OFS=":"}$1!~/[A-Za-z]+/{sum_min += $4;min=sum_min%60;hours+=$3;sum_hour=int(hours+(sum_min/60))}END{print sum_hour,min}' hh_checker
------------------------------------------------------------------------------------------------
TO OUTPUT INTO CSV
------------------------------------------------------------------------------------------------
awk -F"\t" '{OFS=","}{print $1,$2,$3}' cheri-final1.csv > zzz.csv
1
# Add comma as separator (as original comma)
2
awk -F"," '{OFS=","}{print $1,$2,$3}' AviationData.csv > Filtered_AviationData_threefields.csv
3
# Add vertical bar as separator (as original vertical bar)
4
awk -F"|" '{OFS="|"}{print $1,$2,$3}' AviationData.vsv > Filtered_AviationData_threefields.vsv
5
# Add tab as separator (changed from ,)
6
awk -F"," '{OFS="\t"}{print $1,$2,$3}' AviationData.csv > Filtered_AviationData_threefields.tsv
------------------------------------------------------------------------------------------------
Using awk
for i in $(cat blah) ; do echo /home/hgbackupdir/restore.pl $(echo $i |awk -F_ '{print $1}') mysql $i ; done------------------------------------------------------------------------------------------------
cat xxx | while read i ; do awk '/$i/{system("touch ~"$2"/.skip_nextdb,hg_backup")}' /etc/userdomains;done
cat xxx | while read i ; do awk '/'$i'/{system("touch ~"$2"/.skip_nextdb,hg_backup")}' /etc/userdomains; done
awk '$7~/hostgator.com/{gsub(".hostgator.com.*$","");print$7}' exim.alerts
br58
gator4093
gator4112
br60
!_[$2]++
------------------------------------------------------------------------------------------------
awk '/Updated:/{flag=1;next}/\#\t+\#/{flag=0}{gsub("^#","*");gsub("\t#","")}flag' amc_0.3.py
Replaced checkown function with
pwd.getpwuid.
Replaced %s with .format
Removed unecessary logic in logfile.
This is actually pulling from the top of the file output below.

#!/usr/bin/python

#########################################
# #
# Script: Archive Malicious Content #
# Author: Andrew Narunsky #
# Version: 0.3 #
# #
#########################################
# #
# Updated: #
# Replaced checkown function with #
# pwd.getpwuid. #
# Replaced %s with .format #
# Removed unecessary logic in logfile. #
# #
#########################################
# #
# To Do: #
# Get hash for files. #
# Add flag for more hash checking. #
# Add flag for timestamp checking. #
# #
#########################################
------------------------------------------------------------------------------------------------
cpmp() { egrep -sc "${1?Please specify a string.}" /usr/local/apache/domlogs/* | awk -F':' '{if ($NF > 0) print $NF,$1}' | sort -n ; } ; cpmp 187.158.5.15
------------------------------------------------------------------------------------------------
4.5 Specifying How Fields Are Separated

Default Field Splitting:	How fields are normally separated.
Regexp Field Splitting:	Using regexps as the field separator.
Single Character Fields:	Making each character a separate field.
Command Line Field Separator:	Setting FS from the command line.
Full Line Fields:	Making the full line be a single field.
Field Splitting Summary:	Some final points and a summary table.
The field separator, which is either a single character or a regular expression, controls the way awk splits an input record into fields. awk scans the input record for character sequences that match the separator; the fields themselves are the text between the matches.
In the examples that follow, we use the bullet symbol (•) to represent spaces in the output. If the field separator is ‘oo’, then the following line:

moo goo gai pan
is split into three fields: ‘m’, ‘•g’, and ‘•gai•pan’. Note the leading spaces in the values of the second and third fields.

The field separator is represented by the predefined variable FS. Shell programmers take note: awk does not use the name IFS that is used by the POSIX-compliant shells (such as the Unix Bourne shell, sh, or Bash).

The value of FS can be changed in the awk program with the assignment operator, ‘=’ (see Assignment Ops). Often, the right time to do this is at the beginning of execution before any input has been processed, so that the very first record is read with the proper separator. To do this, use the special BEGIN pattern (see BEGIN/END). For example, here we set the value of FS to the string ",":

awk 'BEGIN { FS = "," } ; { print $2 }'
Given the input line:

John Q. Smith, 29 Oak St., Walamazoo, MI 42139
this awk program extracts and prints the string ‘•29•Oak•St.’.

Sometimes the input data contains separator characters that don’t separate fields the way you thought they would. For instance, the person’s name in the example we just used might have a title or suffix attached, such as:

John Q. Smith, LXIX, 29 Oak St., Walamazoo, MI 42139
------------------------------------------------------------------------------------------------
awk -F : '{split($0, d, " |:"); m=(match("JanFebMarAprMayJunJulAugSepOctNovDec",d[2])+2)/3; ; if (d[3]=="") t=mktime(d[8]" "m" "d[4]" "d[5]" "d[6]" "d[7]);else t=mktime(d[7]" "m" "d[3]" "d[4]" "d[5]" "d[6]);c=(strftime("%s") - 5184000);}t>c{checksus = "grep -L SUSPEND /var/cpanel/users/"$NF" 2>/dev/null";checkoff = "ls -1 /home{,1,2,3,4}/"$NF"/public_html/{,*/}{OFERT,geoip,*pay*pal,combo-familia,barclays,santander,identificacao}* 2>/dev/null | head -1"; if (checksus|getline notsus) ; else notsus = 0 ; if (checkoff | getline off) ; else off = 0 ; if (notsus && off) print"\n"$0"\n"off ; close(notsus);close(checkof)}END{print""}' /var/cpanel/accounting.log

Fri Dec 23 15:28:27 2016:CREATE:root:root:retroman.us:192.185.128.171:retroma3
/home2/retroma3/public_html/includes/paypalconfig.php

Sat Dec 24 21:28:30 2016:CREATE:root:root:kosakatadesign.com:192.185.128.171:kosakata
/home2/kosakata/public_html/tools/geoip:

Sun Dec 25 18:53:54 2016:CREATE:root:root:wikeshop.net:192.185.128.171:wikeshop
/home1/wikeshop/public_html/tools/geoip:

Mon Dec 26 20:23:34 2016:CREATE:root:root:murddyeffy.com:192.185.128.172:murddyef
/home1/murddyef/public_html/includes/paypalconfig.php

Tue Dec 27 11:23:03 2016:CREATE:root:root:casasbahia-sslblindados.com:192.185.128.171:casasbb8
/home1/casasbb8/public_html/home/OFERTAS.zip

Wed Dec 28 12:12:08 2016:CREATE:root:root:casas-bahias.com:192.185.128.171:casasbc0
/home1/casasbc0/public_html/images/santander.jpg
------------------------------------------------------------------------------------------------
awk -F \" 'BEGIN{OFS=","}/^[0-9]/{udata=$0}$2=="status"{status=$4}{if($2=="reason"&&$4!="")print udata,status,$4;else if($2=="reason"&&$4=="")print udata,status}' /home/${user}/.cpanel/backup_status
------------------------------------------------------------------------------------------------
Example list
------------------------------------------------------------------------------------------------
[root@gator2007 ~]# cat list2
1708371 1684080 platinum gator2007.hostgator.com
1635639 1684091 mrpierre gator2007.hostgator.com
1708386 1684095 ldoogs1 gator2007.hostgator.com
1708442 1684145 kraegerb gator2007.hostgator.com
1708478 1684182 nongtae gator2007.hostgator.com
1708567 1684275 siammap gator2007.hostgator.com
------------------------------------------------------------------------------------------------
Example Output
------------------------------------------------------------------------------------------------
cat list2 | while read bid pkgid user hname; do nas=$(awk -F \" 'BEGIN{OFS=","}/^[0-9]/{udata=$0}$2=="status"{status=$4}{if($2=="reason"&&$4!="")print udata,status,$4;else if($2=="reason"&&$4=="")print udata,status}' /home/${user}/.cpanel/backup_status) ; echo "$bid,$pkgid,$user$nas"; done
1708371,1684080,platinum,failed,disk
1635639,1684091,mrpierre,finished
1708386,1684095,ldoogs1,finished
1708442,1684145,kraegerb,finished
1708478,1684182,nongtae,failed,inode
1708567,1684275,siammap,finished