//Tips tagged awk
Take the '| sh' off the end before running to check the commands that will be run.
Latest tips by RSS
Click here to subscribe
Follow Shell-Fu on Twitter
Click here to follow
Follow Shell-Fu on identi.ca
Click here to follow
This piece of code lists the size of every file and subdirectory of the current directory, much like du -sch ./* except the output is sorted by size, with larger files and directories at the end of the list. Useful to find where all that space goes.
du -sk ./* | sort -n | awk 'BEGIN{ pref[1]="K"; pref[2]="M"; pref[3]="G";} { total = total + $1; x = $1; y = 1; while( x > 1024 ) { x = (x + 1023)/1024; y++; } printf("%g%s\t%s\n",int(x*10)/10,pref[y],$2); } END { y = 1; while( total > 1024 ) { total = (total + 1023)/1024; y++; } printf("Total: %g%s\n",int(total*10)/10,pref[y]); }'
To find out the number of files of each type in your current directory try the following:
(You may want to add this as an alias rather than type it in each time!)
find ${*-.} -type f | xargs file | awk -F, '{print $1}' | awk '{$1=NULL;print $0}' | sort | uniq -c | sort -nr
5 PHP script text
2 data
2 Zip archive data
2 GIF image data
1 PNG image data
(You may want to add this as an alias rather than type it in each time!)
Simulate a never-ending compilation so you have an excuse for why you're browsing the net (see http://xkcd.com/303/):
When you want to kill it, CTRL-C won't work. You have to suspend it (CTRL-Z) then kill the job (kill %1).
while true; do awk '{ print ; system("let R=$RANDOM%10; sleep $R") }' compiler.log; done
When you want to kill it, CTRL-C won't work. You have to suspend it (CTRL-Z) then kill the job (kill %1).
lynx -dump http://whatismyip.com | awk '/^Your/ {print $5}'This is a way to monitor "/var/log/messages" or any file for certain changes.
The example below actively monitors "stuff" for the word "now" and as soon as "now" is added to the file, the contents of msg are sent by email
The example below actively monitors "stuff" for the word "now" and as soon as "now" is added to the file, the contents of msg are sent by email
$ tail -f stuff | awk ' /now/ { system("mail -s \"Now Occured\" mail@foo.com < msg") }'Sort a file by line length:
cat test.txt | awk '{ printf "%06d|%s\n",length($0),$0}' | sort | cut -b'8-'
Use the following to see the commands you use most often based on your shell history:
history | awk '{print $2}' | sort | uniq -c | sort -rn | headThe command below will print a ascii art graph of connections from you are making to different hosts
You may need to check the output from "netstat -an" to check the host is in the 5th column, if not change the first "awk" to the right column number.
netstat -an | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c \
| awk '{ printf("%s\t%s\t",$2,$1) ; for (i = 0; i < $1; i++) {printf("*")}; print ""}'
You may need to check the output from "netstat -an" to check the host is in the 5th column, if not change the first "awk" to the right column number.
lynx -dump http://www.spantz.org | grep -A999 "^References$" | tail -n +3 | awk '{print $2 }'
A simple shell script to get the latest stable version of the linux kernel:
#!/bin/bash
kernelV=`finger finger@kernel.org | grep 'stable version' | awk '{print $NF}'`
wget -c http://www.kernel.org/pub/linux/kernel/v2.6/linux-$kernelV.tar.bz2
ps | grep processName | grep -v grep | awk '{print "kill -9 " $2}' | sh
Take the '| sh' off the end before running to check the commands that will be run.
Resolve all your conflicted files. Use with caution!!
svn st|awk '/^C/{ print $2; }'|xargs svn resolvedMail somebody about space running low in some path (ksh, bash):
PATHS="/export/home /home"
AWK=/usr/bin/awk
DU="/usr/bin/du -ks"
GREP=/usr/bin/grep
DF="/usr/bin/df -k"
TR=/usr/bin/tr
SED=/usr/bin/sed
CAT=/usr/bin/cat
MAILFILE=/tmp/mailviews$$
MAILER=/bin/mailx
mailto="all@company.com"
for path in $PATHS
do
DISK_AVAIL=`$DF $path | $GREP -v "Filesystem" | $AWK '{print $5}'|$SED 's/%//g'`
if [ $DISK_AVAIL -gt 90 ];then
echo "Please clean up your stuff\n\n" > $MAILFILE
$CAT $MAILFILE | $MAILER -s "Clean up stuff" $mailto
fi
done
I use the following to list non-system users. It should be portable though won't work on systems without the getent command.
alias lsusers='getent passwd | tr ":" " " | awk "\$3 >= $(grep UID_MIN /etc/login.defs | cut -d " " -f 2) { print \$1 }" | sort'#!/bin/bash
# by dj.r4iden
echo "Your ip Address is" `lynx --source http://www.formyip.com/ |grep The | awk {'print $5'}`
# by dj.r4iden
echo "Your ip Address is" `lynx --source http://www.formyip.com/ |grep The | awk {'print $5'}`
Kick all users other than you from your box and keep them out.
watch -d 'w | awk 'NR==4 {print "/dev/"$2}' | xargs fuser -k'Use awk to change the file extension for a group of files. For example to change all .htm files to .php:
This can be tested first by leaving the '| sh' off to give a list of the commands that will be executed.
ls *htm | awk -F. '{print "mv "$0" "$1".php"}' | sh
This can be tested first by leaving the '| sh' off to give a list of the commands that will be executed.
The following command will concatenate pairs of lines from a file, with a comma separating each line.
For example, if FILE is of the form
A
B
C
D
the command would print
A,B
C,D
This works by changing ORS (Output Record Seperator) using a ternary construct ( expr ? iftrue : iffalse ).
NR in awk is the current line number; % is the modulo operator, and 2 is the argument to it. Thus, if the current line number is modulo 2, the ORS is ",", and if the line number is not modulo 2, ORS is "\n".
awk 'ORS=NR%2?",":"\n"' FILE
For example, if FILE is of the form
A
B
C
D
the command would print
A,B
C,D
This works by changing ORS (Output Record Seperator) using a ternary construct ( expr ? iftrue : iffalse ).
NR in awk is the current line number; % is the modulo operator, and 2 is the argument to it. Thus, if the current line number is modulo 2, the ORS is ",", and if the line number is not modulo 2, ORS is "\n".
Use this command to list files that have been updated today in the current directory.
This method is an alternative to using find with the mtime option (see tip 199) and can be a more intuitive way of locating files modified on a specific date, for example:
'ls --full-time' can also be used and the matching criteria modified to find files modified in a particular month, hour, day, between 09:00 and 17:00 each day or anything else as required.
ls -l|awk '/'$(date +%Y-%m-%d)'/{print $NF}'
This method is an alternative to using find with the mtime option (see tip 199) and can be a more intuitive way of locating files modified on a specific date, for example:
ls -l|awk '/2009-04-28/{print $NF}'
'ls --full-time' can also be used and the matching criteria modified to find files modified in a particular month, hour, day, between 09:00 and 17:00 each day or anything else as required.
nmap -iR 20 -sL -n|awk '{print $2}'|grep '^[0-9]'|uniq
Counts files in the current directory and subdirectory
found is vserver-tools
alias lll='for i in *; do echo "`ls -1aRi $i | awk "/^[0-9]+ / { print $1 }" | sort -u | wc -l` $i" ; done | sort -n'
found is vserver-tools
My little "iso2cd" alias. Not clean, but handy. The Burning device will be auto detected.
example call:
iso2cd debian_lenny_final.iso
alias iso2cd="cdrecord -s dev=`cdrecord --devices 2>&1 | grep "\(rw\|dev=\)" | awk {'print $2'} | cut -f'2' -d'=' | head -n1` gracetime=1 driveropts=burnfree -dao -overburn -v"
example call:
iso2cd debian_lenny_final.iso
alias iso2cd="cdrecord -s dev=`cdrecord --devices 2>&1 | grep "\(rw\|dev=\)" | awk {'print $2'} | cut -f'2' -d'=' | head -n1` gracetime=1 driveropts=burnfree -dao -overburn -v"
The command below can be used to sort the lines of a file by ascending order (shortest first). Change 'sort -n' to 'sort -nr' to sort in descending order (longest first).
cat /etc/passwd | awk '{print length, $0}' | sort -n | awk '{$1=""; print $0 }'Convert mac addresses such as 000000abde00 into 00:00:00:ab:de:00
awk '{for(i=10;i>=2;i-=2)$0=substr($0,1,i)":"substr($0,i+1);print}' macaddress_list
sed 's/\(..\)/\1:/g;s/:$//' macaddress_list
// sil at infiltrated.net
awk '{for(i=10;i>=2;i-=2)$0=substr($0,1,i)":"substr($0,i+1);print}' macaddress_list
sed 's/\(..\)/\1:/g;s/:$//' macaddress_list
// sil at infiltrated.net
file -N * | awk -F":" '{type[$2]++}END{ for (i in type) print type[i],i }'