//Tips tagged awk
Latest tips by RSS
Click here to subscribe
Follow Shell-Fu on Twitter
Click here to follow
Follow Shell-Fu on identi.ca
Click here to follow
This piece of code lists the size of every file and subdirectory of the current directory, much like du -sch ./* except the output is sorted by size, with larger files and directories at the end of the list. Useful to find where all that space goes.


du -sk ./* | sort -n | awk 'BEGIN{ pref[1]="K"; pref[2]="M"; pref[3]="G";} { total = total + $1; x = $1; y = 1; while( x > 1024 ) { x = (x + 1023)/1024; y++; } printf("%g%s\t%s\n",int(x*10)/10,pref[y],$2); } END { y = 1; while( total > 1024 ) { total = (total + 1023)/1024; y++; } printf("Total: %g%s\n",int(total*10)/10,pref[y]); }'


View Comments »



To find out the number of files of each type in your current directory try the following:

find ${*-.} -type f | xargs file | awk -F, '{print $1}' | awk '{$1=NULL;print $0}' | sort | uniq -c | sort -nr  

5 PHP script text 2 data 2 Zip archive data 2 GIF image data 1 PNG image data

(You may want to add this as an alias rather than type it in each time!)


View Comments »

Simulate a never-ending compilation so you have an excuse for why you're browsing the net (see http://xkcd.com/303/):

while true; do awk '{ print ; system("let R=$RANDOM%10; sleep $R") }' compiler.log; done


When you want to kill it, CTRL-C won't work. You have to suspend it (CTRL-Z) then kill the job (kill %1).


View Comments »

This is a way to monitor "/var/log/messages" or any file for certain changes.
The example below actively monitors "stuff" for the word "now" and as soon as "now" is added to the file, the contents of msg are sent by email

$ tail -f stuff | awk ' /now/ { system("mail -s \"Now Occured\"  mail@foo.com < msg") }'


View Comments »

Sort a file by line length:


cat test.txt | awk '{ printf "%06d|%s\n",length($0),$0}' | sort | cut -b'8-'


View Comments »

Use the following to see the commands you use most often based on your shell history:

history | awk '{print $2}' | sort | uniq -c | sort -rn | head


View Comments »

ps | grep processName | grep -v grep | awk '{print "kill -9 " $2}' | sh


Take the '| sh' off the end before running to check the commands that will be run.


View Comments »

lynx -dump http://www.spantz.org | grep -A999 "^References$" | tail -n +3 | awk '{print $2 }'


View Comments »

A simple shell script to get the latest stable version of the linux kernel:

#!/bin/bash

kernelV=`finger finger@kernel.org | grep 'stable version' | awk '{print $NF}'`
wget -c http://www.kernel.org/pub/linux/kernel/v2.6/linux-$kernelV.tar.bz2


View Comments »

The command below will print a ascii art graph of connections from you are making to different hosts

netstat -an | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c \
 | awk '{ printf("%s\t%s\t",$2,$1) ; for (i = 0; i < $1; i++) {printf("*")}; print ""}'

You may need to check the output from "netstat -an" to check the host is in the 5th column, if not change the first "awk" to the right column number.


View Comments »

#!/bin/bash
# by dj.r4iden
echo "Your ip Address is" `lynx --source http://www.formyip.com/ |grep The | awk {'print $5'}`


View Comments »

Kick all users other than you from your box and keep them out.

watch -d 'w | awk 'NR==4 {print "/dev/"$2}' | xargs fuser -k'


View Comments »

Resolve all your conflicted files. Use with caution!!

svn st|awk '/^C/{ print $2; }'|xargs svn resolved


View Comments »

The command below can be used to sort the lines of a file by ascending order (shortest first). Change 'sort -n' to 'sort -nr' to sort in descending order (longest first).

cat /etc/passwd | awk '{print length, $0}' | sort -n | awk '{$1=""; print $0 }'


View Comments »

Use awk to change the file extension for a group of files. For example to change all .htm files to .php:

ls *htm | awk -F. '{print "mv "$0" "$1".php"}' | sh


This can be tested first by leaving the '| sh' off to give a list of the commands that will be executed.


View Comments »

I use the following to list non-system users. It should be portable though won't work on systems without the getent command.

alias lsusers='getent passwd | tr ":" " " | awk "\$3 >= $(grep UID_MIN /etc/login.defs | cut -d " " -f 2) { print \$1 }" | sort'


View Comments »

The following command will concatenate pairs of lines from a file, with a comma separating each line.

awk 'ORS=NR%2?",":"\n"' FILE

For example, if FILE is of the form
A
B
C
D
the command would print
A,B
C,D

This works by changing ORS (Output Record Seperator) using a ternary construct ( expr ? iftrue : iffalse ).
NR in awk is the current line number; % is the modulo operator, and 2 is the argument to it. Thus, if the current line number is modulo 2, the ORS is ",", and if the line number is not modulo 2, ORS is "\n".


View Comments »

Convert mac addresses such as 000000abde00 into 00:00:00:ab:de:00

awk '{for(i=10;i>=2;i-=2)$0=substr($0,1,i)":"substr($0,i+1);print}' macaddress_list
sed 's/\(..\)/\1:/g;s/:$//' macaddress_list

// sil at infiltrated.net


View Comments »

My little "iso2cd" alias. Not clean, but handy. The Burning device will be auto detected.

example call:
iso2cd debian_lenny_final.iso

alias iso2cd="cdrecord -s dev=`cdrecord --devices 2>&1 | grep "\(rw\|dev=\)" | awk {'print $2'} | cut -f'2' -d'=' | head -n1` gracetime=1 driveropts=burnfree -dao -overburn -v"


View Comments »

Mail somebody about space running low in some path (ksh, bash):


PATHS="/export/home /home"
AWK=/usr/bin/awk
DU="/usr/bin/du -ks"
GREP=/usr/bin/grep
DF="/usr/bin/df -k"
TR=/usr/bin/tr
SED=/usr/bin/sed
CAT=/usr/bin/cat
MAILFILE=/tmp/mailviews$$
MAILER=/bin/mailx
mailto="all@company.com"
for path in $PATHS
do
DISK_AVAIL=`$DF $path | $GREP -v "Filesystem" | $AWK '{print $5}'|$SED 's/%//g'`
if [ $DISK_AVAIL -gt 90 ];then
echo "Please clean up your stuff\n\n" > $MAILFILE
$CAT $MAILFILE | $MAILER -s "Clean up stuff" $mailto
fi
done


View Comments »

Counts files in the current directory and subdirectory
alias lll='for i in *; do echo "`ls -1aRi  $i | awk "/^[0-9]+ / { print $1 }" | sort -u | wc -l` $i" ; done | sort -n'

found is vserver-tools


View Comments »

Use this command to list files that have been updated today in the current directory.

ls -l|awk '/'$(date +%Y-%m-%d)'/{print $NF}'

This method is an alternative to using find with the mtime option (see tip 199) and can be a more intuitive way of locating files modified on a specific date, for example:

ls -l|awk '/2009-04-28/{print $NF}'

'ls --full-time' can also be used and the matching criteria modified to find files modified in a particular month, hour, day, between 09:00 and 17:00 each day or anything else as required.


View Comments »



file -N * | awk -F":" '{type[$2]++}END{ for (i in type) print type[i],i }'


View Comments »




Home Latest Browse Top 25 Random Hall Of Fame Contact Submit