0

Tip #246   Dealing with duplicate lines

You can use the 'uniq' command to handle duplicate lines in a file: 

$ cat tst
a
a
b
c
c
d

$ uniq -d tst        //show duplicated lines
a
c

$ uniq -u tst        //show unique lines
b
d

$ uniq tst           //remove duplicates
a
b
c
d

 
alias aptitude at awk bash bc cal cat cd colrm comm cp csh curl cut date dd df dialog diff dirname dpkg du fc find fuser grep gs gzip history iconv kill ksh last less ln ls lsof lynx m4 md5sum mkdir mkfifo mkisofs mv mysql nc netstat openssl OSX perl ping popd ps pushd python read redirection rm scp screen sed sort ssh stat sudo svn tail tar tee test top tr uniq vim wc wget xargs