//Tips tagged sed
Remove comments and blank lines from file
Latest tips by RSS
Click here to subscribe
Follow Shell-Fu on Twitter
Click here to follow
Follow Shell-Fu on identi.ca
Click here to follow
Converts the symbolic permissions to octal (ie: numbers) when using 'ls -l':
$ls -l | sed -e 's/--x/1/g' -e 's/-w-/2/g' -e 's/-wx/3/g' -e 's/r--/4/g' \ -e 's/r-x/5/g' -e 's/rw-/6/g' -e 's/rwx/7/g' -e 's/---/0/g' -755 1 jrl jrl 111943 2003-10-21 19:57 logscan -644 1 jrl jrl 35468 2003-11-23 16:13 htfoo -700 1 jrl jrl 3100672 2004-05-15 17:00 mutt -644 1 jrl jrl 10162 2005-02-22 14:14 joinstep2.php -777 1 jrl jrl 41079 2005-04-21 13:02 setistats d755 2 jrl jrl 47 2007-10-26 14:41 rf -700 1 jrl jrl 104 2008-02-05 11:26 getcIf you're going to use this, you may want to make an alias rather than type it in each time!
Add the following alias and function to your profile to be able to copy and paste files at the command line:
You can see below how this can be used:
ccopy(){ cp $1 /tmp/ccopy.$1; }
alias cpaste="ls /tmp/ccopy* | sed 's|[^\.]*.\.||' | xargs -I % mv /tmp/ccopy.% ./%"
You can see below how this can be used:
blackbird:~/tst tks1$ ls 1.txt 2.txt t1.tss t2.tss t3.tss blackbird:~/tst tks1$ ccopy 1.txt blackbird:~/tst tks1$ ccopy 2.txt
blackbird:~/tst tks1$ cd ../tst2 blackbird:~/tst2 tks1$ ls
blackbird:~/tst2 tks1$ cpaste
blackbird:~/tst2 tks1$ ls 1.txt 2.txt
YouTube is great, but the embedded flash player doesn't offer a lot in the way of customizing your viewing experience. There are quite a few options for downloading flv video files from YouTube, but adding the following function to your bashrc will let you stream them directly to your choice of media player.
This should work with any URL of the form http://www.youtube.com/watch?v=[video id here], for example:
function mtube {
video_id=$(curl -s $1 | sed -n "/watch_fullscreen/s;.*\(video_id.\+\)&title.*;\1;p");
mplayer -fs $(echo "http://youtube.com/get_video.php?$video_id");
}
This should work with any URL of the form http://www.youtube.com/watch?v=[video id here], for example:
mtube http://www.youtube.com/watch?v=evQ0QNi2CzA
Add commas to all numeric strings in a file, changing "1234567" to "1,234,567"
sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta' filename
To create a set of backed up files with the current date added at the end of the file name try the following:
This will run the following commands:
ls *txt | sed "s/.*/cp & &.$(date "+%Y%d%m")/"
This will run the following commands:
cp 1.txt 1.txt.20082703
cp 2.txt 2.txt.20082703
cp 3.txt 3.txt.20082703
The following alias will print the directory structure from the current directory in tree format.
alias dirf='find . -type d | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/"'
Selective content replace on files. For example to replace '<?' with '<?php' in all PHP files:
find . -name '*.php' -exec sed -ie 's#<?#<?php#' {} \;Mass-renaming files using find and sed:
(this example will rename all .php3 files to .php)
find -name "*.php3" | sed 's/\(.*\).php3$/mv "&" "\1.php"/' | sh
(this example will rename all .php3 files to .php)
Prefer Perl over Sed? No problem! To use Perl as a Sed-like program:
Have a Perl script ready? No problem!
perl -pe 's/foo/bar/; etc'
Have a Perl script ready? No problem!
perl -p foo.pl
Add the following sed commands to cal to get a calendar with the current date marked:
cal | sed "s/^/ /;s/$/ /;s/ $(date +%e) / $(date +%e | sed 's/./#/g') /"
sed '/ *#/d; /^ *$/d' file
Remove comments and blank lines from file
The following command creates in the /usr/project directory, a copy of the current working directory structure:
find . -type d -print|sed 's@^\.\{0,1\}@/usr/project@' | sed 's/ /\\ /' | xargs mkdir -pSometimes when using sed you find that you need to match across line endings, this can be achieved by getting sed to match the first line and then pulling a second line into the buffer with the N command.
For example, if we have a file:
And want to change 'hello world' to 'hello shell-fu' we need to replace across lines. This can be done with the following command:
Here sed first looks for lines which end with 'hello' then reads the next line, finally replacing 'hello\nworld' with 'hello\nshell-fu'.
This also has a lot of other uses, for example converting double line spaced files to single:
For example, if we have a file:
$ cat foo This is a sample hello world file.
And want to change 'hello world' to 'hello shell-fu' we need to replace across lines. This can be done with the following command:
:~$ cat foo | sed '/hello$/N;s/hello\nworld/hello\nshell-fu/' This is a sample hello shell-fu file.
Here sed first looks for lines which end with 'hello' then reads the next line, finally replacing 'hello\nworld' with 'hello\nshell-fu'.
This also has a lot of other uses, for example converting double line spaced files to single:
cat doublespace | sed '/^$/N;s/\n$//g'
sed = filename | sed 'N;s/\n/\t/' > filename.numbered
Displays a random xkcd comic. Requires ImageMagick.
wget http://dynamic.xkcd.com/comic/random/ -O -| grep <img src="http://imgs.xkcd.com/comics | sed s/<img src="// | sed s/"[a-z]*.*// | wget -i - -O -| display
Mail somebody about space running low in some path (ksh, bash):
PATHS="/export/home /home"
AWK=/usr/bin/awk
DU="/usr/bin/du -ks"
GREP=/usr/bin/grep
DF="/usr/bin/df -k"
TR=/usr/bin/tr
SED=/usr/bin/sed
CAT=/usr/bin/cat
MAILFILE=/tmp/mailviews$$
MAILER=/bin/mailx
mailto="all@company.com"
for path in $PATHS
do
DISK_AVAIL=`$DF $path | $GREP -v "Filesystem" | $AWK '{print $5}'|$SED 's/%//g'`
if [ $DISK_AVAIL -gt 90 ];then
echo "Please clean up your stuff\n\n" > $MAILFILE
$CAT $MAILFILE | $MAILER -s "Clean up stuff" $mailto
fi
done
Go to [http://www.bbc.co.uk/cgi-perl/weather/search/new_search.pl] and search for your location. Copy the link to the 3 day forecast feed, for example [http://feeds.bbc.co.uk/weather/feeds/rss/5day/world/0105.xml]. Then just put that link into the following command to get a quick command line weather forecast.
wget -q -O - http://feeds.bbc.co.uk/weather/.../XXXXX.xml | grep title | sed -e "s/<[^>]*>//g" -e "s/°//g" | egrep "^[A-Z]"
Example:
$ wget -q -O - http://feeds.bbc.co.uk/weather/feeds/rss/5day/world/0105.xml | grep title | sed -e "s/<[^>]*>//g" -e "s/°//g" | egrep "^[A-Z]"
BBC - Weather Centre - Forecast for San Francisco, United States of America Tuesday: sunny, Max Temp: 22C (72F), Min Temp: 13C (55F) Wednesday: sunny, Max Temp: 22C (72F), Min Temp: 15C (59F) Thursday: sunny, Max Temp: 25C (77F), Min Temp: 14C (57F)
Print the contents of a file from a given regular expression to another
This will print the contents of the file from the line that matches /start/ until the line that matches /end/
sed -n '/start/,/end/ p' file
This will print the contents of the file from the line that matches /start/ until the line that matches /end/
Convert mac addresses such as 000000abde00 into 00:00:00:ab:de:00
awk '{for(i=10;i>=2;i-=2)$0=substr($0,1,i)":"substr($0,i+1);print}' macaddress_list
sed 's/\(..\)/\1:/g;s/:$//' macaddress_list
// sil at infiltrated.net
awk '{for(i=10;i>=2;i-=2)$0=substr($0,1,i)":"substr($0,i+1);print}' macaddress_list
sed 's/\(..\)/\1:/g;s/:$//' macaddress_list
// sil at infiltrated.net
The following isn't particularly pretty and should be considered a work in progress, but it's quite fun.
Get examples of ways a command can be used direct from shell-fu by adding the following alias:
This pulls out the tips tagged by the given command. (Make sure you tag any tips you submit!)
Get examples of ways a command can be used direct from shell-fu by adding the following alias:
function examples { lynx -width=$COLUMNS -nonumbers -dump "http://www.shell-fu.org/lister.php?tag=$1" | \
sed -n '/^[a-zA-Z]/,$p' | egrep -v '^http|^javas|View Comm|HIDE|] \+|to Share|^ +\*|^ +[HV][a-z]* l|^ .*efu.*ep.*!$' | \
sed -e '/^ *__*/N;s/\n$//g' | less -r; }
This pulls out the tips tagged by the given command. (Make sure you tag any tips you submit!)
This bit of sed will print the contents of a file until the first line which doesn't contain the specified expression. A useful alternative to 'head' when you're not sure how much of the file you need.
sed -n '/Hello/!q; p'
SSH_COMPLETE=( $(cat ~/.ssh/known_hosts | \
cut -f 1 -d ' ' | \
sed -e s/,.*//g | \
uniq | \
egrep -v [0123456789]) )
complete -o default -W "${SSH_COMPLETE[*]}" sshAdd a leading angle bracket and space to each line (quote a message)
sed 's/^/> /' filename
I'm working on a project to access data from my corporate Active Directory server using ldapsearch. The version of ldapsearch that comes with Red Hat Enterprise Server spews out LDIF in column truncated format. That is to say it inserts a carriage return at column 80 of the output.
This sed goodie unwraps the ldif output.
UNWRAP=' /^ / {; H; d; }; /^ /! {; x; s/\n //; }; '
This sed goodie unwraps the ldif output.
UNWRAP=' /^ / {; H; d; }; /^ /! {; x; s/\n //; }; '
Use the following command to give a history listing without the numbers for easier copy and pasting:
history | sed 's/^[ 0-9]* //'