simple sed examples


“sed” is a powerful tool to automatically edit strings or streams (sed = Stream EDitor). It comes with every modern Linux or Unix system and you can use in many different cases. It’s syntax is not very intuitive but it is definitely worth it to learn it’s basics .

Example use cases:

  • Recursively replace a string in all subfolders (for example replace Version Number from 1.2.3.5 to 2.0.0.1)
  • Replace a destination folder in all install scripts (for example /usr/app-customer1 to /opt/customer2)
  • Escape input from a user and remove all “dangerous” characters
  • Convert strings in ETL processes
  • Remove newlines from a stream to process the lines as arguments

If you want to learn more about “sed” I would recommend that you read the following 2 books – they will give you a good start:

1. sed and awk Pocket Reference (Pocket Reference (O’Reilly)) and also:
2. Introducing Regular Expressions. Without knowing regular expressions, sed will not be very useful to you.

For an even shorter introduction I will provide you with some useful examples, that I use in my day to day work.

Examples

To describe my examples I created an example.txt. The file looks like this:

[email protected]:~$ cat example.txt
line 1
This is some string in LINE 2
Lalalalaline 3

^ empty line 4 described on line 5
another line 6
and THIS IS the LAST LinE 7

basic sed usage

Sed works with streams. So you can pipe your commands output or contents of files directly into sed and manipulate the data as required. You can save the output in new files or also edit a file “inplace”.

sed-usage

The most common usage is probably this (open a console / terminal / putty window):

sed 's/SEARCH/REPLACE/' in.txt >out.txt

This will open the file “in.txt” and look for the string SEARCH line by line.
If it finds it it will replace it with REPLACE (all case sensitive) and the output will be redirected to the file “out.txt”.
If you omit the “>out.txt” part the results will just be printed on your console.

This will replace the string “Lalala” found on line 3 in the example.txt with “XXXXXX” and output the results on stdout:

[email protected]:~$ sed 's/Lalala/XXXXXX/' example.txt
line 1
This is some string in LINE 2
XXXXXXlaline 3

^ empty line 4 described on line 5
another line 6
and THIS IS the LAST LinE 7

replace multiple strings in multiple files recursively

Let’s assume you have hundreds of files of source code and each contains a copyright header that you want to change in all files at once (in all subfolders).

For the example I created the following structure and “Copyright 2015” shall become “Copyright 2016” in all *.py files:

[email protected]:~/src$ find .

./subfolder
./subfolder/file3.py
./exclude_me.txt
./file2.py
./file1.py

The sed command to replace the string recursively in all files will look like this:

find src -type f -name "*.py" -exec \
sed -i 's/Copyright 2015/Copyright 2016/g' {} +

The command above is searching the folder “src” for all files that match “*.py”.

For each file it will run sed to replace “Copyright 2015 with Copyright 2016”.

The “-i” option will cause sed to “inplace” edit the file (it will directly edit and save the files). This can be quite dangerous – so make sure you are using the correct root path for the command and that your search pattern is not too simple. If you would just replace 15 with 16 you can be quite sure that you will modify your code in some other places like “if i < 15 then …” and if you set the path to / you can destroy your whole system as it would also search for 15 in system files and try to modify them. So you better always do a test run without the “-i” option to check what would happen …