9.9 KiB
9.9 KiB
AWK and SED Examples
These examples explore the combined use of AWK and SED for text processing tasks.
Example 1: AWK and SED for Text Transformation
Input File (input.txt)
John Doe,25,Engineer
Jane Smith,30,Designer
Bob Johnson,22,Developer
Goal
We want to transform the CSV data into a formatted table with headers.
Solution
awk -F',' '{printf "%-15s %-3s %-10s\n", $1, $2, $3}' input.txt | sed '1i\
Name Age Job\
--------------- --- ----------'
Output
Name Age Job
--------------- --- ----------
John Doe 25 Engineer
Jane Smith 30 Designer
Bob Johnson 22 Developer
Example 2: AWK and SED for Selective Printing
Input File (input.txt)
apple
banana
cherry
date
Goal
Print lines containing "a" or "e" using AWK and SED.
Solution
awk '/a|e/' input.txt | sed -n '/a\|e/p'
Output
apple
banana
date
Example 3: AWK and SED for Filtering and Sorting
Input File (input.txt)
apple,5
banana,2
cherry,8
date,1
Goal
Filter lines with the second column greater than 2, sort by the second column, and add a header.
Solution
awk -F',' '$2 > 2' input.txt | sort -t',' -k2n | sed '1i\
Fruit,Quantity'
Output
Fruit,Quantity
apple,5
cherry,8
Example 4: AWK and SED for Text Extraction
Input File (input.txt)
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris.
Goal
Extract sentences containing the word "sed" and replace "sed" with "AWK".
Solution
awk 'BEGIN{RS="\\."} /sed/' input.txt | sed 's/sed/AWK/g'
Output
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua
Example 5: AWK and SED for HTML Tag Extraction
Input File (input.html)
<div class="container">
<p>This is a paragraph.</p>
<a href="https://example.com">Visit our website</a>
</div>
Goal
Extract and list all HTML tags present in the file.
Solution
sed -n 's/<[^>]*>//p' input.html | awk '{print tolower($0)}'
Output
div class="container"
p
a href="https://example.com"
/a
Example 6: AWK and SED for JSON Formatting
Input File (input.json)
{
"name": "John Doe",
"age": 30,
"city": "New York"
}
Goal
Format the JSON to be human-readable using AWK and SED.
Solution
awk -v RS= '{$1=$1}1' input.json | sed 's/":"/": "/g; s/{/{\n/g; s/}/\n}/g'
Output
{
"name": "John Doe",
"age": 30,
"city": "New York"
}
Example 7: AWK and SED for Log Analysis
Input File (logfile.txt)
2023-01-15 10:23:15 INFO User login successful
2023-01-15 10:30:42 ERROR Invalid password attempt
2023-01-15 11:05:18 INFO User logout
Goal
Extract and format log entries containing "ERROR".
Solution
awk '/ERROR/' logfile.txt | sed 's/\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\} [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}\) \(ERROR .*\)/Timestamp: \1\n\2\n/'
Output
Timestamp: 2023-01-15 10:30:42
ERROR Invalid password attempt
Example 8: AWK and SED for CSV Transformation
Input File (data.csv)
Name,Age,Occupation
John,28,Engineer
Jane,35,Designer
Bob,22,Developer
Goal
Transform the CSV data into a tab-separated format.
Solution
awk -F',' '{printf "%s\t%s\t%s\n", $1, $2, $3}' data.csv | sed '1i\
Name\tAge\tOccupation'
Output
Name Age Occupation
John 28 Engineer
Jane 35 Designer
Bob 22 Developer
Example 9: AWK and SED for XML Tag Removal
Input File (data.xml)
<root>
<item>Apple</item>
<item>Banana</item>
<item>Cherry</item>
</root>
Goal
Remove all XML tags and extract the content.
Solution
sed 's/<[^>]*>//g' data.xml | awk 'NF'
Output
Apple
Banana
Cherry
Example 10: AWK and SED for Markdown Table Conversion
Input File (table.md)
| Name | Age | Occupation |
|-------|-----|-------------|
| John | 28 | Engineer |
| Jane | 35 | Designer |
| Bob | 22 | Developer |
Goal
Convert the Markdown table into a CSV format.
Solution
sed -n '/|/p' table.md | awk -F'|' '{gsub(/^[ \t]+|[ \t]+$/, "", $2); gsub(/^[ \t]+|[ \t]+$/, "", $3); gsub(/^[ \t]+|[ \t]+$/, "", $4); printf "%s,%s,%s\n", $2, $3, $4}'
Output
Name,Age,Occupation
John,28,Engineer
Jane,35,Designer
Bob,22,Developer
Example 11: AWK and SED for HTML List Extraction
Input File (list.html)
<ul>
<li>Apple</li>
<li>Banana</li>
<li>Cherry</li>
</ul>
Goal
Extract and list items from an HTML unordered list.
Solution
sed -n '/<li>/s/<li>\(.*\)<\/li>/\1/p' list.html | awk '{print "- " $0}'
Output
- Apple
- Banana
- Cherry
Example 12: AWK and SED for YAML to JSON Conversion
Input File (data.yaml)
name: John Doe
age: 30
city: New York
Goal
Convert YAML data into JSON format.
Solution
sed -n '/^[^#]/s/: /: "/p' data.yaml | awk '{printf "%s", $0} END {print "\""}' | sed 's/"/\\"/g' | awk 'BEGIN {print "{"} {print " " $0} END {print "}"}'
Output
{
name: "John Doe",
age: "30",
city: "New York"
}
Example 13: AWK and SED for Extracting URLs
Input File (text_with_urls.txt)
Visit our website: https://example.com
For more information, check out: https://documentation.com
Goal
Extract and list all URLs from the text.
Solution
sed -n 's/.*\(https\?:\/\/[^ ]*\).*/\1/p' text_with_urls.txt | awk '!a[$0]++'
Output
https://example.com
https://documentation.com
Example 14: AWK and SED for Data Filtering and Sorting
Input File (data.txt)
Name|Age|Occupation
John|28|Engineer
Jane|35|Designer
Bob|22|Developer
Goal
Filter entries with Age greater than 25 and sort by Age.
Solution
awk -F'|' '$2 > 25' data.txt | sort -t'|' -k2n | sed '1i\
Name|Age|Occupation'
Output
Name|Age|Occupation
Jane|35|Designer
John|28|Engineer
Example 15: AWK and SED for Line Reversal
Input File (reverse_me.txt)
Line 1
Line 2
Line 3
Goal
Reverse the order of lines in the file.
Solution
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' reverse_me.txt | sed '1i\
Reversed Lines'
Output
Reversed Lines
Line 3
Line 2
Line 1
Example 16: AWK and SED for Word Frequency Analysis
Input File (text_analysis.txt)
apple orange banana
banana apple orange
orange banana apple
Goal
Count the frequency of each word and sort the results.
Solution
awk '{for(i=1; i<=NF; i++) words[$i]++} END {for(word in words) print word, words[word]}' text_analysis.txt | sort -k2n -r | sed '1i\
Word Frequency'
Output
Word Frequency
banana 3
apple 3
orange 3
Example 17: AWK and SED for JSON Pretty Printing
Input File (json_data.json)
{"name":"John","age":30,"city":"New York"}
Goal
Format the JSON data for better readability.
Solution
awk -v RS= '{$1=$1}1' json_data.json | sed 's/":"/": "/g; s/{/{\n/g; s/}/\n}/g'
Output
{
"name": "John",
"age": 30,
"city": "New York"
}
Example 18: AWK and SED for Extracting Email Addresses
Input File (emails.txt)
Contact us at: info@example.com
For support, email support@example.com
Goal
Extract and list all email addresses from the text.
Solution
sed -n 's/.*\b\([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}\)\b.*/\1/p' emails.txt | awk '!a[$0]++' | sed '1i\
Email Addresses'
Output
Email Addresses
info@example.com
support@example.com
Example 19: Convert Markdown Headings to HTML
Input (markdown_headings.md)
# Heading 1
## Heading 2
### Heading 3
Goal
Convert Markdown headings to HTML headings.
Solution
sed -n 's/^# \(.*\)$/\<h1\>\1\<\/h1\>/p; s/^## \(.*\)$/\<h2\>\1\<\/h2\>/p; s/^### \(.*\)$/\<h3\>\1\<\/h3\>/p' markdown_headings.md
Output
<h1>Heading 1</h1>
<h2>Heading 2</h2>
<h3>Heading 3</h3>
Example 20: Convert Markdown Lists to HTML
Input (markdown_lists.md)
- Item 1
- Item 2
- Subitem A
- Subitem B
Goal
Convert Markdown lists to HTML lists.
Solution
awk '/^-/ {gsub(/^-/, "<li>"); print "<ul>"; print $0} !/^-/ {gsub(/^ /, ""); print $0 "</li>"} END {print "</ul>"}' markdown_lists.md
Output
<ul>
<li>Item 1</li>
<li>Item 2<ul><li>Subitem A</li><li>Subitem B</li></ul></li>
</ul>
Example 21: Convert Bold and Italic Markdown to HTML
Input (markdown_bold_italics.md)
**Bold Text** *Italic Text*
Goal
Convert Markdown bold and italic text to HTML.
Solution
sed -e 's/\*\*\(.*\)\*\*/\<strong\>\1\<\/strong\>/g' -e 's/\*\([^*]*\)\*/\<em\>\1\<\/em\>/g' markdown_bold_italics.md
Output
<strong>Bold Text</strong> <em>Italic Text</em>
Example 22: Reverse the Order of Lines
Input (original_order.txt)
Line 1
Line 2
Line 3
Goal
Reverse the order of lines in the file.
Solution
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' original_order.txt
Output
Line 3
Line 2
Line 1
Example 23: Sort Lines Alphabetically
Input (unsorted_lines.txt)
Banana
Apple
Cherry
Goal
Sort lines alphabetically.
Solution
sort unsorted_lines.txt
Output
Apple
Banana
Cherry
Example 24: Sort Lines by Line Length
Input (lines_by_length.txt)
Short
VeryLongLineWithManyCharacters
MediumLength
Goal
Sort lines by their length.
Solution
awk '{ print length, $0 | "sort -n"}' lines_by_length.txt | cut -d ' ' -f2-
Output
Short
MediumLength
VeryLongLineWithManyCharacters