701 lines
9.9 KiB
Markdown
701 lines
9.9 KiB
Markdown
# AWK and SED Examples
|
|
|
|
These examples explore the combined use of AWK and SED for text processing tasks.
|
|
|
|
## Example 1: AWK and SED for Text Transformation
|
|
|
|
### Input File (input.txt)
|
|
|
|
```
|
|
John Doe,25,Engineer
|
|
Jane Smith,30,Designer
|
|
Bob Johnson,22,Developer
|
|
```
|
|
|
|
### Goal
|
|
|
|
We want to transform the CSV data into a formatted table with headers.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk -F',' '{printf "%-15s %-3s %-10s\n", $1, $2, $3}' input.txt | sed '1i\
|
|
Name Age Job\
|
|
--------------- --- ----------'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Name Age Job
|
|
--------------- --- ----------
|
|
John Doe 25 Engineer
|
|
Jane Smith 30 Designer
|
|
Bob Johnson 22 Developer
|
|
```
|
|
|
|
## Example 2: AWK and SED for Selective Printing
|
|
|
|
### Input File (input.txt)
|
|
|
|
```
|
|
apple
|
|
banana
|
|
cherry
|
|
date
|
|
```
|
|
|
|
### Goal
|
|
|
|
Print lines containing "a" or "e" using AWK and SED.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk '/a|e/' input.txt | sed -n '/a\|e/p'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
apple
|
|
banana
|
|
date
|
|
```
|
|
|
|
## Example 3: AWK and SED for Filtering and Sorting
|
|
|
|
### Input File (input.txt)
|
|
|
|
```
|
|
apple,5
|
|
banana,2
|
|
cherry,8
|
|
date,1
|
|
```
|
|
|
|
### Goal
|
|
|
|
Filter lines with the second column greater than 2, sort by the second column, and add a header.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk -F',' '$2 > 2' input.txt | sort -t',' -k2n | sed '1i\
|
|
Fruit,Quantity'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Fruit,Quantity
|
|
apple,5
|
|
cherry,8
|
|
```
|
|
|
|
## Example 4: AWK and SED for Text Extraction
|
|
|
|
### Input File (input.txt)
|
|
|
|
```
|
|
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
|
|
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
|
|
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris.
|
|
```
|
|
|
|
### Goal
|
|
|
|
Extract sentences containing the word "sed" and replace "sed" with "AWK".
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk 'BEGIN{RS="\\."} /sed/' input.txt | sed 's/sed/AWK/g'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua
|
|
```
|
|
|
|
## Example 5: AWK and SED for HTML Tag Extraction
|
|
|
|
### Input File (input.html)
|
|
|
|
```html
|
|
<div class="container">
|
|
<p>This is a paragraph.</p>
|
|
<a href="https://example.com">Visit our website</a>
|
|
</div>
|
|
```
|
|
|
|
### Goal
|
|
|
|
Extract and list all HTML tags present in the file.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
sed -n 's/<[^>]*>//p' input.html | awk '{print tolower($0)}'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
div class="container"
|
|
p
|
|
a href="https://example.com"
|
|
/a
|
|
```
|
|
|
|
## Example 6: AWK and SED for JSON Formatting
|
|
|
|
### Input File (input.json)
|
|
|
|
```json
|
|
{
|
|
"name": "John Doe",
|
|
"age": 30,
|
|
"city": "New York"
|
|
}
|
|
```
|
|
|
|
### Goal
|
|
|
|
Format the JSON to be human-readable using AWK and SED.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk -v RS= '{$1=$1}1' input.json | sed 's/":"/": "/g; s/{/{\n/g; s/}/\n}/g'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
{
|
|
"name": "John Doe",
|
|
"age": 30,
|
|
"city": "New York"
|
|
}
|
|
```
|
|
|
|
## Example 7: AWK and SED for Log Analysis
|
|
|
|
### Input File (logfile.txt)
|
|
|
|
```
|
|
2023-01-15 10:23:15 INFO User login successful
|
|
2023-01-15 10:30:42 ERROR Invalid password attempt
|
|
2023-01-15 11:05:18 INFO User logout
|
|
```
|
|
|
|
### Goal
|
|
|
|
Extract and format log entries containing "ERROR".
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk '/ERROR/' logfile.txt | sed 's/\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\} [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}\) \(ERROR .*\)/Timestamp: \1\n\2\n/'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Timestamp: 2023-01-15 10:30:42
|
|
ERROR Invalid password attempt
|
|
```
|
|
|
|
## Example 8: AWK and SED for CSV Transformation
|
|
|
|
### Input File (data.csv)
|
|
|
|
```
|
|
Name,Age,Occupation
|
|
John,28,Engineer
|
|
Jane,35,Designer
|
|
Bob,22,Developer
|
|
```
|
|
|
|
### Goal
|
|
|
|
Transform the CSV data into a tab-separated format.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk -F',' '{printf "%s\t%s\t%s\n", $1, $2, $3}' data.csv | sed '1i\
|
|
Name\tAge\tOccupation'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Name Age Occupation
|
|
John 28 Engineer
|
|
Jane 35 Designer
|
|
Bob 22 Developer
|
|
```
|
|
|
|
## Example 9: AWK and SED for XML Tag Removal
|
|
|
|
### Input File (data.xml)
|
|
|
|
```xml
|
|
<root>
|
|
<item>Apple</item>
|
|
<item>Banana</item>
|
|
<item>Cherry</item>
|
|
</root>
|
|
```
|
|
|
|
### Goal
|
|
|
|
Remove all XML tags and extract the content.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
sed 's/<[^>]*>//g' data.xml | awk 'NF'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Apple
|
|
Banana
|
|
Cherry
|
|
```
|
|
|
|
## Example 10: AWK and SED for Markdown Table Conversion
|
|
|
|
### Input File (table.md)
|
|
|
|
```markdown
|
|
| Name | Age | Occupation |
|
|
|-------|-----|-------------|
|
|
| John | 28 | Engineer |
|
|
| Jane | 35 | Designer |
|
|
| Bob | 22 | Developer |
|
|
```
|
|
|
|
### Goal
|
|
|
|
Convert the Markdown table into a CSV format.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
sed -n '/|/p' table.md | awk -F'|' '{gsub(/^[ \t]+|[ \t]+$/, "", $2); gsub(/^[ \t]+|[ \t]+$/, "", $3); gsub(/^[ \t]+|[ \t]+$/, "", $4); printf "%s,%s,%s\n", $2, $3, $4}'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Name,Age,Occupation
|
|
John,28,Engineer
|
|
Jane,35,Designer
|
|
Bob,22,Developer
|
|
```
|
|
|
|
## Example 11: AWK and SED for HTML List Extraction
|
|
|
|
### Input File (list.html)
|
|
|
|
```html
|
|
<ul>
|
|
<li>Apple</li>
|
|
<li>Banana</li>
|
|
<li>Cherry</li>
|
|
</ul>
|
|
```
|
|
|
|
### Goal
|
|
|
|
Extract and list items from an HTML unordered list.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
sed -n '/<li>/s/<li>\(.*\)<\/li>/\1/p' list.html | awk '{print "- " $0}'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
- Apple
|
|
- Banana
|
|
- Cherry
|
|
```
|
|
|
|
## Example 12: AWK and SED for YAML to JSON Conversion
|
|
|
|
### Input File (data.yaml)
|
|
|
|
```yaml
|
|
name: John Doe
|
|
age: 30
|
|
city: New York
|
|
```
|
|
|
|
### Goal
|
|
|
|
Convert YAML data into JSON format.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
sed -n '/^[^#]/s/: /: "/p' data.yaml | awk '{printf "%s", $0} END {print "\""}' | sed 's/"/\\"/g' | awk 'BEGIN {print "{"} {print " " $0} END {print "}"}'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
{
|
|
name: "John Doe",
|
|
age: "30",
|
|
city: "New York"
|
|
}
|
|
```
|
|
|
|
## Example 13: AWK and SED for Extracting URLs
|
|
|
|
### Input File (text_with_urls.txt)
|
|
|
|
```
|
|
Visit our website: https://example.com
|
|
For more information, check out: https://documentation.com
|
|
```
|
|
|
|
### Goal
|
|
|
|
Extract and list all URLs from the text.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
sed -n 's/.*\(https\?:\/\/[^ ]*\).*/\1/p' text_with_urls.txt | awk '!a[$0]++'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
https://example.com
|
|
https://documentation.com
|
|
```
|
|
|
|
## Example 14: AWK and SED for Data Filtering and Sorting
|
|
|
|
### Input File (data.txt)
|
|
|
|
```
|
|
Name|Age|Occupation
|
|
John|28|Engineer
|
|
Jane|35|Designer
|
|
Bob|22|Developer
|
|
```
|
|
|
|
### Goal
|
|
|
|
Filter entries with Age greater than 25 and sort by Age.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk -F'|' '$2 > 25' data.txt | sort -t'|' -k2n | sed '1i\
|
|
Name|Age|Occupation'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Name|Age|Occupation
|
|
Jane|35|Designer
|
|
John|28|Engineer
|
|
```
|
|
|
|
## Example 15: AWK and SED for Line Reversal
|
|
|
|
### Input File (reverse_me.txt)
|
|
|
|
```
|
|
Line 1
|
|
Line 2
|
|
Line 3
|
|
```
|
|
|
|
### Goal
|
|
|
|
Reverse the order of lines in the file.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' reverse_me.txt | sed '1i\
|
|
Reversed Lines'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Reversed Lines
|
|
Line 3
|
|
Line 2
|
|
Line 1
|
|
```
|
|
|
|
## Example 16: AWK and SED for Word Frequency Analysis
|
|
|
|
### Input File (text_analysis.txt)
|
|
|
|
```
|
|
apple orange banana
|
|
banana apple orange
|
|
orange banana apple
|
|
```
|
|
|
|
### Goal
|
|
|
|
Count the frequency of each word and sort the results.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk '{for(i=1; i<=NF; i++) words[$i]++} END {for(word in words) print word, words[word]}' text_analysis.txt | sort -k2n -r | sed '1i\
|
|
Word Frequency'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Word Frequency
|
|
banana 3
|
|
apple 3
|
|
orange 3
|
|
```
|
|
|
|
## Example 17: AWK and SED for JSON Pretty Printing
|
|
|
|
### Input File (json_data.json)
|
|
|
|
```json
|
|
{"name":"John","age":30,"city":"New York"}
|
|
```
|
|
|
|
### Goal
|
|
|
|
Format the JSON data for better readability.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk -v RS= '{$1=$1}1' json_data.json | sed 's/":"/": "/g; s/{/{\n/g; s/}/\n}/g'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
{
|
|
"name": "John",
|
|
"age": 30,
|
|
"city": "New York"
|
|
}
|
|
```
|
|
|
|
## Example 18: AWK and SED for Extracting Email Addresses
|
|
|
|
### Input File (emails.txt)
|
|
|
|
```
|
|
Contact us at: info@example.com
|
|
For support, email support@example.com
|
|
```
|
|
|
|
### Goal
|
|
|
|
Extract and list all email addresses from the text.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
sed -n 's/.*\b\([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}\)\b.*/\1/p' emails.txt | awk '!a[$0]++' | sed '1i\
|
|
Email Addresses'
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Email Addresses
|
|
info@example.com
|
|
support@example.com
|
|
```
|
|
|
|
## Example 19: Convert Markdown Headings to HTML
|
|
|
|
### Input (markdown_headings.md)
|
|
|
|
```markdown
|
|
# Heading 1
|
|
## Heading 2
|
|
### Heading 3
|
|
```
|
|
|
|
### Goal
|
|
|
|
Convert Markdown headings to HTML headings.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
sed -n 's/^# \(.*\)$/\<h1\>\1\<\/h1\>/p; s/^## \(.*\)$/\<h2\>\1\<\/h2\>/p; s/^### \(.*\)$/\<h3\>\1\<\/h3\>/p' markdown_headings.md
|
|
```
|
|
|
|
### Output
|
|
|
|
```html
|
|
<h1>Heading 1</h1>
|
|
<h2>Heading 2</h2>
|
|
<h3>Heading 3</h3>
|
|
```
|
|
|
|
## Example 20: Convert Markdown Lists to HTML
|
|
|
|
### Input (markdown_lists.md)
|
|
|
|
```markdown
|
|
- Item 1
|
|
- Item 2
|
|
- Subitem A
|
|
- Subitem B
|
|
```
|
|
|
|
### Goal
|
|
|
|
Convert Markdown lists to HTML lists.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk '/^-/ {gsub(/^-/, "<li>"); print "<ul>"; print $0} !/^-/ {gsub(/^ /, ""); print $0 "</li>"} END {print "</ul>"}' markdown_lists.md
|
|
```
|
|
|
|
### Output
|
|
|
|
```html
|
|
<ul>
|
|
<li>Item 1</li>
|
|
<li>Item 2<ul><li>Subitem A</li><li>Subitem B</li></ul></li>
|
|
</ul>
|
|
```
|
|
|
|
## Example 21: Convert Bold and Italic Markdown to HTML
|
|
|
|
### Input (markdown_bold_italics.md)
|
|
|
|
```markdown
|
|
**Bold Text** *Italic Text*
|
|
```
|
|
|
|
### Goal
|
|
|
|
Convert Markdown bold and italic text to HTML.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
sed -e 's/\*\*\(.*\)\*\*/\<strong\>\1\<\/strong\>/g' -e 's/\*\([^*]*\)\*/\<em\>\1\<\/em\>/g' markdown_bold_italics.md
|
|
```
|
|
|
|
### Output
|
|
|
|
```html
|
|
<strong>Bold Text</strong> <em>Italic Text</em>
|
|
```
|
|
|
|
## Example 22: Reverse the Order of Lines
|
|
|
|
### Input (original_order.txt)
|
|
|
|
```
|
|
Line 1
|
|
Line 2
|
|
Line 3
|
|
```
|
|
|
|
### Goal
|
|
|
|
Reverse the order of lines in the file.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' original_order.txt
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Line 3
|
|
Line 2
|
|
Line 1
|
|
```
|
|
|
|
## Example 23: Sort Lines Alphabetically
|
|
|
|
### Input (unsorted_lines.txt)
|
|
|
|
```
|
|
Banana
|
|
Apple
|
|
Cherry
|
|
```
|
|
|
|
### Goal
|
|
|
|
Sort lines alphabetically.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
sort unsorted_lines.txt
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Apple
|
|
Banana
|
|
Cherry
|
|
```
|
|
|
|
## Example 24: Sort Lines by Line Length
|
|
|
|
### Input (lines_by_length.txt)
|
|
|
|
```
|
|
Short
|
|
VeryLongLineWithManyCharacters
|
|
MediumLength
|
|
```
|
|
|
|
### Goal
|
|
|
|
Sort lines by their length.
|
|
|
|
### Solution
|
|
|
|
```bash
|
|
awk '{ print length, $0 | "sort -n"}' lines_by_length.txt | cut -d ' ' -f2-
|
|
```
|
|
|
|
### Output
|
|
|
|
```
|
|
Short
|
|
MediumLength
|
|
VeryLongLineWithManyCharacters
|
|
```
|
|
|