Learning AWK: A Practical Beginner-Friendly Guide

A hands-on tutorial for using AWK to inspect, filter, and process text files from the command line

Bash

AWK

Command Line

Data Science

Text Processing

Tutorial

A runnable, example-driven introduction to AWK for filtering rows, extracting columns, and processing genomics and log files from the terminal.

Author

Nivedita Bhadra

Published

May 1, 2026

Diagram showing a table of rows and columns being processed by awk, extracting a single highlighted field

The core idea behind awk: it reads a file line by line, splits each line into fields, and lets you filter or extract on those fields directly from the terminal.

awk is a text-processing language particularly well-suited to structured text files. It’s commonly used to:

Filter rows based on conditions
Extract and manipulate specific columns
Transform text into a desired format
Perform simple calculations directly from the command line

It operates line by line, automatically splitting each line into fields — which makes it very effective for handling tabular data such as CSV or TSV files. In practical terms, awk combines ideas from a few different tools: conditional filtering similar to a SQL WHERE clause, basic programming logic like Python, and quick calculations often done in spreadsheet software — all directly in the terminal, without loading data into a separate environment.

Every example below is a real, executable code cell — run against small sample files created as we go, so the output you see is genuine awk output, not illustrative text.

AWK Bash Terminal Text Processing

1 Why AWK is Essential

It works directly on very large files (from gigabytes to terabytes)
It does not require loading data into memory
It is fast and efficient for line-by-line processing
It is well-suited for preprocessing data before a machine learning pipeline

2 Basic Syntax

The basic structure of an awk command consists of a pattern and an action:

awk 'pattern { action }' file

pattern — a condition that determines which lines to process (optional)
action — what to do with those lines

If no pattern is given, the action applies to every line. In the simplest case, awk '{print}' file.txt prints each line as-is — equivalent to displaying the full file. This simple structure is the foundation for everything else awk can do.

%%bash
cat > file.txt << 'EOF'
Alice 25
Bob 30
Charlie 28
EOF

awk '{print}' file.txt

Alice 25
Bob 30
Charlie 28

3 Understanding Fields and Columns

By default, awk treats each line as a record and splits it into fields based on whitespace. Each field is accessed with a positional variable — $1 for the first column, $2 for the second, and $0 for the entire line.

Using the same file.txt from above, extracting just the names (first column):

%%bash
awk '{print $1}' file.txt

Alice
Bob
Charlie

4 Setting Delimiters

Real-world data isn’t always whitespace-separated — commas in CSV files or tabs in TSV files are common. Tell awk how to split each line with the -F option: -F "," for comma-separated values, -F "\t" for tab-separated files (very common in genetics and bioinformatics).

4.1 CSV Example

%%bash
cat > data.csv << 'EOF'
name,age,score
Alice,25,88
Bob,30,64
Charlie,28,73
EOF

awk -F "," '{print $1, $3}' data.csv

name score
Alice 88
Bob 64
Charlie 73

5 Row Filtering

One of the most useful features of awk is filtering rows based on conditions — numeric comparisons, exact text matches, combined logical conditions, or regular expressions — all without modifying the original file.

5.1 Filter Numeric Values

%%bash
awk '$2 > 25' file.txt

Bob 30
Charlie 28

5.2 Filter Exact Match

%%bash
awk '$1 == "Alice"' file.txt

Alice 25

5.3 Multiple Conditions

%%bash
awk '$2 > 25 && $2 < 35' file.txt

Bob 30
Charlie 28

5.4 String Matching (Regex)

%%bash
awk '$1 ~ /A/' file.txt

Alice 25

6 Built-in Variables

awk provides several built-in variables — NR (record/line number), NF (number of fields), and $0 (the full line) are the most commonly used.

%%bash
awk '{print NR, $0}' file.txt

1 Alice 25
2 Bob 30
3 Charlie 28

7 Calculations with AWK

Since awk processes data line by line, it’s straightforward to apply arithmetic to specific columns — multiplying values, summing a column, or computing an average.

7.1 Multiply a Column

%%bash
awk '{print $1, $2*2}' file.txt

Alice 50
Bob 60
Charlie 56

7.2 Sum a Column

%%bash
awk '{sum += $2} END {print sum}' file.txt

7.3 Average

%%bash
awk '{sum += $2} END {print sum/NR}' file.txt

27.6667

8 BEGIN and END Blocks

awk provides two special blocks that run before and after processing the input: BEGIN runs once before any lines are read (useful for initializing variables or printing headers), and END runs once after all lines have been processed (useful for summarizing results).

%%bash
awk '
BEGIN {print "Start"}
{print $1}
END {print "End"}
' file.txt

Start
Alice
Bob
Charlie
End

9 Formatting Output with `printf`

While print is convenient, printf gives more control over formatting — similar to printf in C or Python — useful for clean, structured output.

%%bash
awk '{printf "%s is %d years old\n", $1, $2}' file.txt

Alice is 25 years old
Bob is 30 years old
Charlie is 28 years old

10 Working with Text (String Functions)

awk has built-in functions for cleaning and transforming text — gsub for find-and-replace, toupper/tolower for case conversion — all applied line by line.

10.1 Replace Text

%%bash
awk '{gsub("Alice", "Alicia"); print}' file.txt

Alicia 25
Bob 30
Charlie 28

10.2 Convert Case

%%bash
awk '{print toupper($1), tolower($1)}' file.txt

ALICE alice
BOB bob
CHARLIE charlie

11 Arrays in AWK

awk supports associative arrays for counting or grouping data dynamically, without predefining their size. Below, each unique name in the first column becomes a key, incremented as awk processes each line, with results printed in the END block.

%%bash
cat > names.txt << 'EOF'
Alice
Bob
Alice
Charlie
Bob
Alice
EOF

awk '{count[$1]++} END {for (name in count) print name, count[name]}' names.txt

Alice 3
Charlie 1
Bob 2

12 More Pattern Matching

Beyond column-based filtering, awk can match text patterns directly — lines starting with a character, ending with a pattern, or containing digits.

%%bash
cat > mixed.txt << 'EOF'
Apple 10
Banana 5
Cherry7
XYZ
EOF

echo "--- Starts with A ---"
awk '/^A/' mixed.txt

echo "--- Contains a digit ---"
awk '/[0-9]/' mixed.txt

--- Starts with A ---
Apple 10
--- Contains a digit ---
Apple 10
Banana 5
Cherry7

13 Multi-Column Logic

A single condition is often not enough. awk combines conditions with && (“and”) and || (“or”) for more precise filtering — for example, selecting rows where one column exceeds a threshold while another stays below a different value.

%%bash
cat > multi.txt << 'EOF'
A 60 90
B 40 110
C 70 95
EOF

awk '$2 > 50 && $3 < 100' multi.txt

A 60 90
C 70 95

14 Real Use Case: Log Files

Since logs are usually plain text with a repeated structure, awk is well suited for extracting specific entries, identifying important messages, and counting occurrences of events like errors or warnings — useful for debugging and monitoring pipeline runs.

%%bash
cat > logfile.txt << 'EOF'
INFO 10:01 job started
ERROR 10:02 missing input file
INFO 10:03 retrying
ERROR 10:04 timeout on node 3
INFO 10:05 job completed
EOF

echo "--- ERROR lines ---"
awk '/ERROR/' logfile.txt

echo "--- Count of ERROR lines ---"
awk '/ERROR/ {count++} END {print count}' logfile.txt

--- ERROR lines ---
ERROR 10:02 missing input file
ERROR 10:04 timeout on node 3
--- Count of ERROR lines ---
2

15 Real Use Case: Genetics Data

awk is especially useful in genetics and bioinformatics, where many file formats (VCF, annotation tables, summary statistics) are plain text and tab-separated. Common tasks include selecting chromosome positions, filtering variants by quality, or extracting values packed into a single INFO field.

Below is a small synthetic VCF-style file (not real genomic data) with a standard INFO field containing allele frequency (AF=), to demonstrate the pattern.

%%bash
printf 'chr1\t12345\trs123\tA\tG\t.\tPASS\tAF=0.45\n' > toy.vcf
printf 'chr1\t12400\trs124\tC\tT\t.\tPASS\tAF=0.003\n' >> toy.vcf
printf 'chr2\t500\trs125\tG\tA\t.\tFAIL\tAF=0.12\n' >> toy.vcf

echo "--- Extract key columns (chrom, pos, id) ---"
awk -F "\t" '{print $1, $2, $3}' toy.vcf

echo "--- Filter PASS variants only ---"
awk -F "\t" '$7 == "PASS"' toy.vcf

--- Extract key columns (chrom, pos, id) ---
chr1 12345 rs123
chr1 12400 rs124
chr2 500 rs125
--- Filter PASS variants only ---
chr1    12345   rs123   A   G   .   PASS    AF=0.45
chr1    12400   rs124   C   T   .   PASS    AF=0.003

15.1 Parsing the `INFO` Field

The INFO field often packs multiple key-value pairs together, separated by semicolons. awk can split this field further to pull out a specific value such as allele frequency.

%%bash
awk -F "\t" '{
  split($8, info, ";")
  for (i in info) {
    if (info[i] ~ /^AF=/) {
      split(info[i], a, "=")
      print $1, $2, a[2]
    }
  }
}' toy.vcf

chr1 12345 0.45
chr1 12400 0.003
chr2 500 0.12

15.2 Filtering Rare Variants

Once allele frequency is extracted, a numeric filter narrows the file down to rare variants (e.g. AF < 0.01) — a common first step before downstream analysis.

%%bash
awk -F "\t" '{
  split($8, info, ";")
  for (i in info) {
    if (info[i] ~ /^AF=/) {
      split(info[i], a, "=")
      if (a[2] < 0.01) print $0
    }
  }
}' toy.vcf

chr1    12400   rs124   C   T   .   PASS    AF=0.003

16 Combining AWK with Other Commands

One of awk’s strengths is how easily it pipes together with other command-line tools — grep to pre-filter lines before handing them to awk, or sort/uniq to summarize awk’s output.

16.1 `grep` + `awk`

%%bash
grep "ERROR" logfile.txt | awk '{print $1, $2}'

ERROR 10:02
ERROR 10:04

16.2 Sort + Count

%%bash
awk '{print $1}' names.txt | sort | uniq -c | sort -nr

      3 Alice
      2 Bob
      1 Charlie

17 Common Mistakes

$0 is the full line; $1, $2, … are individual fields.
Forgetting the delimiter is the most common mistake with non-whitespace-separated files:

# Wrong  — no delimiter set, so a CSV line is treated as one field
awk '{print $1}' data.csv

# Correct — tell awk the field separator explicitly
awk -F "," '{print $1}' data.csv

Always use straight quotes (' and ") in awk scripts — curly/smart quotes (from copy-pasting out of a word processor or slideshow) will cause a syntax error.

18 Quick Reference

Task	Command
Print entire file	`awk '{print}' file.txt`
Print a column	`awk '{print $1}' file.txt`
Set delimiter	`awk -F "," '{print $1}' file.csv`
Filter numeric	`awk '$2 > 25' file.txt`
Filter exact match	`awk '$1 == "Alice"' file.txt`
Multiple conditions	`awk '$2 > 25 && $2 < 35' file.txt`
Regex match	`awk '$1 ~ /A/' file.txt`
Line number / field count	`awk '{print NR, NF}' file.txt`
Sum a column	`awk '{sum += $2} END {print sum}' file.txt`
BEGIN / END	`awk 'BEGIN{...} {...} END{...}' file.txt`
Formatted output	`awk '{printf "%s: %d\n", $1, $2}' file.txt`
Replace text	`awk '{gsub("A","B"); print}' file.txt`
Count frequency	`awk '{count[$1]++} END{for (k in count) print k, count[k]}' file.txt`
Combine with grep	`grep "ERROR" file.txt \\| awk '{print $1}'`
Sort + count	`awk '{print $1}' file.txt \\| sort \\| uniq -c \\| sort -nr`

19 Next Steps

From here, worth exploring: multi-file awk scripts stored in a .awk file and run with awk -f script.awk, using awk inside larger shell pipelines for QC automation, and comparing awk against pandas/data.table for cases where the data does need to fit in memory.

awk -f scripts Shell Pipelines grep sort/uniq