Filters & Text Manipulation Tools
1.
Printing:
pr-sh
% ls /home/cs476
| pr -h "--cs 476-" -l 20
-2 -n -d | more -20
-h header, -l page length, -2 columns, -n add line numbers, -d double space
2. File Comparison: compare-sh
% paste f1 f2
a
a
b
b
c
d
e
% cmp
f1 f2
f1 f2 differ: char 5, line 3
% comm
f1 f2
a
b
c
d
e
% comm -12 f1
f2
a
b
-12 suppress columns 1
and 2.
% diff f1 f2
3c3,4
< c
---
> d
> e
3.
Head & Tail:
headtail-sh
% cat
f1
a
b
c
% head -n
1 f1
a
% tail -n
1 -f f1
c
/usr/bin/tail options:
-n no of lines, -f force wait -r
reverse lines
Exercise: getting the middle middle-sh
echo "Usage: middle-sh
<head> <tail> <file>"
length=`wc -l < $3`
top=`expr $length - $2`
bottom=`expr $top - $1`
cat $3 | head -n
$top |
tail -n $bottom
4.
Cut & Paste:
cutpaste-sh
% cat fields
Wahab, Hussein
Maly, Kurt
% cut -d, -f2 fields
> F
Hussein
Kurt
-d delimiter,
-f field number
% cut -d, -f1 fields > L
% cat L
Wahab
Maly
% paste -d " " F L > names
% cat names
Hussein Wahab
Kurt Maly
% cat names emails
Hussein Wahab
Kurt Maly
wahab@cs.odu.edu
maly@cs.odu.edu
% paste
names
emails
Hussein Wahab wahab@cs.odu.edu
Kurt Maly maly@cs.odu.edu
% cat info
Hussein Wahab
wahab@cs.odu.edu
x4512
Kurt Maly
maly@cs.odu.edu
x3915
% paste
-s -d":;\n"
info > info2
-s concatenate separate lines, -d lines
delimiters
% cat info2
Hussein Wahab:wahab@cs.odu.edu;x4512
Kurt Maly:maly@cs.odu.edu;x3915
5.
Sort:
sort-sh
% cat sortdata
Wahab,Hussein
Maly,Kurt
Wahab,Hussein
Maly,Kurt
% sort
sortdata
Maly,Kurt
Maly,Kurt
Wahab,Hussein
Wahab,Hussein
% sort
-u sortdata
Maly,Kurt
Wahab,Hussein
% sort
-t, -k 2 sortdata
Wahab,Hussein
Wahab,Hussein
Maly,Kurt
Maly,Kurt
% paste f1 f2
a
a
b b
c d
e
% sort -u -m f1 f2
a
b
c
d
e
-u unique, -t field separtator, -k field number –m merge
6. Uniq: uniq-sh
% cat grades
1: A
2: B
3: C
4: D
5: F
6: A
7: A
8: B
9: F
% cut -d: -f2 grades | sort | uniq -c
3 A
2 B
1 C
1 D
2 F
-c count
the occurrence of each value
7. Translate: translate-sh
% cat fields |
tee /dev/tty | tr 'a-z' 'A-Z' | tee /dev/tty
| tr 'A-Z' 'a-z'
Wahab, Hussein
Maly, Kurt
WAHAB, HUSSEIN
MALY, KURT
wahab, hussein
maly, kurt
8. Egrep:
egrep-sh
% ls -l /home/cs476 |
egrep "^d.*[M|m]ail.*"
drwx------ 2
cs476 cs476 512 Oct 22 1999 Mail
drwx------ 2
cs476 cs476 512 Dec 9 2003 mail
drwx------ 2
cs476 cs476 512 Nov 3 1998 nsmail
% ypcat passwd | egrep
"^cs[5-9][0-9]+:"
| cut
-d: -f1 | sort -u | tee /dev/tty |
wc -l
9. sed: sed-sh
% sed
“s/^/000/; s/A/Excellent/; s/B/Very Good/; s/C/Good/; s/D/Pass/;
s/F/Fail/” grades
Note:
You may put the sed
commands in a file, e.g., sedscript,
and use:
% sed –f sedscript grades.
Exercise: getting the middle midsed-sh
echo
"Usage: midsed-sh <head> <tail>
<file>"
length=`wc -l < $3`
bottom=`expr $length - $2 + 1`
sed "1,$1
d; $bottom,$length d" $3
1. Selecting and
printing lines:
ex1-sh
Courses accounts:
·
print login
and name:
% ypcat passwd
| awk
-F: ' /^cs[5-9][0-9]+/
{ print $1, $5
} '
cs656 cs656 Class Account
cs554 ajay's grad Networking class
cs775 CS775 Grader Account
-F Field separator
Faculty accounts:
·
print login and name:
% ypcat passwd
| awk -F: ' $4 == 13
{
print
$1 "--> " $5 } '
Shen --> Stewart Shen
Wahab --> Dr. wahab
·
print all fields except the password field:
% ypcat passwd
| awk -F: ' $4 == 13
{ $2=
"" ; {print} } '
·
print line
number, login and name:
% ypcat passwd
| awk -F:
' $4 == 13
{ print ++count
“: “ $1 "-->
" $5 } '
5: shen --> Stewart Shen
11: Wahab -->
Dr. Wahab
Exercise: getting file name from a path pathtofile-sh
echo $1 | awk -F/ '{print
$NF}'
Example:
% pathtofile-sh /home/wahab/public_html
public_html
ex2-sh:
ypcat passwd | awk -F: -f ex2-awk
ex2-awk:
$4 == 13 { print ++count,
$1 "--> " $5 }
-f file containing instructions
Usage:
% ex2-sh
3. Begin and END: ex3-sh & ex3-awk
ex3-sh:
ypcat
passwd | awk
-F: -f ex3-awk
ex3-awk:
system
("date");
printf " HOME = %s \n",
ENVIRON["HOME"]
$4 == 13 {
print ++count, $1 "--> " $5
}
printf
"Total number is %d\n", count
printf "PATH = %s \n", ENVIRON["PATH"]
Usage:
% ex3-sh
Examples:
ê
Selecting and printing
lines: sel-print-pl
system (“date”);
print ("BEGIN - finding cs accounts
\n");
open ( PW, "ypcat
passwd | ");
while (<PW>) {
if (
/^cs[4-9][0-9]+:/
) {
split (/:/);
print
("$. --> $_[0]
$_[4] \n " );
$count++;
}
print ("END - total: $count
\n");
$. Current line number,
$_[i] Content of the ith field
$_
Content of
entire current line.
Usage:
% sel-print-pl
ê Selecting
fields: pathtofile-pl
$path = $ARGV[0];
$nf = @fields = split (/\//, $path);
print (@fields[$nf-1]) ;
% pathtofile-sh /home/wahab/public_html
public_html
ê Translate
& substitute: tr-pl & substitute-pl
open(INFILE,
"fields");
print("BEGIN - translate \n");
while
(<INFILE>) {
tr
/a-z/A-Z/;
print;
}
% more fields
Wahab, Hussein
Maly, Kurt
% tr-pl
BEGIN - list
WAHAB, HUSSEIN
MALY, KURT
END - list:
#! /usr/bin/perl
open(INFILE, "grades");
print("BEGIN - substitute \n");
% more grades
1: A
2: B
3: C
4: D
5: F
6: A
7: A
8: B
9: F
% substitute-pl
BEGIN - list
0001: Excellet
0002: Very Good
0003: Good
0004: Pass
0005: Fail
0006: Excellet
0007: Excellet
0008: Very Good
0009: Fail
END - list:
ê Grade
count: grade_count-pl
open(INFILE, "grades");
while (<INFILE>) {
chomp();
split (/:/);
$grade = $_[1] ;
$gradelist{$grade}++
;
}
foreach $grade
(sort ( keys %gradelist ) ) {
print (" $grade ---> $gradelist{$grade} \n") ;
}
ê Comperhensive
Example: login-pl
Each user has name and password saved in a file. The program
allows the user 3 times to enter the correct password and then send email to
administrator of the violation.
#!/usr/bin/perl
$ADMIN_EMAIL = cs476 ;
$MAX_TRIALS = 3;
init_words();
print "what is your name? ";
$name = <STDIN>;
chomp($name);
print "Hello, $name!\n";
print "What is the secret word? ";
chomp ($guess = <STDIN>);
while (1) {
if ($words{$name} eq
$guess) {
print "Welcome $name\n";
last;
}
elsif
(++$trial < $MAX_TRIALS){
print "Wrong, try again. What is the
secret word? ";
chomp ($guess = <STDIN>);
}
else {
print
"$name, you tried $MAX_TRIALS times, mail sent to admin.\n";
open (MAIL, " | Mail –s
\“login violation\” $ADMIN_EMAIL");
print
MAIL "bad news:
$name guessed $MAX_TRIALS times\n";
close
MAIL;
last;
}
}
sub init_words
{
$filename = <passwd.file>;
open (WORDSLIST, $filename)|| die
"can't open $filename: $!";
while ($name = <WORDSLIST>) {
chomp ($name);
$word =
<WORDSLIST>;
chomp ($word);
$words{$name} = $word;
}
close WORDSLIST;
}
Usage:
% login-pl