UNIX and LINUX part 16

The following example matches any line containing the word Apples.
nawk '/Apples/' fruits

Escape Sequences
The following sequences of characters are referred to as escape sequences. Escape sequences specify a notation for characters that are not easily produced from your keyboard and/or cause problems when displayed to your screen. The nawk and echo commands use escape sequences. Although only nawk uses the escape sequences in regular expressions.
Sequence
Description
\b Matches a backspace.
\f Matches a form feed.
\n Matches a new-line.
\r Matches a carriage return.
\t Matches a tab.
\0ddd Matches the ASCII character for the given octal ASCII code.
\c Matches the literal character c. Use \ to generate a \.
Expression Patterns
Expression patterns perform number or string comparisons. Nawk provides eight comparison operators. Six of these operators are relational operators used to perform number or string comparison. The other two are string operators used to perform string comparisons.
String Operators may be used to compare a regular expression pattern to a specific field or the entire input line. The following two operators are supported:
Operator
Function
/RE/ Matches if the current line contains the specified regular expression. Same as the $0 ~ /RE/ command.
expr ~ RE Matches if the string value of expr contains the regular expression RE. For example,
nawk '$1 ~ /[Aa]pples/' fruits

matches lines where field 1 contains "Apples" or "apples."
expr !~ RE Matches if the string value of expr does not contain the regular expression expr. For example,
nawk '$3 !~ /[Aa]pples/' fruits

matches lines where field 1 does not contain the word "Apples" or "apples."
Relational Operators Relational operators compare strings and numbers. The following six operators are used to compare values.
Operator
Function
lv > rv True if lv is greater than rv. For example,
nawk 'NF > 3' fruits

prints each line containing more than three fields.
lv >= rv True if lv is greater than or equal to rv. For example,
nawk ' $1 >= "M" ' fruits

prints each line where the first field begins with a character that has a higher ASCII code than capital M. Such as N, O, P, Q, etc.
lv < rv True if lv is less than rv.
lv <= rv True if lv is less than or equal to rv.
lv == rv True if lv is equal to rv. For instance,
nawk '$1 == "Oranges" ' fruits

prints each line where the first field equals the string "Oranges."
lv != rv True if lv is not equal to rv.
Compound Patterns
A compound pattern is an expression consisting of multiple expression patterns combined with Boolean operators. The Boolean operators are || (OR), && (AND), ! (NOT), and use of parentheses for grouping the operators. If the result of a compound pattern is true, the current input line is matched and the associated action is executed.
Boolean Operators The Boolean operators can be used to combine regular and relational expressions. The following table describes the Boolean operators.
Operator
Function
! Negation. True if expression is NOT true. For example,
nawk '! /Apples/' fruits

prints each line that does not contain the string "Apples."
|| Logical OR. True if either expression is true. For instance,
nawk '$3 > "3,000,000" || $1 == "FRUIT"' fruits

prints the header line and each line with bushel counts of more than "3,000,000." Note 3,000,000 is a string, not a number, because of the commas.
&& Logical AND. True if both expressions are true. For example,
nawk '$3 > "3,000,000" && NF == 5' fruits

prints the header line and each line containing exactly three fields.
( ) Perform comparisons enclosed together(for grouping). For instance,
nawk '( $1 == "Apples" || $1 == "Oranges" ) &&
( NR == 4 )' fruits

prints lines where the first field is either "Apples" or "Oranges" and there are exactly four fields.
Range Patterns
Range patterns contain two patterns separated by a comma (,). A range pattern matches each line from the occurrence of the first pattern to the occurrence of the second pattern. Both patterns may appear on the same line, thus only the one line is matched. If no input line matches the first pattern, no lines are selected. If no input line matches the second pattern, all lines from the first pattern to the end of the file are selected. For example, the following command selects all of the lines from one that contains Apples to one that contains Cherries.
nawk '/Apples/,/Cherries/' fruits

ACTIONS
The action part of an nawk program performs specified actions when a pattern is matched. If no action is specified the default print action is used. The default is the same as specifying { print }, which is equivalent to { print $0 }.
Comments
The nawk command ignores any characters on a line following a # sign. Thus you can document your nawk programs using the number sign.
Constants
Two types of constants are used by nawk, numeric and string. String constants are created by enclosing a sequence of characters in double or single quote marks. For example,
$1 == "FRUIT"

UNIX and LINUX part 15

If an example is given for syntax reasons only, the filename infile is used.
PROGRAMS
The nawk command interprets a set of commands known as the nawk program. The program can be specified as an argument on the shell command line or stored in a file. In either case the program consists of one or more pattern-action statements. The general form of the pattern-action statement is:
pattern { action }

For example,
nawk 'NF < 5 { print $1 }' fruits

prints the first field of a line if less than five fields exist on the line. Fields are space or tab separated words. The built-in variable NF contains the number of fields in the current line.
Nawk reads each line of input automatically. If the pattern matches the input line (record) in part or whole, then the following action is performed. If no pattern is specified, then the action is performed for every input record. The action may consist of multiple programming statements discussed later. The action must be enclosed by '{}' for nawk to function properly. The general format is:
{ action }

For example,
nawk '{ print }' fruits

prints every line of the file infile. Thus you wrote a very simple version of the cat command. This is the default pattern-action statement of nawk.
If no action is specified, each matching input record is displayed on the standard output. The general format is:
pattern

For example,
nawk 'NF == 4' fruits

prints each line consisting of four fields.
WRITING PROGRAMS
It is very common to write small nawk programs in-line; that is, on the command line or even spread over a couple of lines. For example,
nawk 'BEGIN { print "Annual Fruit Report" }
{ print }
END { print "More fruit to Grow" }' fruits

prints the banner line Annual Fruit Report, every line of the fruits file, and a trailer line More fruit to Grow. This line can be typed from your terminal or placed in a shell script.
NOTE:
In-line nawk programs are limited in length to the command line buffer of your system. This buffer varies; some systems are set to 512 characters of information while others are set to 10240 characters. The general rule of thumb for the length of in-line programs is that they should never exceed 2000 characters (approximately one screenful of characters).
The same nawk program written in a file format would be invoked by the command line:
nawk -f fruitprog fruits

The fruitprog file would contain the following lines.
BEGIN { print "Annual Fruit Report" }
{ print }
END { print "More fruit to Grow" }

Notice the single quotes are removed; the nawk command itself is not needed for the input files.
PATTERNS
Patterns are used to select lines from the input. If a pattern matches a string on an input line, the associated action is performed. It is possible to write an nawk program that does not have a pattern, such as
nawk '{ print $1 }' fruits

The default pattern selects all lines from the input.
NOTE:
The BEGIN and END patterns both require an action.
The BEGIN Pattern
The BEGIN pattern does not match any input. Instead, the action associated with the BEGIN pattern is executed before any input is read by nawk. This allows for initialization of variables, the printing of headers, and other coding that needs to be done before the first line of input is read by nawk. The general format is:
nawk 'BEGIN { initialization code; headers; etc. }' infile

The following example prints a header line.
nawk 'BEGIN { print "Fruit Shipped From Primary State" }' fruits

The END pattern
The END pattern does not match any input. Instead, the action associated with the END pattern is executed after all of the input has been read by nawk. The general format is:
nawk 'END { wrap up code }' infile

The following example prints the total number of lines read by nawk.
nawk 'END { print NR }' infile

The built-in variable NR contains the Number of Records read.
Regular Expression Patterns
The following table contains each regular expression and the task that it performs when used inside a pattern. Regular Expressions are often referred to as REs in UNIX terminology. Thus we use the RE notation for uniformity and briefness.
Metacharacters
Metacharacters are the special characters used in regular expression patterns that have special meanings. Metacharacters are often referred to as special or magic characters.
Regular expression patterns must be enclosed in slashes. For example, to match any records (lines) containing the string cpu you would use /cpu/ as the pattern. The string inside the slashes may be any valid combination of the following list.
Special RE
Description
c Matches the character c if c is not a special regular expression character.
\ Escapes the meaning of a metacharacter.
^ Matches the beginning of the line.
$ Matches the end of the line.
. Matches any single character other than the new-line.
[class] A character class. Matches any one character in the class.
[c1-c2] Matches any one of the ASCII characters in the range defined within the brackets.
[^class] Does NOT match any of the ASCII characters listed within the brackets. Ranges may be specified.
| Alternation of regular expressions. Matches either one or the other of the regular expressions provided.
(!) Concatenation operation. Normally the parentheses are omitted. Allows for control of precedence in the interpretation of the regular expressions specified.
RE* Matches zero or more occurrences of the preceding regular expression.
RE+ Matches one or more occurrences of the preceding regular expression.
RE? Matches zero or one occurrence of the preceding regular expression.
// The null RE refers to the last RE defined.

UNIX and LINUX part 14

Module 6
awk/nawk
DESCRIPTION
The external awk program is an interpretive programming language. Interpretive means awk executes the program as it reads the code, unlike the C programming language which must be compiled before it can be executed. Awk is somewhat of a cross between the egrep command and the C programming language. Like egrep it searches for a regular expression in the input. Like C it can be programmed to perform almost any data manipulation processing.
NOTE:
The new version of awk is named nawk (new awk). Some vendors provide only the old version of awk, some provide nawk under the name of awk, while others provide awk and nawk. System V Release 4.0 provides awk and nawk. It is advisable to use nawk unless of course you have to consider portability to systems that do not support nawk. This module describes the nawk program.
Nawk lends itself well to generating reports, transforming data, retrieving information, and validating data. It is designed to handle data in rows and columns, but it is also useful in locating specific data within free formatted data streams. Its features include:
* Structures, operators, and syntax resembling the C programming language
* Regular expression pattern comparison
* Multisource input. It can read from files, pipes, and the keyboard in the same program. It can be interactive.
* Multidestination output. It can write to multiple files, pipes, and the terminal.
* Formatted output using the C printf function
* Automatically declares typeless variables (a variable may be string or number)
* Automatically parses input lines into fields (columns) and records (lines)
COMMAND FORMAT
Following is the general format of the nawk command.
nawk [ -Fs ] program [ VAR=VAL ] [ file_list ] [ - ]
nawk [ -Fs ] -f prog_file [ VAR=VAL ] [ file_list ] [ - ]

Options
The following list describes the options and their arguments that may be used to control how nawk functions.
-Fs Set field separator to regular expression s; default is white space (spaces and/or tabs).
-f prog_file The -f informs nawk that the next argument is the name of the file containing the nawk commands. The file prog_file contains the nawk program (commands). This allows you to write large nawk programs without exceeding the shell's or nawk's input buffer limit on the command line. The input buffer on most systems is between 512 and 10240 bytes. To find out your system's limit, type grep "ARG_MAX" /usr/include/limits.h and press Return. On BSD systems, type egrep "NCARGS" /usr/include/sys/param.h and press Return.
Arguments
The following list describes the arguments that may be passed to the nawk command.
'program' A single argument containing the nawk program (commands). The argument should be enclosed in single quotes to keep the shell from interpreting it.
VAR=VAL Command line variables. These variables are passed to the nawk program for internal use.
file_list One or more files containing data to be scanned and processed by nawk.
- The standard input is read as input. This may be intermixed with files in the file_list.
If no files are specified for input, nawk reads from the standard input.
DATA FILE
The examples throughout this module assume you have a file named fruits in your HOME directory. The file should contain the following data.
FRUIT STATE BUSHELS PRICE STATUS
Apples Washington 6,700,000 10.59
Oranges Florida 5,900,000 11.69
Peaches Texas 3,600,000 13.79 Incomplete
Pears California 3,100,000 12.89
Raisins California 2,300,000 15.00
Cherries Missouri 2,100,000 17.49
Pineapple Hawaii 3,900,000 12.99
Coconuts Hawaii 2,600,000 14.39 Incomplete

UNIX and LINUX part 13

Module 5
apropos (BSD)
DESCRIPTION
The external apropos command searches for keywords in the header lines of all UNIX Reference Manual entries. There is one manual entry for each UNIX command distributed with your system. These entries are often referred to as "man pages." The header line for each command entry is a brief one-line description of the command.
If you need to use a command but do not know its name, you can use apropos to search for the command by specifying keywords. You specify keywords on the apropos command line. Each keyword is searched for separately in the header lines of all manual entries. If a keyword is located in the title (header line) of a man page, the command's title is displayed. The search ignores the case of letters.
A match occurs even if the keyword is a part of a word in the title. For example, if you specify
apropos command

all titles with the word "command" and "commands" would be displayed.
COMMAND FORMAT
Following is the general format of the apropos command.
apropos keyword [ ... ]

Arguments
The following list describes the arguments that may be passed to the apropos command.
keyword A word you want to search for in the title of all the manual sections.
FURTHER DISCUSSION
The apropos command is the same as the man -k keyword command. If apropos displays a line for a keyword that begins with name(section) ..., you can display the entire manual page by entering the command:
man section name

To display the title line of each manual page that contains the keyword "editor" you specify a apropos command like:
apropos editor

which should return the following output:
a,out (5) - assembler and link editor output format
ed, red (1) - basic line editor
ex, edit (1) - line editor
ld (1) - link editor
sed (1) - stream text editor
vi, view (1) - visual display editor based on ex(1)

The output from your system may vary depending on which utilities have been implemented and installed on your particular system.
DIAGNOSTICS AND BUGS
If apropos fails to work, it is possible the /usr/lib/whatis file has not been created. The system administrator creates this file by changing directories to /usr/lib and executing mkwhatis or makewhatis, depending on your system's command name.
RELATED COMMANDS
Refer to the man and whatis commands described in modules 87 and 155.
RELATED FILES
The apropos command looks in a file containing the titles for all of the manual documents, usually the /usr/lib/whatis.
APPLICATIONS
The apropos command provides a way for you to search for a command to perform a certain function based on keywords. For example, if you needed to display the contents of a file, you could use apropos and specify several keywords that referred to displaying contents of files. For example, you might try the command,
apropos print display show type file

The trick to making apropos work is being flexible in your choice of keywords. Flexible means you should try to remember as many synonyms for a word as you can.
There are trade-offs in how you use keywords. If you use too many or too broad of a word, such as file, you may get pages of output. On the other hand, if you use a very selective set of keywords, you may get no output. The basic advice is to try a few different keywords to get the general idea of how the titles of the man pages are written. Unfortunately, experience is still the best teacher, especially when UNIX man pages are involved.
TYPICAL OPERATION
In this activity you use the apropos command to locate commands related to text. Begin at the shell prompt.
1. Type apropos text and press Return. Depending on your system, the output should resemble the following:
cj> apropos text
nawk(1) - text pattern scanning and processing language
csplit(1) - context split
ed, red (1) - text editor
edit(1) - text editor (variant of ex for casual users)
end, etext, edata(3C) - last locations in program
ex(1) - text editor
fspec(4) - format specification in text files
lex(1) - generate programs for lexical analysis of text
neqn(1) - format mathematical text for nroff
nroff(1) - format text
plock(2) - lock process, text, or data in memory
sed(1) - stream text editor

UNIX and LINUX part 12

Tracked aliases
Tracked aliases are not true aliases. A tracked alias is the actual command name and the full pathname used to access an existing command. It is called tracked alias because the path is aliased for the command name and the new alias tracks directly to the command. This keeps the shell from having to search the entire PATH for a command, thus providing faster command execution.
You can turn command tracking on by using the set command. Type
cj> set -o trackall

to inform the shell to track all commands issued to it. For example, the following sequence of commands illustrates the automatic tracking of commands.
cj> set -o trackall
cj> alias ls # list alias of ls command
ls alias not found # no alias exists for ls
cj> ls # execute the ls command
...
cj> alias ls # list alias of ls command
ls=/bin/ls # tracked alias has been set to full path

You can track individual commands by using the -t option. For example,
cj> alias -t ls

would cause the same results as the above steps. But the shell would not automatically set every command you issue to a tracked alias.
RETURN CODES
Alias returns a zero (true) unless a name is given that has not been set to a string.
APPLICATIONS
The alias command is used to create shorthand versions of longer commands. It is very useful for hard-to-remember commands and for those commands you use repeatedly throughout the day. Remember to place your aliases in your .profile or your ENV file (.kshrc) so they are set each time you log in.
The unalias command is used to remove previously set aliases.
TYPICAL OPERATION
In this activity you use the alias command to create various aliases. Begin at the shell prompt.
1. Type alias "I= -F" and press Return.
C Shell
Type alias I "Is -F" and press Return.
2. Now try your new command by typing I and pressing Return. Notice the output of your l command is a multicolumn list with the executable (*) and directory (/) files being flagged.
3. List all aliases you have set by typing alias and pressing Return.
In this activity you use the unalias command to remove the previously defined l alias command. Begin at the shell prompt.
1. Remove the l alias by typing unalias I and pressing Return.
2. Type alias I and press Return to list out your l alias. Notice the l alias is no longer defined.
3. Turn to Module 64 to continue the learning sequence.

UNIX and LINUX part 1

CORRECTING TYPING MISTAKES
The UNIX System reads each individual character and temporarily saves or buffers them until you press Return or Line Feed. It then interprets the entire line and performs the requested operation.
The UNIX System performs full duplex input/output (I/O) with your terminal. This means you can type characters in to UNIX while UNIX is displaying information out to your terminal. This will result in a messy, hard-to-read screen, but the data will flow both directions, and the input flow will be interpreted in the correct sequence. You may also type ahead of the UNIX System's ability to read the characters, referred to as UNIX's read-ahead feature. The number of read-ahead characters you may type is limited but usually it is more than you will ever need (256 to 4096).
Default Correction Keys
The @(^U-csh), #(DEL-csh) and \ characters have special meaning to UNIX. The @(^U-csh) character deletes the current input line and is referred to as the line kill character. The #(DEL-csh) erases the last character and is referred to as the erase or backspace character. Multiple #(DEL-csh) erases back to the beginning of the current line. The \ character escapes the meaning of any special character. Therefore \# prints a # and does not perform a backspace.
Resetting Correction Keys
The @(^U-csh) and #(DEL-csh) keys can easily be reset. The standard reset values are Ctrl-X (while pressing and holding the Ctrl key, press the X key) used in place of @, and Ctrl-H (Backspace) used instead of #. Most csh users (who are usually on BSD systems) do not reset the ^U and DEL keys.
1. Display the current settings that UNIX interprets as special keys on your terminal by typing stty and pressing Return. Notice the erase and kill default settings. Depending on your system administrator, your settings may differ. The output format for stty may differ if you are on a BSD system.
cj> stty
speed 38400 baud; -parity cread
erase = #; kill = @; swtch = ^';
-inpck icrnl -ixon onlcr tab3
echo echoe echok

BSD (Berkeley)
cj> stty everything
2. Redefine the erase and kill characters by typing stty erase "^h" kill "^x" and pressing Return. The "^" (caret) is typed as Shift-6; do not type Ctrl-H. Notice stty does not display a response.
3. Display the new settings by typing stty and pressing Return.
cj> stty
speed 38400 baud; -parity cread
erase = ^h; kill = ^x; swtch = ^';
-inpck icrnl -ixon onlcr tab3
echo echoe echok

4. Type echo Ti#his is a #n example. at your shell prompt and press Return. Use the appropriate erase key (Ctrl-H/Backspace) in place of the # keys. Notice that when you press your Backspace key the cursor moves back one space.
5. Type echo Another eamxpl@echo Another example and press Return. Use the appropriate line kill key (Ctrl-X) in place of the @ key. Notice that when you press your kill key the cursor jumps to the beginning of the next line; the old line is ignored and a new line of input is read.
BSD (Berkeley)
The new BSD device driver will erase the current line and move your cursor to the beginning of the current line.
6. Type cat/etc/group and press Return. Be ready to immediately press Ctrl-S to stop the output. The Ctrl-S does not appear on your terminal. To resume scrolling of the output press Ctrl-Q.
7. If you have I/O problems with your terminal, check the setting with stty and refer to the stty command (Module 126) for valid parameters. Don't hesitate to request help from a knowledgable source about your terminal and the tty device driver. This is usually a difficult part of UNIX to get a handle on.
KNOWING WHICH UNIX YOU ARE USING
To know which type or version of UNIX you are using, use the uname command. Type uname and press Return. If the name of your system is displayed, you are probably on System V UNIX. If the shell returns a message like "uname: command not found," you are probably on a BSD system.
Another check to perform is to type ls/usr/ucb and press Return. If a listing is returned, you are probably on a BSD-type system. If a message like "/usr/ucb not found" is returned, you are probably on a System V system.
These are not guaranteed checks but they are fairly reliable. If both return info then you are on a hybrid SV and BSD system.
KNOWING WHICH SHELL YOU ARE USING
To know which shell you are using, use the ps command.
On System V type ps and press Return.
BSD (Berkeley)
On BSD type ps -u and press Return.
The right-hand column contains the names of the programs currently executing. If one is csh, you are using the C shell; sh is the Bourne shell, and ksh is the Korn shell.
Another alternative is to type echo $0 and press Return. If you are using the sh or ksh, the corresponding string is displayed on your terminal. If you are using the csh, nothing is displayed

A SAMPLE SESSION WITH UNIX USER COMMANDS

DESCRIPTION
This module provides basic information about getting started using UNIX. It then leads you on a "hands-on" tour of the most commonly used UNIX commands. Because this module is intended to get you familiar with UNIX, it does not provide in-depth explanations. If you wish to know more about a command while you are using it, refer to the module for the given command. The third module provides additional information about features of the UNIX operating system.
The following is a brief outline of information contained in this module. The information discussed in the first four sections is usually set by the system administrator when your login account is created.
* Terminal setup
* Communications between UNIX and your terminal
* Logging in to the UNIX System
* Correcting typing mistakes
* Setting/changing your password
* Knowing which version of UNIX you are using
* Knowing which shell you are using
* UNIX environment
* Executing commands
* Common commands
* Text editing with vi/ex
* Communicating with others
* Logging out of the UNIX System
BEFORE YOU START
Before you can begin using the UNIX operating system you must have a login account. You should request the login account from the system administrator. If you do not have a system administrator, you must create a login account yourself. Check your System Administrator's Guide or Reference Manual for instructions on how to do this.
You will also need a terminal connected to your system. There are several ways for terminals to communicate with the UNIX System: via direct wire, modem, or terminal servers. Your system administrator should connect your terminal to the system.
LOGGING IN TO THE UNIX SYSTEM
Once you have a "login:" prompt on your terminal and a login account, you can log in to the UNIX System. Throughout this book we assume /u1/ts/mylogin is your login (HOME) directory. After you log in and know your HOME directory substitute it in place of /u1/ts/mylogin. We also use the "mylogin" login name; you should replace it with your login name.
1. At the login: prompt type mylogin and press Return.
cj login: mylogin

TIP: Make sure your CAPS key is not activated. If you have a LOGIN: prompt in all capital letters, press Ctrl-D and wait for a new login: prompt to appear, then try to log in to the system.
NOTE:
Your login name must contain one lowercase character. If you do not type at least one lowercase character, UNIX assumes your terminal cannot generate lowercase ASCII characters and treats all characters as uppercase for the remainder of your login session.
2. Type iamuser2 and press Return at the passwd: prompt to log in to the system. Be patient! Depending on your system and its load factor, it could take from a few seconds to a few minutes for the system to respond.
cj login: mylogin
passwd:

NOTE:
Your password will not be printed, or "echoed," as you type it. The UNIX System disables the output so nosy people are not able to read your password.
3. If your login was successful, you should see information displayed on your screen. If your login attempt was unsuccessful, your display will resemble the following,
cj login: mylogin
passwd:
Login incorrect.
login:

NOTE:
If, after a couple of attempts, you cannot get logged in, check your login name and password with the ones your system administrator gave you. If you are still unable to gain access into the system, contact your system administrator, he/she likes to hear from frustrated users!
Messages from the system
The following list is a possible sequence of what may appear on your screen once you type the correct password. You may be required to type information to complete the login sequence. This depends on your local system.
* General information about the local UNIX System is displayed on your terminal from the message of the day file (/etc/motd).
* System displays or lists unread news items.
* System displays or informs you of mail you have received.
* You may be requested to set your terminal type.
* System displays various other information the system administrator has set up.
* A prompt from the UNIX shell (command interpreter) appears. The default prompt is a dollar sign ($).