Linux/UNIX Shells and Shell Scripting
David Morgan
U. C. L. A. Extension - X417.31
see syllabus for email address

 



Administrative links

Syllabus

Grade reports

Course outline


Informational links

Remote Unix access with ssh

vi - the Visual Editor

Line termination

System calls

bash man page

Cheat sheet - bash shell

Cheat sheet - awk

BASH programming Introduction HOW-TO

Advanced Bash-Scripting Guide


FALL 2018
Tuesdays 6:30pm-10:00pm
UCLA Extension Lindbrook Center

10920 Lindbrook Drive

This Website (http://uclaextension.dmorgan.us/morgan_david/unex.html) will be used extensively to communicate with you. Announcements, grade reports, and assignments will be posted here. The site can be viewed from an internet-connected browser anywhere. You are responsible for awareness of the information posted here.

Course outline - with approximate weekly topic coverage. All corresponing readings, , homework assignments, and in-class slides I will use are here. You can preview or review the slide lessons at any time or view them while I am presenting them in class.

Syllabus - containing my email address, specification of required textbook, recommendation of other useful books, and some policy information.
-------------------------------------------------------------------------------------------

Homework -
do - the modification to the Advanced Bash Scripting Guide's word frequency script, course outline section 5 -  - due in your "assignments" subdirectory by end-of-day Wednesday  November 14
read - per course outline, section 5, Quigley chapter 5 (sed).
read - per course outline, section 6, Quigley chapter 14 (programming bash)
anticipate/upcoming - assignment as described in course outline section 6, applying sed to Quigley's "datebook" file.  - due on server by end-of-day Wednesday November 21
 (11/13)

Quoting text - a couple reasons to do it

1a) make it a single token

[root@unexgate sed]# cat show-args 
echo hello
echo "1st arg is: $1"
echo "2nd arg is: $2"
echo goodbye
[root@unexgate sed]# 
[root@unexgate sed]# ./show-args aaa bbb ccc
hello
1st arg is: aaa
2nd arg is: bbb
goodbye
[root@unexgate sed]# ./show-args "aaa bbb" ccc
hello
1st arg is: aaa bbb
2nd arg is: ccc
goodbye
[root@unexgate sed]#

1b) make it a single token

[root@unexgate ~]# cat holidays

Thanksgiving day is for turkeys
July fourth is for fireworks
Christmas eve is for gifts

[root@unexgate ~]# cat holidays | grep July fourth 
grep: fourth: No such file or directory
[root@unexgate ~]# 
[root@unexgate ~]# cat holidays | grep "July fourth"
July fourth is for fireworks
[root@unexgate ~]#


2) make sure it'll be a string (even a null one)

[root@unexgate sed]# VAR=something
[root@unexgate sed]#
[root@unexgate sed]# test "$VAR" = "something"; echo $?
0
[root@unexgate sed]# test $VAR = "something"; echo $?
0
[root@unexgate sed]#
[root@unexgate sed]# VAR=
[root@unexgate sed]# test "$VAR" = "something"; echo $?
1
[root@unexgate sed]# test $VAR = "something"; echo $?
-bash: test: =: unary operator expected
2
[root@unexgate sed]#

Many small specialists - according to the 2nd slide we viewed in this course programs in Unix are supposed to "do one thing well." In that vein I noted this:

"One of the great contributions of Unix to the computing world is that process creation is cheap and easy. This encourages the practice of writing small programs that each do a part of a larger job, and then combining them to collaborate on the completion of that task. Because programming complexity grows much faster than linearly with program size, small programs are much easier to write, debug, and understand than large ones." Shell Scripting, Robbins & Beebe, p353

It can be argued, in relative terms, whether process creation is cheap. Famously for example, threads are useful because they are so much cheaper than processes. But compared to historical and competing operating systems Unix's fork/exec is cheap and, to the point, "cheap enough." That removes the barrier to hitching (perhaps pipelining) a team of small programs together in place of a monolithic one. (11/6)

Homework -
do - the arrays for bubble sort programming exercise, course outline section 5 -  - due in your "assignments" subdirectory by end-of-day Wednesday November 7
(10/30)

Grades - published at link entitled "Grade reports" at left. (10/30)

Homework -
finish - the bash version of the "party program" if you have not yet done so. See homework posting below dated 10/15.  - due in your "assignments" subdirectory by end-of-day Sunday October  28
do - the safe-remote-poweroff programming exercise, course outline section 5; we will discuss in class next week, there are different ways to do it, you can tell us about challenges solved and unsolved   - due in your "assignments" subdirectory by end-of-day Tuesday October  30  (10/23)

Grades - published at link entitled "Grade reports" at left. (10/23)

Command line fu - have fu. (just for fun)   (10/23)

$-
a special shell variable. Its contents are several lower and/or uppercase letters. These are the option letters of the set command that are currently "on", those bash options that are in effect. For documentation of these cryptic letters, try "help set | less" and additionally find the section in the main bash man page headed "OPTIONS". (10/23)

Regular expression humor - you think you got problems? (10/19)

Homework
please see the Homework posting below dated 10/9
note that a due date has been formally assigned there for the grep exercise
do - the "party program" exercise at the link in course outline section 2 entitled "comparing shells". You can do it on unexgate. The file you're supposed to obtain is local there. It's /home/public/the-party-program-csh-ksh.zip. It's readable so you could cp it to your own directory. Leave the bash version of the program (the objective of the exercise) in your "assignments" directory.
read forward - take a look at the material indicated in course outline section 4 (10/15)

A comment about interpreter scripts' interpreters and how they handle their file argument

What happens in an interpreter script to the lines after the initial shebang (first), if the shebang is " #!/bin/mv " as opposed to " #!/bin/bash " ?  Suppose the script file's name is myscript, and it is called as:

myscript  argument

Then an exec( ) function call will be constructed that is the same as would be constructed if you gave the following commands:

/bin/mv  myscript  argument

(as opposed to,   /bin/bash  myscript  argument    )

Then, what happens to the lines in the myscript file depends on how the commands (mv and bash) treat their first argument.  mv takes it as the name of a file to rename. In doing so it does not delve into the file's content at all. bash takes it as the name of a file over  whose contents, viewed as lines, it must iterate in a loop. When called, bash is content-conscious and mv is content-oblivious when a file's name shows up as their first argument. That's why mv is not hindered by a file with multiple lines and just renames it same as if the file held just the single shebang line only. (10/15)

a bash "exec" redirection use case -
prepare a file with 10 lines in it for reading:
 
seq 1 10 | xargs -I + echo "line +" > file

read them out:
  while read line;do echo "$line";done < file

read them out, stopping after each line for user go-ahead:
  while read line;do echo "$line"; read -p "Press any key" -n 1;done < file

Does not work. Instead:
  exec 3<file
  while read -u 3 line;do echo "$line"; read -p "Press any key" -n 1;done

Arranges for the 2 consumers of input (the two "read"s) to get it from different places.

from http://wiki.bash-hackers.org/howto/redirection_tutorial 

Another: you want the output of entire portions of a script (but not of the whole script) captured into a file. The script has many individual lines within it that generate output. You know how to put file redirection ( >> ) on each line, but there are too many lines. At the top of the portion(s) whose output you want diverted, divert it. At the bottom, undivert it:

  echo put \"blue suede shoes\" in a file

  exec   9>&1   1> somefile
  echo One for the money
  echo Two for the show
  echo Three to get ready
  echo Now go, cat go.
  exec   1>&9   9>&-

  echo enjoy your song it's in the file

(10/15)

Homework
read - the material indicated in course outline sections 2 and 3
do - the I/O redirection programming exercise, course outline section 3. Please name the script you create "redirection" and put it in the "assignments" subdirectory of your home directory.- due in your "assignments" subdirectory by end-of-day Wednesday October 17 (provided we are able to cover it in class tonight)
do - the grep exercise from the text book, course outline section 4  due in your "assignments" subdirectory by end-of-day Wednesday October 24 (10/9)

Different "echo"s - I distinguished between the built-in, shell-resident code for "echo" and the binary-file-resident one. They are two different things, though they both do largely the same thing. This note appears in the echo man page, which pertains to the binary-file echo:

"NOTE: your shell may have its own version of echo, which usually
supersedes the version described here. Please refer to your shell's
documentation for details about the options it supports."

This echo was written by Brian Fox and Chet Ramey, the ones who wrote bash itself and hence presumably its built-in echo. No doubt therefore they are highly similar. The echo's that may be built in to the korn shell or c shell will be different, and those found as binary executables in other environments may be different too, written by somebody else yet again. echo is one of the commands that is least standardized, hence least portable. Many people suggest avoiding echo and relying on the printf command instead, whose operational behavior and syntax is more uniformly standardized. (10/9)

Warning - the "datebook" file on the textbook's CD may be corrupted. Some editions deliver a file that uses Microsoft line termination convention. That is, they separate lines with hex 0D 0A character pairs, instead of the single character 0A that unix utilities expect.

You need to fix the file, or you will get wrong results when doing the homework. There is a utility for this purpose called dos2unix. You could fix the file by:

 dos2unix  datebook

If you don't have dos2unix, another way to do it is with tr (the "translate" command). Something like:

 cat  datebook  |  tr  -d  "\r"  >  datebook.temp
 mv  -f  datebook.temp  datebook

And a third way uses sed:

sed -i 's/.$//' datebook

which says, replace the character before the end of the line with nothing. The 0A being interpreted as "the end of the line," that character is its preceding 0D. Replacing it with nothing is getting rid of it. The -i option applies the changes to the original file, in-place.

Applying the correction to an uncorrupted file is harmless, it leaves it uncorrupted. So you might, conservatively, do it on your "datebook" file just in case. Alternatively, download it from the link entitled "Practice files" below. It delivers a zip file, and the files inside the zip are good. (10/9)

Clarification about file descriptor representation - A former student and I talked about file descriptors. It led me to realize that my slides and many sources represent the descriptor table in a potentially confusing way (below, uppper left). The student thought that stdin represents what descriptor 0 connects to, just as /somefile represents what descriptor 3 connects to. They are presented in exactly the same way, after all. No, descriptor 0 doesn't connect to stdin. Nothing connects to stdin. There is no stdin to connect to, since it's nothing but a nickname or synonym for 0. As well, stdout and stderr are synonyms for 1 and 2. Whatever you choose to call them, what they connect to must be some material entity, like a device or file. It makes no sense that they would connect to their name! Better, my revised table below upper right. Princeton Professor Rexford's slides (below) used a good representation in that she shows only the descriptors and their devices, omitting names altogether. The names are an unneeded distraction.

(10/9)

The definitive list of regular expression metacharacters - is in your textbook.
 it's on page 69
 it's on page 83
 it's on page 95
 it's on page 101
the only question is, which one is definitive? The only answer is, none. One is vanilla, the next is grep, another is egrep, and also GNU grep. That's grep. Not vi. Not sed. Not perl.

When you talk about regular experssions, you have to ask "whose?" or "as implemented in which product?"  (10/9)

Hotshot sytle or legibility? - it's a debate. I think I would argue that the latter has more value in shell scripting. Here are two functionally equivalent code snippets. The first one is:

grep -q "$TARGET" "$FILE" && echo "$TARGET" is in "$FILE"

Can you read it? Here is the second one:

if grep -q "$TARGET" "$FILE"
then
    echo "$TARGET" is in "$FILE"
fi

Easier or harder to read? (10/9)

Homework
read - the material indicated in course outline section 1. 
anticipate - upcoming actvities
 1 - 2nd part of lab exercise at course outline section 1 link entitled "primitive shells"
 where we compile bash from source code
 2 - exercise at course outline section 2 link entitled "comparing shells" where we
 write functionally equivalent scripts under 3 different shells; this is textbook author's
 "party program" from chapter 2.
(10/2)

Primitive shell capability comparison results matrix -
Here are the results of trying different features in each of the 4 primitive shells tested in the hands-on in-class lab exercise performed in class.
The point is that shells are merely programs, different ones behave somewhat differently, in accord with the features their authors have chosen to program into them. (10/2)

Steven Bourne - originator of the sh, gives a talk about the shell in 2015. More historical than technical. Listen if interested. (10/1)

How to access your individual account on the class Remote Unix system.

Handout - explaining use of the computers in our classroom.

Structure of a unix process - an anchor reference for much of the discussion about operating system and shell. Keep it in mind throughout the course.

 

Patches to bash source code - applying these will be a step in the bash compilation exercise we will complete next week. There were 11, but now there are 17 patches. Examining release dates, numbers 12-17 are all from September 24, 2014, or later. That's the day information about the shellshock bash bug went public. These latter patches no doubt address shellshock. Bash users on older distribution versions may not have the option of an online update using yum or apt-get, as the maintainers of the repositories have end-of-life'd those versions. For those folks, applying these patches and compiling bash afresh will be their only fix. It sounds complicated but you will do it next week. The procedure is also well documented in a blog post by Steve Jenkins (see the link, in section 1 of the course outline, entitled "bash from source to address the shellshock bash bug"). (9/25)

Practice files from Quigley chapter 3. (9/25)

Someday you too can be heroic.

What is a shell? - "A Unix shell is both a command interpreter and a programming language. As a command interpreter, the shell provides the user interface to the rich set of GNU utilities. The programming language features allow these utilities to be combined."
  --Bash Reference Manual  
Here is a tutorial website that covers the basic functionality of a shell nicely.  (9/25)

Detailed evolution of bash and bash features version-by-version, among other information in this FAQ. (9/25)

bash maintainer Chet Ramey interviewed.
Q: "What prompted the language's name: why was a pun created on the Bourne Shell?"
A: "The FSF has a penchant for puns, and this one seemed appropriate, I suppose. The name predates my involvement."
bash got started in the late '80s. Ramey became primary maintainer in 1990.

``The lyf so short, the craft so long to lerne.'' - Chaucer
 (from Ramey's email signature)

(9/25)

bash authors queried about shellshock (9/25)

Cheat sheet/reference guide - for the bash shell, compiled by a student from the bash man page (help file). (9/25)

Exercise - about diff and patch. (9/25)

GNU bash homepage (9/25)

Request - please don't change the passwords on the "root" or "student" accounts of the classroom workstations.

Using ssh (secure shell). ssh is an important tool you will use for interacting with remote computers. For that you will need an ssh client. There are a number of ssh client alternatives.

Running linux at home.
Bootable Fedora 23 on USB for you to burn - I made a suitable, persistent image recently. If you bring an 8GB or larger USB flash drive to class (empty of anything you care about) we can burn my image to your drive for you to boot from in the future.

 

 

Bibliography - of shell scripting information resources
These are not random. They are curated. All of them are good. I particularly recommend and call your attention to Cooper, Robins/Beebe, and Friedl.

Advanced Bash-Scripting Guide, An in-depth exploration of the art of shell scripting; Mendel Cooper

Classic Shell Scripting  Arnold Robbins and Nelson A. F. Beebe, O'Reilly & Associates, 2005

UNIX Shells by Example Ellie Quigley; Prentice Hall, 4th edition, 2005 (our textbook)

Learning the bash Shell Cameron Newham and Bill Rosenblatt, O'Reily & Associates, 3rd edition,  2005

Introducing Regular Expressions Michael Fitzgerald, O'Reilly & Associates, 2012

Mastering Regular Expressions Jeffrey E. F. Friedl, O'Reilly & Associates, 3rd edition, 2006

Sed and Awk Dale Dougherty, O'Reilly & Associates, 1997

The AWK Programming Language, Alfred Aho, Brian Kernighan, Peter Weinberger, Addison-Wesly Publishing Company, 1988

GNU manuals:
  bash:  https://www.gnu.org/software/bash/manual/bashref.html
  find: http://www.gnu.org/software/findutils/manual/find.html
  grep: http://www.gnu.org/software/grep/manual/
  sed: http://www.gnu.org/software/sed/manual/sed.html
  bc: http://www.gnu.org/software/bc/manual/bc.html

Milestones in the history of computation

Tommy Flowers


Colossus - 1944

 

 


Eniac - 1946