Contents

1. Commands, shells, processes, files.

1.1. Overview

Today, we’ll first talk about the course organization.
Then we’ll discuss the fundamentals of the Linux operating system: kernel, processes, file system, user accounts, and basic commands.

1.2. Course topics

  1. Commands, shells, processes

  2. Installation and upgrade

  3. Server virtualization with KVM

  4. Linux package management

  5. Networking

  6. Network File System (NFS)

  7. System Run levels and process scheduling (at, cron)

  8. Linux and Windows interoperability

  9. Building Linux applications and packages

  10. Shell and Python scripting

  11. Security

  12. High Performance Computing (HPC): OpenMP

  13. HPC: MPI

  14. HPC: GPU computing with Nvidia CUDA

1.3. Course agenda

  • Lectures and practical recitations on the lab computers remotely.

  • Midterm exam (mid March 2021);

  • Final exam (May 2021).

1.4. Unix history milestones and foundation of Linux

  • 1969: a UNIX like OS on PDP 7 was written by Ken Thompson in Bell Labs It was single user, non-multitasking, written in assembler;

  • 1973: C programming language by Dennis Ritchie in Bell Labs;

  • 1975: First widely available version of UNIX (written in C);

  • Late seventies: two main stream UNIX distributions - System V from Bell Labs and Berkeley release.

  • Eighties: commercial UNIX operating systems: DEC, Sun, IBM, etc.

  • 1985: GNU, the Free Software Foundation, is founded (Richard Stallman)

  • 1987: The first GNU C compiler, gcc, becomes freely available via ftp.

  • 1991: Starting of the Free Family Berkely like OS, BSD, (NetBSD, FreeBSD, OpenBSD).

  • 1991: Finnish graduate student Linus Torvalds announces release of Linux based on GNU tools.

  • Now days: numerous distributions of Linux (Links to an external site.); the fastest developing OS; accepted by some commercial hardware and software vendors (IBM, Oracle, HP, Amazon, Google).

  • Has become the leading OS in High Performance Computing, Cloud Computing, mobile OS and apps, microcontroller devices.

'''From https://en.wikipedia.org/wiki/File:Unix_history-simple.svg'''
from IPython.display import display, Image
display(Image(filename='img/Unix_history-simple.png'))
_images/Lesson_1_4_0.png

1.5. What makes Linux different from traditional Unix

  • GNU Public License (GPL) for the Kernel and OS applications: free to use; get the source; develop and disclose your source.

  • The same Kernel and GNU C libraries (glibc) for different distributions: if you can run an application on one kind of Linux, its a matter of setting libraries to run it on the other distribution.

  • Runs on various computer architectures: x86, x86_64, ARM, POWERPC, SPARC, ALPHA, MIPS, HPPA, etc.

  • The fastest developing OS.

  • Various vendor support.

  • Strong community support.

1.6. Why would you run Linux?

Available free kernel, modifiable for specific tasks. Vendors can easily port their software into Linux, for example, for microcontrollers and file systems.

Various developers can work on the same software without intellectual property issues. Open source software allows upgrading and changing software by the people using it.

No license fee for every version of the software on any computer - no charge for scalability.

It is your choice what commercial software you need to buy.

1.8. Linux on a typical Server/Desktop/Laptop (Intel Core i7) hardware

  • From the file system, Linux kernel is loaded into the RAM.

  • The kernel controls the hardware and processes.

  • On the diagram:

    • PCH - Platform Controller Hub

    • DMI - Direct Memory Interface

    • LPC - Low Pin Count; a simple interface to slower I/O devices

title

1.9. Linux kernel and applications

In general, Linux implies rather just a Kernel than a whole OS.

Linux OS distributions include: generic or customized Kernel and generic or customized GNU software (libraries, shells, and applications).

Source codes of the Kernel, shells and GNU applications are freely available to anyone.

Traditional tools for Linux administration and development: C - programming language Shell scripting, Perl, Python

Linux kernel

1.10. Login to a Linux terminal

Login to Linux System

1.12. User accounts

In order to login to a system a user has to authenticate with his/her credentials: user name and password.

Authentication types:

  • local (password/shadow/group) and

  • Domain based (NIS, Kerberos, LDAP).

For local authentication, a user should have an account on the system.

Account File

Entry

/etc/passwd

mike:x:1001:1001:Michael Whites:/home/mike:/bin/bash

/etc/shadow

mike:\(1\)zXCV7fz8ii84grbZhj:14087:0:99999:7:::

/etc/group

mike:x:1001:mike

/etc/group

admin:x:112:hostadm,mike

Accounts can be created only by a superuser (root).

Commands to create user accounts:

adduser (interactive command)

useradd (can be included in a script).

1.12.1. Exercise

Create a new user account by following the instructor.

Become root: 

sudo -s

Create a new user account, mike: 

adduser mike

Verify the account existence: 

id mike

Change password of the user: 

passwd mike

1.13. Linux shells

When you login to a Linux system, you get a command shell.

Shells are listed in /etc/shells. Default Linux shell: bash

System commands, scripts and applications run in the shell - they become child processes of the shell.

Shell variables are local to the shell.

A shell variable becomes an environment variable after executing command export on it: 

svar=VAR_1  #initialize shell variable
export svar #becomes environment variable
env | grep svar

Environment variables are inherited by the child shells and processes.

Commands executed in the shell should be either built-in shell commands, or addressed with the full path, or located in the PATH environment variable. 

echo $PATH

1.13.1. Exercise

Create a new shell variable, svar: 

svar=VAR_1
echo $svar

Start a new child shell and see if svar is defined there:

bash
echo $svar

Exit from the child shell, export the variable, and see if it is defined in a child shell:

exit
export svar
bash
echo $svar

1.14. Files and directories

fishy

  • The cylinders define partition boundaries.

  • The sectors define data blocks.

  • A file system is created within a disk partition.

  • Files are created within a file system.

Files and Directories

fishy

INODES contain information about files

  • a pointer to the file data blocks

  • link count

  • the owner

  • the group

  • mode

  • size

  • last access time

  • last modification time

  • last update time (attributes)

Type of files

  • Regular or ordinary files

  • Directories

  • Symbolic links

  • Device files

  • Directory and Inodes

1.15. File Permissions and Ownerships

1.15.1. Permissions

Octal

Binary

Permission

Meaning

0

000

none

All turned off

1

001

–x

Execute

2

010

-w-

Write

3

011

-wx

Write, execute

4

100

r–

Read

5

101

r-x

Read, execute

6

110

rw-

Read, write

7

111

rwx

Read, write, execute

1.15.2. File Ownerships: USER (u), GROUP (g), OTHERS (o)

Changing permissions Permissions are applied for USER, GROUP and OTHERS (rwx rwx rwx)

chmod 660 testf.txt
ls -l  testf.txt

-rw-rw----    1 mike   staff          0 Jan 18 10:26 testf.txt

umask sets default permission for files and directories. For example,

umask 022

perm/umask

directory

file

permission

777

666

umask

-022

-022

permission

755

644

1.16. Assigning permissions and ownerships on files (Exercise)

Create a new directory EX1 and step into it:

mkdir EX1
cd EX1

Check umask and create a new file, mf1.txt. See the file attributes with command ls.

umask
touch mf1.txt
ls -l mf1.txt

Change umask and create anothe file, mf2.txt

umask 022
touch mf2.txt
ls -l mf2.txt

Create another file, mf3.exe and make it executable:

touch mf3.exe
chmod 755 mf3.exe
ls -l mf3.exe

Try runing files mf2.txt and mf3.exe, and see which one is runnable:

./mf2.txt
./mf3.exe

As you can see only the executable file can run.
Changing the user ownership on file mf1.txt:

sudo chown mike mf1.txt
ls -l mf1.txt

Note, you need to be the root user when changing the file ownership.

Changing the group ownership on file mf2.txt:

sudo chown :mike mf2.txt
ls -l mf2.txt

Changing both the user and group ownership on file mf3.exe:

sudo chown mike:mike mf3.exe
ls -l mf*

1.17. Sticky bit (Exercise)

Sticky bit on a directory protects files in the directory from been modified/removed by non their owners.

For example /tmp directory on a Unix/Linux system has a sticky bit set

ls -ld /tmp

drwxrwxrwt 4 root root 4096 2018-07-25 16:29 /tmp

Open a new terminal and become root:

sudo -s

Create a new directory, temp, and give it a world writable permissions

mkdir temp
chmod 777 temp

In the directory, user root creates a new empty file:

cd temp
touch ex3

In the other terminal,try removing this file as user hostadm:

cd temp
rm ex3

The file has been deleted.

Change the permission on temp director in the root user terminal:

chmod 1777 temp

Then repeat the steps above with creating and removing file ex3 in directory temp

1.18. Processes

Program running on a system is a process. Linux is a multi-processing (multi-tasking) system.

Process types:

  • User Processes. A user process is one that is initiated by a regular user account and runs in user space.

  • Daemon Processes. A daemon process is an application that is designed to run in the background, usually related to a service.

  • Kernel Threads. Kernel processes execute only in kernel space.

Privilege levels. Kernel space and user space:

_images/2560px-Priv_rings.svg.png

1.20. Process termination and renice command

  • Program running on a system is a process. Linux is a multi-processing (multi-tasking) system.

  • Attributes: Lifetime, PID, PPID, UID, GID, env variables, CWD.

  • Process with PID=1 is init

  • Monitoring: ps (for full listing, use ps -ef or ps -aux), top, pstree

  • Signal a running processes with PID=1009:

    kill -HUP 1009    #Hang up; re-read config files
kill -1   1009
kill -9  1009     #Kill; stop unconditionally
kill -KILL 1009
kill -15   1009    #Terminate gracefully
kill -TERM 1009
kill -TSTP 1009   #Suspend; can be continued
kill -18   1009

  • Process can be started with lower and higher priority through nice (range: -20 highest, +19 lowest)

    nice -10 matlab
nice --10 matlab

  • Process can be re-niced at run time:

    renice 5 1009
renice -5 1009

Exercise

When administrating a Linux system, it is often needed to terminate processes that consume a lot of resources (CPU and/or RAM), and slow down everything else.

  • Install package gimp

    install the pkg

    apt-get install gimp

    Start gimp application on a “background”:

    gimp &

  • Find out the PID of the process:

    pgrep gimp

    The output shows the PID of the process, for exmple 4198.
    Terminate process 4198:

    kill -15 4198

    An alternative way to kill an application is by using pkill command:

    pkill -15 gimp

    If option -15 above doesn’t work, try -9

  • Reference: Understanding Linux processes

1.21. Process data streams

  • Most of the Linux system processes have three data streams:

    standard input “0” (stdin)

    standard output “1” (stdo)

    standard error “2” (stderr)

  • The standard output and standard error are directed to the screen of your monitor; the standard input is read from a keyboard.

    It is possible to redirect the standard output and error into files, for example

    ps -ef 1 > stdo.out

    or

    ps -ef > stdo.out

    To redirect both stdo and stderr to the same file:

    command > output.txt 2>&1

    For example,

    ps -ef > output.txt 2>&1

    If you need to discard the stdo stream, you can re-direct it to /dev/null:

    ps -ef 1>/dev/null


    It is also possible to re-direct the standart output of one process into the standard input of the other using pipes "|":

    ps -ef | less

1.22. Background processes


Processes with Input/Output detached from the terminal are called background processes.

Exercise
In a terminal window, run gimp graphics software:

gimp

Suspend the interactive process, gimp, and move it to the background: 
# Ctrl-Z
jobs
bg %1

Now the shell can be used for running the other commands and applications. Move the background job (with its ID) back to the foreground and terminate gimp process:

fg %1
# Ctrl-C
Computing Environment Course 2. Installation