Investigating and Managing Processes

Overview

1. Process

  • A process is a set of instructions loaded into memory.
  • Numeric Process ID (PID) used for identification.
  • UID, GID and SELinux context determines filesystem access.
  • The Linux Kernel tracks every aspect of a process by its PID under /proc/PID.

Listing Process

  • The ps command is used to view the process information.
  • By Default, shows processes from the current terminal

Options

  • a: Shows processes from all the terminals.
  • x: Shows all the processes owned by you, or shows all the processes when used together with the a option (such as: ps ax) Including processes that are not controlled by a terminal Such as Daemon processes, This shows up as ? in the tty column of the output

  • u: Shows process owner information.
  • f: Shows process parentage

  • o: Shows custom information Such as pid, tty, stat, nice, %cpu, %mem, time, comm, command, euser, ruser

Examples:

[mitesh@Matrix ~]$ ps
[mitesh@Matrix ~]$ ps a
[mitesh@Matrix ~]$ ps x
[mitesh@Matrix ~]$ ps u
[mitesh@Matrix ~]$ ps f
[mitesh@Matrix ~]$ ps xo pid,tty,stat,%cpu,%mem,time,command,euser,ruser
[mitesh@Matrix ~]$ ps axo pid,tty,stat,%cpu,%mem,time,command,euser,ruser

Process Status

  • Every process has a state property, which describes whether the process is actively using the cpu (Running), in memory but not doing anything (Sleeping), waiting for a resource to become available (Uninterruptable Sleep) or terminated but not flushed from the process list (Zombie).
  • Running and Sleeping are normal, but the presence of Uninterruptable Sleep or Zombie processes may indicate problems lurking on your system.

Uninterruptable Sleep

  • Process is sleeping and can not be woken up until an event occurs.
  • It can not be woken up by a signal.
  • Typically, the result of I/O operations, such as a failed network connections (For NFS Hard Mounts).

Zombie

  • Just before a process dies, it sends a signal to its parent and waits for an acknowledgment before terminating.
  • Even if the parent process does not immediately acknowledge this signal, all resources except for the process identity number (PID) are released.
  • Zombie processes are cleared from the system during the next system reboot And do not adversely affect system performance.

Finding Process

# Most Flexible
[mitesh@Matrix ~]$ ps axo pid,tty,comm | grep 'cups'
1516 ?        cupsd
3066 pts/1    eggcups

# By predefined patterns: pgrep
[mitesh@Matrix ~]$ pgrep -U root
[mitesh@Matrix ~]$ pgrep -G mitesh

[mitesh@Matrix ~]$ pgrep cups
1516
3066

# By exact program name:	pidof
[mitesh@Matrix ~]$ pidof cupsd
1516

2. Signals

  • Signals are simple messages that can be communicated to processes with commands like kill.
  • Sent directly to processes, no user interface required.
  • Programs associate actions with each signal.

  • Signals are specified by name or number when sent man 7 signals shows complete list

  • Signal 1 HUP (SIGHUP) Re-read Configuration Files
  • Signal 9 KILL (SIGKILL) Terminate Immediately
  • Signal 15 TERM (SIGTERM) Terminate Cleanly
  • Signal 18 CONT (SIGCONT) Continue If Stopped
  • Signal 19 STOP (SIGSTOP) Stop Process

Sending Signals to Process

  • By PID: kill [signal] pid ...
  • By Pattern: pkill [signal] pattern
  • By Name: killall [signal] command ...

  • kill can send many signals, but processes only respond to those signals whose they have been programmed to recognize.
  • For Example: Most services are programmed to reload their configuration when they receive a HUP(1) signals.

  • Some processes are terminated when they completed their tasks.
    Interactive applications may need the user to issue a quit command.
    In other cases, processes may need to be terminated with Ctrl+c, which sends an INT(2) signal to the process.

  • The process is shutdown cleanly means Terminate child process first & Complete any pending I/O operations.

NOTE!: The KILL(9) signal should be used only if a process will not respond to a Ctrl+c or a TERM(15) signals. Using KILL(9) signal on a routine basis may cause zombie processes and lost data.


# The following are all identical and will send default TERM(15) signal to the process with PID number 3705
[mitesh@Matrix ~]$ kill 3705
[mitesh@Matrix ~]$ kill -15 3705
[mitesh@Matrix ~]$ kill -TERM 3705
[mitesh@Matrix ~]$ kill -SIGTERM 3705

3. Scheduling Priority

  • Every running process has a scheduling priority: A ranking among running processes determining which should get the attention of the processor.

  • Priority is affected by a process’ nice value.
  • The nice value range from -20 to 19 ( Default is 0 )
    • -20: Highest CPU
    • 19: Lowest CPU

Altering Scheduling Priority

  • Niceness value may be altered…

  • When starting a process

[mitesh@Matrix ~]$ nice -n 5 command
  • After Starting the process
[mitesh@Matrix ~]$ renice 5 -p PID

NOTE!: Only root may decrease nice value. Non-privileged users start a process at any positive nice value but cannot lower it once raised.

[mitesh@Matrix ~]$ nice -n 10 myprog
[mitesh@Matrix ~]$ renice 15 -p PID
[root@Matrix ~]# renice -19 -p PID

Process Management Tools

CLI - top, htop
  • Display list of processes running on your system, updated every 3 seconds.
  • You can use keystrokes to kill, renice and change the sorting order of processes.
  • Use ? key to view the complete list of hotkeys.
  • You can exit top by pressing the q key.
GUI - gnome-system-monitor
  • The gnome-system-monitor, which can be run from the console Or by selecting Applications -> System Tools -> System Monitor

  • Display real time process information
  • Allows killing, re-nicing, sorting

4. Job Control

Background Process

  • Append the ampersand to the command line: firefox &

Suspended Running Program

  • Use `Ctrl+z
  • Send STOP(19) signal

Manage Background Or Suspended Jobs

  • List Job Numbers and Names: jobs
  • Resume in the Background: bg [%jobnum]
  • Resume in the Foreground: fg [%jobnum]
  • Send a Signal: kill [SIGNAL] [%jobnum]

Examples:

[mitesh@Matrix ~]$ ping 127.0.0.1 &> /dev/null
^Z
[1]+  Stopped                 ping 127.0.0.1 &>/dev/null

[mitesh@Matrix ~]$ bg
[1]+ ping 127.0.0.1 &>/dev/null &

[mitesh@Matrix ~]$ firefox &
[2] 4162

NOTE!: The number next to [2] after backgrounding firefox is the PID

[mitesh@Matrix ~]$ jobs
[1]-  Running                 ping 127.0.0.1 &>/dev/null &
[2]+  Running                 firefox &

NOTE!: The + or -`` signs next to the job numbers tells which job is the default
+ sign is the default job


5. Scheduling Process

  • One time jobs use at, Recurring jobs use crontab
/-----------------------------------------------------------------------\
|									|
|	Create		|	at time		|	crontab -e	|
|	List		|	at -l		|	crontab -l	|
|	Details		|	at -c jobnum	|			|
|	Remove		|	at -d jobnum	|	crontab -r	|
|	Edit		|			|	crontab -e	|
|									|
\-----------------------------------------------------------------------/
  • Non-redirected output is mailed to the user
  • The root can modify jobs for other users

at command

  • Scheduling One Time Job with at command
  • One Command Per Line
  • Terminated With Ctrl+d

Options

/-----------------------------------------------------------------------\
|	at 8:00am December 7			at 7 am Thursday	|
|	at midnight + 23 minutes		at now + 5 minutes	|
\-----------------------------------------------------------------------/
/-------------------------------------------------------------------------------------------------------\
|	Command		Alias		        Meanning					        |
|--------------------------------------------------------------------------------------------------------
|													|
|	atq	|	at -l		|	Lists the jobs currently pending.			|
|	atrm	|	at -d jobnum	|	Deletes the job.					|
|		|	at -c jobnum	|	Cats the full environment for the specified job.	|
|													|
\-------------------------------------------------------------------------------------------------------/

Example:

[mitesh@Matrix ~]$ at 0200
at> date
at> cal
at> <EOT>
job 1 at 2011-08-26 02:00


[mitesh@Matrix ~]$ atq
[mitesh@Matrix ~]$ at -l
1	2011-08-26 02:00 a mitesh

crontab command

  • Scheduling Recurring Jobs with crontab command
  • The cron mechanism is controlled by a process named crond.
  • This process runs every minute and determines if an entry in user’s cron tables need to be executed.

  • The crontabs are stored in /var/spool/cron/
  • The root can modify the jobs for other users with crontab -u username and any of the other options, such as -e.

Crontab File Format

  • Comment lines begin with #.
  • One entry per line, no limit to line length.
  • Entry consist of five space-delimited fields followed by a command name.
  • Fields are Minute, Hour, Day Of Month, Month, Day Of week.
  • An asterisk (*) in a field represent all valid values.
  • Multiple values are separated by commas.
  • See man 5 crontab for more details
/-----------------------------------------------------------------------\
|									|
|	Minute		|	0-59					|
|	Hour		|	0-23					|
|	Day Of Month	|	1-31					|
|	Month		|	1-12 (Or Jan, Feb, Mar, Etc)		|
|	Day Of Week	|	0-7  (Or Sun, Mon, Tue, Etc)		|
|				(0 or 7 = Sunday, 1 = Monday)		|
|									|
\-----------------------------------------------------------------------/
 # * * * * *  command to execute
 # │ │ │ │ │
 # │ │ │ │ │
 # │ │ │ │ └───── day of week (0 - 6) (0 to 6 are Sunday to Saturday, or use names; 7 is Sunday, the same as 0)
 # │ │ │ └────────── month (1 - 12)
 # │ │ └─────────────── day of month (1 - 31)
 # │ └──────────────────── hour (0 - 23)
 # └───────────────────────── min (0 - 59)

Example:

[mitesh@Matrix ~]$ crontab -e
#Min	Hour	DOM	Month	DOW	Command
0	0	31	10	*	mail -s "boo" $LOGNAME < boo.txt
0	2	*	*	*	netstat -tulpn | diff - /media/cdrom/baseline
0	4	*	*	1,3,5	find ~ -name core | xargs rm -f {}
crontab in details
  • The cron mechanism is controlled by a process named crond.
  • This process runs every minute and determines if an entry in user’s cron tables need to be executed.

  • The crontabs are stored in /var/spool/cron/`
  • The root can modify the jobs for other users with crontab -u username and any of the other options, such as -e.
Crontab File Format
  • Comment lines begin with #.
  • One entry per line, no limit to line length.
  • Entry consist of five space-delimited fields followed by a command name.
  • Fields are Minute, Hour, Day Of Month, Month, Day Of week.
  • An asterisk (*) in a field represent all valid values.
  • Multiple values are separated by commas.
  • Special Time Specification Nicknames: @reboot, @yearly, @annually, @monthly, @weekly, @daily, @hourly
  • See man 5 crontab for more details

Examples:

[mitesh@Matrix ~]$ crontab -e
#Min    Hour    DOM     Month   DOW     Command
0       0       31      10      *       mail -s "boo" $LOGNAME < boo.txt
0       2       *       *       *       netstat -tulpn | diff - /media/cdrom/baseline
0       4       *       *       1,3,5   find ~ -name core | xargs rm -f {}

*/2	*	*	*	*	echo "Every 2 Minutes" &> /dev/tty1
*/5	*	*	*	*	echo "Every 5 Minutes" &> /dev/tty1
@reboot					echo "Runs Once After Reboot" &> /dev/tty1

[mitesh@Matrix ~]$ echo '*/15 8-17 * * 1-5 echo Breaktime' | crontab
The Cron Access Control
/-----------------------------------------------------------------------------------------------------------------------------------------------\
|	---------------		--------------		Only root can install the crontab files.						|
|																		|
|	/etc/cron.allow		--------------		The root & All The Listed users in cron.allow can install the crontab files.		|
|																		|
|	---------------		/etc/cron.deny		All The users except The users in cron.deny can install the crontab files.		|
|																		|
|	/etc/cron.allow		/etc/cron.deny		The cron.deny file is ignored.								|
|							The root & All The Listed users in cron.allow can install the crontab files.		|
\-----------------------------------------------------------------------------------------------------------------------------------------------/

NOTE!: Denying A User Through The Use Of Above Files Does Not Disable Their Installed crontab.

System Crontab Files

  • Different Format Than User Crontab Files
  • Default System Crontab File Is /etc/crontab
  • The /etc/cron.d/ Directory Contains The Additional System Crontab Files

Example:

[mitesh@Matrix ~]$ cat /etc/crontab
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

#run-parts
01	*	*	*	*	root	run-parts	/etc/cron.hourly
02	4	*	*	*	root	run-parts	/etc/cron.daily
02	4	*	*	0	root	run-parts	/etc/cron.weekly
42	4	1	*	*	root	run-parts	/etc/cron.monthly

NOTE!: The System Crontab Files Are Different From The Users Crontab Files
In The System Crontab Files Sixth Field Is A Username, Which will Be Used To Execute The Commands.

The run-parts Is A Shell Script (/usr/bin/run-parts).
The run-parts Shell Scripts Take One Argument - A Directory Name And Invokes All Of The Program In That Directory.

Thus, At 4:02 Every Morning, All Of The Executables In The /etc/cron.daily/ Directory Will Be Run As The root User.

Default Daily Cron Jobs
  • The /etc/cron.daily Are Usually Used For:

    • Clean Up Temporary Directories
    • Update mlocate & whatis Database
    • Perform Other Housekeeping Tasks
    A) The tmpwatch:
    • Deletes All Files In /tmp Directory Which Is Not Accessed For 240 Hours (10 Days)
    • Deletes All Files In /var/tmp Directory Which Is Not Accessed For 720 Hours (30 Days)
    B) The logrotate:
    • Keeps Log Files From Getting Too Large
    • Rotates Log Files On
    • Predefined Intervals (Weekly)
    • When Reach The Predefined Size
    • Old Files Are Optionally Compressed

    • Configuration Files:
      • /etc/logrotate.conf (Global Configuration)
      • /etc/logrotate.d/ (Override Global Configuration)

Example: The /var/log/messages Is Rotated Weekly To /var/log/messages-yyyymmdd

The Anacron System
  • The Anacron Runs The Missed Cron Jobs When The System Boots.
  • The Anacron Command Is Used To Run The Missed Daily,Weekly & Monthly Cron Jobs.

Example:

  • According To The /etc/crontab File
  • At 4:02 Every Morning, All Of The Executables In The /etc/cron.daily/ Directory Will Be Run As root User.
  • Now Suppose Your Laptop Is Almost Always Off At The 4:02 AM, Then The mlocate & whatis Database Is Never Be Updated.

Configuration File

  • /etc/anacrontab
  • Field1: If The Cron Jobs Not Been Run For The Specified No Of Days
  • Field2: Wait For The Specified No Of Minutes Before Runs
  • Field3: Job Identifier
  • Field4: The Cron Job To Run

Examples:

[mitesh@Matrix ~]$ cat /etc/anacrontab
# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22

# Period In Days	Delay In Minutes	Job-Identifier	Command
1		5			cron.daily	nice run-parts /etc/cron.daily
7		25			cron.weekly	nice run-parts /etc/cron.weekly
@monthly 	45			cron.monthly	nice run-parts /etc/cron.monthly
How Anacron Works
  • According To The /etc/crontab File
  • The 1st Command To Run Is 0anacron.
  • The 0anacron Command Sets The Last Run Timestamp In A /var/spool/anacron/cron.{daily,weekly,monthly} Files.

  • On The System Boot Up, The Anacron Commands Runs.
  • The /etc/anacrontab File Specify How Often The Commands In cron.daily/ cron.weekly/ and cron.monthly/ Should Be Runs.
  • If These Commands Are Not Runs In This Time Then
  • The Anacron Command Waits For The Specified No Of Minutes In The /etc/anacrontab File & Then Runs The Commands

6. Grouping Commands

  • Two ways to group commands

Compound

  • Example: date; who | wc -l
  • Commands run back to back

Subshell

  • Commands inside parentheses are run in their own instance of bash, called subshell.
  • Example: (date; who | wc -l)
  • All output is sent to a single STDOUT and STDERR

  • Suppose you want to maintain a count of the number of users logged on, Along with a time/date stamp, in a log file.

Examples:

[mitesh@Matrix ~]$ date >> logfile
[mitesh@Matrix ~]$ who | wc -l >> logfile

[mitesh@Matrix ~]$ date; who | wc -l
Tue Aug 30 14:04:31 IST 2011
3

[mitesh@Matrix ~]$ date; who | wc -l >> logfile
Tue Aug 30 14:05:08 IST 2011

[mitesh@Matrix ~]$ (date; who | wc -l) >> logfile

7. Exit Status

  • Processes report success or failure with an exit status.
  • 0 for success
  • 1-255 for failure

  • $? stores the exit status of the most recent command
  • exit [num] terminates and set status to num

Examples:

[mitesh@Matrix ~]$ ping -c1 -w1 localhost &> /dev/null
[mitesh@Matrix ~]$ echo $?
0

[mitesh@Matrix ~]$ ping -c1 -w1 station999 &> /dev/null
[mitesh@Matrix ~]$ echo $?
2

8. Conditional Execution Operators

  • * Commands can be run conditionally based on exit status.
  • && Represents conditional AND THEN
  • || Represents conditional OR ELSE

NOTE!: When executing two commands separated by &&,
The 2nd command runs if the 1st command exit successfully (Exit status 0).
When executing two commands separated by ||,
The 2nd command runs if the 1st command fails (Exit status in the range of 1 to 255).



Examples:

[mitesh@Matrix ~]$ grep -q 'no_such_user' /etc/passwd || echo "No such user"
No such user

[mitesh@Matrix ~]$ ping -c1 -w2 localhost &> /dev/null \
> && echo "Localhost is up" \
> || (echo "Localhost is unreachable"; exit 1)
Localhost is up
[mitesh@Matrix ~]$ echo $?
0


[mitesh@Matrix ~]$ ping -c1 -w2 station999 &> /dev/null \
> && echo "Station999 is up" \
> || (echo "station999 is unreachable"; exit 1)
station999 is unreachable
[mitesh@Matrix ~]$ echo $?
1
#!/bin/bash
for x in $(seq 1 10)
do
  echo adding test$x
  (
    echo -ne "test$x\t"
    useradd test$x 2>&1 > /dev/null && mkpasswd test$x
  ) >> /tmp/userlog
done
echo 'cat /tmp/userlog to see new passwords'

9. test command

  • The test command evaluates true or false scenarios to simplify conditional execution.
  • Returns 0 for true
  • Returns 1 for false

NOTE!: Strings should be compared using a Mathematical Operator, While Numbers are compared using an Abbreviation (-eq).



Examples:

# Long Form
$ test "$A"  =  "$B" && echo "Strings are equal"
$ test "$A" -eq "$B" && echo "Integers are equal"

# Shorthand
$ [ "$A"  =  "$B" ] && echo "Strings are equal"
$ [ "$A" -eq "$B" ] && echo "Integers are equal"

File Tests

  • Use the following command for complete list man test
/-----------------------------------------------------------------------------------------------\
|												|
|	-d FILE		|	FILE exists and is a directory					|
|	-e FILE		|	FILE exists							|
|	-f FILE		|	FILE exists and is a regular file				|
|	-h FILE		|	FILE exists and is a symbolic link (same as -L)			|
|	-L FILE		|	FILE exists and is a symbolic link (same as -h)			|
|	-r FILE		|	FILE exists and read permission is granted			|
|	-s FILE		|	FILE exists and has a size greater than zero			|
|	-w FILE		|	FILE exists and write permission is granted			|
|	-x FILE		|	FILE exists and execute (or search) permission is granted	|
|	-O FILE		|	FILE exists and is owned by the effective user ID		|
|	-G FILE		|	FILE exists and is owned by the effective group ID		|
|												|
\-----------------------------------------------------------------------------------------------/

Example:

$ [ -f ~/lib/functions ] && source ~/lib/functions

10. Scripting If Statements

  • Every process reports an exit status.
  • 0 for success
  • 1-255 for failure

  • Execute instructions based on a exit status of the command.
#!/bin/bash
if ping -c1 -w2 station1 &> /dev/null
then
  echo "Station1 is up"
elif grep "station1" ~/maintenance.txt &> /dev/null
then
  echo "Station1 is undergoing maintenance"
else
  echo "Station1 is unexpectedly DOWN!"
  exit 1
fi
  • The exit status can be checked within the body of the if as shown in the example, Or you can assign the exit status to a variable using a subshell, as in:

    • STATUS=$(test -x /bin/ping6)
  • The if structure can be combined with conditional operator

#!/bin/bash
if test -x /bin/ping6; then
  ping6 -c1 ::1 &> /dev/null && echo "IPv6 stack is up"
elif test -x /bin/ping; then
  ping -c1 127.0.0.1 &> /dev/null && echo "No IPv6, but IPv4 stack is up"
else
  echo "Oops! This should not happen."
  exit 255
fi

NOTE!: This script checks for the IPv6 version of the ping command (ping6) exists.
If it does, it uses ping6 to send a test packet to the system’s IPv6 Loopback Interface.
Else The script checks for the IPv4 version of the ping command (ping) exists.
If it does, it uses ping to send a test packet to the system’s IPv4 Loopback Interface.
If neither IPv6 nor IPv4 exists,
Something is probably wrong and a non-zero return code is issued along with a warning message.

  • FOR GOOD EXAMPLE OF REAL WORLD SCRIPTS, LOOK AT THE SCRIPTS IN /etc/init.d/*


Newsletter

Get updated when I create new content.
Unsubscribe whenever. Never any spam.

Note

I'm Tama. I turn down every ad, affiliate, and sponsor request I get. I write free resources that help people. If you enjoy my content, please consider supporting what I do.

Support my work