Linux

Linux Introduction

Linux is a Unix like operating system. It is open source and free. We might sometimes use the word "Unix" instead of Linux.

A user can interact with Linux either using a 'graphical interface' or using the 'command line interface'.

Learning to use the command line interface has a bigger learning curve than the graphical interface but the former can be used to automate very easily. Also, most of the server side work is generally done using the command line interface.

Linux Operating System

The operating system is made of three parts:

The Programs

A user executes programs. AngryBird is a program that gets executed by the kernel, for example. When a program is launched, it creates processes. Program or process will be used interchangeably.

The Kernel

The Kernel handles the main work of an operating system:

1. Allocates time & memory to programs
2. Handles File System
3. Responds to various Calls

The Shell

A user interacts with the Kernel via the Shell. The console as opened in the previous slide is the shell. A user writes instructions in the shell to execute commands. Shell is also a program that keeps asking you to type the name of other programs to run.

Linux Files & Processes

Everything in Unix is either a file or a process.

Process

When you run a program, a process is created. Every process is identified by a number called process ID. To check the processes you are running, execute "ps" command on the shell. You can think of the process ID to be a sequence number given by the operating system. It may be different at different execution of the same program.

File

A file is a sequence of data. A file could be created by users using word processors or text editors or by the program to keep the information. A program is kept in the form of a file and when it is run by the kernel, it loads as a process.

A file is generally written on the disk so that it exists even after the computer restarts. It is saved in a disk - either hard disk drive (HDD - cheaper and slower) or solid state drive (SSD - faster but costlier).

A file is identified by a name called file path. In Unix, everything is represented as file:

1.    Devices such as Mouse, Keyboard

2.    Programs are saved as file

3.    Disk and Monitor

The Directory Structure

A file is kept inside a directory. A directory can have a file or another directory inside it. It is like an inverted tree.

The top-level directory is "/" called root. "/" directory does not have a parent. /A/B means B is inside A which is inside the top-level directory "root" denoted by /.

List Files and Directory

To see the list of files use the command: ls

Relative & Absolute Paths - Change the Directory

There are two ways to represent a file/directory path:

Absolute: This way of representing a file/directory is independent of the current directory of the user. Such paths start with "/". Example: The absolute path is the path of the directory from the root directory, like the path of 'john' directory, is actually '/home/john' as you can see in the tree structure

Relative: Relative to the current working directory. Example: Consider that you're currently in directory projects, the relative path from directory projects to directory john would be ../../../home/john it is like tracing the path back to / then to john.

You can change the directory using the cd command. Every user is given a separate home directory.

Home Directory & Current Directory

Inside the console, you are always in a directory. On login, by default, you land in your home directory.

To see the present working directory use: pwd

To change the directory to your home directory use only cd command without any arguments.

Create Directory

You can create the directory using mkdir command in the following way:

mkdir <directoryname>

INSTRUCTIONS

·         Use the console on the right-hand side. If the console is not visible then please click on the "Console" button on the right-hand side

·         Type cd ~ command to go to your home directory

·         Create a directory with the name xyz in your home directory.

Note - In case, you are stuck, please see this video: Demo - Create Directory - Video

Delete a file

To delete a file you can use rm command. To delete the directory use rm -r

Create File

A file is a sequence of bytes and represents data. It is found in a directory. A file could contain any kind of data: an executable program, data representing movies, music, pictures or plain text.

You can create an empty file using touch command. Please note that the extension of the file doesn't matter much in Unix.

Create File

A file is a sequence of bytes and represents data. It is found in a directory. A file could contain any kind of data: an executable program, data representing movies, music, pictures or plain text.

You can create an empty file using touch command. Please note that the extension of the file doesn't matter much in Unix.

Create Text File Using Nano

You can use a text editor to edit or create a text file. There are many editors in Linux such as nano, pico, vi, emacs. Let's try to use nano for creating a file

1.    Launch nano to edit:

2.  nano myfirstfile.txt

3.    Type the following text into it:

4.  He walked into his exile

5.    Press Control+x and afterward press y to Save Changes

6.    Hit enter to exit the nano editor

Copy File

To copy files and directories we use the cp command

INSTRUCTIONS

Please make the copy of myfirstfile.txt to myfirstfile_copy.txt by using the command

cp myfirstfile.txt myfirstfile_copy.txt

Two file names : If the command contains two file names, then it copy the contents of 1st file to the 2nd file. If the 2nd file doesn’t exist, then first it creates one and content is copied to it. But if it existed then it is simply overwritten without any warning. So be careful when you choose destination file name.

cp Src_file Dest_file

Copy a file into another directory

1.    Create a directory with a name myproject using the command:

2.  mkdir myproject

3.    Copy the file myfirstfile.txt created in the previous exercise to myproject directory

cp myfirstfile.txt myproject

Copy a directory into another directory

1.    Create a directory "src"

2.  mkdir src

3.    Create a file in "src"

4.  touch src/myfile.txt

5.    Create a directory "proj"

6.  mkdir proj

7.    Copy the src into "proj"

cp -r src proj

1.    Two file names : If the command contains two file names, then it copy the contents of 1st file to the 2nd file. If the 2nd file doesn’t exist, then first it creates one and content is copied to it. But if it existed then it is simply overwritten without any warning. So be careful when you choose destination file name.

2.  cp Src_file Dest_file

Suppose there is a directory named geeksforgeeks having a text file a.txt.
Example:

$ ls

a.txt

 

$ cp a.txt b.txt

 

$ ls

a.txt  b.txt

3.    One or more arguments : If the command has one or more arguments, specifying file names and following those arguments, an argument specifying directory name then this command copies each source file to the destination directory with the same name, created if not existed but if already existed then it will be overwritten, so be careful !!.

4.  cp Src_file1 Src_file2 Src_file3 Dest_directory

Suppose there is a directory named geeksforgeeks having a text file a.txt, b.txt and a directory name new in which we are going to copy all files.
Example:

$ ls

a.txt  b.txt  new

 

Initially new is empty

$ ls new

 

$ cp a.txt b.txt new

 

$ ls new

a.txt  b.txt

Note: For this case last argument must be a directory name. For the above command to work, Dest_directory must exist because cp command won’t create it.

5.    Two directory names : If the command contains two directory names, cp copies all files of the source directory to the destination directory, creating any files or directories needed. This mode of operation requires an additional option, typically R, to indicate the recursive copying of directories.

6.  cp -R Src_directory Dest_directory

In the above command, cp behavior depend upon whether Dest_directory is exist or not. If the Dest_directory doesn’t exist, cp creates it and copies content of Src_directory recursively as it is. But if Dest_directory exists then copy of Src_directory becomes sub-directory under Dest_directory.

Options:

There are many options of cp command, here we will discuss some of the useful options:
Suppose a directory named geeksforgeeks contains two files having some content named as a.txt and b.txt. This scenario is useful in understanding the following options.

$ ls geeksforgeeks

a.txt  b.txt

 

$ cat a.txt

GFG

 

$ cat b.txt

GeeksforGeeks

1. -i(interactive): i stands for Interactive copying. With this option system first warns the user before overwriting the destination file. cp prompts for a response, if you press y then it overwrites the file and with any other option leave it uncopied.

$ cp -i a.txt b.txt

cp: overwrite 'b.txt'? y

 

$ cat b.txt

GFG

2. -b(backup): With this option cp command creates the backup of the destination file in the same folder with the different name and in different format.

$ ls

a.txt  b.txt

 

$ cp -b a.txt b.txt

 

$ ls

a.txt  b.txt  b.txt~

3. -f(force): If the system is unable to open destination file for writing operation because the user doesn’t have writing permission for this file then by using -f option with cp command, destination file is deleted first and then copying of content is done from source to destination file.

$ ls -l b.txt

-r-xr-xr-x+ 1 User User 3 Nov 24 08:45 b.txt

 

User, group and others doesn't have writing permission.

 

Without -f option, command not executed

$ cp a.txt b.txt

cp: cannot create regular file 'b.txt': Permission denied

 

With -f option, command executed successfully

$ cp -f a.txt b.txt

4. -r or -R: Copying directory structure. With this option cp command shows its recursive behavior by copying the entire directory structure recursively.
Suppose we want to copy geeksforgeeks directory containing many files, directories into gfg directory(not exist).

$ ls geeksforgeeks/

a.txt  b.txt  b.txt~  Folder1  Folder2

 

Without -r option, error

$ cp geeksforgeeks gfg

cp: -r not specified; omitting directory 'geeksforgeeks'

 

With -r, execute successfully

$ cp -r geeksforgeeks gfg

 

$ ls gfg/

a.txt  b.txt  b.txt~  Folder1  Folder2

 

5. -p(preserve): With -p option cp preserves the following characteristics of each source file in the corresponding destination file: the time of the last data modification and the time of the last access, the ownership (only if it has permissions to do this), and the file permission-bits.
Note: For the preservation of characteristics you must be the root user of the system, otherwise characteristics changes.

$ ls -l a.txt

-rwxr-xr-x+ 1 User User 3 Nov 24 08:13 a.txt

 

$ cp -p a.txt c.txt

 

$ ls -l c.txt

-rwxr-xr-x+ 1 User User 3 Nov 24 08:13 c.txt

As we can see above both a.txt and c.txt(created by copying) have same characteristics.

Examples:

Copying using * wildcard: The star wildcard represents anything i.e. all files and directories. Suppose we have many text document in a directory and wants to copy it another directory, it takes lots of time if we copy files 1 by 1 or command becomes too long if specify all these file names as the argument, but by using * wildcard it becomes simple.

Initially Folder1 is empty

$ ls

a.txt  b.txt  c.txt  d.txt  e.txt  Folder1

 

$ cp *.txt Folder1

 

$ ls Folder1

a.txt  b.txt  c.txt  d.txt  e.txt

 

Move Files & Directories

mv command is used to move or rename files and directories.

INSTRUCTIONS

1.    Create a file myfirstfile.txt.

2.  nano myfirstfile.txt

3.    Add the text a clever fox into the editor and press control+x, followed by y, and then enter to save the file and come out of the nano editor.

4.    Rename myfirstfile.txt to firstfile.txt.

mv myfirstfile.txt firstfile.txt

Delete Files & Directories

To delete a file or a folder, use rm command

INSTRUCTIONS

Delete the file mywork/firstfile.txt created in the previous step.

rm command in Linux with examples

rm stands for remove here. rm command is used to remove objects such as files, directories, symbolic links and so on from the file system like UNIX. To be more precise, rm removes references to objects from the filesystem, where those objects might have had multiple references (for example, a file with two different names). By default, it does not remove directories.

This command normally works silently and you should be very careful while running rm command because once you delete the files then you are not able to recover the contents of files and directories.

Syntax:

 

rm [OPTION]... FILE...

Let us consider 5 files having name a.txt, b.txt and so on till e.txt.

$ ls

a.txt  b.txt  c.txt  d.txt  e.txt

Removing one file at a time

$ rm a.txt

 

$ ls

b.txt  c.txt  d.txt  e.txt

 

Removing more than one file at a time

$ rm b.txt c.txt

 

$ ls

d.txt  e.txt

Note: No output is produced by rm, since it typically only generates messages in the case of an error.

Options:

1. -i (Interactive Deletion): Like in cp, the -i option makes the command ask the user for confirmation before removing each file, you have to press y for confirm deletion, any other key leaves the file un-deleted.

$ rm -i d.txt

rm: remove regular empty file 'd.txt'? y

 

$ ls

e.txt

 

2. -f (Force Deletion): rm prompts for confirmation removal if a file is write protected. The -f option overrides this minor protection and removes the file forcefully.

$ ls -l

total 0

-r--r--r--+ 1 User User 0 Jan  2 22:56 e.txt

 

$ rm e.txt

rm: remove write-protected regular empty file 'e.txt'? n

 

$ ls

e.txt

 

$ rm -f e.txt

 

$ ls

 

 

Note: -f option of rm command will not work for write-protect directories.

3. -r (Recursive Deletion): With -r(or -R) option rm command performs a tree-walk and will delete all the files and sub-directories recursively of the parent directory. At each stage it deletes everything it finds. Normally, rm wouldn’t delete the directories but when used with this option, it will delete.

Below is the tree of directories and files:

$ ls

A

 

$ cd A

 

$ ls

B  C

 

$ ls B

a.txt  b.txt

 

$ ls C

c.txt  d.txt

 

Now, deletion from A directory(as parent directory) will be done as:

$ rm *

rm: cannot remove 'B': Is a directory

rm: cannot remove 'C': Is a directory

 

$ rm -r *

 

$ ls

 

 

Every directory and file inside A directory is deleted.

4. –version: This option is used to display the version of rm which is currently running on your system.

$ rm --version

rm (GNU coreutils) 8.26

Packaged by Cygwin (8.26-2)

Copyright (C) 2016 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later .

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

 

Written by Paul Rubin, David MacKenzie, Richard M. Stallman,

and Jim Meyering.

Applications of wc Command

Delete file whose name starting with a hyphen symbol (-): To remove a file whose name begins with a dash (“-“), you can specify a double dash (“–“) separately before the file name. This extra dash is necessary so that rm does not misinterpret the file name as an option. Let say their is a file name -file.txt, to delete this file write command as:

$ ls

-file.txt

 

$ rm -file.txt

rm: unknown option -- l

Try 'rm ./-file.txt' to remove the file '-file.txt'.

Try 'rm --help' for more information.

 

$ rm -- -file.txt

 

$ ls

 

Seeing Inside File - cat, tail, head

To see what is inside a text file, you can use either cat, tail or head command.

Using cat you can see the whole content of the file:

cat myfirstfile_copy.txt

Do not use this command to look inside a huge file. For the huge file, you can use the more command which would display the content of a file in a paginated way:

more myfirstfile_copy.txt

tail shows you the last few lines of a file

tail myfirstfile_copy.txt

By default tail shows you only last 10 lines, you can change it using the command line option. For example, to see the last 20 lines, you can use

tail -20 myfirstfile_copy.txt

Say you want to see the web server's (Apache, Nginx) access log of your website. Logs will keep on getting appended to access.log file when visitors visit your website. Now you can run below command and keep it running. It will keep on printing newly added lines in the access.log.

tail -f access.log

If you are interested in the first few lines, you can use the head command. By default head shows you only first 10 lines, you can change it using the command line option. For example, to see the first 20 lines, you can use

head -20 myfirstfile_copy.txt

Tail command in Linux with examples

It is the complementary of head command.The tail command, as the name implies, print the last N number of data of the given input. By default it prints the last 10 lines of the specified files. If more than one file name is provided then data from each file is precedes by its file name.

Syntax:

tail [OPTION]... [FILE]...

Let us consider two files having name state.txt and capital.txt contains all the names of the Indian states and capitals respectively.

 

$ cat state.txt

Andhra Pradesh

Arunachal Pradesh

Assam

Bihar

Chhattisgarh

Goa

Gujarat

Haryana

Himachal Pradesh

Jammu and Kashmir

Jharkhand

Karnataka

Kerala

Madhya Pradesh

Maharashtra

Manipur

Meghalaya

Mizoram

Nagaland

Odisha

Punjab

Rajasthan

Sikkim

Tamil Nadu

Telangana

Tripura

Uttar Pradesh

Uttarakhand

West Bengal

Without any option it display only the last 10 lines of the file specified.
Example:

$ tail state.txt

Odisha

Punjab

Rajasthan

Sikkim

Tamil Nadu

Telangana

Tripura

Uttar Pradesh

Uttarakhand

West Bengal

Options:

1. -n num: Prints the last ‘num’ lines instead of last 10 lines. num is mandatory to be specified in command otherwise it displays an error. This command can also be written as without symbolizing ‘n’ character but ‘-‘ sign is mandatory.

$ tail -n 3 state.txt

Uttar Pradesh

Uttarakhand

West Bengal

       OR

$ tail -3 state.txt

Uttar Pradesh

Uttarakhand

West Bengal

Tail command also comes with an ‘+’ option which is not present in the head command. With this option tail command prints the data starting from specified line number of the file instead of end. For command: tail +n file_name, data will start printing from line number ‘n’ till the end of the file specified.

$ tail +25 state.txt

Telangana

Tripura

Uttar Pradesh

Uttarakhand

West Bengal

2. -c num: Prints the last ‘num’ bytes from the file specified. Newline count as a single character, so if tail prints out a newline, it will count it as a byte. In this option it is mandatory to write -c followed by positive or negative num depends upon the requirement. By +num, it display all the data after skipping num bytes from starting of the specified file and by -num, it display the last num bytes from the file specified.
Note: Without positive or negative sign before num, command will display the last num bytes from the file specified.

With negative num

$ tail -c -6 state.txt

Bengal

        OR

$ tail -c 6 state.txt

Bengal

 

With positive num

$ tail -c +263 state.txt

Nadu

Telangana

Tripura

Uttar Pradesh

Uttarakhand

3. -q: It is used if more than 1 file is given. Because of this command, data from each file is not precedes by its file name.

Without using -q option

$ tail state.txt capital.txt

  state.txt

Odisha

Punjab

Rajasthan

Sikkim

Tamil Nadu

Telangana

Tripura

Uttar Pradesh

Uttarakhand

West Bengal

  capital.txt

Dispur

Patna

Raipur

Panaji

Gandhinagar

Chandigarh

Shimla

Srinagar

Ranchi

 

 

With using -q option

$ tail -q state.txt capital.txt

Odisha

Punjab

Rajasthan

Sikkim

Tamil Nadu

Telangana

Tripura

Uttar Pradesh

Uttarakhand

West BengalDispur

Patna

Raipur

Panaji

Gandhinagar

Chandigarh

Shimla

Srinagar

Ranchi

Bengaluru

4. -f: This option is mainly used by system administration to monitor the growth of the log files written by many Unix program as they are running. This option shows the last ten lines of a file and will update when new lines are added. As new lines are written to the log, the console will update with the new lines. The prompt doesn’t return even after work is over so, we have to use the interrupt key to abort this command. In general, the applications writes error messages to log files. You can use the -f option to check for the error messages as and when they appear in the log file.

$ tail -f logfile

5. -v: By using this option, data from the specified file is always preceded by its file name.

$ tail -v state.txt

==> state.txt <==

Odisha

Punjab

Rajasthan

Sikkim

Tamil Nadu

Telangana

Tripura

Uttar Pradesh

Uttarakhand

West Bengal

6. –version: This option is used to display the version of tail which is currently running on your system.

 

$ tail --version

tail (GNU coreutils) 8.26

Packaged by Cygwin (8.26-1)

Copyright (C) 2016 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later .

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

 

Written by Paul Rubin, David MacKenzie, Ian Lance Taylor,

and Jim Meyering.

 

Applications of tail Command

1. How to use tail with pipes(|): The tail command can be piped with many other commands of the unix. In the following example output of the tail command is given as input to the sort command with -r option to sort the last 7 state names coming from file state.txt in the reverse order.

$ tail -n 7 state.txt

Sikkim

Tamil Nadu

Telangana

Tripura

Uttar Pradesh

Uttarakhand

West Bengal

 

$ tail -n 7 state.txt | sort -r

West Bengal

Uttarakhand

Uttar Pradesh

Tripura

Telangana

Tamil Nadu

Sikkim

It can also be piped with one or more filters for additional processing. Like in the following example, we are using cat, head and tail command and whose output is stored in the file name list.txt using directive(>).

$ cat state.txt | head -n 20 | tail -n 5  > list.txt

 

$ cat list.txt

Manipur

Meghalaya

Mizoram

Nagaland

Odisha

 

What is happening in this command let’s try to explore it. First cat command gives all the data present in the file state.txt and after that pipe transfers all the output coming from cat command to the head command. Head command gives all the data from start(line number 1) to the line number 20 and pipe transfer all the output coming from head command to tail command. Now, tail command gives last 5 lines of the data and the output goes to the file name list.txt via directive operator.
2. Print line between M and N lines

This article is contributed by Akash Gupta. If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to contribute@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.

Please write comments if you find anything incorrect, or you want to share more information about the topic discussed above.

Use find command

Sometimes you might want to search for a particular file based on various attributes of a file such as size or name etc. The find command comes very handily for such use cases.

For example, to find all the text files in the current directory, you can use this command

find . -name '*.txt'

find ./ -name '*.txt'

find command in Linux with examples

The find command in UNIX is a command line utility for walking a file hierarchy. It can be used to find files and directories and perform subsequent operations on them. It supports searching by file, folder, name, creation date, modification date, owner and permissions. By using the ‘-exec’ other UNIX commands can be executed on files or folders found.

Syntax :

$ find [where to start searching from]

 [expression determines what to find] [-options] [what to find]

Options :

 

·         -exec CMD: The file being searched which meets the above criteria and returns 0 for as its exit status for successful command execution.

·         -ok CMD : It works same as -exec except the user is prompted first.

·         -inum N : Search for files with inode number ‘N’.

·         -links N : Search for files with ‘N’ links.

·         -name demo : Search for files that are specified by ‘demo’.

·         -newer file : Search for files that were modified/created after ‘file’.

·         -perm octal : Search for the file if permission is ‘octal’.

·         -print : Display the path name of the files found by using the rest of the criteria.

·         -empty : Search for empty files and directories.

·         -size +N/-N : Search for files of ‘N’ blocks; ‘N’ followed by ‘c’can be used to measure size in characters; ‘+N’ means size > ‘N’ blocks and ‘-N’ means size < 'N' blocks.

·         -user name : Search for files owned by user name or ID ‘name’.

·         \(expr \) : True if ‘expr’ is true; used for grouping criteria combined with OR or AND.

·         ! expr : True if ‘expr’ is false.

Use grep command

If you want to locate a word in files, you can use the grep command. Grep lists all the lines from files in which a particular word exists. Examples of grep

grep myword file1 file2

If you want to search in files recursively - inside every subdirectory of a directory, use the following command

grep -r myword directory

If you want to search case insensitive, use -i switch

grep -i myword file1 file2

Practical Use Cases

grep is a very powerful tool. It can somewhat behave like where clause in SQL queries.

Few Examples are:

1. On a web server, you could filter only the errors from a log file
2. Say you have a directory containing the temperature of various cities and you are looking for temperatures of the city having the name as nyc, you could easily do: grep nyx tempdirectory/*

3.  grep command in Unix/Linux

4.    The grep filter searches a file for a particular pattern of characters, and displays all lines that contain that pattern. The pattern that is searched in the file is referred to as the regular expression (grep stands for globally search for regular expression and print out).
Syntax:

5.  grep [options] pattern [files]

6.  Options Description

7.  -c : This prints only a count of the lines that match a pattern

8.  -h : Display the matched lines, but do not display the filenames.

9.  -i : Ignores, case for matching

10.-l : Displays list of a filenames only.

11.-n : Display the matched lines and their line numbers.

12.-v : This prints out all the lines that do not matches the pattern

13.-e exp : Specifies expression with this option. Can use multiple times.

14.-f file : Takes patterns from file, one per line.

15.-E : Treats pattern as an extended regular expression (ERE)

16.-w : Match whole word

17.-o : Print only the matched parts of a matching line,

18. with each such part on a separate output line.

19. Sample Commands

20. Consider the below file as an input.

21.  

22.

23.$cat > geekfile.txt

24.unix is great os. unix is opensource. unix is free os.

25.learn operating system.

26.Unix linux which one you choose.

27.uNix is easy to learn.unix is a multiuser os.Learn unix .unix is a powerful.

28. 1. Case insensitive search : The -i option enables to search for a string case insensitively in the give file. It matches the words like “UNIX”, “Unix”, “unix”.

29.$grep -i "UNix" geekfile.txt

30. Output:

31.unix is great os. unix is opensource. unix is free os.

32.Unix linux which one you choose.

33.uNix is easy to learn.unix is a multiuser os.Learn unix .unix is a powerful.

34. 2. Displaying the count of number of matches : We can find the number of lines that matches the given string/pattern

35.$grep -c "unix" geekfile.txt

36. Output:

37.2

38. 3. Display the file names that matches the pattern : We can just display the files that contains the given string/pattern.

39.$grep -l "unix" *

40. 

41.or

42.

43.$grep -l "unix" f1.txt f2.txt f3.xt f4.txt

44. Output:

45.geekfile.txt

46. 4. Checking for the whole words in a file : By default, grep matches the given string/pattern even if it found as a substring in a file. The -w option to grep makes it match only the whole words.

47.$ grep -w "unix" geekfile.txt

48. Output:

49.unix is great os. unix is opensource. unix is free os.

50.uNix is easy to learn.unix is a multiuser os.Learn unix .unix is a powerful.

51. 5. Displaying only the matched pattern : By default, grep displays the entire line which has the matched string. We can make the grep to display only the matched string by using the -o option.

52.$ grep -o "unix" geekfile.txt

53. Output:

54.  

55.

56.unix

57.unix

58.unix

59.unix

60.unix

61.unix

62.
6. Show line number while displaying the output using grep -n : To show the line number of file with the line matched.

63.$ grep -n "unix" geekfile.txt

64. Output:

65.1:unix is great os. unix is opensource. unix is free os.

66.4:uNix is easy to learn.unix is a multiuser os.Learn unix .unix is a powerful.

67. 7. Inverting the pattern match : You can display the lines that are not matched with the specified search sting pattern using the -v option.

68.$ grep -v "unix" geekfile.txt

69. Output:

70.learn operating system.

71.Unix linux which one you choose.

72. 8. Matching the lines that start with a string : The ^ regular expression pattern specifies the start of a line. This can be used in grep to match the lines which start with the given string or pattern.

73.$ grep "^unix" geekfile.txt

74. Output:

75.unix is great os. unix is opensource. unix is free os.

76. 9. Matching the lines that end with a string : The $ regular expression pattern specifies the end of a line. This can be used in grep to match the lines which end with the given string or pattern.

77.$ grep "os$" geekfile.txt

78. 10.Specifies expression with -e option. Can use multiple times :

79.$grep –e "Agarwal" –e "Aggarwal" –e "Agrawal" geekfile.txt

80. 11. -f file option Takes patterns from file, one per line.

81.$cat pattern.txt

82. 

83.Agarwal

84.Aggarwal

85.Agrawal

86.$grep –f pattern.txt  geekfile.txt

87. This article is contributed by Akshay Rajput. If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to contribute@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.

88. Please write comments if you find anything incorrect, or you want to share more information about the topic discussed above.

89.  To find out all the lines from the file big.txt located in /cxldata folder that contain a keyword "Aylmer", we can use the following command:

90. grep Aylmer /cxldata/big.txt

Use wc command

To find the number of characters, words, and lines, use wc command.

If you are only looking for number of lines you could use:

wc -l

wc command in Linux with examples

wc stands for word count. As the name implies, it is mainly used for counting purpose.

·         It is used to find out number of lines, word count, byte and characters count in the files specified in the file arguments.

·         By default it displays four-columnar output.

·         First column shows number of lines present in a file specified, second column shows number of words present in the file, third column shows number of characters present in file and fourth column itself is the file name which are given as argument.

Syntax:

wc [OPTION]... [FILE]...

Let us consider two files having name state.txt and capital.txt containing 5 names of the Indian states and capitals respectively.

 

$ cat state.txt

Andhra Pradesh

Arunachal Pradesh

Assam

Bihar

Chhattisgarh

 

$ cat capital.txt

Hyderabad

Itanagar

Dispur

Patna

Raipur

Passing only one file name in the argument.

$ wc state.txt

 5  7 63 state.txt

       OR

$ wc capital.txt

 5  5 45 capital.txt

Passing more than one file name in the argument.

$ wc state.txt capital.txt

  5   7  63 state.txt

  5   5  45 capital.txt

 10  12 108 total

Note : When more than file name is specified in argument then command will display four-columnar output for all individual files plus one extra row displaying total number of lines, words and characters of all the files specified in argument, followed by keyword total.

Options:

1. -l: This option prints the number of lines present in a file. With this option wc command displays two-columnar output, 1st column shows number of lines present in a file and 2nd itself represent the file name.

With one file name

$ wc -l state.txt

5 state.txt

 

With more than one file name

$ wc -l state.txt capital.txt

  5 state.txt

  5 capital.txt

 10 total

2. -w: This option prints the number of words present in a file. With this option wc command displays two-columnar output, 1st column shows number of words present in a file and 2nd is the file name.

With one file name

$ wc -w state.txt

7 state.txt

 

With more than one file name

$ wc -w state.txt capital.txt

  7 state.txt

  5 capital.txt

 12 total

3. -c: This option displays count of bytes present in a file. With this option it display two-columnar output, 1st column shows number of bytes present in a file and 2nd is the file name.

With one file name

$ wc -c state.txt

63 state.txt

 

With more than one file name

$ wc -c state.txt capital.txt

 63 state.txt

 45 capital.txt

108 total

4. -m: Using -m option ‘wc’ command displays count of characters from a file.

With one file name

$ wc -m state.txt

63 state.txt

 

With more than one file name

$ wc -m state.txt capital.txt

 63 state.txt

 45 capital.txt

108 total

5. -L: The ‘wc’ command allow an argument -L, it can be used to print out the length of longest (number of characters) line in a file. So, we have the longest character line Arunachal Pradesh in a file state.txt and Hyderabad in the file capital.txt. But with this option if more than one file name is specified then the last row i.e. the extra row, doesn’t display total but it display the maximum of all values displaying in the first column of individual files.
Note: A character is the smallest unit of information that includes space, tab and newline.

With one file name

$ wc -L state.txt

17 state.txt

 

With more than one file name

$ wc -L state.txt capital.txt

 17 state.txt

 10 capital.txt

 17 total

6. –version: This option is used to display the version of wc which is currently running on your system.

 

$ wc --version

wc (GNU coreutils) 8.26

Packaged by Cygwin (8.26-1)

Copyright (C) 2016 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later .

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

 

Written by Paul Rubin and David MacKenzie.

Applications of wc Command

1. To count all files and folders present in directory: As we all know ls command in unix is used to display all the files and folders present in the directory, when it is piped with wc command with -l option it display count of all files and folders present in current directory.

$ ls gfg

a.txt

b.txt 

c.txt 

d.txt 

e.txt 

geeksforgeeks 

India

 

$ ls gfg | wc -l

7

2. Display number of word count only of a file: We all know that this can be done with wc command having -w option, wc -w file_name, but this command shows two-columnar output one is count of words and other is file name.

$ wc -w  state.txt

7 state.txt

So to display 1st column only, pipe(|) output of wc -w command to cut command with -c option. Or use input redirection(<).

$ wc -w  state.txt | cut -c1

7

      OR

$ wc -w < state.txt

7

This article is contributed by Akash Gupta. If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to contribute@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.

Please write comments if you find anything incorrect, or you want to share more information about the topic discussed above.

Permissions - Overview

Permissions in Unix are an integral part of the operating system.

You can see the details of a file using ls -la command:

ls -la myfile

And you can see the details of all files in the current directory by using:

ls -la

Every file has an owner (also referred to as the user), a group and permissions. A user can be part of many groups. A group can have many users.

The permission attributes of a file signify who can do what.

Practical applications of ‘ls’ command in Linux

ls is a Linux shell command that lists directory contents of files and directories.Some practical examples of ls command are shown below.

1. Open Last Edited File Using ls -t

ls -t : It sorts the file by modification time, showing the last edited file first. head -1 picks up this first file.To open the last edited file in the current directory use the combination of ls and head commands as shown below.

 

 

[Note: This will open the last file you edited (i.e second.txt)]

2. Display One File Per Line Using ls -1

3. Display All Information About Files/Directories Using ls -l

$ ls -l : To show long listing information about the file/directory.

-rw-rw-r– 1 maverick maverick 1176 Feb 16 00:19 1.c

1st Character – File Type: First character specifies the type of the file.
In the example above the hyphen (-) in the 1st character indicates that this is a normal file. Following are the possible file type options in the 1st character of the ls -l output.
Field Explanation

·         – normal file

·         d : directory

·         s : socket file

·         l : link file

·         Field 1 – File Permissions: Next 9 character specifies the files permission. The every 3 characters specifies read, write, execute permissions for user(root), group and others respectively in order. Taking above example, -rw-rw-r– indicates read-write permission for user(root) , read permission for group, and no permission for others respectively. If all three permissions are given to user(root), group and others, the format looks like -rwxrwxrwx

·         Field 2 – Number of links: Second field specifies the number of links for that file. In this example, 1 indicates only one link to this file.

·         Field 3 – Owner: Third field specifies owner of the file. In this example, this file is owned by username ‘maverick’.

·         Field 4 – Group: Fourth field specifies the group of the file. In this example, this file belongs to ”maverick’ group.

·         Field 5 – Size: Fifth field specifies the size of file in bytes. In this example, ‘1176’ indicates the file size in bytes.

·         Field 6 – Last modified date and time: Sixth field specifies the date and time of the last modification of the file. In this example, ‘Feb 16 00:19’ specifies the last modification time of the file.

·         Field 7 – File name: The last field is the name of the file. In this example, the file name is 1.c.

4. Display File Size in Human Readable Format Using ls -lh
ls -lh (h stands for human readable form) : To display file size in easy to read format. i.e i.e M for MB, K for KB, G for GB.

5. Display Directory Information Using ls -ld

 

 

When you use “ls -l” you will get the details of directories content. But if you want the details of the directory then you can use -d option as., For example, if you use ls -l /etc will display all the files under the etc directory. But, if you want to display the information about the /etc/ directory, use -ld option as shown below.
$ ls -l /etc

$ ls -ld /etc

6. Order Files Based on Last Modified Time Using ls -lt

ls -lt : To sort the file names displayed in the order of last modification time.You will be finding it handy to use it in combination with -l option.

7. Order Files Based on Last Modified Time (In Reverse Order) Using ls -ltr

$ ls -ltr : To sort the file names in the last modification time in reverse order. This will be showing the last edited file in the last line which will be handy when the listing goes beyond a page.

8. Display Hidden Files Using ls -a (or) ls -A

$ ls -a : To show all the hidden files in the directory, use ‘-a option’. Hidden files in Unix starts with ‘.’ in its file name.It will show all the files including the ‘.’ (current directory) and ‘..’ (parent directory).

 

 

$ ls -A : To show the hidden files, but not the ‘.’ (current directory) and ‘..’ (parent directory).

[Note: . and .. are not displayed here]

9. Display Files Recursively Using ls -R

$ ls /etc/apt

$ ls -R /etc/apt : To show all the files recursively. When you do this from /, it shows all the unhidden files in the whole file system recursively.

10. Display File Inode Number Using ls -i

Sometimes you may want to know the inone number of a file for internal maintenance. Use -i option as shown below to display inone number. Using inode number you can remove files that has special characters in it’s name.

$ ls -i

 

 

$ ls -i /etc/apt

11. Hide Control Characters Using ls -q

ls -q : To print question mark instead of the non graphics control characters.

12. Display File UID and GID Using ls -n

$ ls -n ~/kv : Lists the output like -l, but shows the uid and gid in numeric format instead of names.

13. Visual Classification of Files With Special Characters Using ls -F

$ ls -F : Instead of doing the ‘ls -l’ and then the checking for the first character to determine the type of file. You can use -F which classifies the file with different special character for different kind of files.

·         / – directory.

·         nothing – normal file.

·         @ – link file.

·         * – Executable file

 

 

14. Visual Classification of Files With Colors Using ls -F

$ ls –color=auto : Recognizing the file type by the color in which it gets displayed is an another kind in classification of file. In the below output directories get displayed in blue, soft links get displayed in green, and ordinary files gets displayed in default color.

Permissions - Using chmod To Change

You can change the permissions of a file using chmod command: chmod permission_cmd myfile

You can allow or disallow the user (u), group (g) or other(o) the following actions: read (r), write (w) and execute (x).

So, if you want to allow the user (the person who owns it) to execute the file:

chmod u+x myfile

And to disallow the user (the person who owns it) to execute the file:

chmod u-x myfile

Say you want to give the rw (read & write) permissions to owner and group of a file, you will have to use the following command:

chmod u+r,u+w,g+r,g+w myfile

    or

chmod u+rw,g+rw myfile

To give members of a group ability to modify a file, use:

chmod g+w myfile

chmod command in Linux with examples

In Unix-like operating systems, the chmod command is used to change the access mode of a file.
The name is an abbreviation of change mode.

Syntax :

chmod [reference][operator][mode] file...

The references are used to distinguish the users to whom the permissions apply i.e. they are list of letters that specifies whom to give permissions. The references are represented by one or more of the following letters:

 

Reference   Class     Description

u          owner      file's owner

 

g          group      users who are members of

                      the file's group

 

o          others     users who are neither the

                      file's owner nor members of

                      the file's group

 

a          all       All three of the above, same as ugo

The operator is used to specify how the modes of a file should be adjusted. The following operators are accepted:

Operator  Description

+         Adds the specified modes to the

          specified classes

 

-         Removes the specified modes from

          the specified classes

 

=         The modes specified are to be made

          the exact modes for the specified

          classes

Note : Putting blank space(s) around operator would make the command fail.

The modes indicate which permissions are to be granted or removed from the specified classes. There are three basic modes which correspond to the basic permissions:

r       Permission to read the file.

w       Permission to write (or delete) the file.

x       Permission to execute the file, or, in

        the case of a directory, search it.

Types of permissions which we will be changing using chmod command :
In linux terminal, to see all the permissions to different files, type ls -l command which lists the files in the working directory in long format. The figure below shows an example to use ls -l and its output :

Let us take a look at above figure. To make things easy to understand some columns and rows are eliminated and extra spaces are added to the permissions column to make it easier to read as shown below:

- rw- rw- r--  mik  mik    assgn1_client.c

- rw- rw- r--  mik  mik    assgn1_server.c

d rwx rwx r-x  mik  mik    EXAM

- rw- rw- r--  mik  mik    raw.c

- rwx r-x r-x  mik  mik    header.sh

... so on...

·         The very first column represents the type of the file i.e. is it a normal file or a
directory where d represents a directory and – represents a normal file.

·         The first set three letters after the file type tell what the Owner of the file, have permissions to do. For example: In assgn1_client.c, has owner’s permission as rw-, which means the owner mik can only read(r) and write(w) the file but cannot execute(x).

·         Note: The 3rd and 4th columns represents the name of the owner of the file and the group to which the owner belongs respectively.

·         The next three letters after the user’s permission are the group’s permissions.
For example: header.sh has group permissions as r-x, which means Other people in the mik group can not write(w) the header.sh script but can only read(r) or execute(x) it.

·         Note that when a directory has the x set, this takes the special meaning of “permitted to search this directory”.

·         The last three letters in the permissions column tell us what the “others” may do. The general practice is to protect the files from external access so that others can’t write any files or directories. They may read(r) or execute(x) it. For example: The assgn1_client.c has others permission as r- – which means it can only be read by other(external) access but cannot be written or executed by them.

Now, let us see how chmod command can be used to change the access mode of a file.

Example 1 :
Let’s change the assgn1_client.c permission so that the owner cannot write(w) in the file but can only read it.

BEFORE: -rw-rw-r--  mik  mik  assgn1_client.c

 

COMMAND: chmod u=r assgn1_client.c

 

AFTER: -r--rw-r--  mik   mik   assgn1_client.c

Before :

After :

Example 2 :
Let’s restrict the permission such that the user cannot search the directory EXAM.

BEFORE: drwxrwxr-x  mik   mik   EXAM

 

COMMAND: chmod u=rw EXAM

 

AFTER: drw-rwxr-x   mik  mik    EXAM

After applying the chmod u=rw EXAM command, the user (owner) cannot change the directory. If the user tries to change the directory, then it shows the message “Permission denied” as shown in the figure below :
Permissions - Numeric

You can also use numbers to represent the permissions.

4 is for Read, 2 is for Write and 1 is for Execute permission.

The following diagram shows if the user has all three permissions rwx and group has only read and execute permission and others have only read permissions, the rwxr-xr-- can be represented by 754.

So, for a file with permissions rwxr-xr-x, the owner has rwx permission which 4+2+1=7 and group and others have r-x which means 4+0+1 = 5. Thus the command you would use to set such permission would be: chmod 755 myfile.

Permissions - Advanced

There is a special user called root on Unix systems which has all the privileges which can permit any user on any file.

Some users who are allowed to act on behalf of the root are called sudoers. This list of sudoers can be edited using sudoedit command. Such users are allowed to run commands as if they are root using: sudo .

You can change the owner of a file using chown and change the group of a file using chgrp command. Please note that changing the owner of the file is possible only with an administrative account.

Process

Every program or command is a sequence of instructions stored as a file. You run it on the shell or by any other means such as double clicking it from user interface.
When it is run, the content of the program is read from file and loaded into the memory and then instructions are executed by operating system.
All the commands that we have been using like ls, cat are program.

Find information about processes

While a program is running, it is called a process. A process is uniquely identified by PID - process id. To find out the information about all the processes, you can use the following commands: pstree, top, ps.

ps lists the processes of the system. Without any arguments, it lists only your processes. To see all of the processes by all users, run

ps aux

The first column mentions the user id who has started the process.

To continuously monitor all processes, use the top command. This command is usually used to monitor which processes are running, taking up most CPU or Memory.

To learn more about command use man command name. So, please go through manual using command

man ps

 

man top

Processes in Linux/Unix

A program/command when executed, a special instance is provided by the system to the process. This instance consists of all the services/resources that may be utilized by the process under execution.

·         Whenever a command is issued in unix/linux, it creates/starts a new process. For example, pwd when issued which is used to list the current directory location the user is in, a process starts.

·         Through a 5 digit ID number unix/linux keeps account of the processes, this number is call process id or pid. Each process in the system has a unique pid.

·         Used up pid’s can be used in again for a newer process since all the possible combinations are used.

·         At any point of time, no two processes with the same pid exist in the system because it is the pid that Unix uses to track each process.

Initializing a process

A process can be run in two ways:

 

1.    Foreground Process : Every process when started runs in foreground by default, receives input from the keyboard and sends output to the screen.
When issuing pwd command

2.  $ ls pwd

Output:

$ /home/geeksforgeeks/root

When a command/process is running in the foreground and is taking a lot of time, no other processes can be run or started because the prompt would not be available until the program finishes processing and comes out.

3.    Backround Process : It runs in the background without keyboard input and waits till keyboard input is required. Thus, other processes can be done in parallel with the process running in background since they do not have to wait for the previous process to be completed.
Adding & along with the command starts it as a background process

4.   $ pwd &

Since pwd does not wants any input from the keyboard, it goes to the stop state until moved to the foreground and given any data input. Thus, on pressing Enter, :
Output:

[1]   +   Done                 pwd

$

That first line contains information about the background process – the job number and the process ID. It tells you that the ls command background process finishes successfully. The se The second is a prompt for another command.

Tracking ongoing processes

ps (Process status) can be used to see/list all the running processes.

$ ps

 

PID       TTY      TIME        CMD

19        pts/1    00:00:00    sh

24        pts/1    00:00:00    ps

For more information -f (full) can be used along with ps

 

$ ps –f

 

UID      PID  PPID C STIME    TTY        TIME CMD

52471     19     1 0 07:20    pts/1  00:00:00f     sh

52471     25    19 0 08:04    pts/1  00:00:00      ps -f

For a single process information, ps along with process id is used

$ ps 19

 

PID       TTY      TIME        CMD

19        pts/1    00:00:00    sh

For a running program (named process) Pidof finds the process id’s (pids)

Fields described by ps are described as:
UID: User ID that this process belongs to (the person running it)
PID: Process ID
PPID: Parent process ID (the ID of the process that started it)
C: CPU utilization of process
STIME: Process start time
TTY: Terminal type associated with the process
TIME: CPU time taken by the process
CMD: The command that started this process

There are other options which can be used along with ps command :
-a: Shows information about all users
-x: Shows information about processes without terminals
-u: Shows additional information like -f option
-e: Displays extended information

Stopping a process

When running in foreground, hitting Ctrl + c (interrupt character) will exit the command. For processes running in background kill command can be used if it’s pid is known.

$ ps –f

 

UID      PID  PPID C STIME    TTY        TIME CMD

52471     19     1 0 07:20    pts/1  00:00:00      sh

52471     25    19 0 08:04    pts/1  00:00:00      ps –f

 

$ kill 19

Terminated

If a process ignores a regular kill command, you can use kill -9 followed by the process ID .

$ kill -9 19

Terminated

Other process commands:

bg: A job control command that resumes suspended jobs while keeping them running in the background
Syntax:

bg [ job ]

For example

 

bg %19

fg: It continues a stopped job by running it in the foreground.
Syntax:

fg [ %job_id ]

For example

fg 19

top: This command is used to show all the running processes within the working environment of Linux.
Syntax:

top

nice: It starts a new process (job) and assigns it a priority (nice) value at the same time.
Syntax:

nice [-nice value]

nice value ranges from -20 to 19, where -20 is of the highest priority.

renice : To change the priority of an already running process renice is used.
Syntax:

renice [-nice value] [process id]

df: It shows the amount of available disk space being used by file systems
Syntax:

df

Output:

Filesystem     1K-blocks      Used Available Use% Mounted on

/dev/loop0      18761008  15246876   2554440  86% /

none                   4         0         4   0% /sys/fs/cgroup

udev              493812         4    493808   1% /dev

tmpfs             100672      1364     99308   2% /run

none                5120         0      5120   0% /run/lock

none              503352      1764    501588   1% /run/shm

none              102400        20    102380   1% /run/user

/dev/sda3      174766076 164417964  10348112  95% /host

free: It shows the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel
Syntax:

free

Output:

             total       used       free     shared    buffers     cached

Mem:       1006708     935872      70836          0     148244     346656

-/+ buffers/cache:     440972     565736

Swap:       262140     130084     132056

Types of Processes

1.    Parent and Child process : The 2nd and 3rd column of the ps –f command shows process id and parent’s process id number. For each user process there’s a parent process in the system, with most of the commands having shell as their parent.

2.    Zombie and Orphan process : After completing its execution a child process is terminated or killed and SIGCHLD updates the parent process about the termination and thus can continue the task assigned to it. But at times when the parent process is killed before the termination of the child process, the child processes becomes orphan processes, with the parent of all processes “init” process, becomes their new ppid.
A process which is killed but still shows its entry in the process status or the process table is called a zombie process, they are dead and are not used.

3.    Daemon process : They are system-related background processes that often run with the permissions of root and services requests from other processes, they most of the time run in the background and wait for processes it can work along with for ex print daemon.
When ps –ef is executed, the process with ? in the tty field are daemon processes

4. Background Processes

So far whenever we ran a command, if it is taking time, we had to wait till it finishes before we could type another command. If a process is taking the time and does not require input from you, you would like to run it in the background.

To run a program in the background, put an & at the end of the command:

mycmd &

 

 

To kill any running process, please use the command:

5.  kill processid

6.    Basically, kill lets you send a signal to the running process. For example, to send the terminate signal, you can use

7.  kill -9 processid

kill command in Linux with Examples

kill command in Linux (located in /bin/kill), is a built-in command which is used to terminate processes manually. kill command sends a signal to a process which terminates the process. If the user doesn’t specify any signal which is to be sent along with kill command then default TERM signal is sent that terminates the process.

Options and examples

1. kill -l :To display all the available signals you can use below command option:

Syntax:

 

$kill -l

Signals can be specified in three ways:

·         By number (e.g. -5)

·         With SIG prefix (e.g. -SIGkill)

·         Without SIG prefix (e.g. -kill)

Note:

·         Negative PID values are used to indicate the process group ID. If you pass a process group ID then all the process within that group will receive the signal.

·         A PID of -1 is very special as it indicates all the processes except kill and init, which is the parent process of all processes on the system.

·         To display a list of running processes use the command ps and this will show you running processes with their PID number. To specify which process should receive the kill signal we need to provide the PID.

Syntax:

$ps

2. kill pid : To show how to use a PID with the kill command.

Syntax:

$kill pid

3. kill -s : To show how to send signal to processes.

Syntax:

kill {-signal | -s signal} pid 

4. kill -L :This command is used to list available signals in a table format.

Syntax:

kill {-l | --list[=signal] | -L | --table} 

In Unix, a job is a group of one or more processes. One of the ways of creating job is when we put a process in the background.

You can see the list of jobs by using the command:

    jobs

The number that you see in square brackets [] is the job id. This is different from the process id. To bring a process to foreground, you can use

    fg %jobid

or simply

    fg jobid

If jobs command lists following:

        [1]-  Stopped         tail -f mycmd.sh

        [2]+  Stopped         top

        [3]   Running         sleep 1000 &

fg command in Linux with examples

fg command in linux used to put a background job in foreground.

Syntax:

fg [job_spec]

job_spec may be:

 

%n : Refer to job number n.
%str : Refer to a job which was started by a command beginning with str.
%?str : Refer to a job which was started by a command containing str.
%% or %+ : Refer to the current job. fg and bg will operate on this job if no job_spec is given.
%- : Refer to the previous job.

Options for fg command :

·         fg [JOB_SPEC] : This command is used to put the mentioned job running inbackground to foreground.

“sleep 500” is a command which is used to create a dummy job which runs for 500 seconds.

More - Interacting with processes

To send a running process to background, first press Ctrl+z to suspend it and then type bg to send the suspended process to background.

When you exit the shell or disconnect, the processes you were running get killed sometimes. To keep a process running in the background even if the shell has been disconnected, use

    nohup

or

    screen

bg command in Linux with Examples

bg command in linux is used to place foreground jobs in background.

Syntax:

bg [job_spec ...]

job_spec may be:

 

%n : Refer to job number n.
%str : Refer to a job which was started by a command beginning with str.
%?str : Refer to a job which was started by a command containing str.
%% or %+ : Refer to the current job. fg and bg will operate on this job if no job_spec is given.
%- : Refer to the previous job.

Options for bg command:

·         bg [JOB_SPEC] : This command is used to put the mentioned job in background. In the below screenshot, we do following :

1.    We use jobs command to list all jobs

2.    We create a process using sleep command, we get its ID as 1.

3.    We put it in background by providing its ID to bg.

‘sleep 500’ is used to create dummy foreground job.

 

Process hierarchy

A process (parent) can execute another process (child). When a parent process is killed all the child processes are automatically assigned to the main system process called init. The init process is the first process that is started on your computer and is numbered as 1. If this process is killed the system will shutdown.

To see the tree of processes, you can use the command

    pstree

A user can kill only the processes that the user has created not the processes of other users. Only root can kill the processes of other users.

Writing first shell script

A shell script is a file containing a list of commands. Let's create a simple command that prints two words:

1. Open a text editor to create a file myfirstscript.sh:

2.  nano myfirstscript.sh

3. Write the following into the editor:

4.  #!/bin/bash

5.  name=linux

6.  echo "hello $name world"

Note: In Unix, the extension doesn't dictate the program to be used while executing a script. It is the first line of the script that would dictate which program to use. In the example above, the program is /bin/bash which is a Unix shell.

1. Press Ctrl +x to save and then y and enter key to exit
2. Now, by default, it would not have executable permission. You can make it executable like this:

3.  chmod +x myfirstscript.sh

4. To run the script, use:

./myfirstscript.sh

Program returns value and arguments

Every program returns a value to the operating system. It is also referred to as the exit status. In Linux, a program returns 0 if successful. Otherwise a non-zero error number.

To check the return value of the previous command you can check the value of a special variable ?

    echo $?

Networking: Sockets and ports

There are some programs such as a web server that need to keep running all the time. Such programs are called services.

You can communicate with such programs even from another computer. These programs or processes bind themselves to a port. In other words, such programs listen on a port. No two programs can listen on the same port. Every computer is a given an IP address which is unique in the network so that other computers or users on other computers can connect to such systems.

For example, a web server listens on 80 port and an email server listens on 25 port.

You can connect and talk to any service using nc command in the following way:

    nc computer_name(or ip_address) port_number

The way a person refers to himself as "I", a computer refers itself as localhost or IP Address 127.0.0.1. it is also called as a loopback address.

Files & Directories: linking

If you want to give multiple names to a single file without copying the content, you can create the link.

1. Create a file:

2.  nano orig_text

3. Put the following text in it:

4.  This is my valuable data

5. Save and Exit: Press CTRL+x and y
6. Check if the file has right data:

7.  cat orig_text

8. Create Link:

9.  ln orig_text mylink

10. Check if it is there:

ls -l mylink

Files & Directories - Symbolic links

A symbolic link points to a file. In case, the original file is deleted, the symbolic link would be pointing to non-existing file.

You can create a symbolic link to a directory too.

1. Create a file:

2.  nano orig_text1

3. Put the following text in it:

4.  This is my not-so-valuable data

5. Save and Exit: Press Ctrl+x and y
6. Check if the file has right data:

7.  cat orig_text1

8. Create Link:

9.  ln -s orig_text1 myslink

10. Check if it is there:

11. ls -l myslink

You should see something like this

lrwxrwxrwx 1 sandeepgiri9034 sandeepgiri9034 10 Dec 14 11:45 myslink -> orig_text1

12. Check the contents of myslink using:

cat myslink

readlink command in Linux with Examples

readlink command in Linux is used to print resolved symbolic links or canonical file names. In simple words whenever we have a symbolic link and we want to know what path it represents. Then, in that case, the readlink command comes into play to show the actual path of the symbolic link.

Syntax:

readlink [OPTION]... FILE...

Example: It will print the print resolved symbolic links or canonical file names of the symbolic link passed with the command as shown below.

 

 

Explanation: As you can see in the above example we have a symbolic link with the name of the desk with the help of readlink command we will be able to identify its actual path.

Options:

·         readlink -f: This option canonicalize by following every symlink in every component of the given name recursively; all but the last component must exist.

Example:

readlink -f desk1

·         readlink -e: This option will canonicalize by following every symlink in every component of the given name recursively, all components must exist.

Example:

readlink -e desk

Explanation: This option is similar to -f option the only difference between -f and -e option is in -e, all components must exist and in -f, the last component must exist.

·         readlink -m : This option canonicalize by following every symlink in every component of the given name recursively, without requirements on components existence.

Example:

readlink -m desk3

·         readlink -n : This option will do not output the trailing delimiter.

Example :

readlink -n desk4

·         readlink -q: This option will execute in quiet mode.

Example:

readlink -q desk

Explanation: With the help of this option the user can read the symbolic links in every component and nothing like errors is being displayed on the console.

·         readlink -s : This option will suppress most error messages.

Example:

readlink -s desk5

·         readlink -v : This option will report error messages if any.

Example:

readlink -v desk6

·         readlink -z : This option will end each output line with NUL, not newline.

Example:

readlink -z desk2

·         readlink –help : This option will display this help and exit.

readlink --help

·         readlink –version : This option will show the version information and exit.

readlink --version

Like pointers in any programming languages, links in Linux are pointers pointing to a file or a directory. Creating links is a kind of shortcuts to access a file. Links allow more than one file name to refer to the same file, elsewhere. There are two types of links :

1. Soft Link or Symbolic links
2. Hard Links

These links behave differently when the source of the link (what is being linked to) is moved or removed. Symbolic links are not updated (they merely contain a string which is the pathname of its target); hard links always refer to the source, even if moved or removed.

For example, if we have a file a.txt. If we create a hard link to the file and then delete the file, we can still access the file using hard link. But if we create a soft link of the file and then delete the file, we can’t access the file through soft link and soft link becomes dangling. Basically hard link increases reference count of a location while soft links work as a shortcut (like in Windows)

Chaining Unix Commands

Say, you want to execute a command only if another command is successful then you use &&.

And if you want a command to be executed if another command has failed, you use ||.

1. Create a file tea_ready:

2.  touch tea_ready

3. The following command would print "Tea is ready" if tea_ready file exists:

4.  ls -l tea_ready && echo "Tea is ready"

5. Delete the file tea_ready:

6.  rm tea_ready

7. Check the command from #2 again:

8.  ls -l tea_ready && echo "Tea is ready"

9. The following will print "Tea is not ready":

10. ls -l tea_ready || echo "Tea is not ready"

11. You can use brackets to avoid confusion:

(ls -l tea_ready && echo "Tea is ready") || echo "Tea is not ready"

Redirecting the output of a program

The output of a program can be saved to a file:

    myprogram > myfile

If myfile does not exist, it will be created. If it exists it will be overwritten.

INSTRUCTIONS

Please follow these steps:

1.    Run the following command to save the output of echo to a file:

2.  echo "hello" > hello.out

3.    Append world to it:

4.  echo "world" >> hello.out

5.    Check by typing:

cat hello.out

 

 

Pipes - Introduction

If we want to send the output of one program to another, we can use the pipe. A pipe is denoted by |.

echo command prints on the standard output whatever argument is passed to it.

echo "Hi"

wc command prints the number of characters, words, and lines out of whatever you type on standard input. Start wc command, type some text and press Ctrl+d to end the input:

wc

hi

how are you

[CTRL + d]

Output-

        2       4      15

Now, if we want to count the number of words, characters in the output of any program, we can pipe the output in the following way:

        echo "Hello, World" | wc

You can also save the results using redirection, in the following way:

        echo "Hello, World" | wc > wc_results

Please execute the above command to succeed in the assessment.

Filters

Pipes are very powerful. They can be chained to solve many problems. So, most of the commands are built in a way that they can be used with pipes as long as they read from standard input (keyboard or pipe) and write to standard output (screen or pipe )

Such programs that can be used with pipes are generally called filters. Command examples of filters are:

wc - for counting the letters, words, and lines in the input

grep - displays only the lines from the input in which keyword (which is passed as argument) is found.

sort - sorts/orders the input lines lexically (alphabetically) by default but can be changed

more - displays the input in a page-wise manner

cat - displays the content of the file passed as an argument

sed - substitute a word with another word: sed 's/word/another_word/'

tr - translate character ranges. For example to lowercase characters in input you can use:

    tr 'A-Z' 'a-z'

uniq - Display the uniq input lines. The input lines need to be sorted. If you want to display frequency, use:

    uniq -c

Word Count Exercise

Step 1:

Check the Data using the cat command. Since the file is big, you can use more to see pagewise

    cat /cxldata/big.txt | more

Step 2:

Replace space with newline such that every line in output contains only single word:

    cat /cxldata/big.txt | sed 's/ /\n/g' |more

For example, after replacing space with a new line in "I am ok" we should get:

I

am

ok

The "/g" is an option of sed which makes replace all occurrences of space instead of only one.

Also, note this command has three programs connected by two pipes. The output of the cat is going to sed and the output of sed is going to more to see the pagewise.

Step 3:

We can sort the words using sort command in the following way

    cat /cxldata/big.txt | sed 's/ /\n/g' | sort|more

Note that we are using the more command just to avoid screen-blindness (too much text scrolling).

Step 4:

We can now, count the words using uniq command

    cat /cxldata/big.txt | sed 's/ /\n/g' | sort|uniq -c|more

Please save the result of the command to a file word_count_results in your home directory

    cat /cxldata/big.txt | sed 's/ /\n/g' | sort|uniq -c > word_count_results

Improved Word Count Using Unix Commands

We can further improve the word frequency count by using more filters.

Improvement 1:

Translate to lower case using

    tr 'A-Z' 'a-z'

Improvement 2:

Remove non-alphanumeric characters using sed with regular expression:

    sed 's/[^0-9a-z]//g'

Improvement 3:

Replace all whitespace (multiple tabs and spaces):

    sed -E 's/[ \t]+/\n/g'

Please note that since we are using regular expressions, we need to specify -E

Improvement 4:

Display most frequent at the top or display the results in reverse numeric sorting:

    sort -nr

Improvement 5:

If the input file is big, the sort command might use too much memory. So, you can force sort command to use less memory say 100 MB:

    sort -S 50M

After all of these improvements, please save the results

    cat /cxldata/big.txt |tr 'A-Z' 'a-z'| sed -E 's/[ \t]+/\n/g'|sed 's/[^0-9a-z]//g' | sort|uniq -c|sort -nr -S 50M > word_count_results_nice

Shell script for WordCount

A shell script is a file which contains the commands separated by a newline.

INSTRUCTIONS

Let's create a script to do the sorting of the data:

1.    Create a file using nano text editor:

2.  nano wordcountscript.sh

3.    The first line of a script should have #! followed by the name of the program to execute the script with. Since we are creating a shell script, we want it to be executed using bash. So, the first line of the program should be:

4.  #!/bin/bash

5.    Add the command in the editor:

6.  tr 'A-Z' 'a-z'| sed -E 's/[ \t]+/\n/g'|sed 's/[^0-9a-z]//g' | sort|uniq -c|sort -nr -S 50M

7.    Save the file by pressing Ctrl+x and y

8.    Now, make this file executable:

9.  chmod +x wordcountscript.sh

10.  Check if it is running:

11. cat /cxldata/big.txt | ./wordcountscript.sh | more

Please recall ./ means current directory and | more will show the result pagewise instead of causing too much scrolling.

Also, note that whatever is the input to the script is also passed to the programs executed in the script.

Usually, a program runs with the same permissions as the user who is running it. The program can read or modify only the files which user is allowed to.

A process or program is something that is running. A process is started by the user. The question is what is a process allowed to do? Can a process do something that the user is not allowed to do? If so, then the user will create programs which can do anything. Therefore, the process is allowed to do only the things a owner of process can do. And who is the owner, the user that started the process.

Permissions of Processes - setuid

In Unix, there is a file /etc/shadow that contains (one-way)encrypted passwords of every user. The user can not see the contents of the file. This is to defend the password cracking programs.

To change the password, the user needs to use the command: passwd. This passwd command first asks you for your old password and encrypts your input and compares it against the value in the file /etc/shadow. If it matches then it updates the password file /etc/shadow with new content.

When you are not allowed to view the /etc/shadow file, how can a program (passwd) do the same when run by you?

This is where the idea of special permission called setuid came into picture. A program file can be given setuid permission such that the program becomes the user who owns the program file instead of the user who is running it.

passwd command in Linux with Examples

passwd command in Linux is used to change the user account passwords. The root user reserves the privilege to change the password for any user on the system, while a normal user can only change the account password for his or her own account.

Syntax:

 passwd [options] [username]

Example:

 

Command: passwd

Command [root]: passwd user1

Note: sudo can be used to invoke root privileges by normal users, and can change the password for root itself. This is particularly helpful when a user is member of admin group (holds a position in sudoers list (/etc/sudoers) and can use commands with sudo) and the root password is not set, which is case with many common distributions of linux.

Command: sudo passwd root

Processing in passwd command:

1.    Verify current user password : Once the user enters passwd command, it prompts for current user password, which is verified against the password stored in /etc/shadow file user. The root user can bypass this step and can directly change the password, so as the forgotten passwords may be recovered.

2.    Verify password aging information : In Linux, a user password can be set to expire after a given period of time. Also, a user can be prohibited to change his/her password for a period. This password aging information (and the password itself) is stored in a file /etc/shadow.

3.    Change the password : After authentication, the user is prompted to enter the new password and verify it by retyping the password.

/etc/shadow file: The shadow file is a list of colon separated values with 9 fields, as shown below:

user1:$6$x8wAJRpP$EWC97sXW5tqac10Q2TQyXkR.1l1jdK4VLK1pkZKmA2mbA6UnSGyo94Pis074viWBA3sVbkCptSZzuP2K.y.an/:17887:0:99999:7:::

·         field 1: User name.

·         field 2: Encrypted Password.

·         field 3: Number of days since January 1, 1970 to when the password was last changed.

·         field 4: Minimum number of days for which password can not be changed. (value 0 means it can be changed anytime).

·         field 5: Number of days after password must be changed. (value 99999 means that the password never expires).

·         field 6: Number of days to warn user for expiring password.

·         field 7: Number of days after password expires that the account is disabled.

·         field 8: The number of days from January 1, 1970 to the date when an account was disabled.

·         field 9: This field is reserved for some possible future use.

passwd options:

·         -d, –delete: This option deletes the user password and makes the account password-less.

·         -e, –expire: This option immediately expires the account password and forces the user to change password on their next login.

·         -h, –help: Display help related to the passwd command.

·         -i, –inactive INACTIVE_DAYS: This option is followed by an integer, INACTIVE_DAYS, which is the number of days after the password expires that the account will be deactivated.

example: passwd -i 3 user1

·         -k, –keep-tokens: This option is used when you only want to change the password if it is expired. It keeps the authentication tokens for the authentication if the password is not yet expired, even if you requested to change it. Note that if the expiry period for a user is set to 99999, then this option will not keep tokens and the password will be changed.

·         -l, –lock: Lock the password of user. This appends the encrypted password of the user with a character ‘!’, and thus making it unable to match with any of input password combinations. This does not disable the account but prevents the user from logging in using a password. Though other authentication methods like ssh keys can be used to login to the account.

·         -n, –mindays MIN_DAYS: Change the minimum number of days between password changes to MIN_DAYS so that the user can’t change the password for MIN_DAYS.

·         -q, –quiet: This option is used for quiet mode. While using this option to change a password, the message “Changing password for $user.”, which usually gets printed before changing a password, does not get echoed.

·         -r, –repository REPO: This option is used to change password for repository named “REPO”.

·         -R, –root CHROOT_DIR: Apply changes in the CHROOT_DIR directory and use the configuration files from the CHROOT_DIR directory. This basically changes the root directory for the passwd process for once, and since CHROOT_DIR is a sub-directory of the root, it can not access the configuration files outside the CHROOT_DIR.

·         -S, –status: Shows the password status (7 fields) of user in the following format:

user1 P 12/22/2018 0 99999 7 3

The first field is the user’s login name. The second field indicates if the user account has a locked password (L), has no Password (NP), or has a usable password (P). The third field gives the date of the last password change. The next four fields are the minimum age, maximum age, warning period, and inactivity period for the password. These ages are expressed in days.

-S [, –status] -a [, –all]: This combination of options shows password status for all users. Note that -a or –all cannot be used without -S option.

·         -u, –unlock: Unlock the password of an account.

·         -w, –warndays WARN_DAYS: This option is used to change the number of days before the password is to expire, to display the warning for expiring password.

·         -x, –maxdays MAX_DAYS Set the maximum number of days for which the password remains valid. After MAX_DAYS, the password will expire and the user will be forced to change password.

Permissions - setting setuid

You can make a program setuid by giving s instead of x permission. If you have written a script x.sh, an example would be:

chmod +s x.sh

1. Start creating file with the name whoownsit_username.sh in /tmp directory:

2.  nano /tmp/whoownsit_$USER.sh

Note: If your user name is sandeep1234, the filename would be whoownsit_sandeep1234.sh

3. Put the following content in the editor in the previous step:

4.  whoami

5. Save it by pressing Ctrl+x and press y
6. Give it setuid permission:

7.  chmod +sx /tmp/whoownsit_$USER.sh

8. Check if you have given correct permission by

9.  ls -l /tmp/whoownsit_$USER.sh

It should display something like this in permissions: rwsrwsr-x

Note: setuid doesn't work in shell scripts. Please see http://www.faqs.org/faqs/unix-faq/faq/part4/section-7.html

Special System commands

sudo

This command makes a program run as root (the system administrator). This command is only allowed to be used by few users. Such users are called sudoers. You can modify the sudoers using

sudo visudo

shutdown

This command makes the system shutdown. You can use the following command to shut down the system immediately

shutdown -h now

The alternative command to shutdown immediately is: halt

Restart the system

To restart the system, you can use the reboot command.

Please note that above commands (shutdown, halt, reboot) can only be run as root.

Where is my program?

To find where is your program located, you can use which command.

For example,

which java

would print /usr/bin/java which means java is a command in the directory /usr/bin.

To further find out, you can use:

ls -l /usr/bin/java

This would display:

lrwxrwxrwx 1 root root 22 May 18  2016 /usr/bin/java -> /etc/alternatives/java

It mean that /usr/bin/java is actually a link to /etc/alternatives/java

Let us try:

ls -l /etc/alternatives/java

It should display something like:

lrwxrwxrwx 1 root root 72 May 18  2016 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre/bin/java

Further, to find out about the content of a file, you can use file command:

file /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre/bin/java

This should display something like

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre/bin/java: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=df3fba6cc0f63b51a2270014a3994480680a8ca0, stripped

This means it is a Linux binary.

Environment variables

Unix shell provides environment variables. To see the entire list of environment variables, use:

set

These environment variables can be used by the shell, programs or commands.

For example, the $PS1 variable is used by the shell to display the prompt. To change your prompt, you can try:

PS1='xxx>>'

The environment variable $PATH is a list of directories separated by a colon. It is used by the shell to find the file corresponding to a command.

Setting Environment variables

You can set the environment variables simply by assignment: MYVAR=VAL

The following list of commands:

MYV=myfile

 

ls $MYV

are equivalent to the single command:

ls myfile