Section 1: The basic shit that linux guy that likes feeling superior doesn't want you to know
Linux Shell
The Linux shell is the place in which you enter commands as needed, shell is also above kernel used by the system for running programs and commands.
Types of Default Shells:
default interactive shell: This is a program a user uses to enter commands, run programs from command line, shell scripts, etc.
default system shell: This is used by the Linux system to run system shell scripts, normally runs at startup
/bin/sh is pointer to system's default system shell which is typically /bin/bash, however some distribution's /bin/sh points to another shell. An example of this is Ubuntu's /bin/sh points to /bin/dash by default.
Types of Default Shells:
default interactive shell - shell program a user uses to enter commands, run programs from command line, shell scripts, etc.
default system shell - used by the Linux system to run system shell scripts, normally at startup
/bin/sh is symlink to the system's default system shell, normally /bin/bash for linux. Some distributions /bin/sh points to another shell, such as /bin/dash to Ubuntu.
There are multiple default interactive shells in Linux, you may choose the one you want to use. Below is a list of common Linux shells:
- bash: This is the most common used shell for user accounts
- sh: The shell upon which bash is based
- tcsh: This shell is based on C shell (csh), a popular shell in some groups but never used by default, no environment variables are used with tcsh
- csh: original C shell, rarely used
- ksh: Korn shell, this shell is supposed to be the best of bash and csh, this shell has a small but dedicated following
- zsh: Z shell, This shell takes evolution further than Ksh and incorporates new features.
- Dozens of other shells exist, these six are just the most noteworthy/used
Login shells are shell programs that are launched automatically when you initiate a text-mode login as opposed to those that run in a terminal or xterm window, or any other terminal emulator.
Commands
System Detail Commands
-
uname (shows the operating system being run)
-
uname - a (gives more information from uname)
-
uptime (shows uptime)
File System Commands
Below are some basic file system commands and their usage:
- cd (change directory)
- pwd (prints working directory)
- echo (displays text entered)
- time pwd (tells how long the system took to execute the pwd command, shows total execution time, user CPU time, and system CPU time.)
- set (displays variety of options relating to bash shell operation, pass options to set to change shell operations.)
- chmod (change permissions of files and folders)
- exit (terminates the shell)
Internal and External Commands
Internal Commands are commands that are directly built into your linux distribution or the kernel itself. These command will run in any directory you're in just by typing in their name(such as man or less).
External Commands are commands that are not directly built into the kernel or linux distribution. These commands must either be added in an environmental variable or be ran from their root directory in order to be used. (such as /home/[user]/Downloads/totalallynotahack)
Example: I need to run echo. If it's a built in command I can just type in echo "text" and it will put this text on the screen. However if echo Isn't an internal command I would have to go to its directory(assume its /bin/echo) and type /bin/echo "text" to achieve this result.
If the external command is in your current directory use ./[filename] to run that file.
The type command is used to check if the command is built in or not, here is the use cases below:
- type (check if command is built in)
- type - a (check if there are multiple command locations.)
Shell Tools/Assistance
Command completion:
Command completion is built into most shells as a very nice way of helping users type commands.
To use it type part of a command or filename and press tab, the shell tries to fill in the rest. If there are many options the shell fills in what it can and stops, if there's only one it fills it in all the way. Avoids typos when trying to use command completion or you'll end up using the wrong command/file.
History
The shell keeps history of every command you type, this saves a lot of time say you need to run a command again or change the arguements for it. Up arrow on keyboard is used to scroll upward through your history on the terminal
Most shells have the ability to search for a command in the history, press ctrl+r to begin a backward search, begin typing characters that should be unique to the command, then keep typing until you find the correct command and press ctrl+r repeatedly until you find it. ctrl+s is used to search forward in command history following the same steps.
NOTE: Commands starting with a space are NOT saved in the history with bash.
Editing features:
- Use ctrl+a or e to move cursor to the start or end of the line respectively.
- Use ctrl+b and f move backwards and forwards within a line as well as the left and right arrow keys.
Use ctrl+left or right arrow moves back and forward in the line same as pressing esc then b or f
Using ctrl+d or delete key deletes the character under the cursor.
- The backspace key deletes character to the left of the cursor.
- Ctrl+k deletes all text from the cursor to the end of the line.
Pressing ctrl+x and then backspace deletes all the text from the cursor to beginning of line.
Pressing ctrl+T transposes the character before cursor with character under the cursor, using esc then t does the same thing.
Pressing esc then U converts text from the cursor to the end of the word to uppercase.
- Pressing Esc and then l converts text from cursor to end of word to lowercase.
Esc and then C converts the letter under the cursor to uppercase only.
-
Launch a full-fledged editor to edit a command by pressing ctrl+x followed by ctrl+e. Bash shell attempts to launch the editor from $FCEDIT or $EDITOR environmental variable.
You can run a command in history by typing ![history number]
Bash history will store password and is readable by anyone with permissions, try not to put passwords in bash prompt or it may be readable by others!!
-
Mkdir [foldername] (makes new empty folder)
-
Touch [filename] (creates empty file in current directory)
-
Ls (lists all files in current directory)
-
Ls -l (lists all files in directory with file permissions)
-
Less (displays text within a file)
-
More (displays text within a file)
Environment variables:
Environmental variables are variables that can be modified by multiple programs that may also have information that applies to a variety of programs. For example $TERM variable conveys the capabilities of the terminal program being used.
You can type echo $[environmentalvariable](PATH for example) to display the current setting
env (list all environment variables. Pipe it to less or more using | to make them all readable in terminal easily.)
unset [environmental variable] (deletes the specified environmental variable. BE VERY CAREFUL DOING THIS)
man -- (short for manual, provides summaries of what a command, file or other feature does.) example: Man [whatever file/progarm you want to learn about] By default man uses the less pager to display information.
- Use esc then V to move back a page
- Use arrow keys to move up or down a line at a time
- Use / key to search for text
- Q to exit less.
Change pager using -P. Example: man -P /bin/more uname
man -k "keyword" (search through man pages for command. Use good keyword choices!!)
Man pages are organized into sections. Programs can be in more than one type of category
Section Number Description
1 Executable programs and shell commands
2 System calls provided by the kernel
3 Library calls provided by the program libraries
4 Devices files(usually stored in /dev)
5 Files formats
6 Games
7 Miscellaneous(macro packages, conventions, and so on)
8 System administration commands(programs run mostly or
exclusively by root)
9 Kernel Routines
Some programs use info pages instead of man pages. info is similar to man but info uses hyptertext format so you can move from section to section of the documentation.
There is also built-in help pages called help:
- help [command]
man, info, and help pages are references, NOT TUTORIALS. These assume you know the command or at least understand linux well. Research the command if you have no understanding of said command.
Input and Output:
In Linux, input and output data is treated as a stream, which is a data entity that can be manipulated. You can redirect input and output streams from keyboard and monitor to go to other sources like files. You can pipe output from one program as input to another, great for putting together multiple programs.
File Descriptors:
All objects are handled as files in Linux, file descriptors are used to identify file objects.
- Standard Input: Programs take keyboard input via standard input, STDIN abbreviated, file descriptor is 0. most cases this is keyboard data.
- Standard Output: Text-mode programs mainly, STDOUT abbreviated. Standard output is normally on-screen text-mode session or terminal emulator like xterm. Descriptor is 1
- Standard Error: abbreviated STDERR, descriptor is 2, carries high-priority info like error messages. Normally sent to same output device as standard output, hard to tell apart. One can be redirected independently of the other, great for just logging errors and displaying only STDOUT to screen.
Use operators to redirect input and output, example: (redirect STDOUT of the echo command)
echo $Path 1>path.txt
This outputs the command to path.txt, operator used to perform this is > and file descriptor for STDOUT is 1
You don't have to use the file descriptor if you're redirecting STDOUT, but the > is still required.
You can also leave out STDIN file descriptor when using correct redirection operator.
Redirection Operators:
Redirection operator Effect
> Creates a new file containing standard output. If the
specified file exists, it's overwritten. No file
descriptor necessary.
>> Appends standard output to existing file. If the
specified file doesn't exist, it's created. No file
descriptor necessary.
2> Creates a new file containing standard error. If the
specified file exists, it's overwritten. File Descriptor
necessary.
2>> Append Standard Error to the existing file. If the
specified file doesn't exist it's created. File
Descriptor necessary.
&> Creates a new file containing both standard output and\
standard error. If the specified file exists, it's
overwritten. No file descriptors necessary.
< Sends the contents of the specified file to be used as
standard input. No file descriptor necessary.
<< Accepts text on the following lines as standard input.
No file descriptor necessary.
<> Causes the specified file to be used for both standard
input and standard output. No file descriptor necessary.
Most operators deal with output because there are two types of output.
< is the most important operator for input
SIDENOTE: Point output or errors to /dev/null to just get rid of the output. This is essentially a black hole.
<< operator implements a here document. This takes text from subsequent lines as standard input unlikely to be used on command line, mainly used in a script for interactive program. Normally used by programs to present the end of input or similar.
Programs frequently work based off another one's output. A great way of doing this is to send first program's STDOUT to second program's STDIN. This isn't efficient and leaves a lot of files on the system though, so a better way is to use data pipes (|). The first program has output and second program takes the output as input. This can be stacked for indefinite usage. Example: echo "Hello" | more
-
tee (allows program's output to be stored and viewed immediately.) Ex: echo $PATH | tee path.txt ;
tee -a (append data to existing text document).
xargs (Used to run commands in conjunction off of each other.) Ex: find / -user Nathan | xargs -d "\n" rm
This user command finds all files in the / directory tree owned by Nathan, then is passed to xargs which calls the rm command. -d "\n" tells xargs to only use newlines as delimiters to the next item
Backtick(the key above tab) is used to call another command whose results are substituted for what is typed in the command line, works well for simple situations but doesn't work well in complex ones.
-
cat (short for concatenate, great for combining files)Ex: cat first.txt second.txt > combined.txt
cat also displays the file if only one file is entered.
-
cat [filename] -E or cat --show-ends (puts $ at the end of each line)
-
cat -n or cat --number (add line numbers to the beginning of every line)
-
cat -b or ** cat --number-nonblank** (numbers lines that contain text only)
-
cat -s or cat --squeeze-blank (compresses blank lines to one blank line)
-
cat -T or cat --show-tabs (displays tabs as ^I)
cat -v or ** cat--show-nonprinting** (displays other characters using carat and M- notations.)
tac (opposite of cat, reverse output)
join (combines two files by matching contents of specified fields within files. Fields are space-separated entries on a line.)
-
join -t char (use a different character as the field separator.)
Example:
file1.txt:
111-1111 John, Smith
555-5555 Steve, John
999-9999 Johnson, Bob
file2.txt:
111-1111 unlisted
555-5555 listed
999-9999 listed
Join file1.txt file2.txt
You can use join -1 [field] -2 [field] [file1] [file2] to join different fields together!
join is a core of manipulation for databases using text, but limited because of formatting requirement
paste (merges files line by line, tabbing lines from each file to separate them)
Great for combining lines that are exactly lined up that don't have the format for join, great for two column data if both files are formatted properly
expand (converts tabs to spaces(yes some programs hate tabs))
-
expand -t num or expand --tabs=num (change the tab spacing value.)
For files that aren't feasible to use ASCII for use od(octal dump) to display files in a base 8(octal) format. The first field in each line is the index of where this line is located in the file. od can also use hex, decimal and ASCII with control characters that are escaped.
-
sort (command to sort output files.)
Ignore Case: sort -f or sort --ignore-case (causes sort to ignore case in sorting, typically sort differentiates between upper and lower case letters)
Month Sort: sort -M or sort --month-sort (causes program to sort by three-letter month abbreviations.)
Numeric Sort: sort -n or sort --numeric-sort (sort by number)
Reverse Sort Order: sort -r or sort --reverse (sorts in reverse order)
Sort Field: specify another field with sort -k filed or sort --key=field (field is the sort field)
split (requires output filename, splits one file into two or more)
Split by Bytes: split -b size or ** split --bytes=size** (breaks input file int pieces of size bytes. Option can split file mid-line)
Split by Bytes in Line-Sized Chunks: break file into files of no more than specified size, split -C=size or split --line-bytes=size
Split by Number of Lines: split -l lines or split --lines=lines (splits the file into chunks with no more tha specified number of lines.)
tr command changes characters from standard input. **tr [options] SET1 [SET2] **
Set1 is characters you want replaced, SET2 is characters you want to replace them with.
tr -t (truncates SET1 to SET2 size)
tr -d (causes program to delete characters from SET1; omit SET2)
unexpand (logical opposite of expand, converts multiple spaces to tabs.) unexpand -t num or --tabs=num (changes the spacing options)
uniq (removes duplicate lines, mainly good after sorting file.)
fmt (tries to clean up paragraphs and fix indentation, default width of 75 characters), can be changed by fmt -width, fmt -w width and fmt --width=width
nl (used to number lines)
Body Numbering Style: nl -b style or nl --body-numbering=style (where style is style format code)
Header and Footer Numbering Style: nl -h style or nl --header-numbering=style (option for header) and nl -f style or nl --footer-numbering=style (for footer)
Page Separator: reset line numbers for new page, identify new page by nl -d=code or nl --section-delimiter=code
nl -p or nl --no-renumber option it doesn't reset with new page
Numbering format with nl -n format or nl --number-format=format (where format is ln, rn, or rz.)
Style code Description
t The default behavior is to number lines that aren't empty. You can
make this default explicit by using a style code of t.
a This style code causes all lines to be numbered, including empty
lines.
n This style code causes all line numbers to be omitted, which may
be desirable for header or footers.
pREGEXP This option causes only lines that match the specified
regular expression(REGEXP) to be numbered.
Example: nl -b a unnumbered > all-numbered.txt
-pr (formats text with header, footer, line width etc so it's ready to be printed properly.)
pr can take output from another program as input, or can pipe its output to another program. By default pr assumes there is an 80-character line length. Words are mono-spaced. Default output has headers for current date and time, original filename, and page number.
pr -numcols or pr --columns=numcols (lines are truncated or run into multiple columns if they don't fit.)
pr -d or pr --double-space (creates double-spaced output from single-spaced file.)
pr -F pr -f or pr--form-feed (outputs a form-feed character between pages. Used for certain printers)
pr -l lines or pr --length=lines (change length of pages in lines.)
pr -h text or pr --header=text (replaces the filename in the header with user-specified text.) pr -t or **--omit-header ** (omits the entire header, use "" for multiple word headers.)
pr -o chars or pr --indent=chars (sets left margin to chars length. Margin size is added to page width, defaults to 72 characters.) -2 chars or --width chars (changes page width.)
-lpr (command used to print files)
Ex: cat -n /home/nathan/test/newtext.txt | pr -d | lpr (numbers files, double spaces file, then prints it)
-head (used to just display the first few lines of a file,) ** head -c num** or --bytes=num (tells head to display num bytes of the file rather than default 10 lines.) head -n num or --lines=num (display num number of lines.)
-tail (just like head but last 10 lines, uses the same -c --bytes, -n and --lines options.) Additional commands: tail -f or --follow (keeps the file open in tail and displays new lines as they're added. Great for log files.) tail --pid=pid (terminates tail when process with PID of pid terminates)
-less (supposedly a better version of more, read file a screen at a time. spacebar moves forward screen at a time, esc followed by v moves back a screen at a time, up and down does one line at a time, search using / followed by search term. n repeats search forward, N repeats it backwards. ? To search backwards in the file, g followed by line number to go to that line, q to exit. Less can ONLY be used as end of pipe due to formatting. H displays internal help system)
-cut (extracts portions of input lines and displays them.) cut -b list or --bytes=list (cuts specified list of bytes from input file,) cut -c or --characters=list option cuts the specified list of characters from input file. cut -f list or --fields=list (option cuts list number of fields from input file.) Field is tab-delimited by default but can be changed with cut -d char or --delim=char, or --delimited=char. cut -s or --only-delimited (changes behavior so only lines delimited are echoed. Frequently used in scripts to extract data from another output.)
-wc (produces word count, as well as line and byte count.) Limit output to: newline count(wc --lines or wc -l), word count(wc --words or wc -w), byte count(wc --bytes or wc -c) or character count(wc --chars or wc -m). Maximum line length(wc --max-line-length or wc -L).
Regular expressions:
Regular expressions are tools for describing or matching patterns in text. Used like plaintext but certain characters show patterns. Extremely important for anything computer-wise.
Explaining regular expressions:
Regular expressions take two forms -- basic and extended. The program you use determines which type is usable, and you pass an option as to what type is to be used if there is more than one type.
"HWaddr" or "Linux" could be a regular expression for an alphabetic string that is the same size or longer which contains that specific regular expression. Example: a regular expression with the string "Linux" matches "Linux", "Ubuntu is Linux", and "Linux me up baby". It does not, however match "Kai is love, Kai is life".
More advanced forms of regular expressions:
- Bracket Expressions: characters in [] brackets, the expression matches any one character within the bracket, example: regular expression b[aeiou]gs matches "bags", "bugs", "begs", "bogs", and "bigs"
- Range Expressions: used with bracket, start and end points separated by a "-" ex: regular expression y[4-7]g can be "y4g" "y5g" "y6g" or "y7g"
- Any Single Character: uses a ., the dot represents any single character except new line. Ex: regular expression y.g can be "yag" "ybg" "ycg", so on and so forth
- Start and End of Line: carat represents start of line, $ represents end of line
- Repetition Operators: how many times an item must exist, * means zero or more occurrences, + sigh for one or more, ? Means zero or one match
- Multiple Possible Strings: | separates two possible matches, Bob|Steve matches either "Bob" or "Steve"
- Parentheses: () surround subexpressions
- Escaping: "\" is used if you want to check for a character that is also a regex operator
-grep -- searches for files that contain specified string and return file name(if text file returns a line of context). Basic syntax is grep [options] regex [files]. grep -c or grep --count (changes number of lines to return,) grep -f file or **grep --file=file (takes input from file instead of terminal/command line.) grep -f or grep --ignore-case (ignores case when searching.) grep -r or grep --recursive (searches all directories and subdirectories not just the specified ones.) rgrep does the same thing. grep -F or grep --fixed-strings (turn off regular expressions, or use fgrep.) grep -E or grep --extended-regexp to use extended regular expressions. Grep can also be used to search through output for material that's important.
-
sed: directly modifies a file's contents, sends changed file to standard output. Two synaxes for files: sed [options] -f script-file [input-file] or sed [options] script-text [input-file]
Command Addresses Meaning
= 0 or 1 Display current line number
a\text 0 or 1 Append text to the file
i\text 0 or 1 Insert text into file
r filename 0 or 1 Append text from filename into the file
c\text range Replace the selected range of lines with
the provided text
s/regex/replacement range Replaces text that matches the regular
expression with replacement
W filename range Write the current patter space to the
specified file
q 0 or 1 Immediately quit the script but print the
current pattern space
Q 0 or 1 Immediately quit the script.
sed works based off addresses, aka line numbers. Commands can operate on no addresses(entire file), one address, aka one line, or two addresses(a range).
Add g to the command string if one line may have multiple instances you need changed. sed is an EXTREMELY advanced topic/command./.
Credit to @Kai for helping me fix and format this :D
This is my first post so please tell me of any formatting errors etc! I really appreciate it