column
Split columns automatically and auto-align in a tabular format
Format whitespace delimited text as table:
mount | column -t:
Format colon delimited text as table:
cat /etc/passwd | column -t -s:
Remove surrounding parenthesis
cat in_file | tr -d '()' > out_file
Remove non-printable characters
... | tr -d '[:cntrl:]'
... | tr -cd '[:print:]'
- https://stackoverflow.com/questions/8914435/awk-sed-how-to-remove-parentheses-in-simple-text-file
switch case
#!/bin/bash
# Must be run in the service's directory.
help_message() {
echo -e "Usage: $0 [apply|destroy|plan|refresh|show]\n"
echo -e "The following arguments are supported:"
echo -e "\tapply \t Refresh the Terraform remote state, perform a \"terraform get -update\", and issue a \"terraform apply\""
echo -e "\tdestroy \t Refresh the Terraform remote state and destroy the Terraform stack"
echo -e "\tplan \t Refresh the Terraform remote state, perform a \"terraform get -update\", and issues a \"terraform plan\""
echo -e "\trefresh \t Refresh the Terraform remote state"
echo -e "\tshow \t Refresh and show the Terraform remote state"
exit 1
}
## Begin script ##
if [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
help_message
fi
[ -z backend.tf ] || createBackendConfig
[ -d ~/.terraform.d/plugin-cache ] || mkdir -p ~/.terraform.d/plugin-cache
ACTION="$1"
case $ACTION in
apply|destroy|plan|refresh|show)
$ACTION $@
;;
****)
echo "That is not a valid choice."
help_message
;;
esac
Validate variables
You can use -z to test whether a variable is unset or empty:
if [[ -z $DB || -z $HOST || -z $DATE ]]; then
echo 'one or more variables are undefined'
exit 1
fi
echo "You are good to go"
I’ve used an extended test [[, which means that I don’t need to use quotes around my variables. I’m assuming that you need all three variables to be defined in order to continue. The exit in the if branch means that the else is superfluous.
The standard way to do it in any POSIX-compliant shell would be like this:
if [ -z "$DB" ] || [ -z "$HOST" ] || [ -z "$DATE" ]; then
echo 'one or more variables are undefined'
exit 1
fi
The important differences here are that each variable check goes inside a separate test and that double quotes are used around each parameter expansion.
Compare directories
# diff --brief -Nr dir1/ dir2/
$ diff --brief -Nr roles-galaxy/nginx roles/nginx2
Files roles-galaxy/nginx/tasks/main.yml and roles/nginx2/tasks/main.yml differ
Files roles-galaxy/nginx/tasks/setup-AmazonLinux2.yml and roles/nginx2/tasks/setup-AmazonLinux2.yml differ
Program detection in shell scripts
POSIX compatible:
command -v <the_command>
For bash specific environments:
hash <the_command> # For regular commands. Or...
type <the_command> # To check built-ins and keywords
Explanation
Avoid which. Not only is it an external process you’re launching for doing very little (meaning builtins like hash, type or command are way cheaper), you can also rely on the builtins to actually do what you want, while the effects of external commands can easily vary from system to system.
Why care?
- Many operating systems have a
whichthat doesn’t even set an exit status, meaning theif which foowon’t even work there and will always report thatfooexists, even if it doesn’t (note that some POSIX shells appear to do this forhashtoo). - Many operating systems make
whichdo custom and evil stuff like change the output or even hook into the package manager.
So, don’t use which. Instead use one of these:
$ command -v foo >/dev/null 2>&1 || { echo >&2 "I require foo but it's not installed. Aborting."; exit 1; }
$ type foo >/dev/null 2>&1 || { echo >&2 "I require foo but it's not installed. Aborting."; exit 1; }
$ hash foo 2>/dev/null || { echo >&2 "I require foo but it's not installed. Aborting."; exit 1; }
(Minor side-note: some will suggest 2>&- is the same 2>/dev/null but shorter - this is untrue. 2>&- closes FD 2 which causes an error in the program when it tries to write to stderr, which is very different from successfully writing to it and discarding the output (and dangerous!))
If your hash bang is /bin/sh then you should care about what POSIX says. type and hash’s exit codes aren’t terribly well defined by POSIX, and hash is seen to exit successfully when the command doesn’t exist (haven’t seen this with type yet). command’s exit status is well defined by POSIX, so that one is probably the safest to use.
If your script uses bash though, POSIX rules don’t really matter anymore and both type and hash become perfectly safe to use. In bash type now has a -P to search just the PATH and hash has the side-effect that the command’s location will be hashed (for faster lookup next time you use it), which is usually a good thing since you probably check for its existence in order to actually use it.
As a simple example, here’s a function that runs gdate if it exists, otherwise date:
gnudate() {
if hash gdate 2>/dev/null; then
gdate "$@"
else
date "$@"
fi
}
Note that hash will only look in PATH. If your user’s PATH does not include sbin, hash will not find the binary that lives there. If you want to run bash code with sudo, you need to invoke bash from sudo: if sudo bash -c ‘hash groupadd’; then …
True if file exists and is executable.
test -x filename [ -x filename ]
hash foo 2>/dev/null: works with zsh, bash, dash and ash.
type -p foo: it appears to work with zsh, bash and ash (busybox), but not dash (it interprets -p as an argument).
command -v foo: works with zsh, bash, dash, but not ash (busybox) (-ash: command: not found).
Also note that builtin is not available with ash and dash.
Create backup file
cp -fp example.txt{,.bak}
Get ISO date
TIMESTAMP="$(date '+%Y-%m-%d')"
Remove specific file extension
# returns template.conf
basename template.conf.j2 .j2
xargs fail-safe
xargs --no-run-if-empty <command>
Bash variable tricks
${parameter:-word}
If parameter is unset or null, the expansion of word is substituted. Otherwise, the value of parameter is substituted.
${parameter:=word}
If parameter is unset or null, the expansion of word is assigned to parameter. The value of parameter is then substituted. Positional parameters and special parameters may not be assigned to in this way.
${parameter:?word}
If parameter is null or unset, the expansion of word (or a message to that effect if word is not present) is written to the standard error and the shell, if it is not interactive, exits. Otherwise, the value of parameter is substituted.
${parameter:+word}
If parameter is null or unset, nothing is substituted, otherwise the expansion of word is substituted.
${parameter:offset}
${parameter:offset:length}
This is referred to as Substring Expansion. It expands to up to length characters of the value of parameter starting at the character specified by offset. If parameter is ‘@’, an indexed array subscripted by ‘@’ or ‘*’, or an associative array name, the results differ as described below. If length is omitted, it expands to the substring of the value of parameter starting at the character specified by offset and extending to the end of the value. length and offset are arithmetic expressions (see Shell Arithmetic).
If offset evaluates to a number less than zero, the value is used as an offset in characters from the end of the value of parameter. If length evaluates to a number less than zero, it is interpreted as an offset in characters from the end of the value of parameter rather than a number of characters, and the expansion is the characters between offset and that result. Note that a negative offset must be separated from the colon by at least one space to avoid being confused with the ‘:-’ expansion.
Here are some examples illustrating substring expansion on parameters and subscripted arrays:
$ string=01234567890abcdefgh
$ echo ${string:7}
7890abcdefgh
$ echo ${string:7:0}
$ echo ${string:7:2}
78
$ echo ${string:7:-2}
7890abcdef
$ echo ${string: -7}
bcdefgh
$ echo ${string: -7:0}
$ echo ${string: -7:2}
bc
$ echo ${string: -7:-2}
bcdef
$ set -- 01234567890abcdefgh
$ echo ${1:7}
7890abcdefgh
$ echo ${1:7:0}
$ echo ${1:7:2}
78
$ echo ${1:7:-2}
7890abcdef
$ echo ${1: -7}
bcdefgh
$ echo ${1: -7:0}
$ echo ${1: -7:2}
bc
$ echo ${1: -7:-2}
bcdef
$ array[0]=01234567890abcdefgh
$ echo ${array[0]:7}
7890abcdefgh
$ echo ${array[0]:7:0}
$ echo ${array[0]:7:2}
78
$ echo ${array[0]:7:-2}
7890abcdef
$ echo ${array[0]: -7}
bcdefgh
$ echo ${array[0]: -7:0}
$ echo ${array[0]: -7:2}
bc
$ echo ${array[0]: -7:-2}
bcdef
If parameter is ‘@’, the result is length positional parameters beginning at offset. A negative offset is taken relative to one greater than the greatest positional parameter, so an offset of -1 evaluates to the last positional parameter. It is an expansion error if length evaluates to a number less than zero.
The following examples illustrate substring expansion using positional parameters:
$ set -- 1 2 3 4 5 6 7 8 9 0 a b c d e f g h
$ echo ${@:7}
7 8 9 0 a b c d e f g h
$ echo ${@:7:0}
$ echo ${@:7:2}
7 8
$ echo ${@:7:-2}
bash: -2: substring expression < 0
$ echo ${@: -7:2}
b c
$ echo ${@:0}
./bash 1 2 3 4 5 6 7 8 9 0 a b c d e f g h
$ echo ${@:0:2}
./bash 1
$ echo ${@: -7:0}
If parameter is an indexed array name subscripted by ‘@’ or ‘*’, the result is the length members of the array beginning with ${parameter[offset]}. A negative offset is taken relative to one greater than the maximum index of the specified array. It is an expansion error if length evaluates to a number less than zero.
These examples show how you can use substring expansion with indexed arrays:
$ array=(0 1 2 3 4 5 6 7 8 9 0 a b c d e f g h)
$ echo ${array[@]:7}
7 8 9 0 a b c d e f g h
$ echo ${array[@]:7:2}
7 8
$ echo ${array[@]: -7:2}
b c
$ echo ${array[@]: -7:-2}
bash: -2: substring expression < 0
$ echo ${array[@]:0}
0 1 2 3 4 5 6 7 8 9 0 a b c d e f g h
$ echo ${array[@]:0:2}
0 1
$ echo ${array[@]: -7:0}
Substring expansion applied to an associative array produces undefined results.
Substring indexing is zero-based unless the positional parameters are used, in which case the indexing starts at 1 by default. If offset is 0, and the positional parameters are used, $@ is prefixed to the list.
${!prefix*}
${!prefix@}
Expands to the names of variables whose names begin with prefix, separated by the first character of the IFS special variable. When ‘@’ is used and the expansion appears within double quotes, each variable name expands to a separate word.
${!name[@]}
${!name[*]}
If name is an array variable, expands to the list of array indices (keys) assigned in name. If name is not an array, expands to 0 if name is set and null otherwise. When ‘@’ is used and the expansion appears within double quotes, each key expands to a separate word.
${#parameter}
The length in characters of the expanded value of parameter is substituted. If parameter is ‘’ or ‘@’, the value substituted is the number of positional parameters. If parameter is an array name subscripted by ‘’ or ‘@’, the value substituted is the number of elements in the array. If parameter is an indexed array name subscripted by a negative number, that number is interpreted as relative to one greater than the maximum index of parameter, so negative indices count back from the end of the array, and an index of -1 references the last element.
${parameter#word}
${parameter##word}
The word is expanded to produce a pattern just as in filename expansion (see Filename Expansion). If the pattern matches the beginning of the expanded value of parameter, then the result of the expansion is the expanded value of parameter with the shortest matching pattern (the ‘#’ case) or the longest matching pattern (the ‘##’ case) deleted. If parameter is ‘@’ or ‘’, the pattern removal operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with ‘@’ or ‘’, the pattern removal operation is applied to each member of the array in turn, and the expansion is the resultant list.
${parameter%word}
${parameter%%word}
The word is expanded to produce a pattern just as in filename expansion. If the pattern matches a trailing portion of the expanded value of parameter, then the result of the expansion is the value of parameter with the shortest matching pattern (the ‘%’ case) or the longest matching pattern (the ‘%%’ case) deleted. If parameter is ‘@’ or ‘’, the pattern removal operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with ‘@’ or ‘’, the pattern removal operation is applied to each member of the array in turn, and the expansion is the resultant list.
${parameter/pattern/string}
The pattern is expanded to produce a pattern just as in filename expansion. Parameter is expanded and the longest match of pattern against its value is replaced with string. If pattern begins with ‘/’, all matches of pattern are replaced with string. Normally only the first match is replaced. If pattern begins with ‘#’, it must match at the beginning of the expanded value of parameter. If pattern begins with ‘%’, it must match at the end of the expanded value of parameter. If string is null, matches of pattern are deleted and the / following pattern may be omitted. If the nocasematch shell option (see the description of shopt in The Shopt Builtin) is enabled, the match is performed without regard to the case of alphabetic characters. If parameter is ‘@’ or ‘’, the substitution operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with ‘@’ or ‘’, the substitution operation is applied to each member of the array in turn, and the expansion is the resultant list.
${parameter^pattern}
${parameter^^pattern}
${parameter,pattern}
${parameter,,pattern}
This expansion modifies the case of alphabetic characters in parameter. The pattern is expanded to produce a pattern just as in filename expansion. Each character in the expanded value of parameter is tested against pattern, and, if it matches the pattern, its case is converted. The pattern should not attempt to match more than one character. The ‘^’ operator converts lowercase letters matching pattern to uppercase; the ‘,’ operator converts matching uppercase letters to lowercase. The ‘^^’ and ‘,,’ expansions convert each matched character in the expanded value; the ‘^’ and ‘,’ expansions match and convert only the first character in the expanded value. If pattern is omitted, it is treated like a ‘?’, which matches every character. If parameter is ‘@’ or ‘’, the case modification operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with ‘@’ or ‘’, the case modification operation is applied to each member of the array in turn, and the expansion is the resultant list.
${parameter@operator}
The expansion is either a transformation of the value of parameter or information about parameter itself, depending on the value of operator. Each operator is a single letter:
Q- The expansion is a string that is the value of parameter quoted in a format that can be reused as input.E- The expansion is a string that is the value of parameter with backslash escape sequences expanded as with the $’…’ quoting mechanism.P- The expansion is a string that is the result of expanding the value of parameter as if it were a prompt string (see Controlling the Prompt).A- The expansion is a string in the form of an assignment statement or declare command that, if evaluated, will recreate parameter with its attributes and value.a- The expansion is a string consisting of flag values representing parameter’s attributes.
If parameter is ‘@’ or ‘’, the operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with ‘@’ or ‘’, the operation is applied to each member of the array in turn, and the expansion is the resultant list.
Curl with certificate and basic authentication
USER="PID1AV4"
PASS="JtyZDzs34TCqamAnTVAkiarEbt-3w6exhwmAaXuu"
AUTH=$(echo -n "${USER}:${PASS}" | base64)
curl -vvv https://api-certs.domain.tld/prod/api/customer \
--header "X-ApplicationName: My Application" \
-H 'Accept: application/json' -H "Authorization: Basic ${AUTH}" -H 'Content-Type: application/json' \
--cert /app/certs/api-gateway-prod.pem \
--key /app/certs/api-gateway-prod.key \
--key-type PEM
SSH
Completely suppress key based authentication / Format for suSSHi:
user@@susshi.dc.domain.tld: account with access granted by suSSHiroot@12.13.14.15: actual target host/user
ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@root@12.13.14.15@susshi.dc.domain.tld
SMTP CLI clients
mailx
echo "This is the message body and contains the message" | mailx -v \
-r "noreply@senderdomain.com" \
-s "This is the subject" \
-S smtp="smtpserver.domain.tld" \
seb@recipientdomain.tld
AWK
If the 7nth field equals “security”, print field 3.
$ zypper lp | awk ' $7=="security" { print $3 }'
openSUSE-2013-992
openSUSE-2014-109
openSUSE-2014-114
openSUSE-2014-78