# Create a json export that can be imported into other systems
bw export --format json --output myorg-bitwarden-export.json --organizationid eba3880c-370e-48f9-880c-90bb67f57b1b
# Create json export that contains additional information about attachments
bw list items --organizationid eba3880c-370e-48f9-880c-90bb67f57b1b > myorg-bitwarden-itemlist.json
# Export those attachments into folders that match the BW item items
bash <(jq -r '.[] | select(.attachments != null) | . as $parent | .attachments[] | "bw get attachment \(.id) --itemid \($parent.id) --output \"./attachments/\($parent.id)/\(.fileName)\""' myorg-bitwarden-itemlist.json)
There also seems to be a community tool called PortWarden
that supports backing up and restoring BW items:
https://github.com/vwxyzjn/portwarden
References:
]]>1 - avg(rate(node_cpu{job="default/node-exporter",mode="idle"}[1m]))
1 - avg(rate(node_cpu_seconds_total{mode="idle"}[1m]))
sum(node_load1{job="default/node-exporter"})
/
sum(node:node_num_cpu:sum)
With the following recording rule:
- expr: |-
count by (instance) (sum by (instance, cpu) (
node_cpu_seconds_total{job="node-exporter"}
))
record: node:node_num_cpu:sum
1 - sum(
node_memory_MemFree_bytes{job="…"} +
node_memory_Cached_bytes{job="…"} +
node_memory_Buffers_bytes{job="…"}
)
/
sum(node_memory_MemTotal_bytes{job="…"})
1e3 = 1000 (it is sometimes used to convert from milliseconds to seconds).
1e3 * sum(
rate(node_vmstat_pgpgin{job="…"}[1m]) +
rate(node_vmstat_pgpgout{job="…"}[1m]))
)
sum(rate(request_duration_seconds_count{job="…"}[1m]))
sum(rate(request_duration_seconds_count{job="…", status_code!~"2.."}[1m]))
histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket{job="…"}[1m])) by (le))
count(device_boot_time* on (instance) group_left(region, firmware) device_info{region="$region", firmware=~"$firmware"})
Here, device_info
is the dummy gauge, always just set to one. You can see that we are selecting this gauge based on the Grafana template values.
These two labels are used to group_left
join on other metrics, for which we want (on
) the instance. The metric is device_boot_time
but in this case the value is not relevant for the metric, as all we are doing using count
to count all devices that have a boot time greater than 0 (which is all devices).
This gives a count of all of the devices in the region that match the region and firmware.
In order to get the All option to work for firmware, a few things are needed:
.*
=~
topk(5, time() - device_boot_time* on (instance) group_left(region, firmware) device_info{region="$region", firmware=~"$firmware"})
topk(10, changes(device_boot_time[1h])
The changes function looks for sudden jumps to the value of a gauge. This can be used to track the number of times a device reboots in an hour. For this demo, our simulated devices reboot 8 times per hour on average. If you use the dashboard selectors to choose Massachusetts as the region, you will see that one device is crashing much more frequently, and the dashboard uses conditional formatting to show this as red.
alerts:
groups:
- name: device_alerts
rules:
- alert: LidLeftOpen
expr: (time() - (lid_open_start_time * lid_open_status)) > 900 and (time() - (lid_open_start_time * lid_open_status)) < 9000
for: 30s
labels:
severity: page
system: mechanical
annotations:
summary: "Lid open on {{ $labels.instance }}"
description: "Lid has remained open for more than 15 minutes on {{ $labels.instance }}"
- alert: DeviceRebooting
expr: changes(device_boot_time[1h]) > 20
for: 2m
labels:
severity: debug
system: software
annotations:
summary: "{{ $labels.instance }} rebooting"
description: "{{ $labels.instance }} has been rebooting more than 20 times an hour"
Metric value should be number of seconds since the epoch. The following will tell you how many hours since the workflow started:
(time() - argo_workflows_custom_start_time_gauge_workflow) / 3600
The general rule for choosing the range is that it should be at least 4x the scrape interval. This is to allow for various races, and to be resilient to a failed scrape.
Source: https://www.robustperception.io/what-range-should-i-use-with-rate
Your services will have a rough tree structure, have a dashboard per service and walk the tree from the top when you have a problem. Similarly for each service, have dashboards per subsystem. Rule of Thumb: Limit of 5 graphs per dashboard, and 5 lines per graph.
Query: rate(foo_metric_bucket[10m])
Legend format: {{le}}
Format as: Heatmap
Name: big_routine_job
Type: Query
Refresh: On Time Range Change
Query: query_result(topk(3, avg_over_time(go_goroutines[$__range])))
Regex: /job="([^"]+)"/
sum(rate(http_requests_total{job="$big_routine_job"}[10m])) by (handler)
rate(requests[5m])
sum(rate(requests[5m])) by (service_name)
sum(rate(requests{service_name="catalogue"}[5m])) by (instance)
Derive average request duration over a rolling 5 minute period
rate(request_duration_sum[5m])
/
rate(request_duration_count[5m])
topk(5,
sum by (image) (
rate(container_cpu_usage_seconds_total{
id=~"/system.slice/docker.*"}[5m]
)
)
)
- alert: "Node CPU Usage"
expr: (100 - (avg by (instance)
(irate(node_cpu{app="prometheus-node-exporter", mode="idle"}[5m]))
* 100)) > 70
for: 30s
annotations:
miqTarget: "ExtManagementSystem"
url: "https://www.example.com/high_node_cpu_fixing_instructions"
description: "{{ $labels.instance }}: CPU usage is above 70%
(current value is: {{ $value }})"
labels:
severity: "warning"
The nodes in your cluster have resources. The most important resources your nodes provide in a Kubernetes cluster are CPU, Memory, Network and Disk. Let’s apply the USE method to all of these.
To calculate the amount of cpu utilization by host in your Kubernetes cluster we want to sum all the modes except for idle, iowait, guest, and guest_nice. The PromQL looks like this:
sum(rate(
node_cpu{mode!=”idle”,
mode!=”iowait”,
mode!~”^(?:guest.*)$”
}[5m])) BY (instance)
A metric that the node-exporter gives us for saturation is the Unix load average. Loosely, the load average is the number of processes running plus the those waiting to run.
node_load1, node_load5 and node_load15 represent the 1, 5 and 15 min load averages. This metric is a gauge and is already averaged for you. As a standalone metric it is somewhat useless with knowing how many CPUs your node has. Is a load average of 10 good or bad? It depends. If you divide the load average by the number of CPUs you have in your cluster, then you get an approximation of the CPU saturation of your system.
node_exporter does not expose a count of node CPUs directly, but if you count just one of the above CPU modes, say “system”, you can get a CPU count by node:
count(node_cpu{mode="system"}) by (node)
Now you can normalize the node_load1 metric by the number of CPUs on the node expressed as a percentage:
sum(node_load1) by (node) / count(node_cpu{mode="system"}) by (node) * 100
The node_exporter does not reveal anything about CPU errors.
It would seem that memory utilization and saturation are a somewhat easier to reason about as the amount of physical memory on a node is known. Again nothing is easy! node_exporter gives us 43 node_memory_* metrics to work with!
The amount of available memory on a Linux system is not just the reported “free” memory metric. Unix systems rely heavily on memory that is not in use by applications to share code (buffers) and to cache disk pages (cached). So one measure of available memory is:
sum(node_memory_MemFree + node_memory_Cached + node_memory_Buffers)
Newer Linux kernels (after 3.14) expose a better free memory metric, node_memory_MemAvailable
.
Divide that by the total memory available on the node and you get a percentage of available memory, then subtract from 1 to get a measure of node memory utilization:
1 - sum(node_memory_MemAvailable) by (node)
/ sum(node_memory_MemTotal) by (node)
We’re running a Java application with a connection string that looks like this with AWS RDS MySQL version 5.7:
jdbc:mysql://${db.endpoint}/${db.schema}?useConfigs=maxPerformance&characterEncoding=utf8&useSSL=false
As you can see all database queries will be sent to the RDS instance in plaintext.
Goals
We want:
We do not require clients to authenticate themselves via client certificates.
MySQL uses yaSSL for secure connections in the following versions:
MySQL uses OpenSSL for secure connections in the following versions:
Amazon RDS for MySQL supports Transport Layer Security (TLS) versions 1.0, 1.1, and 1.2. The following table shows the TLS support for MySQL versions.
MySQL version | TLS 1.0 | TLS 1.1 | TLS 1.2 |
---|---|---|---|
MySQL 8.0 | Supported | Supported | Supported |
MySQL 5.7 | Supported | Supported | Supported for MySQL 5.7.21 and later |
MySQL 5.6 | Supported | Supported for MySQL 5.6.46 and later | Supported for MySQL 5.6.46 and later |
To validate the RDS server identity we need the RDS CA certificate chain that can be downloaded from here:
You can then test the CA chain with the mysql client:
$ wget https://truststore.pki.rds.amazonaws.com/eu-central-1/eu-central-1-bundle.pem
$ /usr/bin/mysql -P ${db_port} -h ${db_endpoint} \
-u ${db_username} -p${db_password} \
--ssl-ca=eu-central-1-bundle.pem --ssl-verify-server-cert contosodb
# newer mysql clients use another parameter instead:
$ /usr/bin/mysql -P ${db_port} -h ${db_endpoint} \
-u ${db_username} -p${db_password} \
--ssl-ca=eu-central-1-bundle.pem --ssl-mode=VERIFY_IDENTITY contosodb
You should always check certificate expiration dates:
$ openssl crl2pkcs7 -nocrl -certfile eu-central-1-bundle.pem \
| openssl pkcs7 -print_certs -noout -text | rg -B3 'Subject:'
Validity
Not Before: Aug 22 17:08:50 2019 GMT
Not After : Aug 22 17:08:50 2024 GMT
Subject: C=US, L=Seattle, ST=Washington, O=Amazon Web Services, Inc., OU=Amazon RDS, CN=Amazon RDS Root 2019 CA
--
Validity
Not Before: Sep 11 19:36:20 2019 GMT
Not After : Aug 22 17:08:50 2024 GMT
Subject: C=US, ST=Washington, L=Seattle, O=Amazon Web Services, Inc., OU=Amazon RDS, CN=Amazon RDS eu-central-1 2019 CA
--
Validity
Not Before: May 21 22:33:24 2021 GMT
Not After : May 21 23:33:24 2121 GMT
Subject: C=US, O=Amazon Web Services, Inc., OU=Amazon RDS, ST=WA, CN=Amazon RDS eu-central-1 Root CA ECC384 G1, L=Seattle
--
Validity
Not Before: May 21 22:23:47 2021 GMT
Not After : May 21 23:23:47 2061 GMT
Subject: C=US, O=Amazon Web Services, Inc., OU=Amazon RDS, ST=WA, CN=Amazon RDS eu-central-1 Root CA RSA2048 G1, L=Seattle
--
Validity
Not Before: May 21 22:28:26 2021 GMT
Not After : May 21 23:28:26 2121 GMT
Subject: C=US, O=Amazon Web Services, Inc., OU=Amazon RDS, ST=WA, CN=Amazon RDS eu-central-1 Root CA RSA4096 G1, L=Seattle
I’ve split up the CA bundle into separate files with their names based on a slugified version of their common name (CN=...
):
Amazon-RDS-Root-2019-CA.pem
Amazon-RDS-eu-central-1-2019-CA.pem
Amazon-RDS-eu-central-1-Root-CA-ECC384-G1.pem
Amazon-RDS-eu-central-1-Root-CA-RSA2048-G1.pem
Amazon-RDS-eu-central-1-Root-CA-RSA4096-G1.pem
Next we need to import them into a Java truststore that can be used by our JDBC connection:
Note: You should be using the newer pkcs12 (aka pfx) format instead of the older jks format for truststores.
#!/bin/bash
TRUSTSTORE_NAME=rds-truststore.pkcs12
TRUSTSTORE_PASSWORD=supersecretpassword
# Add certs to truststore
for CA in *.pem; do
echo "Store: [${TRUSTSTORE_NAME}] - Importing [${CA}]"
keytool -import -noprompt -trustcacerts -file "${CA}" -alias "${CA%%.*}" -storetype PKCS12 -keystore "${TRUSTSTORE_NAME}" -storepass "${TRUSTSTORE_PASSWORD}"
done
This will import all available *.pem files into a newly created truststore called rds-truststore.pkcs12
.
Place the truststore in your application’s environment - I placed it in /srv/aws-rds/truststore.pkcs12
. Update the connection string and restart/redeploy the application:
# https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-connp-props-security.html
# newer v8 format
#db.url=jdbc:mysql://${db.endpoint}/${db.schema}?useConfigs=maxPerformance&characterEncoding=utf8&sslMode=VERIFY_CA&trustCertificateKeyStoreType=PKCS12&trustCertificateKeyStoreUrl=file:///srv/aws-rds/truststore.pkcs12&trustCertificateKeyStorePassword=supersecretpassword
# older v5.1 format
db.url=jdbc:mysql://${db.endpoint}/${db.schema}?useConfigs=maxPerformance&characterEncoding=utf8&useSSL=true&verifyServerCertificate=true&trustCertificateKeyStoreType=PKCS12&trustCertificateKeyStoreUrl=file:///srv/aws-rds/truststore.pkcs12&trustCertificateKeyStorePassword=supersecretpassword
Everything should be starting up smoothly. You should see established DB connections in your RDS metrics. To verify that TLS is being used run the following query on the database server:
SELECT sbt.variable_value AS tls_version, t2.variable_value AS cipher,
processlist_user AS user, processlist_host AS host
FROM performance_schema.status_by_thread AS sbt
JOIN performance_schema.threads AS t ON t.thread_id = sbt.thread_id
JOIN performance_schema.status_by_thread AS t2 ON t2.thread_id = t.thread_id
WHERE sbt.variable_name = 'Ssl_version' AND t2.variable_name = 'Ssl_cipher'
ORDER BY tls_version;
Empty tls_version and cipher columns mean that no transport encryption is used:
+-------------+-----------------------------+--------------------------------+---------------+
| tls_version | cipher | user | host |
+-------------+-----------------------------+--------------------------------+---------------+
| | | ... | ... |
| | | foobar-dev-01-user-backup | 10.200.96.121 |
| | | foobar-dev-03-user | 10.200.96.34 |
| | | foobar-int-01-user | 10.200.98.126 |
| TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 | myapp-laboratory-user | 10.200.96.116 |
| TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 | myapp-laboratory-user | 10.200.96.116 |
| TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 | myapp-laboratory-user | 10.200.96.116 |
+-------------+-----------------------------+--------------------------------+---------------+
Ref:
]]>If you are not familiar with these terms, I would strongly recommend reading the article from Google’s SRE book on Service Level Objectives first.
In summary:
From the above, it’s clear that we must have service metrics to tell us when the service is considered (un)available. There are several approaches for this:
Methodology | RED | USE |
---|---|---|
Meaning | Rate Errors Duration |
Utilization Saturation Errors |
Used for | request driven things like endpoints | resources like queues, caches, CPUs, disks |
Services tend to fall into a few broad categories in terms of the SLIs they find relevant:
User-facing serving systems, such as frontends, generally care about availability, latency, and throughput. In other words: Could we respond to the request? How long did it take to respond? How many requests could be handled?
Storage systems often emphasize latency, availability, and durability. In other words: How long does it take to read or write data? Can we access the data on demand? Is the data still there when we need it? See Data Integrity: What You Read Is What You Wrote for an extended discussion of these issues.
Big data systems, such as data processing pipelines, tend to care about throughput and end-to-end latency. In other words: How much data is being processed? How long does it take the data to progress from ingestion to completion? (Some pipelines may also have targets for latency on individual processing stages.)
All systems should care about correctness: was the right answer returned, the right data retrieved, the right analysis done? Correctness is important to track as an indicator of system health, even though it’s often a property of the data in the system rather than the infrastructure per se, and so usually not an SRE responsibility to meet.
For most services, the most straightforward way of representing risk tolerance is in terms of the acceptable level of unplanned downtime. Unplanned downtime is captured by the desired level of service availability, usually expressed in terms of the number of “nines” we would like to provide: 99.9%, 99.99%, or 99.999% availability. For serving systems, this metric is traditionally calculated based on the proportion of system uptime:
Using this formula over the period of a year, we can calculate the acceptable number of minutes of downtime to reach a given number of nines of availability:
\[availability = {uptime \over (uptime + downtime)}\]For distributed systems a time-based metric for availability is usually not meaningful because we are looking across globally distributed services. Sufficient fault isolation makes it very likely that we are serving at least a subset of traffic for a given service somewhere in the world at any given time (i.e., we are at least partially “up” at all times). Therefore, instead of using metrics around uptime, we define availability in terms of the request success rate.
Aggregate availability shows how this yield-based metric is calculated over a rolling window (i.e., proportion of successful requests over a one-day window).
\(availability = {successful requests \over total requests}\) [availability = {successful requests \over total requests}]
Comparison | Time-based Availability | Aggregate Availability |
---|---|---|
System Type | Consumer | Infrastructure |
Used for | Monoliths, user facing systems | distributed services, batch, pipeline, storage, caching layers and transactional systems |
Measured as | proportion of system uptime | request success rate |
Question to ask yourself | How much time was the system unavailable? |
What's the amount of errors across all service instances? Helps with analyzing system performance by revealing bottlenecks or errors |
Methodology to use | RED | USE |
Most often, we set quarterly availability targets for a service and track our performance against those targets on a weekly, or even daily, basis. This strategy lets us manage the service to a high-level availability objective by looking for, tracking down, and fixing meaningful deviations as they inevitably arise. See Service Level Objectives for more details.
Now that we have an idea what SLIs/SLOs/SLAs are it’s time to use a
Expression: [Metric Identifier][Operator][Metric Value]
Expression: [Success Objective][SLI][Period]
Data Tier (a message bus system)
Tier | Capability | SLI | SLO |
---|---|---|---|
Ingest/Routing | Ingest | Percent of well-formed payloads accepted | 99.9% |
Ingest/Routing | Routing | Time to deliver message to correct destination | 99.5% of messages in under 5 seconds |
Horizontally Scaled Data Tier | Query Data | Latency | |
Horizontally Scaled Data Tier | Query Data | Correctness/Error rate |
Note: SLAs are service level agreements: an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain. The consequences are most easily recognized when they are financial—a rebate or a penalty—but they can take other forms. An easy way to tell the difference between an SLO and an SLA is to ask “what happens if the SLOs aren’t met?”: if there is no explicit consequence, then you are almost certainly looking at an SLO.
I recently ran into a problem with the Bitwarden CLI. We use it in our company but another company that I am doing contract work for also stores secrets in their own Bitwarden instance.
Now to my dismay I learned that the bw CLI can only be connected to a single server and to switch servers you have to actively run bw logout
.
Constantly running…
bw logout
bw config server https://vault.somedomain.tld
bw login
…is way to cumbersome so I wrote a small helper script to simplify the process.
Note: this requires fzf
and jq
.
In your Bitwarden CLI data dir you will find a data.json
. That file represents the vault that is encrypted with your master password.
Steps:
Now every time you run bwctx
you can pick which company’s vault you want to work with.
Run bwctx list
to see which vault is currently active.
Here’s an example
$ bw config server https://vault.example.org/
$ bw login
# now connected to https://vault.example.org/ with
# an encrypted data.json in the BW data dir
$ cd "$HOME/Library/Application Support/Bitwarden CLI"
$ mkdir eo
$ mv data.json ./eo/
$ bw config server https://vault.contoso.com/
$ bw login --sso # this server uses SSO so use OAUTH2 for the login
$ mkdir cc
$ mv data.json ./cc/
# create config.json and add bwctx to your PATH
$ bwctx
switching to: eo
$ bwctx list
current config: ./eo/data.json
Store somewhere in your PATH, typically as /usr/local/bin/bwctx
:
#!/bin/bash
# Linux: "$HOME/.config/Bitwarden CLI"
VAULT_PATH="$HOME/Library/Application Support/Bitwarden CLI"
CONF_PATH="${VAULT_PATH}/config.json"
cd "${VAULT_PATH}"
if [ "$#" -ge "1" ]; then
echo "current config: $(readlink data.json)"
exit 0
fi
ACCOUNT=$(cat "${CONF_PATH}" | jq -r '.[].account' | fzf)
if [[ ! -z "${ACCOUNT}" ]]; then
echo "switching to: ${ACCOUNT}"
ln -sf "./${ACCOUNT}/data.json" data.json
fi
[
{"server": "https://vault.contoso.com/", "account": "cc"},
{"server": "https://vault.example.org/", "account": "eo"}
]
Library/Application Support/Bitwarden CLI
▶ tree
.
├── config.json
├── data.json -> ./eo/data.json
├── eo
│ └── data.json
└── cc
└── data.json
tee /path/to/file <<EOF
${variable}
EOF
tee /path/to/file <<'EOF'
${variable}
EOF
tee /path/to/file <<-EOF
${variable}
EOF
tee -a /path/to/file <<EOF
${variable}
EOF
tee /path/to/file <<EOF >/dev/null
${variable}
EOF
sudo
as wellsudo -u USER tee /path/to/file <<EOF
${variable}
EOF
Reference:
]]>kubectl wait
to wait on an arbitrary JSON path.
Note: jsonpath expressions that lead to a nested object or list are not supported
# --for=jsonpath='{}'=value
# Wait for the pod "busybox1" to contain the status phase to be "Running".
kubectl wait --for=jsonpath='{.status.phase}'=Running pod/busybox1
kubectl wait --for=jsonpath='{.status.containerStatuses[0].ready}'=true pod/busybox1
kubectl wait --for=jsonpath='{.spec.containers[0].ports[0].containerPort}'=80 pod/busybox1
kubectl wait --for=jsonpath='{.spec.nodeName}'=knode0 pod/busybox1
Ref:
]]>:screenshot --dpr [multiplier]
, e.g. :screenshot --dpr 2
A trailing slash on the source avoids creating an additional directory level at the destination.
rsync -avz src/ dest # content of ./src/ transferred to ./dest/
rsync -avz src dest # content of ./src/ transferred to ./dest/src/
Some people have mentioned using the --partial
flag works, it needs to be mentioned however that it only resumes when the --append
or --append-verify
flag is used when resuming.
--partial
creates a hidden file of the file that has not finished the sync process, the file is kept when you interrupt syncing. It continues to complete the file when you use --append
after resuming, otherwise when not using --append
, the incomplete hidden file is kept and remains incomplete.
Conclusion: You can just interrupt rsync --partial
using Ctrl
+ C
if you use rsync --append
when resuming
- Lists files without copying them
rsync --dry-run src dest
- Show progress per file:
rsync --progress src dest
- Show global progress:
rsync --info=progress2 src dest
Settings in $HOME/.ssh/config
are also respected by rsync making commonly accessed systems far easier to use.
rsync -avz -e "ssh -l <user>" <src> <user>@<server>:<target>
Non standard SSH port
rsync -e "ssh -p 2222" ...
Use specific ssh key
rsync -e "ssh -i $HOME/.ssh/id_rsa_for_rsync" ...
Tunnel through a jump host with key agent forwarding:
rsync -e "ssh -A -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand=\"ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -W %h:%p ${SSH_USER}@${BASTION_HOST}\"" ./deployment/ ${SSH_USER}@${TARGET_HOST}:/var/www/${ENV}/
Run with elevated permissions:
rsync --rsync-path 'sudo rsync' ...
Copy remote->local, keep attributes, use compression, be verbose and show human readable units:
rsync -avzh -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" \
--progress 'preview-host:/var/www/html/mb/app/*' /tmp/
Copy local->remote, keep attributes, be verbose, use root permissions on the target, delete files that are not in the source path and use checksums to compare files:
rsync --rsync-path 'sudo rsync' -avP \
-og --chown=root:root \
--checksum --delete \
./deployment/ ubuntu@${TARGET_HOST}:/opt/app/${ENV}/
--exclude=important_file.txt
- Can be used to omit files or directories from being synced.--exclude=backups/ --include=backups/most_recent
- Inside the exclusion we can explicitly include certain file, folders or patterns that fall inside the broader exclude.In the following example, we are excluding the node_modules and tmp directories which are located inside the src_directory:
rsync -a --exclude=node_modules --exclude=.DS_Store --exclude=tmp /src_directory/ /dst_directory/
The second option is to use the --exclude-from
argument and specify the files and directories you want to exclude in a file.
rsync -a --exclude-from='/exclude-file.txt' /src_directory/ /dst_directory/
/exclude-file.txt
file1.txt
.DS_Store
node_modules
tmp
Exclude filters can be written in a condensed form:
# Before:
rsync -a --exclude 'file1.txt' --exclude 'dir1/*' --exclude 'dir2' src_directory/ dst_directory/
# After:
rsync -a --exclude={'file1.txt','dir1/*','dir2'} src_directory/ dst_directory/
'*.jpg*'
It is little trickier to exclude all other files and directories except those that match a certain pattern. Let’s say you want to exclude all other files and directories except the files ending with .jpg
.
One option is to use the following command:
rsync -a -m \
--include='*.jpg' \
--include='*/' \
--exclude='*' \
src_directory/ dst_directory/
When using multiple include/exclude option, the first matching rule applies.
--include='*.jpg'
- First we are including all .jpg
files.--include='*/'
- Then we are including all directories inside the in src_directory
directory. Without this rsync will only copy *.jpg
files in the top level directory.-m
- Removes empty directories.# Automatically delete source files after successful transfer
rsync --remove-source-files -zvh backup.tar /tmp/backups/
Since --remove-source-files
does not remove directories, issue the following commands to move files over ssh:
rsync -avh --progress --remove-source-files /source/* user@server:/target \
&& find /source -type d -empty -delete
rsync ... > /tmp/rsyncbackup.log 2> /tmp/rsyncbackup.errors.log
-o, --owner preserve owner (super-user only)
-g, --group preserve group
--devices preserve device files (super-user only)
--specials preserve special files
-D same as --devices --specials
-t, --times preserve modification times
-p, --perms preserve file/directory permissions
-l, --links copy symlinks as symlinks
-u, --update skip files that are newer on the receiver
-C, --cvs-exclude auto-ignore files in the same way CVS does
--progress show progress during transfer
--stats give some file-transfer stats
--list-only list the files instead of copying them
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
-a
, --archive
- Archive mode, equivalent to -rlptgoD
. This option tells rsync
to syncs directories recursively, transfer special and block devices, preserve symbolic links, modification times, group, ownership, and permissions. Sometimes you will have to supplement the -a
parameter with:
-X
- Preserve extended attributes, e.g. SELinux contexts may be stored as such attributes on distributions like CentOS/RedHat where these are used by default.-A
- Preserve ACLs (Access Control Lists)-z
, --compress
- This option will force rsync
to compresses the data as it is sent to the destination machine. Use this option only if the connection to the remote machine is slow.-P
- equivalent to --partial --progress
. When this option is used rsync
will show a progress bar during transfer and to keep the partially transferred files. It is useful when transferring large files over slow or unstable network connections. Without -P
or --partial
, if the connection drops during a transfer, the file is deleted and you will have to restart from scratch.--delete
- Delete files in the destination that don’t exist anymore in the source location. Used when you want to keep an exact replica of the source files/directories. Without this option, files that have been deleted in the source won’t be deleted on destination, which is preferable for most backup schemas. Keep in mind that the --delete
parameter exposes you to the risk of losing the entire backup, if used inappropriately (e.g., if you use the wrong source directory or an empty one). An option like --max-delete=3
so that rsync never deletes more than 3 files can reduce the amount of data you might lose. The number can be adjusted according to your use case.-q
, --quiet
- Use this option if you want to suppress non-error messages.-e
- This option allows you to choose a different remote shell. By default, rsync
is configured to use ssh.-v
- Verbose mode prints more statistics: what files are currently copied/transferred and summary about bytes transferred and speedup ratio.-r
- Copy every object contained in directories and subdirectories. Without this option, directories are skipped and only files are copied. E.g., rsync -v root@example.com:/etc/* /tmp
would only copy files from /etc/
. When you are copying/transferring a single directory, you have to use this option or the -a
parameter, otherwise nothing happens, the directory is simply skipped.-h
- Show “human readable” numbers: instead of statistics being shown in bytes, they will be displayed in megabytes, kilobytes, etc., because 9.82M
is easier to read than 9,821,016
.When files are getting significantly bigger on the other side Thin Provisioning (TP) is probably enabled on the source system - a method of optimizing the efficiency of available space in Storage Area Networks (SAN) or Network Attached Storages (NAS).
E.g.: The source file was only 10GB because of TP being enabled, and when transferred over using rsync without any additional configuration, the target destination was receiving the full 100GB of size. rsync could not do the magic automatically, it had to be configured.
The flag that does this work is -S
or -sparse
and it tells rsync to handle sparse files efficiently. And it will do what it says! It will only send the sparse data so source and destination will have a 10GB file.
There appear to be font rendering issues in electron apps on some machines. This only affects electron apps that were installed as snap packages and will mess up the user interface partially or completely. Often people will recommend deleting that app’s font cache but that does not really solve the problem.
How much you’re affected really depends on the system’s fonts library so install the following fonts to resolve the problem:
noto-fonts-emoji
or bdf-unifont
ttf-dejavu
Source:
]]>