API tools faq
paste
ameinild

Ask Ubuntu - Rsync session

ameinild
Oct 14th, 2020 (edited)
373
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Bash 8.93 KB | None | 0 0
raw download clone embed print report
  1. # Agenda #
  2. : '
  3. - Basic SSH usage
  4. - Basic Rsync CLI usage
  5. - Basic usage of Rsync with cron
  6. - Using SSHFS to mount remote folders
  7. - Taste of advanced Rsync script (if there is time)
  8. '
  9.  
  10. ## Example objects used ##
  11. : '
  12. For the purpose of this session, the following example objects will be used:
  13. User:           artur
  14. Host:           myserver
  15. Source dirs:    /home/artur/documents, /home/artur/pictures, /home/artur/videos
  16. Destination:    /mnt/backup
  17. '
  18.  
  19. # Basic SSH usage #
  20. : '
  21. Secure Shell (SSH) is a cryptographic network protocol for operating network services securely over an unsecured network. SSH can be used for many things, including:
  22. - For login to a shell on a remote host
  23. - For setting up automatic (passwordless) login to a remote host
  24. - To backup and copy files efficiently and securely using SCP, SFTP or Rsync
  25. - Plus many other things
  26.  
  27. SSH operates with a keypair of a private and public key, so you can login remotely without using password (you can check the .ssh folder in your homedir to see if you already have a keypair with: ls ~/.ssh)
  28. '
  29.  
  30. : 'Creating an SSH keypair'
  31. ssh-keygen
  32.  
  33. : 'For me, this would create the files id_rsa (private key) and id_rsa.pub (public key) in the folder /home/artur/.ssh'
  34.  
  35. : 'Copy the public key to the remote host (may require password login)'
  36. ssh-copy-id artur@myserver
  37.  
  38. : 'After this, you can login to a remote shell (if SSH server is running on the remote host) without a password using'
  39. ssh artur@myserver
  40.  
  41. : '
  42. You can also add the public key manually to the file /home/artur/.ssh/authorized_keys on the remote host
  43.  
  44. !!! NEVER give out your PRIVATE key !!! (the file only named id_rsa, and the largest of the 2 files)
  45.  
  46. This post explain some more details:
  47. https://askubuntu.com/questions/46930/how-can-i-set-up-password-less-ssh-login/46935
  48. '
  49.  
  50. # Basic Rsync CLI usage #
  51. : 'General usage'
  52. rsync [Options] [Source(s)] [Destination]
  53.  
  54. : 'Syncing locally (ignore the options for now)'
  55. rsync -zav /home/artur/documents /mnt/backup
  56.  
  57. : 'Syncing from local to remote'
  58. rsync -zav /home/artur/documents artur@myserver:/mnt/backup
  59.  
  60. : 'Syncing from remote to local'
  61. rsync -zav artur@myserver:/home/artur/documents /mnt/backup
  62.  
  63. : '
  64. The username is the user to log in to the remote host with SSH. Typically, this would be the normal user (in this case artur) or root.
  65.  
  66. The hostname of myserver can be either an IP address or a FQDN (fully qualified domain name). For instance, if I had local DNS, and my domain name was arturmeinild.net, then a FQDN could be: myserver.arturmeinild.net
  67.  
  68. Adding a public SSH key to the remote host is the preferred way to login - alternatively a password file can be used.
  69. '
  70.  
  71. ## Useful CLI options (inspired by Grsync) ##
  72. : '
  73. -t, --times                 preserve modification times
  74. -p, --perms                 preserve permissions
  75. -o, --owner                 preserve owner
  76. -g, --group                 preserve group
  77.  
  78.    --delete                delete extraneous files from destination dir
  79. -x, --one-file-system       do not cross filesystem boundaries
  80. -v, --verbose               increase verbosity
  81.    --progress              show progress during transfer
  82.    --ignore-existing       skip updating files that exist on receiver
  83.    --size-only             skip files that match in size
  84. -u, --update                skip files that are newer on receiver
  85.    --modify-window=1       compare mod-times with reduced accuracy (Windows compatibility)
  86.  
  87. -r, --recursive             recurse into directories
  88. -l, --links                 copy symlinks as symlinks (preserve)
  89. -n, --dry-run               perform a trial run with no changes made
  90. -z, --compress              compress file data during transfer
  91. -a, --archive               archive mode; equals -rlptgoD (-D are more special filetypes)
  92.    --password-file=FILE    read remote-access password from FILE
  93. '
  94.  
  95. # Basic usage of Rsync with cron #
  96. : '
  97. There are generally 3 different ways to run a command or script with cron:
  98. - Add a cronjob definition in crontab
  99. - Add a cronjob definition file in /etc/cron.d/
  100. - Put a script file directly in /etc/cron.hourly/, /etc/cron.daily/, /etc/cron.weekly/ or /etc/cron.monthly/
  101.  
  102. Each method can have advantages for certain usecases, or sometimes it just comes down to personal preference.
  103. In the following examples we will add cronjob definitions under /etc/cron.d/, which have the following format:
  104. '
  105. [Minute] [Hour] [Day of month] [Month] [Day of week] [User] [Command]
  106.  
  107. : '
  108. The time and date fields are: (first 5 fields)
  109.  Field          Allowed values
  110.  -----          --------------
  111.  Minute         0-59
  112.  Hour           0-23
  113.  Day of month   1-31
  114.  Month          1-12
  115.  Day of week    0-7 (0 or 7 is Sunday)
  116.  
  117. In addition, for all fields * is wildcard meaning any value.
  118. '
  119.  
  120. ## Example cronjob definitions ##
  121. : 'For a cronjob that runs rsync for a single folder each day at 22:30, this would be the cronjob definition to put in a file under /etc/cron.d/ (create a text file with any name you like - and notice the PATH must be included in the job file as well)'
  122. PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
  123. 30 22 * * * artur rsync -zav /home/artur/documents /mnt/backup
  124.  
  125. : 'You can also chain more commands with a semicolon to several destinations'
  126. PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
  127. 30 22 * * * artur rsync -zav /home/artur/documents /mnt/backup; rsync -zav /home/artur/documents artur@myserver:/mnt/backup
  128.  
  129. ## Cron exercise ##
  130. : 'Edit a new file in /etc/cron.d/ as root'
  131. sudo nano /etc/cron.d/crontest
  132.  
  133. : 'Add the following lines, where you replace [Minute] and [Hour] so the script runs 15 minutes from now in your Timezone, and [User] with your username'
  134. PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
  135. [Minute] [Hour] * * * [User] echo "Hello" > /home/[User]/hello.txt
  136.  
  137. : 'Save the file - and let us see if the file hello.txt is created in about 15 minutes!'
  138.  
  139. ## A simple Rsync script ##
  140. : 'However, if you have several files and folders or need to sync to multiple destinations, it might be better to write a simple bash script. I could create a bash script in my homedir: /home/artur/rsyncbackup.sh (remember to make it executable with: chmod +x ~/rsyncbackup.sh)'
  141. #!/bin/bash
  142.  
  143. rsync -zav /home/artur/{documents,pictures,videos} /mnt/backup
  144. rsync -zav /home/artur/{documents,pictures,videos} artur@myserver:/mnt/backup
  145.  
  146. : 'And then this script is added to the cronjob file instead'
  147. PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
  148. 30 22 * * * artur /home/artur/rsyncbackup.sh
  149.  
  150. : '
  151. A nice tool to help with cron expressions is grontab.guru:
  152. https://crontab.guru/
  153. '
  154.  
  155. # Using SSHFS to mount remote folders #
  156. : 'Installing SSHFS'
  157. sudo apt install sshfs
  158.  
  159. : 'Mounting remote folder (folder /mnt/backup/remote must exist and be empty)'
  160. sshfs artur@myserver:/mnt/backup /mnt/backup/remote
  161.  
  162. : 'Now the remote folder /mnt/backup on myserver is available in /mnt/backup/remote on the local machine. Now we can use a local rsync command to actually perform a remote sync'
  163. rsync -zav /home/artur/{documents,pictures,videos} /mnt/backup/remote
  164.  
  165. : '
  166. One important aspect of SSHFS is that it cannot cross between users - so if you mount a folder with the user artur, only this user can access the contents, regardless of read permissions.
  167. '
  168.  
  169. # Taste of advanced Rsync script #
  170. : '
  171. I am currently writing a script that reads some simple configuration files for syncing specific files and folders to multiple locations - both locally and remote.
  172.  
  173. The script works with 2 different types of files: Homefiles and Rootfiles.
  174. - Homefiles are files or directories in my homedir that must be synced.
  175. - Rootfiles are various config files and directories owned by root that must be synced.
  176.  
  177. The file format is like this:
  178. '
  179. # Homefiles for Rsync Backup
  180. ::LOCAL-ONLY::
  181. .gnupg
  182. .ssh
  183. .tmux.conf
  184.  
  185. ::REMOTE-ONLY::
  186.  
  187. ::LOCAL+REMOTE::
  188. .config
  189. .bash_aliases
  190. .bashrc
  191. bash
  192.  
  193. ::END::
  194.  
  195. : '
  196. As you can see, the config file syntax directs the files into groups, depending on whether they should be synced only locally, only remotely, or both.
  197. .gnupg, .ssh, .config and bash are directories
  198. .tmux.conf, .bash_aliases and .bashrc are files
  199.  
  200. The script reads another config file that defines:
  201. - Destination paths for homefiles and rootfiles when the category is ::LOCAL::
  202. - A list of remote hosts to connect to when the category is ::REMOTE::
  203.  
  204. The script then has the following functionality:
  205. - Checks if the remote hosts are online
  206. - Reads the definition file for first homefiles then rootfiles, and for each line executes the relevant rsync command to:
  207.  * Sync locally
  208.  * Sync remotely (logging in as either normal user or root)
  209.  * Sync both locally and remotely
  210. - Logging nicely formatted output while doing so (log function not finished yet)
  211.  
  212. I run this script under /etc/cron.daily - so I get automatic backup and sync of important config files
  213.  
  214. This is just to give an idea about what you can do with a more customized use of Rsync.
  215. '
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!