In this article, we’ll get a basic understating of excluding files and directories with Rsync command line utility. Also includes a few useful examples using this feature.

Exclude files and directories with rsync

The most basic method for excluding files and directories with Rsync is by using the --exclude option followed by the name of the file or directory you want to exclude. For example, if you want to exclude all files with the “.log” extension, you can use the following command:

rsync -av --exclude='*.log' source/ destination/ 

You can also exclude specific directories by including the entire path, like this:

rsync -av --exclude='path/to/directory' source/ destination/ 

Exclude all written in text file

Another way to exclude files and directories is by using a separate file called an “exclude file”. This file contains a list of excluded patterns, one per line, and Rsync will read the file and apply all the patterns specified in it.

rsync -av --exclude-from='exclude.txt' source/ destination/ 

Here `exclude.txt` file containing all the patterns of files or directories to be excluded

The patterns in the exclude file can be a shell glob (e.g., *.log) or a regular expression (if the --exclude-from option is replaced with --exclude-from-file).

You can also use a `.gitignore` file to specify files to be excluded, which is a good way to exclude version control files.

rsync -av --exclude-from='.gitignore' source/ destination/ 

Include specific files and exclude other

It is also possible to include and exclude files at the same time by using the --include option followed by the name of the file or directory you want to include, and the --exclude option for everything else. For example, the following command will include all files with the “.txt” extension and exclude all other files:

rsync -av --include='*.txt' --exclude='*' source/ destination/ 

It’s important to note that the order of the --include and --exclude options is significant. If you want to include a specific file or directory and then exclude others, the --include option should come first.

Exclude existing files on destination

Another thing to consider is to exclude files that are already present in the destination and the destination is a backup. You can use --ignore-existing options to achieve this.

rsync --ignore-existing -av source/ destination/ 

You can read more about it: https://tecadmin.net/rsync-command-to-copy-missing-files-only/

Exclude large files

Another consideration when using Rsync to exclude files and directories is performance. Excluding large files or directories can save a significant amount of time and space when transferring or synchronizing data. For example, if you’re performing a backup, you may want to exclude large media files or backups of backups.

rsync --ignore-existing --size-range='+100M' -av source/ destination/

There are also many other options available with rsync that can be used to further fine-tune the file transfer process. For example, you can use the --exclude option to exclude certain files or directories from the transfer, or the --dry-run option to see what files would be copied without actually performing the transfer.

Wrap Up

Overall, the ability to exclude files and directories with Rsync is a powerful and essential feature that can greatly improve the efficiency, accuracy, and security of your backups, synchronization, and data management tasks. By using the methods and best practices discussed in this article, you can take full advantage of this feature and ensure that your Rsync commands are working as efficiently and effectively as possible.

The post Rsync Command to Exclude Files and Directories: An Ultimate Guide appeared first on TecAdmin.

Copy Missing Files with Rsync

To copy only missing files using rsync, you can use the `--ignore-existing` option. This tells rsync to skip files that already exist in the destination location and only copy files that are missing. For example, to copy all files from the directory “source” to the directory “destination”, using the `--ignore-existing` option, you would run the following command:

rsync --ignore-existing -av source/ destination/ 

The -a option tells rsync to preserve file attributes and the -v option verbose mode which will show you the progress of the transfer and the files that are being copied.

Copy Modified Files with Rsync

It’s also possible to copy only the new or modified files and exclude the older files that already exist in the destination. For this you can use -u or --update along with -r or --recursive option to transfer the directory recursively.

rsync -ur source/ destination/ 

Copy Files Modified in Last N Days

You can also specify that the rsync should only copy files that have been modified in the last N days. For example, to copy only files that have been modified in the last 7 days, you would use the --max-age option along with the number of days, like this:

rsync --ignore-existing --max-age=7 -av source/ destination/

Copy Files By Size with Rsync

It’s also possible to specify only to copy the files that are larger or smaller than a certain size using --size-range option, which takes the argument of either +SIZE or -SIZE. For example, if you want to copy only files that are larger than 100MB, you would use the --size-range option like this:

rsync --ignore-existing --size-range='+100M' -av source/ destination/

There are also many other options available with rsync that can be used to further fine-tune the file transfer process. For example, you can use the --exclude option to exclude certain files or directories from the transfer, or the --dry-run option to see what files would be copied without actually performing the transfer.

Conclusion

In conclusion, the rsync command is a powerful tool for copying files and directories and can be used to copy only the files that are missing or have been modified in the destination location. By using the various options available with rsync, you can further fine-tune the file transfer process to suit your specific needs.

The post Rsync Command to Copy Missing Files (Examples) appeared first on TecAdmin.

This guide will walk you through the steps of using Rsync over a non-standard SSH port, including how to configure the remote server and the Rsync command line.

Using Rsync with Non-standard SSH Port

The next step is to configure Rsync to use the non-standard port. You can do this by using the “-e” option, which allows you to specify an alternate remote shell to use for communication.

The syntax for using Rsync with a non-standard SSH port is as follows:

# Syntax
rsync -avz -e 'ssh -p [non-standard port number]' [source] [destination]

For example, if you want to synchronize the directory “SRC” on your local machine to the remote server on port 2232, the command would be:

rsync -avz -e "ssh -p 2232" SRC/ user@remote.host:/DEST/ 

This command tells Rsync to use the “ssh -p 2222” command as the remote shell, which connects to the remote server on port 2222. The “-a” option stands for “archive” mode, which preserves the permissions, ownership, timestamps and so on. The “-v” option stands for verbose mode and will print messages that give you detailed information of the process, -e option tells rsync to use ssh as a remote shell, and -z compresses the data during transfer.

Configuring the Remote Server

You can configure the remote server to listen on a non-standard SSH port. To do this, you will need to edit the SSH server configuration file, typically located at /etc/ssh/sshd_config.

In the configuration file, locate the line that starts with “Port” and change the default port number (22) to the desired non-standard port number. For example, if you want to use port 2232, the line should read “Port 2232”.

Save the changes and exit the configuration file. Restart the SSH server for the changes to take effect.

Wrap Up

With the remote server and Rsync configured to use the non-standard SSH port, you can now run the Rsync command to synchronize the files and directories.

It’s important to note that if you are using firewall on the server and client side you may need to open the non-standard port for ssh and also for rsync.

And that’s it! By following these steps, you should now be able to use Rsync over a non-standard SSH port. This can be useful for increased security, or for when you need to work around network restrictions.

The post Rsync Over a Non-Standard SSH Port: A Beginner’s Guide appeared first on TecAdmin.

Rsync is typically employed for backing up large amounts of data or transferring data between two computers. Rsync supports local-to-local, local-to-remote, and remote-to-local file syncing. However, remote-to-remote file syncing is not supported.

In this post, we will look at some common uses for the rsync command-line tool.

Syntax

rsync [OPTION...] SOURCE... [DESTINATION]

• Here SOURCE can be a local directory like “/opt/backup” or a remote location like “user@example.com:/opt/remotedir/”
• Also the DESTINATION can be referred to as a local directory or remote system directory
• Both SOURCE and DESTINATION can refer to a local directory.
• Only one SOURCE or DESTINATION can be the remote. You can’t use both as remote locations.

Rsync Command Options

Rsync offers a large number of useful options that control each and every aspect of its behavior and make it more flexible for synchronizing files. Below is the list of some frequently used Rsync command options:

• -v, -vv -vvv, --verbose – This option is used for verbose logs on standard output. The default rsync runs silently. The higher the number of “v” increases the logging level.
• -a, --archive – Is the most common used option with rsync command. The archive mode is equivalent to the options -rlptgoD . Basically, it includes all necessary options like, recursively, preserving file permissions, symbolic links, file ownership, and timestamps.
• -z, -–compress – Use this option to transfer data in compressed format. Which is useful to save bandwidth.
• -h, –-human-readable – Use this option to print all the outputs in a human-readable format.
• --delete – Use this option only if you need to remove files on the destination, which does not exist on the source.
• --exclude=PATTERN – Exclude files from rsync matching the given pattern.

  rsync --exclude '*.jpg' SRC/ DEST/
  

• --include=PATTERN – Include files in rsync matching the given pattern.

  rsync  --include '*.jpg' --include '*.txt' --exclude '*' SRC/ DEST/
  

Rsync Examples

Here are 12 practical examples of the Rsync command in Unix/Linux systems.

Copy/Sync files and directories local-to-local

Rsync allows synchronizing files between directories on the local system. For example, I have a few backup files on my local machine under the /mnt/backup directory. I want to copy/sync all the files to the USB drive that is mounted on /mnt/disk1. Use the following command to synchronize all files and directories from /opt/backup to /mnt/disk1.

rsync -avhz /opt/backup /mnt/disk1/ 

sending incremental file list
backup/
backup/tecadmin.22Mar2022.sql
backup/tecadmin.22Mar2022.tar.gz
backup/tecadmin.23Mar2022.sql
backup/tecadmin.23Mar2022.tar.gz

sent 1.09G bytes  received 96 bytes  23.39M bytes/sec
total size is 1.47G  speedup is 1.35



Copy/Sync files and directories local-to-remote

Using rsync, we can quickly copy/synchronize files from local to remote systems. For example, I need to transfer a website to a new server. This required website content to be a copy of the remote systems. Use the below command to copy all files from the local /var/www/html directory to the remote (192.168.1.100) systems /var/www/html directory.

rsync -avhz /var/www/html root@192.168.1.100:/var/www/html/ 

The above command is helpful for a system administrator, who copies files between two systems on regular basis.

Copy/Sync files and directories remote-to-local

The Rsync command also allows the source as a remote directory, but then the destination must be local. Use the source directory from the remote host and the destination must be the local directory. For example, I need to copy all backup files stored under /backups directory on remote to the local /opt/backups directory.

rsync -avhz root@192.168.1.100:/backups /opt/backups 

Using Rsync over SSH

The Rsync connects to remote systems using the Rsync daemon directly via TCP. We can instruct rsync to use Secure shell (SSH) to connect remote systems rather than creating a TCP connection.

Use -e option to specify the remote shell. The below command will use ssh as a remote shell:

rsync -avhz -e ssh /src root@192.168.1.100:/dest/  

If the SSH server on the destination is running on non-standard port use this tutorial to connect rsync over non-standard port.

Ignore files and directories already exists on destination

Use the Rsync command with --ignore-existing option to ignore all files, that already exist on the destination. For example, you want to schedule a backup script to copy all files to the backup disk daily. In that case, you don’t want to copy all files again and again. This option will copy those files, that are not available at the destination.

rsync -avhz --ignore-existing  /opt/backups /mnt/disk1 

Update file only if the source is newer

Use --update option to update the file on remote only if there is a newer version is local. The Rsync will create a file on the destination if not exists. Also, the update file on the destination of a local timestamp is newer than the remote system file.

rsync --update -avhz /opt/backups /mnt/disk1/ 

Remove file from source after sync

Use --remove-source-files option to remove a file from the source after successful transfer to the destination. For example, you need to schedule a backup of log files to the remote system. Also, wanted to remote log the file from the local system after successfully copied to the remote. This will help to claim disk space.

rsync --remove-source-files -avhz /var/log root@192.168.1.100:/backup/logs/ 

Exclude specific file or directory with Rsync

You can specify a file or pattern with --exclude=PATTERN option to exclude from rsync.

rsync -avh --exclude '*.zip' --exclude '*.log' /src user@remote.host:/dest 

Include Specific files with Rsync

The default Rsync includes all files under the source directory tree. Now, you need to include only specific files to synchronize to the destination.

Here you can specify --include=PATTERN to include specific files and exclude all files with * .

rsync -avhz -include '*.jpg' --include '*.txt' --exclude '*' /src user@remote.host:/dest 

Display Progress with Rsync

Use --progress option to display progress during the file transfer between source and destination with the rsync command.

rsync -avh --progress /src user@remote.host:/dest 

Chnage file permissions with Rsync

You can instruct the Rsync to change the files owner and group owners on the destination. Use option --chown=USER:GROUP with Rsync to change file permission.

rsync -avh --chown=USER:GROUP /src user@remote.host:/dest 

Similarly, you can use the –chmod=CHMOD option to change file or directory permissions on the destination.

rsync -avh --chmod=755 /src user@remote.host:/dest 

Rsync dry run only

Use --dry-run option to execute dry run only. It will show you similar results as the original command but nothing will update on the destination. Basically, we can use this to check, what the Rsync will update before actually running it.

rsync -avh --dry-run /src user@remote.host:/dest 

Conclusion

Rsync is a great utility for transferring large amounts of data between two machines. You may also use Rsync to back up the entire system at a remote backup location in this article. You’ve learned some practical Rsync command examples in this article that are great for daily tasks.

The post Rsync Commad in Linux With 12 Practical Examples appeared first on TecAdmin.