This topic describes the basic principles of file transfer and common tools that used to transfer files on UNIX-like and Linux platforms. You can choose an appropriate method to transfer files based on your needs.

Basic principles of file transfer

File transfer, also known as file data communication, is a form of information transfer that transfers file data between data sources and data destinations. During a file transfer, the operating system extracts file data to memory for temporary storage and copies the data to a data destination. An encryption operation adds a layer of security to a file, and a copy operation transfers the file as a whole to a data destination. The file does not need to be decrypted during the copy operation. The file needs to be decrypted only when it is packed into a compressed package and the package needs to be opened. A large file cannot be transferred instantly in one piece from one server to another because file transfer is a continuous process. If the transfer is unexpectedly interrupted, the file cannot be transferred to the destination because the file is transferred as a whole. If multiple files are transferred, these files are transferred one by one in sequence. If the transfer is interrupted, the file that is being transferred cannot be transferred to the destination, but the files that are already transferred to the destination are not affected. A compressed package is considered as one file regardless of how many files exist in the package.

Netcat, Secure Copy (SCP), and rsync are tools that can be used to transfer files. The following section describes the tools and how to use these tools:

Netcat

Netcat is a feature-rich networking utility that provides optimal capability to transfer files. It can establish TCP connections, send UDP packets, scan TCP and UDP ports, and handle both IPv6 and IPv4 packets.

Parameter description

The following table describes the main parameters of Netcat.

Parameter or option Description
-C Keeps the source server and the destination server connected to each other.
-d
-g <Gateways> Specifies router-hop communication gateways. Up to eight gateways can be specified.
-G <Number of indicators> Specifies the number of source routing indicators. The value must be a multiple of 4.
-i <Latency> Specifies the time interval for transferring data and scanning communication ports. Unit: seconds.
-l Enables the listen mode to control incoming data.
-o <Output file> Specifies the name of the file in which transferred data is converted and saved in hexadecimal format.
-p <Communication port> Specifies the communication port that is used by the local server.
-r Specifies the communication port on the local server to use for connection to the remote server.
-s <IP address> Specifies the IP address of the local source server.
-u Uses the UDP transmission protocol.
-v Shows the command execution process.
-w <Timeout period> Specifies the time to wait before the connection is established. Unit: seconds.
-z Enables the zero input/output mode, which is used only for scanning communication ports.
-n Uses IP addresses instead of Domain Name System (DNS) servers.

Sample commands

NC stands for Netcat. Sample commands of NC:
  • Scan port 21 to port 24 (In this example, the IP address 192.168.2.34 is used.)
    nc -v -w 2 192.168.2.34 -z 21-24
    Sample output:
    nc: connect to 192.168.2.34 port 21 (tcp) failed: Connection refused
    Connection to 192.168.2.34 22 port [tcp/ssh] succeeded!
    nc: connect to 192.168.2.34 port 23 (tcp) failed: Connection refused
    nc: connect to 192.168.2.34 port 24 (tcp) failed: Connection refused
  • Copy files from 192.168.2.33 to 192.168.2.34
    • Run the following command on the server whose IP address is 192.168.2.34: nc-l 1234 > test.txt.
    • Run the following command on the server whose IP address is 192.168.2.33: nc192.168.2.34 < test.txt.
  • Use memcached by running NC commands
    • Run the following command to store data: printf "set key 0 10 6rnresultrn" |nc 192.168.2.34 11211.
    • Run the following command to obtain data: printf "get keyrn" |nc 192.168.2.34 11211.
    • Run the following command to delete data: printf "delete keyrn" |nc 192.168.2.34 11211.
    • Run the following command to check the status of the server whose IP address is 192.168.2.34: printf "statsrn" |nc 192.168.2.34 11211.
    • Run the following command to simulate a top command to check the status of the server whose IP address is 192.168.2.34: watch "echo stats" |nc 192.168.2.34 11211.
    • Run the following command to clear the cache:
      printf "flush_allrn" |nc 192.168.2.34 11211       #Proceed with caution when you run the command. After the cache is cleared, it cannot be restored.

SCP

SCP is a utility that allows you to securely transfer files between two servers (local to local, local to remote, remote to local, or remote to remote) based on the SSH protocol. The use of SCP commands is similar to that of Remote Copy (RCP) commands. We recommend that you use SCP commands because they are more secure than RCP commands.
  • SCP prompts for passwords or security tokens when they are required for authentication.
  • SCP commands rely on SSH to transfer data and use the same authentication mechanism as SSH to provide the same security protection.

    SSH is a reliable protocol that provides security for remote logon sessions and other network services. The SSH protocol can prevent information disclosure during remote management. SCP is a utility based on SSH so that servers that transfer data by using SCP commands must support SSH.

Characteristics

SCP has the following characteristics:
  • Similar to RCP, SCP can retain the file attributes of a specific file system and retain sub-directories that need to be recursively copied.
  • SCP can transfer files more confidentially. Because of this characteristic, you are prompted to enter passwords and configure SSH-related settings when you use SCP to transfer files. SCP is less convenient but it is a better choice for users who want to transfer files securely.

Parameter description

The following table describes the main parameters of SCP.
Parameter or option Description
-v Produces verbose output that can be used to debug connection, authentication, or configuration issues.
-B Enables the batch transfer mode. In this mode, no security tokens or passwords are required during data transfer.
-C Compresses files or directories during the copy process.
-P Specifies the SSH port to use if port 22 is not the default SSH port.
-r Copies the whole directory recursively.
-4 Forcefully uses only IPv4 addresses.
-6 Forcefully uses only IPv6 addresses.

Sample commands

Sample commands of SCP:
  • Generate an Rivest-Shamir-Adleman (RSA) key.
    When you run an SCP command, you are prompted for passwords. If you do not want to enter usernames or passwords each time you run an SCP command to copy files between two servers, you can configure SSH settings. Sample output

    The preceding command is run to generate an RSA key. Each time you are prompted for passwords and paths for storing keys, press the Enter key to use the default paths and blank passwords. This way, the generated public key is stored in /.ssh/id_rsa.pub and the private key is stored in /.ssh/id_rsa. Then, copy the content of the public key to the /.ssh/authorized_keys file on the server that you want to access so that you do not need to enter passwords the next time you access the server.

  • Copy files between two Linux servers
    Basic command syntax:
    scp [Optional parameter] file_source file_target
    • Run one of the following four commands to copy a local file to a remote server:
      scp local_file remote_username@remote_ip:remote_folder
      scp local_file remote_username@remote_ip:remote_file
      scp local_file remote_ip:remote_folder
      scp local_file remote_ip:remote_file
      Note Command description:
      • In the first and second commands, usernames are specified. After the commands are run, you must enter the corresponding passwords. In the third and fourth commands, no usernames are specified. After the commands are run, you must enter usernames and passwords.
      • In the first and third commands, directories on the remote servers are specified. After the commands are run, local files are copied to the specified directories.
      • In the second and fourth commands, file names are specified. After the commands are run, local files are copied to the remote servers and use the specified file names.
    • Run the following command to copy a file from a remote server to a local server.

      To copy a file from a remote server to a local server, exchange the positions of the last two parameters in the command used to copy a local file to a remote server. Sample command:

      scp remote_username@remote_ip:remote_folder local_file
  • Copy directories between two Linux servers
    Basic command syntax:
    scp -r file_source file_target
    • Run one of the following two commands to copy a local directory to a remote server:
      scp -r local_file remote_username@remote_ip:remote_folder
      scp -r local_file remote_ip:remote_folder
      Note In the first command, a username is specified. After the command is run, you must enter the corresponding password. In the second command, no username is specified. After the command is run, you must enter a username and a password.
    • Run the following command to copy a directory from a remote server to a local server.

      To copy a directory from a remote server to a local server, exchange the positions of the last two parameters in the command used to copy a local directory to a remote server.

      scp -r remote_username@remote_ip:remote_folder local_file

rsync

rsync is a utility for transferring and synchronizing files in UNIX and Linux operating systems. rsync, a replacement for RCP, can transfer files and directories over Remote Shell Protocol (RSH) or SSH and can be run as a daemon. In rsync daemon mode, the rsync server enables port 873 to connect to local rsync clients. During a client connection, the rsync server validates the security token. If the security token is valid, files can be transferred. The first time the source server is connected to the destination server, all files on the source server are transferred to the destination server. For subsequent connections, only incremental data of the files is transferred.

Installation methods

Run one of the following commands to install rsync.
Note You can use the installation package manager that comes with each distribution to install rsync.
sudo apt-get  install  rsync      #Install rsync online in Debian or Ubuntu operating systems.
slackpkg  install  rsync          #Install rsync online by using Slackware packages.
yum install rsync                 #Install rsync in Fedora or Red Hat operating systems.
Run the following command to install rsync by compiling the source code:
wget http://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz
tar xf rsync-3.0.9.tar.gz
cd rsync-3.0.9
./configure && make && make install

Parameter description

The following table describes the main parameters of rsync.
Parameter or option Description
-v Enables the verbose output mode.
-a Enables the archive mode, which tells rsync to copy files recursively and retain all file attributes. This parameter is equivalent to -rlptgoD.
-r Copies sub-directories recursively.
-l Retains symbolic links.
-p Retains the permissions of the file.
-t Retains the time information of the file.
-g Retains the group information of the file.
-o Retains the owner information of the file.
-D Retains the information of the server on which the file is located.
-H Retains hard links.
-S Handles sparse files to save space for Data Storage and Transfer (DST).
-z Compresses backup files during transfer.

Working modes

rsync has the following six working modes:
  • Run the following command to copy a local file. In this example, a file in the /home/coremail directory is copied to the /cmbak directory.
    rsync -avSH /home/coremail/ /cmbak/
  • Run the following command to copy data from a local server to a remote server:
    rsync -av /home/coremail/ 192.168.11.12:/home/coremail/
  • Run the following command to copy data from a remote server to a local server:
    rsync -av 192.168.11.11:/home/coremail/ /home/coremail/
  • Run the following command to copy files from a remote rsync daemon to a local server:
    rsync -av root@172.16.78.192::www /databack
  • Run the following command to copy files from a local server to a remote rsync daemon. This working mode is used when the DST path information contains the :: delimiter.
    rsync -av /databack root@172.16.78.192::www
  • Run the following command to obtain the list of files on a remote server. In this working mode, data is also transferred by using rsync, except that you do not need to specify the information of the local server.
    rsync -v rsync://192.168.11.11/data

Configuration file description

The following section describes the configuration file of rsync:

cat/etc/rsyncd.conf              #The following section shows the content of the rsync configuration file:
port = 873                       #Specify a port number.
uid = nobody                     #Specify the user identity (UID) of the daemon process when the module transfers files.
gid = nobody                     #Specify the group identity (GID) of the daemon process when the module transfers files.
use chroot = no                  #Specify whether to use chroot to change directories in file systems for running processes.
max connections = 10             #Specify the maximum number of concurrent connections.
strict modes = yes               #Specify whether to check the permissions on the password file.
pid file = /usr/local/rsyncd/rsyncd.pid       #Specify the process ID (PID) file.
lock file = /usr/local/rsyncd/rsyncd.lock     #Specify the lock file that supports the maximum number of concurrent connections. Default value: /var/run/rsyncd.lock.
motd file = /usr/local/rsyncd/rsyncd.motd     #Specify the file that defines the information of the rsync server. You can specify the content of the rsyncd.motd file.
log file = /usr/local/rsyncd/rsync.log        #Specify the log of the rsync server.
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300
[conf]                                   #The following section describes custom parameters.
path = /usr/local/nginx/conf             #Specify the directory to be backed up.
comment = Nginx conf
ignore errors                            #Ignore specific I/O errors.
read only = no                           #Specify the upload mode. If this parameter is set to no, files can be uploaded by using the rsync client. If this parameter is set to yes, files can only be read by using the rsync client.
write only = no                          #Specify the download mode. If this parameter is set to no, files can be downloaded by using the rsync client. If this parameter is set to yes, files cannot be downloaded by using the rsync client.
hosts allow = 192.168.2.0/24             #Specify IP addresses that can connect to the server.
hosts deny = *                           #Specify IP addresses that cannot connect to the server.
list = false                             #Specify whether to use the module list upon requests.
uid = root
gid = root
auth users = backup                      #Specify a connection username. The username specified here has nothing to do with Linux system usernames.
secrets file = /etc/rsyncd.pass          #Specify the password file for authentication.