Common tools used to transfer files on UNIX-like and Linux platforms - Elastic Compute Service

If you want to install software, back up and restore data, or import and export data for Elastic Compute Service (ECS) instances, you can use common file transfer tools on UNIX-like and Linux platforms. This topic describes the basic principles of file transfer and common tools that can be used to transfer files, such as Netcat, Secure Copy (SCP), and rsync. You can select an appropriate method to transfer files based on your business requirements.

Basic principles of file transfer

File transfer, also known as file data communication, is a form of information transfer that transfers file data between data sources and data destinations. During file transfer, the operating system extracts file data to memory for temporary storage and copies the data to a data destination. An encryption operation adds a layer of security to a file, and a copy operation transfers the file as a whole to a data destination. The file does not need to be decrypted during the copy operation. The file needs to be decrypted only if it is packed into a compressed package and the package needs to be opened. Take note of the following items:

A large file cannot be instantly transferred in one piece from one server to another because file transfer is a continuous process. If the transfer is interrupted, the file cannot be transferred to the destination because the file is transferred as a whole.
If multiple files are transferred, the files are transferred in sequence. If the transfer is interrupted, the file that is being transferred cannot be transferred to the destination, but the files that are already transferred to the destination are not affected. A compressed package is considered as one file regardless of the number of files contained in the package.

Netcat, SCP, and rsync are tools that can be used to transfer files. The following section describes the tools and how to use these tools.

Netcat

Netcat is a networking tool that provides various features and powerful file transfer capabilities. Netcat can establish TCP connections, send UDP packets, scan TCP and UDP ports, and handle IPv6 and IPv4 packets.

Parameter description

The following table describes the parameters of Netcat.

Parameter	Description
-C	Maintains the connection between the source server and the destination server.
-g <Gateways>	Specifies router-hop communication gateways. Up to eight gateways can be specified.
-G <Number of indicators>	Specifies the number of source routing indicators. The value must be a multiple of 4.
-i <Latency>	Specifies the time interval for transferring data and scanning communication ports. Unit: seconds.
-l	Enables the listen mode to control incoming data.
-o <Output file>	Specifies the name of the file in which transferred data is converted and saved in hexadecimal format.
-p <Communication port>	Specifies the communication port that is used by the local server.
-r	Specifies the communication port on the local server to use for connection to the remote server.
-s <IP address>	Specifies the IP address of the local source server.
-u	Uses the UDP transmission protocol.
-v	Shows the command execution process.
-w <Timeout period>	Specifies the time to wait before the connection is established. Unit: seconds.
-z	Enables the zero input/output mode, which is used only for scanning communication ports.
-n	Uses IP addresses instead of Domain Name System (DNS) servers.

Sample commands

NC stands for Netcat. Sample commands of NC:

Scan port 21 to port 24. In this example, the IP address 192.168.2.34 is used.

nc -v -w 2 192.168.2.34 -z 21-24

Sample output:

nc: connect to 192.168.2.34 port 21 (tcp) failed: Connection refused
Connection to 192.168.2.34 22 port [tcp/ssh] succeeded!
nc: connect to 192.168.2.34 port 23 (tcp) failed: Connection refused
nc: connect to 192.168.2.34 port 24 (tcp) failed: Connection refused

Copy files from 192.168.2.33 to 192.168.2.34
- Run the following command on the server whose IP address is 192.168.2.34: nc-l 1234 > test.txt.
- Run the following command on the server whose IP address is 192.168.2.33: nc 192.168.2.34 < test.txt
Use memcached by running NC commands
- Run the following command to store data: printf "set key 0 10 6rnresultrn" |nc 192.168.2.34 11211.
- Run the following command to obtain data: printf "get keyrn" |nc 192.168.2.34 11211.
- Run the following command to delete data: printf "delete keyrn" |nc 192.168.2.34 11211.
- Run the following command to check the status of the server whose IP address is 192.168.2.34: printf "statsrn" |nc 192.168.2.34 11211.
- Run the following command to simulate a top command to check the status of the server whose IP address is 192.168.2.34: watch "echo stats" |nc 192.168.2.34 11211.
- Run the following command to clear the cache:
```
printf "flush_allrn" |nc 192.168.2.34 11211       #Exercise caution when you run the command. After the cache is cleared, it cannot be restored.
```

SCP

SCP is a tool that allows you to securely transfer files between two servers (local to local, local to remote, remote to local, or remote to remote) based on the SSH protocol. The use of SCP commands is similar to that of Remote Copy (RCP) commands. We recommend that you use SCP commands because they are more secure than RCP commands.

Note

For more information about SCP, see Use SFTP or SCP to upload files from or download files to a local computer that runs a Linux or macOS operating system.

If authentication is required, SCP prompts you for passwords or security tokens.
SCP commands require SSH to transfer data and use the same authentication mechanism as SSH to provide the same security protection.
SSH is a reliable protocol that provides security for remote logon sessions and other network services. The SSH protocol can prevent information disclosure during remote management. SCP is a tool based on SSH. Servers that transfer data by using SCP commands must support SSH.

Characteristics

SCP has the following characteristics:

Similar to RCP, SCP can retain the file attributes of a file system and retain sub-directories that need to be recursively copied.
Compared with RCP, SCP is less convenient but provides more secure file transfer. You are prompted to enter passwords and configure SSH-related settings when you use SCP to transfer files.

Parameter description

The following table describes the parameters of SCP.

Parameter	Description
-v	Produces verbose output that can be used to debug connection, authentication, or configuration issues.
-B	Enables the batch transfer mode. In this mode, no security tokens or passwords are required during data transfer.
-C	Compresses files or directories during the copy process.
-P	Specifies the SSH port to use if port 22 is not the default SSH port.
-r	Recursively copies the whole directory.
-4	Forcibly uses only IPv4 addresses.
-6	Forcibly uses only IPv6 addresses.

Sample commands

Sample commands of SCP:

Generate a Rivest-Shamir-Adleman (RSA) key.
When you run an SCP command, you are prompted for passwords. If you do not want to enter usernames or passwords each time you run an SCP command to copy files between two servers, you can configure SSH settings.
The preceding command is run to generate an RSA key. Each time you are prompted for passwords and paths for storing keys, press the Enter key to use the default paths and blank passwords. This way, the generated public key is stored in /.ssh/id_rsa.pub and the private key is stored in /.ssh/id_rsa. Then, copy the content of the public key to the /.ssh/authorized_keys file on the server that you want to access. This way, you do not need to enter passwords the next time you access the server.

Copy files between two Linux servers

Basic command syntax:

scp [Optional parameter] file_source file_target

Run one of the following commands to copy a local file to a remote server.

No.	Command	Username and password	Remote directory or file
1	`scp local_file remote_username@remote_ip:remote_folder`	A username is specified. After the command is run, you must enter the corresponding password.	A directory on the remote server is specified. After the command is run, the local file is copied to the specified directory.
2	`scp local_file remote_username@remote_ip:remote_file`	A username is specified. After the command is run, you must enter the corresponding password.	A file name is specified. After the command is run, the local file is copied to the remote server and uses the specified file name.
3	`scp local_file remote_ip:remote_folder`	No username is specified. After the command is run, you must enter a username and a password.	A directory on the remote server is specified. After the command is run, the local file is copied to the specified directory.
4	`scp local_file remote_ip:remote_file`	No username is specified. After the command is run, you must enter a username and a password.	A file name is specified. After the command is run, the local file is copied to the remote server and uses the specified file name.

To use the fourth command to copy the test.txt file to a remote server whose public IP address is 120.XXX.XXX.XXX and set the file name to test02.txt, perform the following steps:

Run the scp test.txt 120.XXX.XXX.XXX:test02.txt command on the local server and enter the logon password of the remote server.
View the file on the remote server.
You can view the test02.txt file on the remote server.

Copy a file from a remote server to a local server
To copy a file from a remote server to a local server, exchange the positions of the last two parameters in the command used to copy a local file to a remote server. Sample command:
```
scp remote_username@remote_ip:remote_folder local_file
```

Copy directories between two Linux servers
Basic command syntax:
```
scp -r file_source file_target
```
- Run one of the following commands to copy a local directory to a remote server:
  - If a username is specified, you must enter the corresponding password after the command is run.
```
scp -r local_file remote_username@remote_ip:remote_folder
```
  - If no username is specified, you must enter a username and a password after the command is run.
```
scp -r local_file remote_ip:remote_folder
```
- Run the following command to copy a directory from a remote server to a local server.
  To copy a directory from a remote server to a local server, exchange the positions of the last two parameters in the command used to copy a local directory to a remote server.
```
scp -r remote_username@remote_ip:remote_folder local_file
```

rsync

rsync is a tool for transferring and synchronizing files in UNIX and Linux operating systems. rsync, a replacement for RCP, can transfer files and directories over Remote Shell Protocol (RSH) or SSH and can be run as a daemon. In rsync daemon mode, the rsync server enables port 873 to connect to local rsync clients. During a client connection, the rsync server validates the security token. If the security token is valid, files can be transferred. The first time the source server is connected to the destination server, all files on the source server are transferred to the destination server. For subsequent connections, only incremental data of the files is transferred.

Installation methods

Run one of the following commands to install rsync.

Note

You can use the installation package manager that comes with each distribution to install rsync.


sudo apt-get  install  rsync      #Install rsync online in Debian or Ubuntu operating systems.
slackpkg  install  rsync          #Install rsync online by using Slackware packages.

Run the following command to install rsync by compiling the source code:

wget http://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz
tar xf rsync-3.0.9.tar.gz
cd rsync-3.0.9
./configure && make && make install

Parameter description

The following table describes the parameters of rsync.

Parameter	Description
-v	Enables the verbose output mode.
-a	Enables the archive mode, which allows rsync to recursively copy files and retain all file attributes. This parameter is equivalent to -rlptgoD.
-r	Recursively copies sub-directories.
-l	Retains symbolic links.
-p	Retains the permissions of the file.
-t	Retains the time information of the file.
-g	Retains the group information of the file.
-o	Retains the owner information of the file.
-D	Retains the information about the server on which the file is located.
-H	Retains hard links.
-S	Handles sparse files to save space for Data Storage and Transfer (DST).
-z	Compresses backup files during transfer.

Working modes

rsync has the following working modes:

Run the following command to copy a local file. In this example, a file in the /home/coremail directory is copied to the /cmbak directory.
```
rsync -avSH /home/coremail/ /cmbak/
```
Run the following command to copy data from a local server to a remote server:
```
rsync -av /home/coremail/ 192.168.11.12:/home/coremail/
```
Run the following command to copy data from a remote server to a local server:
```
rsync -av 192.168.11.11:/home/coremail/ /home/coremail/
```
Run the following command to copy files from a remote rsync daemon to a local server:
```
rsync -av root@172.16.78.192::www /databack
```
Run the following command to copy files from a local server to a remote rsync daemon. This working mode is used when the DST path information contains the :: delimiter.
```
rsync -av /databack root@172.16.78.192::www
```
Run the following command to obtain the list of files on a remote server. In this working mode, data is also transferred by using rsync, except that you do not need to specify the information about the local server.
```
rsync -v rsync://192.168.11.11/data
```

Configuration file description

The following section describes the configuration file of rsync:

cat/etc/rsyncd.conf              #The following section shows the content of the rsync configuration file:
port = 873                       #Specify a port number.
uid = nobody                     #Specify the user identity (UID) of the daemon process when the module transfers files.
gid = nobody                     #Specify the group identity (GID) of the daemon process when the module transfers files.
use chroot = no                  #Specify whether to use chroot to change directories in file systems for running processes.
max connections = 10             #Specify the maximum number of concurrent connections.
strict modes = yes               #Specify whether to check the permissions on the password file.
pid file = /usr/local/rsyncd/rsyncd.pid       #Specify the process ID (PID) file.
lock file = /usr/local/rsyncd/rsyncd.lock     #Specify the lock file that supports the maximum number of concurrent connections. Default value: /var/run/rsyncd.lock.
motd file = /usr/local/rsyncd/rsyncd.motd     #Specify the file that defines the information about the rsync server. You can specify the content of the rsyncd.motd file.
log file = /usr/local/rsyncd/rsync.log        #Specify the log of the rsync server.
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300
[conf]                                   #The following section describes custom parameters.
path = /usr/local/nginx/conf             #Specify the directory that you want to back up.
comment = Nginx conf
ignore errors                            #Ignore specific I/O errors.
read only = no                           #Specify the upload mode. If you set this parameter to no, files can be uploaded by using the rsync client. If you set this parameter to yes, files can only be read by using the rsync client.
write only = no                          #Specify the download mode. If you set this parameter to no, files can be downloaded by using the rsync client. If you set this parameter to yes, files cannot be downloaded by using the rsync client.
hosts allow = 192.168.2.0/24             #Specify IP addresses that can connect to the server.
hosts deny = *                           #Specify IP addresses that cannot connect to the server.
list = false                             #Specify whether to use the module list upon requests.
uid = root
gid = root
auth users = backup                      #Specify a connection username. The specified username is independent of Linux system usernames.
secrets file = /etc/rsyncd.pass          #Specify the password file for authentication.

References

For information about how to use SFTP to transfer files, see Use SFTP or SPC to upload files from or download files to a local computer that runs a Linux or macOS operating system.
If you use Workbench to connect to a Linux instance, you can use Workbench to manage files in your Linux instance in a visualized manner, such as uploading and downloading files. For more information, see Use Workbench to upload a file to a Linux instance.