blob: 3a115f3879ae29cbc9f675b5e94bb45675650099 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
|
# mscp
[](https://github.com/upa/mscp/actions/workflows/build-ubuntu.yml) [](https://github.com/upa/mscp/actions/workflows/build-macos.yml) [](https://github.com/upa/mscp/actions/workflows/test.yml)
`mscp`, a variant of `scp`, copies files over multiple ssh (sftp)
sessions. Multiple threads in mscp transfer (1) multiple files
simultaneously and (2) a large file in parallel. It may shorten the
waiting time for transferring a lot of/large files over networks.
You can use `mscp` like `scp`, for example, `mscp example.com:srcfile
/tmp/dstfile`. Remote hosts only need to run `sshd` supporting the
SFTP subsystem, and you need to be able to ssh to the hosts (as
usual).
Differences from `scp` are:
- remote glob on remote shell expansion is not supported.
- remote to remote copy is not supported.
- `-r` option is not needed.
- and any other differences I have not noticed and implemented...
## Install
- homebrew
```console
brew install upa/tap/mscp
```
- Linux: Download a package for your environment from
[Releases page](https://github.com/upa/mscp/releases).
## Build from source
mscp depends on [libssh](https://www.libssh.org/).
- macOS
```console
brew install libssh
```
- ubuntu
```console
sudo apt-get install libssh-dev
```
- rhel
```console
sudo yum install libssh-devel
```
Clone and build this repositoy.
```console
git clone https://github.com/upa/mscp.git
cd mscp
mkdir build && cd build
cmake .. && make
# install the mscp binary to CMAKE_INSTALL_PREFIX/bin (usually /usr/local/bin)
make install
```
## Run
- Usage
```shell-session
$ mscp
mscp v0.0.0: copy files over multiple ssh connections
Usage: mscp [vqDCHdh] [-n nr_conns]
[-s min_chunk_sz] [-S max_chunk_sz]
[-b sftp_buf_sz] [-B io_buf_sz]
[-l login_name] [-p port] [-i identity_file]
[-c cipher_spec] source ... target
-n NR_CONNECTIONS number of connections (default: half of # of cpu cores)
-s MIN_CHUNK_SIZE min chunk size (default: 64MB)
-S MAX_CHUNK_SIZE max chunk size (default: filesize / nr_conn)
-b SFTP_BUF_SIZE buf size for sftp_read/write (default 131072B)
-B IO_BUF_SIZE buf size for read/write (default 131072B)
Note that this value is derived from
qemu/block/ssh.c. need investigation...
-v increment verbose output level
-q disable output
-D dry run
-l LOGIN_NAME login name
-p PORT port number
-i IDENTITY identity file for publickey authentication
-c CIPHER cipher spec, see `ssh -Q cipher`
-C enable compression on libssh
-H disable hostkey check
-d increment ssh debug output level
-h print this help
```
- Example: copy an 8GB file on tmpfs over a 100Gbps link
- Two Intel Xeon Gold 6130 machines directly connected with Intel E810 100Gbps NICs.
```shell-session
$ mscp /tmp/test.img 10.0.0.1:/tmp/
[===============================================================] 100% 8GB/8GB 3.02GB/s
```
- `-v` options increment verbose output level.
```shell-session
$ mscp test 10.0.0.1:
[===============================================================] 100% 13B/13B 2.41KB/s
$ mscp -v test 10.0.0.1:
file test/test.txt (local) -> ./test/test.txt (remote) 9B
file test/test2/2.txt (local) -> ./test/test2/2.txt (remote) 2B
file test/1.txt (local) -> ./test/1.txt (remote) 2B
copy start: test/test.txt
copy start: test/1.txt
copy start: test/test2/2.txt
copy done: test/1.txt
copy done: test/test2/2.txt
copy done: test/test.txt
[===============================================================] 100% 13B/13B 2.51KB/s
$ mscp -vv -n 4 test 10.0.0.1:
connecting to 10.0.0.1 for checking destinations...
file test/test.txt (local) -> ./test/test.txt (remote) 9B
file test/test2/2.txt (local) -> ./test/test2/2.txt (remote) 2B
file test/1.txt (local) -> ./test/1.txt (remote) 2B
connecting to 10.0.0.1 for a copy thread...
connecting to 10.0.0.1 for a copy thread...
connecting to 10.0.0.1 for a copy thread...
connecting to 10.0.0.1 for a copy thread...
copy start: test/test.txt
copy start: test/1.txt
copy start: test/test2/2.txt
copy done: test/test.txt
copy done: test/test2/2.txt
copy done: test/1.txt
[===============================================================] 100% 13B/13B 3.27KB/s
```
Note: mscp is still under development, and the author is not
responsible for any accidents on mscp.
|
