blob: b1cb254a5b1fdf51ace30b03ef234a200d6658ef (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
|
# mscp
[](https://github.com/upa/mscp/actions/workflows/build-ubuntu.yml) [](https://github.com/upa/mscp/actions/workflows/build-macos.yml) [](https://github.com/upa/mscp/actions/workflows/test.yml)
`mscp`, a variant of `scp`, copies files over multiple ssh (SFTP)
connections. Multiple threads and connections in mscp transfer (1)
multiple files simultaneously and (2) a large file in parallel. It
would shorten the waiting time for transferring a lot of/large files
over networks.
You can use `mscp` like `scp`, for example, `mscp
user@example.com:srcfile /tmp/dstfile`. Remote hosts only need to run
standard `sshd` supporting the SFTP subsystem, and you need to be able
to ssh to the hosts (as usual). `mscp` does not require anything else.
Differences from `scp`:
- remote glob on remote shell expansion is not supported.
- remote to remote copy is not supported.
- `-r` option is not needed.
- and any other differences I have not implemented and noticed...
## Install
- homebrew
```console
brew install upa/tap/mscp
```
- Linux
Download a package for your environment from [Releases
page](https://github.com/upa/mscp/releases).
## Build from source
mscp depends on [libssh](https://www.libssh.org/).
- macOS
```console
brew install libssh
```
- ubuntu
```console
sudo apt-get install libssh-dev
```
- rhel
```console
sudo yum install libssh-devel
```
Clone and build this repositoy.
```console
git clone https://github.com/upa/mscp.git
cd mscp
mkdir build && cd build
cmake .. && make
# install the mscp binary to CMAKE_INSTALL_PREFIX/bin (usually /usr/local/bin)
make install
```
## Run
- Usage
```console
$ mscp
mscp v0.0.0: copy files over multiple ssh connections
Usage: mscp [vqDCHdh] [-n nr_conns]
[-s min_chunk_sz] [-S max_chunk_sz]
[-b sftp_buf_sz] [-B io_buf_sz] [-a nr_ahead]
[-l login_name] [-p port] [-i identity_file]
[-c cipher_spec] source ... target
```
- Example: copy an 8GB file on tmpfs over a 100Gbps link
- Two Intel Xeon Gold 6130 machines directly connected with Intel E810 100Gbps NICs.
```console
$ mscp /tmp/test.img 10.0.0.1:/tmp/
[=====================================================] 100% 8GB/8GB 3.02GB/s
```
- `-v` option increments verbose output level.
```console
$ mscp test 10.0.0.1:
[=====================================================] 100% 13B/13B 2.41KB/s
$ mscp -v test 10.0.0.1:
file test/test.txt (local) -> ./test/test.txt (remote) 9B
file test/test2/2.txt (local) -> ./test/test2/2.txt (remote) 2B
file test/1.txt (local) -> ./test/1.txt (remote) 2B
copy start: test/test.txt
copy start: test/1.txt
copy start: test/test2/2.txt
copy done: test/1.txt
copy done: test/test2/2.txt
copy done: test/test.txt
[=====================================================] 100% 13B/13B 2.51KB/s
$ mscp -vv -n 4 test 10.0.0.1:
connecting to 10.0.0.1 for checking destinations...
file test/test.txt (local) -> ./test/test.txt (remote) 9B
file test/test2/2.txt (local) -> ./test/test2/2.txt (remote) 2B
file test/1.txt (local) -> ./test/1.txt (remote) 2B
connecting to 10.0.0.1 for a copy thread...
connecting to 10.0.0.1 for a copy thread...
connecting to 10.0.0.1 for a copy thread...
connecting to 10.0.0.1 for a copy thread...
copy start: test/test.txt
copy start: test/1.txt
copy start: test/test2/2.txt
copy done: test/test.txt
copy done: test/test2/2.txt
copy done: test/1.txt
[=====================================================] 100% 13B/13B 3.27KB/s
```
- Full usage
```console
$ mscp -h
mscp v0.0.0: copy files over multiple ssh connections
Usage: mscp [vqDCHdh] [-n nr_conns]
[-s min_chunk_sz] [-S max_chunk_sz]
[-b sftp_buf_sz] [-B io_buf_sz] [-a nr_ahead]
[-l login_name] [-p port] [-i identity_file]
[-c cipher_spec] source ... target
-n NR_CONNECTIONS number of connections (default: half of # of cpu cores)
-s MIN_CHUNK_SIZE min chunk size (default: 64MB)
-S MAX_CHUNK_SIZE max chunk size (default: filesize / nr_conn)
-b SFTP_BUF_SIZE buf size for sftp_read/write (default 131072B)
-B IO_BUF_SIZE buf size for read/write (default 131072B)
Note that the default value is derived from
qemu/block/ssh.c. need investigation...
-b and -B affect only local to remote copy
-a NR_AHEAD number of inflight SFTP read commands (default 16)
-v increment verbose output level
-q disable output
-D dry run
-l LOGIN_NAME login name
-p PORT port number
-i IDENTITY identity file for publickey authentication
-c CIPHER cipher spec, see `ssh -Q cipher`
-C enable compression on libssh
-H disable hostkey check
-d increment ssh debug output level
-h print this help
```
Note: mscp is still under development, and the author is not
responsible for any accidents due to mscp.
## Build with Async Write
Asynchronous SFTP write improves local-to-remote copy throughput. The
following procedure is how to build mscp with libssh with a
`sftp_async_write` patch based on
https://github.com/limes-datentechnik-gmbh/libssh (see [Re: SFTP Write
async](https://archive.libssh.org/libssh/2020-06/0000004.html))
```console
# install required package
sudo apt install libkrb5-dev
# clone this repositoy
git clone https://github.com/upa/mscp -b async-write
cd mscp
# build patched libssh
git clone https://git.libssh.org/projects/libssh.git/ --depth=10 -b libssh-0.10.4
cd libssh && git apply ../patch/libssh-0.10.4.patch
mkdir build && cd build
cmake .. -DCMAKE_INSTALL_PREFIX=../../libssh-installed
make && make install
# build mscp with the patched libssh
mv ../.. # mv to mscp dir
mkdir build && cd build
# on ubuntu
cmake .. -DLIBSSH_PATH=$(pwd)/../libssh-installed -DWITH_ASYNC_WRITE=1
# on macOS (intel)
cmake .. -DLIBSSH_PATH=$(pwd)/../libssh-installed -DWITH_ASYNC_WRITE=1 \
-DOPENSSL_ROOT_DIR=/usr/local/opt/openssl
make
```
|
