blob: 87abdda174b2a776a2f19e20551b905a23f6cb06 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
|
# mscp
[](https://github.com/upa/mscp/actions/workflows/build-ubuntu.yml) [](https://github.com/upa/mscp/actions/workflows/build-macos.yml) [](https://github.com/upa/mscp/actions/workflows/test.yml)
`mscp`, a variant of `scp`, copies files over multiple ssh (SFTP)
connections. Multiple threads and connections in mscp transfer (1)
multiple files simultaneously and (2) a large file in parallel. It
would shorten the waiting time for transferring a lot of/large files
over networks.
You can use `mscp` like `scp`, for example, `mscp
user@example.com:srcfile /tmp/dstfile`. Remote hosts only need to run
standard `sshd` supporting the SFTP subsystem (e.g. openssh-server),
and you need to be able to ssh to the hosts as usual. `mscp` does not
require anything else.
Differences from `scp` on usage:
- remote glob on remote shell expansion is not supported.
- remote to remote copy is not supported.
- `-r` option is not needed.
- and any other differences I have not implemented and noticed.
## Install
- homebrew
```console
brew install upa/tap/mscp
```
- Linux
Download a package for your environment from [Releases
page](https://github.com/upa/mscp/releases).
## Build
mscp depends on a patched [libssh](https://www.libssh.org/). The
patch introduces asynchronous SFTP Write, which is derived from
https://github.com/limes-datentechnik-gmbh/libssh (see [Re: SFTP Write
async](https://archive.libssh.org/libssh/2020-06/0000004.html)).
Currently macOS and Linux (Ubuntu, CentOS, Rocky) are supported.
```console
# clone this repository
git clone https://github.com/upa/mscp.git
cd mscp
# prepare patched libssh
git submodule update --init
patch -d libssh -p1 < patch/libssh-0.10.4.patch
# install build dependency
bash ./scripts/install-build-deps.sh
# configure mscp
mkdir build && mv build
cmake ..
## in macOS, you may need OPENSSL_ROOT_DIR for cmake:
cmake .. -DOPENSSL_ROOT_DIR=$(brew --prefix)/opt/openssl@1.1
# build
make
# install the mscp binary to CMAKE_INSTALL_PREFIX/bin (usually /usr/local/bin)
make install
```
## Run
- Usage
```console
$ mscp
mscp v0.0.5: copy files over multiple ssh connections
Usage: mscp [vqDCHdNh] [-n nr_conns] [-m coremask]
[-s min_chunk_sz] [-S max_chunk_sz] [-a nr_ahead] [-b buf_sz]
[-l login_name] [-p port] [-i identity_file]
[-c cipher_spec] [-M hmac_spec] source ... target
```
- Example: copy a 15GB file on memory over a 100Gbps link
- Two Intel Xeon Gold 6130 machines directly connected with Intel E810 100Gbps NICs.
- Default `openssh-server` runs on the remote host.
```console
$ mscp /var/ram/test.img 10.0.0.1:/var/ram/
[======================================] 100% 15GB/15GB 1.7GB/s 00:00 ETA
```
```console
# with some optimizations. top speed reaches 3.0GB/s.
$ mscp -n 5 -m 0x1f -c aes128-gcm@openssh.com /var/ram/test.img 10.0.0.1:/var/ram/
[======================================] 100% 15GB/15GB 2.4GB/s 00:00 ETA
```
- `-v` option increments verbose output level.
```console
$ mscp test 10.0.0.1:
[======================================] 100% 26B /26B 6.3KB/s 00:00 ETA
```
```console
$ mscp -vv test 10.0.0.1:
number of connections: 7
connecting to 10.0.0.1 for checking destinations...
file test/testdir/asdf (local) -> ./test/testdir/asdf (remote) 9B
file test/testdir/qwer (local) -> ./test/testdir/qwer (remote) 5B
file test/test1 (local) -> ./test/test1 (remote) 6B
file test/test2 (local) -> ./test/test2 (remote) 6B
we have only 4 chunk(s). set number of connections to 4
connecting to 10.0.0.1 for a copy thread...
connecting to 10.0.0.1 for a copy thread...
connecting to 10.0.0.1 for a copy thread...
copy start: test/test1
copy start: test/test2
copy done: test/test1
copy start: test/testdir/asdf
copy done: test/test2
copy start: test/testdir/qwer
copy done: test/testdir/qwer
copy done: test/testdir/asdf
[======================================] 100% 26B /26B 5.2KB/s 00:00 ETA
```
- Full usage
```console
$ mscp -h
mscp v0.0.5: copy files over multiple ssh connections
Usage: mscp [vqDCHdNh] [-n nr_conns] [-m coremask]
[-s min_chunk_sz] [-S max_chunk_sz] [-a nr_ahead] [-b buf_sz]
[-l login_name] [-p port] [-i identity_file]
[-c cipher_spec] [-M hmac_spec] source ... target
-n NR_CONNECTIONS number of connections (default: floor(log(cores)*2)+1)
-m COREMASK hex value to specify cores where threads pinned
-s MIN_CHUNK_SIZE min chunk size (default: 64MB)
-S MAX_CHUNK_SIZE max chunk size (default: filesize/nr_conn)
-a NR_AHEAD number of inflight SFTP commands (default: 32)
-b BUF_SZ buffer size for i/o and transfer
-v increment verbose output level
-q disable output
-D dry run
-l LOGIN_NAME login name
-p PORT port number
-i IDENTITY identity file for public key authentication
-c CIPHER cipher spec
-M HMAC hmac spec
-C enable compression on libssh
-H disable hostkey check
-d increment ssh debug output level
-N disable tcp nodelay (default on)
-h print this help
```
Note: mscp is still under development, and the author is not
responsible for any accidents due to mscp.
|
