blob: 0fdd0f562ee94d701dbef7f2db0cbe53cc54ec43 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
|
# mscp
`mscp`, a variant of `scp`, copies files over multiple ssh (sftp)
sessions. Multiple threads in mscp transfer (1) multiple files
simultaneously and (2) a large file in parallel.
The usage of mscp is similar to scp, e.g., `mscp example.com:srcfile
/tmp/dstfile`, except remote glob on remote shell expansion and `-r`
is not needed on mscp.
## Build
mscp depends on [libssh](https://www.libssh.org/).
- macOS
```console
brew install libssh
```
- ubuntu
```console
sudo apt-get install libssh-dev
```
Clone and build this repositoy.
```console
git clone https://github.com/upa/mscp.git
cd mscp
mkdir build
cd build
cmake ..
make
```
## Run
- Usage
```shell-session
./mscp -h
mscp: copy files over multiple ssh connections
Usage: mscp [CvqDdh] [-n nr_conns] [-s min_chunk_sz] [-S max_chunk_sz]
[-b sftp_buf_sz] [-B io_buf_sz]
[-l login_name] [-p port] [-i identity_file]
[-c cipher_spec] source ... target
-n NR_CONNECTIONS number of connections (default: half of # of cpu cores)
-s MIN_CHUNK_SIZE min chunk size (default: 64MB)
-S MAX_CHUNK_SIZE max chunk size (default: filesize / nr_conn)
-b SFTP_BUF_SIZE buf size for sftp_read/write (default 131072B)
-B IO_BUF_SIZE buf size for read/write (default 131072B)
Note that this value is derived from
qemu/block/ssh.c. need investigation...
-v increment verbose output level
-q disable output
-D dry run
-l LOGIN_NAME login name
-p PORT port number
-i IDENTITY identity file for publickey authentication
-c CIPHER cipher spec, see `ssh -Q cipher`
-C enable compression on libssh
-d increment ssh debug output level
-h print this help
```
- Copy an 8GB file over a 100Gbls link
- Two Intel Xeon Gold 6130 machines directly connected with Intel E810 100Gbps NICs.
```shell-session
$ ./mscp /tmp/test.img 10.0.0.1:/tmp/
[===============================================================] 100% 8GB/8GB 3.02GB/s
$
```
- `-v` options increment verbose output level.
```shell-session
$ ./mscp test 10.0.0.1:
[===============================================================] 100% 13B/13B 2.41KB/s
$ ./mscp -v test 10.0.0.1:
file test/test.txt (local) -> ./test/test.txt (remote) 9B
file test/test2/2.txt (local) -> ./test/test2/2.txt (remote) 2B
file test/1.txt (local) -> ./test/1.txt (remote) 2B
copy start: test/test.txt
copy start: test/1.txt
copy start: test/test2/2.txt
copy done: test/1.txt
copy done: test/test2/2.txt
copy done: test/test.txt
[===============================================================] 100% 13B/13B 2.51KB/s
$ ./mscp -vv -n 4 test 10.0.0.1:
connecting to 10.0.0.1 for checking destinations...
file test/test.txt (local) -> ./test/test.txt (remote) 9B
file test/test2/2.txt (local) -> ./test/test2/2.txt (remote) 2B
file test/1.txt (local) -> ./test/1.txt (remote) 2B
connecting to 10.0.0.1 for a copy thread...
connecting to 10.0.0.1 for a copy thread...
connecting to 10.0.0.1 for a copy thread...
connecting to 10.0.0.1 for a copy thread...
copy start: test/test.txt
copy start: test/1.txt
copy start: test/test2/2.txt
copy done: test/test.txt
copy done: test/test2/2.txt
copy done: test/1.txt
[===============================================================] 100% 13B/13B 3.27KB/s
```
Note: mscp is still under development, and the author is not
responsible for any accidents on mscp.
|
