diff options
| -rw-r--r-- | README.md | 119 |
1 files changed, 119 insertions, 0 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..0fdd0f5 --- /dev/null +++ b/README.md @@ -0,0 +1,119 @@ +# mscp + +`mscp`, a variant of `scp`, copies files over multiple ssh (sftp) +sessions. Multiple threads in mscp transfer (1) multiple files +simultaneously and (2) a large file in parallel. + +The usage of mscp is similar to scp, e.g., `mscp example.com:srcfile +/tmp/dstfile`, except remote glob on remote shell expansion and `-r` +is not needed on mscp. + +## Build + +mscp depends on [libssh](https://www.libssh.org/). + +- macOS + +```console +brew install libssh +``` + +- ubuntu + +```console +sudo apt-get install libssh-dev +``` + +Clone and build this repositoy. + +```console +git clone https://github.com/upa/mscp.git + +cd mscp +mkdir build +cd build +cmake .. +make +``` + + +## Run + +- Usage + +```shell-session +./mscp -h +mscp: copy files over multiple ssh connections + +Usage: mscp [CvqDdh] [-n nr_conns] [-s min_chunk_sz] [-S max_chunk_sz] + [-b sftp_buf_sz] [-B io_buf_sz] + [-l login_name] [-p port] [-i identity_file] + [-c cipher_spec] source ... target + + -n NR_CONNECTIONS number of connections (default: half of # of cpu cores) + -s MIN_CHUNK_SIZE min chunk size (default: 64MB) + -S MAX_CHUNK_SIZE max chunk size (default: filesize / nr_conn) + -b SFTP_BUF_SIZE buf size for sftp_read/write (default 131072B) + -B IO_BUF_SIZE buf size for read/write (default 131072B) + Note that this value is derived from + qemu/block/ssh.c. need investigation... + -v increment verbose output level + -q disable output + -D dry run + + -l LOGIN_NAME login name + -p PORT port number + -i IDENTITY identity file for publickey authentication + -c CIPHER cipher spec, see `ssh -Q cipher` + -C enable compression on libssh + -d increment ssh debug output level + -h print this help +``` + +- Copy an 8GB file over a 100Gbls link + - Two Intel Xeon Gold 6130 machines directly connected with Intel E810 100Gbps NICs. + +```shell-session +$ ./mscp /tmp/test.img 10.0.0.1:/tmp/ +[===============================================================] 100% 8GB/8GB 3.02GB/s +$ +``` + +- `-v` options increment verbose output level. + +```shell-session +$ ./mscp test 10.0.0.1: +[===============================================================] 100% 13B/13B 2.41KB/s + +$ ./mscp -v test 10.0.0.1: +file test/test.txt (local) -> ./test/test.txt (remote) 9B +file test/test2/2.txt (local) -> ./test/test2/2.txt (remote) 2B +file test/1.txt (local) -> ./test/1.txt (remote) 2B +copy start: test/test.txt +copy start: test/1.txt +copy start: test/test2/2.txt +copy done: test/1.txt +copy done: test/test2/2.txt +copy done: test/test.txt +[===============================================================] 100% 13B/13B 2.51KB/s + +$ ./mscp -vv -n 4 test 10.0.0.1: +connecting to 10.0.0.1 for checking destinations... +file test/test.txt (local) -> ./test/test.txt (remote) 9B +file test/test2/2.txt (local) -> ./test/test2/2.txt (remote) 2B +file test/1.txt (local) -> ./test/1.txt (remote) 2B +connecting to 10.0.0.1 for a copy thread... +connecting to 10.0.0.1 for a copy thread... +connecting to 10.0.0.1 for a copy thread... +connecting to 10.0.0.1 for a copy thread... +copy start: test/test.txt +copy start: test/1.txt +copy start: test/test2/2.txt +copy done: test/test.txt +copy done: test/test2/2.txt +copy done: test/1.txt +[===============================================================] 100% 13B/13B 3.27KB/s +``` + +Note: mscp is still under development, and the author is not +responsible for any accidents on mscp. |