diff options
| author | Ryo Nakamura <upa@haeena.net> | 2024-01-13 19:06:56 +0900 |
|---|---|---|
| committer | Ryo Nakamura <upa@haeena.net> | 2024-01-14 18:07:17 +0900 |
| commit | 1479607efe0be21e174bb977db5ad4f988547ffa (patch) | |
| tree | 459dbc32e1e75f31d84becc1f98d13c622304dcc /README.md | |
| parent | 6f4038a48053da7074ad6bea429289329c0a9e3a (diff) | |
add manpage for mscp
doc/mscp.rst is generate from mscp.1 by make generate-mscp-rst.
README is also updateded to reference doc/mscp.rst.
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 124 |
1 files changed, 17 insertions, 107 deletions
@@ -3,17 +3,20 @@ [](https://github.com/upa/mscp/actions/workflows/build-ubuntu.yml) [](https://github.com/upa/mscp/actions/workflows/build-macos.yml) [](https://github.com/upa/mscp/actions/workflows/test.yml) -`mscp`, a variant of `scp`, copies files over multiple ssh (SFTP) -connections. Multiple threads and connections in mscp transfer (1) -multiple files simultaneously and (2) a large file in parallel. It -would shorten the waiting time for transferring a lot of/large files -over networks. +`mscp`, a variant of `scp`, copies files over multiple SSH (SFTP) +connections by multiple threads. It enables transferring (1) multiple +files simultaneously and (2) a large file in parallel, reducing the +transfer time for a lot of/large files over networks. -You can use `mscp` like `scp`, for example, `mscp -user@example.com:srcfile /tmp/dstfile`. Remote hosts only need to run -standard `sshd` supporting the SFTP subsystem (e.g. openssh-server), -and you need to be able to ssh to the hosts as usual. `mscp` does not -require anything else. +You can use `mscp` like `scp`, for example: + +```shell-session +$ mscp user@example.com:srcfile /tmp/dstfile +``` + +Remote hosts only need to run standard `sshd` supporting the SFTP +subsystem (e.g. openssh-server), and you need to be able to ssh to the +hosts as usual. `mscp` does not require anything else. https://github.com/upa/mscp/assets/184632/19230f57-be7f-4ef0-98dd-cb4c460f570d @@ -62,7 +65,7 @@ chmod 755 /usr/local/bin/mscp ## Build -mscp depends on a patched [libssh](https://www.libssh.org/). The +mscp depends on a patched [libssh](https://www.libssh.org/). The patch introduces asynchronous SFTP Write, which is derived from https://github.com/limes-datentechnik-gmbh/libssh (see [Re: SFTP Write async](https://archive.libssh.org/libssh/2020-06/0000004.html)). @@ -94,105 +97,12 @@ make # install the mscp binary to CMAKE_INSTALL_PREFIX/bin (usually /usr/local/bin) make install ``` + Source tar balls (`mscp-X.X.X.tar.gz`, not `Source code`) in [Releases page](https://github.com/upa/mscp/releases) contains the patched version of libssh. So you can start from cmake with it. -## Run - -- Usage - -```console -$ mscp -mscp v0.0.8: copy files over multiple ssh connections - -Usage: mscp [vqDHdNh] [-n nr_conns] [-m coremask] [-u max_startups] - [-s min_chunk_sz] [-S max_chunk_sz] [-a nr_ahead] [-b buf_sz] - [-l login_name] [-p port] [-i identity_file] - [-c cipher_spec] [-M hmac_spec] [-C compress] source ... target -``` - -- Example: copy a 15GB file on memory over a 100Gbps link - - Two Intel Xeon Gold 6130 machines directly connected with Intel E810 100Gbps NICs. - - Default `openssh-server` runs on the remote host. - -```console -$ mscp /var/ram/test.img 10.0.0.1:/var/ram/ -[======================================] 100% 15GB/15GB 1.7GB/s 00:00 ETA -``` - -```console -# with some optimizations. top speed reaches 3.0GB/s. -$ mscp -n 5 -m 0x1f -c aes128-gcm@openssh.com /var/ram/test.img 10.0.0.1:/var/ram/ -[======================================] 100% 15GB/15GB 2.4GB/s 00:00 ETA -``` - -- `-v` option increments verbose output level. - -```console -$ mscp test 10.0.0.1: -[=======================================] 100% 49B /49B 198.8B/s 00:00 ETA -``` - -```console -$ mscp -vv test 10.0.0.1: -file: test/test1 -> ./test/test1 -file: test/testdir/asdf -> ./test/testdir/asdf -file: test/testdir/qwer -> ./test/testdir/qwer -file: test/test2 -> ./test/test2 -we have only 4 chunk(s). set number of connections to 4 -connecting to localhost for a copy thread... -connecting to localhost for a copy thread... -connecting to localhost for a copy thread... -copy start: test/test1 -copy start: test/test2 -copy start: test/testdir/asdf -copy start: test/testdir/qwer -copy done: test/test1 -copy done: test/test2 -copy done: test/testdir/qwer -copy done: test/testdir/asdf -[=======================================] 100% 49B /49B 198.1B/s 00:00 ETA -``` - -- Full usage - -```console -$ mscp -h -mscp v0.0.9-11-g5802679: copy files over multiple ssh connections - -Usage: mscp [vqDHdNh] [-n nr_conns] [-m coremask] [-u max_startups] - [-s min_chunk_sz] [-S max_chunk_sz] [-a nr_ahead] [-b buf_sz] - [-l login_name] [-p port] [-F ssh_config] [-i identity_file] - [-c cipher_spec] [-M hmac_spec] [-C compress] source ... target - - -n NR_CONNECTIONS number of connections (default: floor(log(cores)*2)+1) - -m COREMASK hex value to specify cores where threads pinned - -u MAX_STARTUPS number of concurrent outgoing connections (default: 8) - -s MIN_CHUNK_SIZE min chunk size (default: 64MB) - -S MAX_CHUNK_SIZE max chunk size (default: filesize/nr_conn) - - -a NR_AHEAD number of inflight SFTP commands (default: 32) - -b BUF_SZ buffer size for i/o and transfer - - -v increment verbose output level - -q disable output - -D dry run. check copy destinations with -vvv - -r no effect - - -l LOGIN_NAME login name - -p PORT port number - -F CONFIG path to user ssh config (default ~/.ssh/config) - -i IDENTITY identity file for public key authentication - -c CIPHER cipher spec - -M HMAC hmac spec - -C COMPRESS enable compression: yes, no, zlib, zlib@openssh.com - -H disable hostkey check - -d increment ssh debug output level - -N enable Nagle's algorithm (default disabled) - -h print this help -``` +## Documentation -Note: mscp is still under development, and the author is not -responsible for any accidents due to mscp. +[manpage](/doc/mscp.rst) is available.
\ No newline at end of file |