1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
|
# mscp: multi-threaded scp
[](https://github.com/upa/mscp/actions/workflows/build-ubuntu.yml)
[](https://github.com/upa/mscp/actions/workflows/build-macos.yml)
[](https://github.com/upa/mscp/actions/workflows/build-freebsd.yml)
[](https://github.com/upa/mscp/actions/workflows/test.yml)
`mscp`, a variant of `scp`, copies files over multiple SSH (SFTP)
connections by multiple threads. It enables transferring (1) multiple
files simultaneously and (2) a large file in parallel, reducing the
transfer time for a lot of/large files over networks.
You can use `mscp` like `scp`, for example:
```shell-session
$ mscp srcfile user@example.com:dstfile
```
Remote hosts only need to run standard `sshd` supporting the SFTP
subsystem (e.g. openssh-server), and you need to be able to ssh to the
hosts as usual. `mscp` does not require anything else.
https://github.com/upa/mscp/assets/184632/19230f57-be7f-4ef0-98dd-cb4c460f570d
--------------------------------------------------------------------
Major differences from `scp` on usage:
- Remote-to-remote copy is not supported.
- `-r` option is not needed to transfer directories.
- Checkpointing for resuming failed transfer is supported.
- and any other differences I have not implemented and noticed.
Paper:
- Ryo Nakamura and Yohei Kuga. 2023. Multi-threaded scp: Easy and Fast File Transfer over SSH. In Practice and Experience in Advanced Research Computing (PEARC '23). Association for Computing Machinery, New York, NY, USA, 320–323. https://doi.org/10.1145/3569951.3597582
## Install
- macOS
```console
# Homebrew
brew install upa/tap/mscp
# MacPorts
sudo port install mscp
```
- Ubuntu
```console
sudo add-apt-repository ppa:upaa/mscp
sudo apt-get install mscp
```
- RHEL-based distributions
```console
sudo dnf copr enable upaaa/mscp
sudo dnf install mscp
```
- Single binary `mscp` for x86_64 (not optimal performance)
```console
wget https://github.com/upa/mscp/releases/latest/download/mscp.linux.x86_64.static -O /usr/local/bin/mscp
chmod 755 /usr/local/bin/mscp
```
## Build
mscp depends on a patched [libssh](https://www.libssh.org/). The
patch introduces asynchronous SFTP Write, which is derived from
https://github.com/limes-datentechnik-gmbh/libssh (see [Re: SFTP Write
async](https://archive.libssh.org/libssh/2020-06/0000004.html)).
We test building mscp on Linux (Ubuntu, Rocky, Alma, and Alpine),
macOS, and FreeBSD.
```console
# clone this repository
git clone https://github.com/upa/mscp.git
cd mscp
# prepare patched libssh
git submodule update --init
patch -d libssh -p1 < patch/$(git --git-dir=./libssh/.git describe).patch
# install build dependency
bash ./scripts/install-build-deps.sh
# configure mscp
mkdir build && cd build
cmake ..
# in macOS, you may need OPENSSL_ROOT_DIR for cmake:
# cmake .. -DOPENSSL_ROOT_DIR=$(brew --prefix)/opt/openssl@1.1
# build
make
# install the mscp binary to CMAKE_INSTALL_PREFIX/bin (usually /usr/local/bin)
make install
```
Source tar balls (`mscp-X.X.X.tar.gz`, not `Source code`) in
[Releases page](https://github.com/upa/mscp/releases) contain the patched version
of libssh. So you can start from cmake with it.
## Documentation
[manpage](/doc/mscp.rst) is available.
|
