使用BBCP来提升跨互联网的数据传输速度

小程序：扫一扫查出行
【扫一扫了解最新限行尾号】
复制小程序

背景介绍：
目前项目在美国东西部以及欧洲都有服务器节点，跨互联网的数据传输速度很不稳定，之前我们主要是通过SCP以及Rsync等方式进行数据传输的。
无意间发现了BBCP这个软件之后，经过测试，效果非常好，速度提升效果很明显，并且传输速度一直比较稳定，同时支持日志以及失败后重试等参数，非常不错。

参考资料：
http://www.slac.stanford.edu/~abh/bbcp/
http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm
https://www.olcf.ornl.gov/kb_articles/transferring-data-with-bbcp/

安装配置：

[aikaiyuan@aikaiyuan~]$ sudo wget http://www.slac.stanford.edu/~abh/bbcp/bin/amd64_rhel60/bbcp -O /usr/bin/bbcp
[aikaiyuan@aikaiyuan~]$ sudo chmod +x /usr/bin/bbcp[dong@west-ads1 ~]$ sudo wget http://www.slac.stanford.edu/~abh/bbcp/bin/amd64_rhel60/bbcp -O /usr/bin/bbcp
[dong@west-ads1 ~]$ sudo chmod +x /usr/bin/bbcp[aikaiyuan@aikaiyuan~]$ which bbcp
/usr/bin/bbcp
[aikaiyuan@aikaiyuan~]$ ssh west-ads1 which bbcp
/usr/bin/bbcp[aikaiyuan@aikaiyuan~]$ cd dong
[aikaiyuan@aikaiyuandong]$ sudo dd if=/dev/zero of=/home/adsymp/dong/file.2g bs=1024M count=2

2+0 records in
2+0 records out
2147483648 bytes (2.1 GB) copied, 45.9129 s, 46.8 MB/s

[aikaiyuan@aikaiyuandong]$ ls -lh
total 2.0G
-rw-r--r-- 1 root root 2.0G Mar 4 06:40 file.2g[aikaiyuan@aikaiyuandong]$ time bbcp -r -P 2 -V -w 8m -s 16 file.2g west-ads1:/home/adsymp/dong/

bbcp: Window size reduced to 245760 bytes.
bbcp: Indexing files to be copied...
bbcp: Copying 0 files in 0 directories.
Source sc2-admin1002.drawbrid.ge using initial send window of 18700
Target iad2-admin1002.drawbrid.ge using initial recv window of 87380
bbcp: Creating /home/adsymp/aikaiyuan/file.2g
bbcp: 140304 06:46:12 0% done; 8.5 MB/s, avg 8.5 MB/s
bbcp: 140304 06:46:14 1% done; 6.9 MB/s, avg 7.5 MB/s
...
bbcp: 140304 06:51:46 99% done; 7.7 MB/s, avg 6.1 MB/s
bbcp: 140304 06:51:48 99% done; 3.3 MB/s, avg 6.1 MB/s
Source cpu=3.643 (sys=3.552 usr=0.091).
File /home/adsymp/aikaiyuan/file.2g created; 2147483648 bytes at 6.0 MB/s
48 buffers used with 0 reorders; peaking at 0.
Source sc2-admin1002.drawbrid.ge using a final send window of 433840
Target cpu=15.149 (sys=14.505 usr=0.644).
Target iad2-admin1002.drawbrid.ge using a final recv window of 2298624
1 file copied at effectively 6.0 MB/sreal 5m42.236s
user 0m0.104s
sys 0m3.567s