正文
python BitTornado P2P分发大文件
小程序:扫一扫查出行
【扫一扫了解最新限行尾号】
复制小程序
【扫一扫了解最新限行尾号】
复制小程序
P2P分发大文件思路
1、将软件包生成种子文件
2、通过saltstack将种子文件分发至每台服务器
3、每台服务器进行种子下载
推荐使用Twitter开源的murder。Twitter用它来分发大文件完成代码更新。
下载地址: https://github.com/lg/murder
本文介绍murder的dist,它是基于python开发的。
murder dist
-
murder_tracker.py 可以理解它为调度中心
-
murder_make_torrent.py 生成种子,并且注册在调度中心
-
murder_client.py 播种和获取文件包(上传、下载)
举例部署结构:
192.168.1.11 调度中心
192.168.1.12 种子生成、第一个播种文件服务器
192.168.1.100-200 下载文件
1、启动调度中心
python murder_tracker.py --port 8998 --dfile data --logfile urder_tracker.log
- --port 监听的端口,默认是8998
- --dfile 存储近期下载信息的文件
- --logfile 日志文件,默认是标准输出
shell启动脚本
#!/bin/bash
#
# Start/Stop murder-tracker
#
# description: murder-tracker
# processname: murder-tracker if [ -f /etc/rc.d/init.d/functions ]; then
. /etc/rc.d/init.d/functions
fi name="murder-tracker"
murder_tracker_bin="/usr/local/dist/murder_tracker.py"
murder_tracker_log="/usr/local/dist/logs/murder_tracker.log"
murder_tracker_data="/usr/local/dist/var/tracker_data"
murder_user=murder find_tracker_process () {
PID=`ps -ef | grep murder_tracker | grep python |grep -v $|grep -v grep|grep -v sh|grep -v root| awk '{ print $2 }'`
echo $PID
} start () {
getent passwd $murder_user >/dev/null || useradd -r -s /sbin/nologin $murder_user
LOG_DIR=`dirname ${murder_tracker_log}`
DATA_DIR=`dirname ${murder_tracker_data}`
if [ ! -d $LOG_DIR ]; then
echo -e "\e[35mLog dir ${LOG_DIR} doesn't exist. Creating\e[0m"
mkdir -p $LOG_DIR
fi
if [ ! -d $DATA_DIR ]; then
echo -e "\e[35mLog dir ${DATA_DIR} doesn't exist. Creating\e[0m"
mkdir -p $DATA_DIR
fi
chown -R $murder_user:$murder_user $DATA_DIR $LOG_DIR find_tracker_process
if [ "$PID" != "" ]; then
echo -e "\e[35m$name is already running!\e[0m"
else
daemon --user $murder_user nohup python $murder_tracker_bin --dfile $murder_tracker_data --logfile $murder_tracker_log > /dev/null >& &
echo -e "\e[35mStarting $name Done\e[0m"
fi
} stop () {
find_tracker_process
if [ "$PID" != "" ]; then
echo -e "\e[35mStopping $name\e[0m"
kill $PID
else
echo -e "\e[35m$name is not running yet\e[0m"
fi
} case $ in
start)
start
;;
stop)
stop
exit
;;
restart)
stop
sleep
start
;;
status)
find_tracker_process
if [ "$PID" != "" ]; then
echo -e "\e[35m$name is running: $PID\e[0m"
exit
else
echo -e "\e[35m$name is not running\e[0m"
exit
fi
;;
*)
echo -e "\e[35mUsage: $0 {start|stop|restart|status}\e[0m"
RETVAL=
esac
exit
2、生成种子文件
python murder_make_torrent.py deploy.tar.gz 192.168.1.11:8998 deploy.torrent
3、创建第一个播种文件服务节点
python murder_client.py seed deploy.torrent deploy.tar.gz 192.168.1.12
- 最后一个参数是本机的IP地址
4、通过saltstack将种子文件deploy.torrent发送到每台服务器上
5、所有节点192.168.1.100-200获取文件包,这些服务器下载文件,同时也承担播种节点
python murder_client.py peer deploy.torrent deploy.tar.gz 192.168.1.xxx
- 最后一个参数是本机的IP地址
备注:这些主机分发时防火墙需要放开,程序中默认使用10000-60000之间端口,BitTornado/download_bt1.py中定义
('minport', 10000, 'minimum port to listen on, counts up if unavailable'),
('maxport', 60000, 'maximum port to listen on'),
('random_port', 1, 'whether to choose randomly inside the port range ' +
'instead of counting up linearly'),
参考博文:http://john88wang.blog.51cto.com/2165294/1793080