正文
Elasticsearch 与 Mongodb 数据同步问题
小程序:扫一扫查出行
【扫一扫了解最新限行尾号】
复制小程序
【扫一扫了解最新限行尾号】
复制小程序
1、mongo-connector工具
首先安装python环境
wget http://www.python.org/ftp/python/3.0.1/Python-3.0.1.tgz
tar -zxvf Python-3.0.1.tgz
cd Python-3.0.1
./configure
make all
make install
看来新装的版本生效了,做个软连接应该就OK。
ln -s /usr/local/bin/python3.0 /usr/bin/python
python -V
安装pip
$ wget https://bootstrap.pypa.io/get-pip.py
$ python get-pip.py
$ pip -V#查看pip版本
安装mongo-connector
如果是 Elasticsearch 5.x
pip install 'mongo-connector[elastic5]'
[Reference] https://github.com/mongodb-labs/mongo-connector
执行命令:
mongo-connector -m 192.9.8.204:27017 -t 192.9.8.223:9200 -d elastic2_doc_manager -n wymlib.ym_literature
-m :mongodb地址,-t:目的地址,-d : Used to specify the path to each doc manager file that will be used. -n:指定数据库
2、elasticsearch-river-mongodb
该工具只支持 elasticsearch 1.7.3 及以下版本
bin/plugin --install com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/2.0.9
mongodb-river同步信息属性配置完整版:
$ curl -XPUT "localhost:9200/_river/${es.river.name}/_meta" -d '
{
"type": "mongodb",
"mongodb": {
"servers":
[
{ "host": ${mongo.instance1.host}, "port": ${mongo.instance1.port} },
{ "host": ${mongo.instance2.host}, "port": ${mongo.instance2.port} }
],
"options": {
"secondary_read_preference" : true,
"drop_collection": ${mongo.drop.collection},
"exclude_fields": ${mongo.exclude.fields},
"include_fields": ${mongo.include.fields},
"include_collection": ${mongo.include.collection},
"import_all_collections": ${mongo.import.all.collections},
"initial_timestamp": {
"script_type": ${mongo.initial.timestamp.script.type},
"script": ${mongo.initial.timestamp.script}
},
"skip_initial_import" : ${mongo.skip.initial.import},
"store_statistics" : ${mongo.store.statistics},
},
"credentials":
[
{ "db": "local", "user": ${mongo.local.user}, "password": ${mongo.local.password} },
{ "db": "admin", "user": ${mongo.db.user}, "password": ${mongo.db.password} }
],
"db": ${mongo.db.name},
"collection": ${mongo.collection.name},
"gridfs": ${mongo.is.gridfs.collection},
"filter": ${mongo.filter}
},
"index": {
"name": ${es.index.name},
"throttle_size": ${es.throttle.size},
"bulk_size": ${es.bulk.size},
"type": ${es.type.name}
"bulk": {
"actions": ${es.bulk.actions},
"size": ${es.bulk.size},
"concurrent_requests": ${es.bulk.concurrent.requests},
"flush_interval": ${es.bulk.flush.interval}
}
}
}'
[Reference] https://github.com/richardwilly98/elasticsearch-river-mongodb