Synchronize Data From MongoDB to Elasticsearch

Your mongo replica set and elasticsearch are hosted on 2 servers. The mongo server is on "mip:mport". The ES server is on "esip:esport".

On another server, install mongo-connector with pip install mongo-connector. If there are some errors, it may be caused by absence of gcc. On Ubuntu, install them with sudo apt-get install build-essential python-dev.

mongo-connector -m mip:mport -t esip:esport -d elastic_doc_manager --admin-username root --password rootPwd

That's it! Now when you insert documents into MongoDB, you can see it on elasticsearch.

We've studied Mongo River and Mongoosatic, neither is better than this.

Note 1 (2015.11.15):

阿里云服务器上，用普通用户es启动es服务(bin/elasticsearch)，同一用户启动mongo-connector后，报 "AttributeError: 'IndicesClient' object has no attribute 'delete_mapping'"错误， es版本为2.0，mongo-connector版本为2.1，将es版本改为1.7，mongo-connector改为1.3后， 仍然出现这一错误，原因是MongoDB中删除了test库中的一个Collection，但ES中仍然有，就会出现这个错误， 解决方法是清空ES的test重新同步一遍： curl -XDELETE http://123.57.188.18:9322/test. Ref: https://github.com/mongodb-labs/mongo-connector/issues/349

Note 2:

Use root to sync is unsafe. We create an user with "clusterManager" role to do this work. Login with root, and run:

use admin
db.createUser(
    {
      user: "clustermgr",
      pwd: "cmpwd",
      roles: [ "clusterManager" ]
    }
)

But run the command mongo-connector -m 119.254.211.28:15515 -t localhost:9200 -d elastic_doc_manager --admin-username clustermgr --password cmpwd will be the error OperationFailure: database error: not authorized for query on local.oplog.rs.