在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称:go-mysql-org/go-mysql-elasticsearch开源软件地址:https://github.com/go-mysql-org/go-mysql-elasticsearch开源编程语言:Go 98.1%开源软件介绍:go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses Call for Committer/MaintainerSorry that I have no enough time to maintain this project wholly, if you like this project and want to help me improve it continuously, please contact me through email ([email protected]). Requirement: In the email, you should list somethings(including but not limited to below) to make me believe we can work together. Your GitHub ID The contributions to go-mysql-elasticsearch before, including PRs or Issues. The reason why you can improve go-mysql-elasticsearch. Install
How to use?
Notice
SourceIn go-mysql-elasticsearch, you must decide which tables you want to sync into elasticsearch in the source config. The format in config file is below:
If you want to sync all table in database, you can use asterisk(*).
RuleBy default, go-mysql-elasticsearch will use MySQL table name as the Elasticserach's index and type name, use MySQL table field name as the Elasticserach's field name. Notice: go-mysql-elasticsearch will use the lower-case name for the ES index and type. E.g, if your table named BLOG, the ES index and type are both named blog. Rule can let you change this name mapping. Rule format in config file is below:
In the example above, we will use a new index and type both named "t" instead of default "t1", and use "my_title" instead of field name "title". Rule field typesIn order to map a mysql column on different elasticsearch types you can define the field type as follows:
Modifier "list" will translates a mysql string field like "a,b,c" on an elastic array type '{"a", "b", "c"}' this is specially useful if you need to use those fields on filtering on elasticsearch. Wildcard tablego-mysql-elasticsearch only allows you determind which table to be synced, but sometimes, if you split a big table into multi sub tables, like 1024, table_0000, table_0001, ... table_1023, it is very hard to write rules for every table. go-mysql-elasticserach supports using wildcard table, e.g:
"test_river_[0-9]{4}" is a wildcard table definition, which represents "test_river_0000" to "test_river_9999", at the same time, the table in the rule must be same as it. At the above example, if you have 1024 sub tables, all tables will be synced into Elasticsearch with index "river" and type "river". Parent-Child RelationshipOne-to-many join ( parent-child relationship in Elasticsearch ) is supported. Simply specify the field name for
Note: you should setup relationship with creating the mapping manually. Filter fieldsYou can use
In the above example, we will only sync MySQL table tfiler's columns Ignore table without a primary keyWhen you sync table without a primary key, you can see below error message.
You can ignore these tables in the configuration like:
Elasticsearch PipelineYou can use Ingest Node Pipeline to pre-process documents before indexing, like JSON string decode, merge fileds and more.
Node: you should create pipeline manually and Elasticsearch >= 5.0. Why not other rivers?Although there are some other MySQL rivers for Elasticsearch, like elasticsearch-river-jdbc, elasticsearch-river-mysql, I still want to build a new one with Go, why?
Todo
DonateIf you like the project and want to buy me a cola, you can through:
Feedbackgo-mysql-elasticsearch is still in development, and we will try to use it in production later. Any feedback is very welcome. Email: [email protected] |
2023-10-27
2022-08-15
2022-08-17
2022-09-23
2022-08-13
请发表评论