ruby on rails - Querying before Elasticsearch index update is finished returns old data

Question

I'm using searchkick gem with option:

 searchkick callbacks: :async

There is a case where I delete a record and the client immediately calls the index endpoint to list all records. But since the ES index was not updated yet by the background job, it gets old data.

I'm looking for ideas on how to solve this. Is there a cheap way to check if an index is being updated?

question from:https://stackoverflow.com/questions/65925676/querying-before-elasticsearch-index-update-is-finished-returns-old-data

Answer 1

This happens cause of refresh_interval property of your index. If not set explicitly, it's default values is 1 (which I am assumging is your case).

From documentation:

By default, Elasticsearch periodically refreshes indices every second, but only on indices that have received one search request or more in the last 30 seconds.

Solution 1

You can use the refresh parameter of Elasticsearch to overcome your problem but at your own risk. I addressed a similar problem here.

curl -X PUT "localhost:9200/test/_doc/1?refresh&pretty" -H 'Content-Type: application/json' -d'
{"test": "test"}
'
curl -X PUT "localhost:9200/test/_doc/2?refresh=true&pretty" -H 'Content-Type: application/json' -d'
{"test": "test"}
'

Solution 2

There is a support to explicitly refresh your index.

curl -X POST "localhost:9200/my-index-000001/_refresh?pretty"

Note: Above mentioned solutions are not cheap, in-fact are expensive ones and can cause performance issues if you have large number of documents inside the index.

Solution 3

(If you have a backend and RDS layer in between of your client and ES) Once you have the response from ES, delete the entries from the final response which are not present in RDS.