Recurring index not working for ElasticSearch 5.3.0
Problem Identification and Cause
All the records for recurring index are stored in Y_RECORD_CHANGE table. Elastic search takes the newer objects from this table and indexes. If all the newer records are indexed this table should be empty.
Check the count of this table in database using the query "select count(*) from y_record_change;". If this count is not zero and is keep on increasing with every minute. You can say that it is due to a corrupt record stuck in this table which is not allowing this to index other records too.
Resolution
You can resolve this in two ways
1.Truncate the y_record_change table which will remove all the records from that table and perform a restart and then re-index all objects.
2. Use the class file attached here - removeFromYrecordChangeTable.class and run it as a scheduled action. This will remove 10k records at a time from y_record_change table. If records are more than 10k , run the scheduled action twice. At last re-index all the objects. (This does not require a restart)
Verification
Check the y_record_change count,it should be zero before re-indexing.