java - spring batch multithread same sort in reader as writer

Question

We are experimenting with a spring batch single thread to spring batch multithread.

The setup is rather easy :

• Reader : Read some data with a specific sorting.
• Processor : fetch some additional values.
• Writer : Write it all to an csv file in order of read.

So we have changed the reader to a JdbcPagingItemReader and converted the sort from

order by firstname, lastname, id;

to

Map<String, Order> sortConfiguration = new HashMap<>();
sortConfiguration.put("firstname", Order.ASCENDING);
sortConfiguration.put("lastname", Order.ASCENDING);
sortConfiguration.put("id", Order.ASCENDING);

the commit-interval is set to 200.
The batch runs fine, but our csv is completly out of order.
I assumed that spring would write in the file after each commit (and hoped that he write page per page in order), but the disorder is greater then chunks 200 lines.
I got for example line 1, 3 and 5 should be together in a thread and line 2 and 4 in another thread.
Is there any option to preserve the order or is the only way to abandon multithread?

question from:https://stackoverflow.com/questions/65916086/spring-batch-multithread-same-sort-in-reader-as-writer

Answer 1

Multi-threading is incompatible with ordering. If you use a multi-threaded step, items will be read, processed and written in an undefined order. This is mentioned in the Multi-threaded Step section of the reference documentation:

The result of the above configuration is that the Step executes by reading, processing,
and writing each chunk of items (each commit interval) in a separate thread of execution.
Note that this means there is no fixed order for the items to be processed, and a chunk
might contain items that are non-consecutive compared to the single-threaded case.