I have surfed the internet for some time before writing the question, and it seems that this problem is not so much popular (not a mention of it anywhere). I use Kafka Streams API in my project for a calculation of something. I won't go into details of the algorithm, because it is not important here at all, what I have a problem with is that old values that are to be pushed to the topic are not pushed there until new values arrive. I guess it is because I use windowing and suspend the window, but I would like to somehow solve the problem (not with hacky solutions like pushing empty events). In this particular case, I need to wait till the end of the window before publishing the value to the output topic. The topology looks as follows:
firstStream.leftJoin(secondStream, ...)
.leftJoin(thirdStream, ...)
.filter(...)
.groupByKey(...)
.windowedBy(TimeWindows.of(Duration.ofSeconds(windowSeconds)).grace(Duration.ofSeconds(graceSeconds)))
.aggregate(...)
.suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded()))
.toStream()
.to(outputTopic)
So, whenever I push an event to the firstStream
, I should receive an output of the transformations in the outputTopic
after the window closes, but actually, I receive the output only when I push a new event after the window closes.
So, my question is whether there is a way to push output after the window closes without sending a new event?
question from:
https://stackoverflow.com/questions/65917910/kafka-streams-api-pushes-old-values-to-a-topic-only-after-a-new-event-arrives 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…