Kafka Msg VS REST Calls

Question

Nowadays in microservice world, i’m seeing alot of design in my workplace that uses kafka messaging when you can achieve similar results using rest api calls between microservices. Technically you can stop using rest api calls altogether and instead use kafka messaging. I really want to know the best practice, pros and cons, when to use api calls between microsevices, when to use kafka messaging.

Lets put a real life example:

I have an inventory service and a vendor service. Everyday vendor service calls the vendor API to get new items and these need to be moved into inventory service. The number of items can be up to 10,000 objects.

For this use case, is it better to :

1. After getting new data from vendor API, call REST API of inventory service to store the new items.

2. After getting new data from vendor API, send them as message to a kafka topic, to be consumed by inventory service

Which way would you choose and what is the consideration

question from:https://stackoverflow.com/questions/57852689/kafka-msg-vs-rest-calls

Answer 1

Gist (for those who want just the gist)

1. • Kafka - Publish & Subscribe (just process the pipeline, will notify once the job is done)

   • REST - Request & Await response (on-demand)

---

2. • Kafka - Publish once - Subscribe n times (by n components).

   • REST - Request once, get the response once. Deal over.

---

3. • Kafka - Data is stored in topic. Seek back & forth (offsets) whenever you want till the topic is retained.

   • REST - Once the response is over, it is over. Manually employ a database to store the processed data.

---

4. • Kafka - Split the processing, have intermediate data stored in intermediate topics (for speed and fault-tolerance)

   • REST - Take the data, process it all at once OR if you wish to break it down, don't forget to take care of your OWN intermediate data stores.

---

5. • Kafka - The one who makes the request typically is not interested in a response (except the response that if the message is sent)

   • REST - I am making the request means I typically expect a response (not just a response that you have received the request, but something that is meaningful to me, some computed result for example!)

Q&A style

Is your data streaming?
If the data keeps on coming and you have a pipeline to execute, Kafka is best.

Do you need a request-response model?
If the user requests for something and they wait for a response, then REST is best.

Kafka (or any other streaming platform) is typically used for pipelines i.e where we have forward flow of data.

Data comes to Kafka and from there it goes through component1, component2 and so on and finally (typically) lands in a database.

To get the information on-demand we need a data store (a database) where we can query and get it. In such a case we provide a REST interface which the user can invoke and get the data they want.

---

Regarding your example,

Everyday vendor service calls the vendor API to get new items and these need to be moved into inventory service

Questions & Answers

Is your vendor API using REST?

Then you need to pull the data and push to Kafka. From there your inventory service (or any other service thereafter) will subscribe to that topic and execute their processing logic.

The advantage here is that you can add any other service which requires vendor data as a consumer to the vendor topic.

Moreover, the vendor data is always there for you even after your inventory service processed it.

If you use REST for this, you need to call the Vendor API for every component that requires vendor data which becomes trivial when used with Kafka

Do you want the inventory to be queried?

Store it in a database after processing through Kafka and provide a REST on top of this. This is needed because Kafka is typically a log, to make the data query-able you would need some database.