I am considering using Apache Kafka as a distributed message publisher to many subscribers. It is the perfect fit for me, since the solution has to scale easily.
The Kafka's documentation states that the message may be acknowledged thus ensuring the message delivery. However, today I came across this article which states that there are scenarios in which the messages may be lost. Then again, the article is only available in Google cache, so I do not know whether it is trustworthy...
So I have one doubt - is there any moment, any scenario, in which the message will be lost? In another words - my main requirement is that each message must reach its destination. Can it be met by using the Apache Kafka? Is it the right tool for this job?
Best Answer
The original of the article you are looking for is here: http://engineering.onlive.com/2013/12/12/didnt-use-kafka/
If you read the full article and the comments you'll see much of the concern is not about the guarantee of at least once delivery, but that it was delivered AND successfully processed by the client. The last couple of comments on the article, including by the original author, seem to indicate he's satisfied with the approach.
You might also find this article of interest - similar concerns:
https://www.mail-archive.com/users%40kafka.apache.org/msg04492.html
And from some of the documentation:
So effectively Kafka guarantees at-least-once delivery by default andallows the user to implement at most once delivery by disablingretries on the producer and committing its offset prior to processinga batch of messages. Exactly-once delivery requires co-operation withthe destination storage system but Kafka provides the offset whichmakes implementing this straight-forward.
Most of the conversations I've seen are not about the guarantee of at least once, but how to go from there to at most once or to exactly once.
Kafka does claim that
For a topic with replication factor N, we will tolerate up to N-1 server failures without losing any messages committed to the log.
It might worth reading the Message Delivery Semantics written in their doc page for a better understanding