Latest articlesIf you’re not a subscriber, here’s what you missed this month.
To receive all the full articles and support ByteByteGo, consider subscribing: Every system that processes data eventually has to answer one question. When is the data complete enough to be moved to the compute stage? A program adding up a day’s sales needs to know whether all of today’s sales have actually arrived. For data stored in a file, the answer is trivial because the file has an end. However, for data that arrives continuously and never stops, there is no clean answer, and how a system resolves that gap is the difference between batch processing and streaming. Batch processing waits for completeness. It collects data up to a natural boundary, a closing time, or a finished file, and then computes over the whole set at once. Streaming prioritizes completeness for speed. It produces answers continuously from data that is still arriving, which means it has to estimate when enough data has come in and handle the cases where that estimate is wrong. This trade-off between completeness and latency is the key consideration when dealing with streaming and batch.
In this article, we will cover the strategies on each side and what each one costs.
Boundaries...Continue reading this post for free in the Substack app
|
Comments
Post a Comment