官术网_书友最值得收藏!

Low latency and stateful processing

Apex is a native streaming architecture. As previously discussed, this allows processing of events as soon as they arrive without artificial delay, which enables real-time use cases with very low latency. Another important capability is stateful processing. Windowing may require a potentially very large amount of computational state. However, state also needs to be tracked in connectors for correct interaction with external systems. For example, the Apex Kafka connector will keep track of partition offsets as part of its checkpointed state so that it can correctly resume consumption after recovery from failure. Similarly, state is required for reading from files and other sources. For sources that don't allow for replay, it is even necessary to retain all consumed data in the connector until it has been fully processed in the DAG.

Stateful stream processors have what is also referred to as continuous operator model. Operators are initialized once, at launch time. Subsequently, as events are processed one by one, state can be accumulated and held in-memory as long as it is needed for the computation. Access to the memory is fast, which allows for very low latency.

So, what about fault tolerance? The platform is responsible for checkpointing the state. It can do so efficiently and provides everything needed to guarantee that state can be restored and is consistent in the event of failure. Unlike the early days of Apache Storm with per tuple acknowledgement overhead and user responsibility for state handling, the next generation streaming architectures provide fault tolerance mechanisms that do not compromise performance and latency. How Apex solves this, will be covered in detail in Chapter 5Fault Tolerance and Reliability.

主站蜘蛛池模板: 年辖:市辖区| 南召县| 武汉市| 谷城县| 宜黄县| 临清市| 绩溪县| 安丘市| 伊春市| 和政县| 炉霍县| 荥阳市| 哈尔滨市| 河曲县| 潞西市| 育儿| 商南县| 霍山县| 溧水县| 仁怀市| 绥芬河市| 开平市| 新平| 德惠市| 兴安县| 辛集市| 浦县| 阳江市| 尼玛县| 晋宁县| 沁水县| 新宁县| 高邑县| 前郭尔| 安新县| 无为县| 廊坊市| 阳泉市| 黔西| 金川县| 中方县|