官术网_书友最值得收藏!

Low latency and stateful processing

Apex is a native streaming architecture. As previously discussed, this allows processing of events as soon as they arrive without artificial delay, which enables real-time use cases with very low latency. Another important capability is stateful processing. Windowing may require a potentially very large amount of computational state. However, state also needs to be tracked in connectors for correct interaction with external systems. For example, the Apex Kafka connector will keep track of partition offsets as part of its checkpointed state so that it can correctly resume consumption after recovery from failure. Similarly, state is required for reading from files and other sources. For sources that don't allow for replay, it is even necessary to retain all consumed data in the connector until it has been fully processed in the DAG.

Stateful stream processors have what is also referred to as continuous operator model. Operators are initialized once, at launch time. Subsequently, as events are processed one by one, state can be accumulated and held in-memory as long as it is needed for the computation. Access to the memory is fast, which allows for very low latency.

So, what about fault tolerance? The platform is responsible for checkpointing the state. It can do so efficiently and provides everything needed to guarantee that state can be restored and is consistent in the event of failure. Unlike the early days of Apache Storm with per tuple acknowledgement overhead and user responsibility for state handling, the next generation streaming architectures provide fault tolerance mechanisms that do not compromise performance and latency. How Apex solves this, will be covered in detail in Chapter 5Fault Tolerance and Reliability.

主站蜘蛛池模板: 格尔木市| 临沧市| 竹北市| 宁津县| 岱山县| 江源县| 越西县| 康定县| 沈丘县| 德安县| 根河市| 陕西省| 清流县| 阳泉市| 靖边县| 富锦市| 大关县| 界首市| 青川县| 新乐市| 崇仁县| 古浪县| 仪征市| 随州市| 江达县| 马公市| 平阳县| 翼城县| 平和县| 日照市| 祁连县| 鹿泉市| 黄大仙区| 花莲县| 曲周县| 门源| 花莲市| 泾川县| 丹凤县| 大埔县| 广昌县|