Real-time threat detection (Capital One)

Capital One is currently the eighth largest bank in the U.S. One of its core areas of business was facing vast and increasing costs for an existing solution to guard against digital threats. The bank set out to find a new solution that would deliver better performance while also being more cost effective.

At the time, Capital One was processing several thousand transactions every second. The bank's innovation team established that the solution must be able to process data within low double-digit milliseconds latency, scale easily, ensure that it runs internal algorithms with zero data loss, and also be highly available. Additionally, the team realized that tackling this challenge would require dynamic and flexible machine learning algorithms in a real-time distributed environment.

The team launched a rigorous process of evaluating numerous streaming technologies including Apache Apex, Apache Flink, Apache Storm, Apache Spark Streaming, IBM Infosphere Streams, Apache Samza, Apache Ignite, and others. The evaluation process involved developing parallel solutions using each of the technologies, and comparing the quantitative results generated by each technology as well as its qualitative characteristics.

At the conclusion of the evaluation, only one technology emerged as being able to meet all of Capital One's requirements. In the team's own words:

"Of all evaluated technologies, Apache Apex is the only technology that is ready to bring the decision making solution to production based on: Maturity, Fault Tolerance, Enterprise-Readiness, and Performance."

With Apache Apex, Capital One was able to:

• Achieve latency in single-digit milliseconds, which is significantly lower than the double digit millisecond latency that the bank set out to achieve and which is a hard requirement for use cases such as online transactions
• Meet the SLA requirements of continuously running the data pipeline applications with
  99.999% uptime on 24x7 basis, with automatic failover
• Reduce the total cost of ownership, based on Apex's ability to run on Hadoop and scale out with commodity grade hardware
• Easily add newer applications and features to accurately detect suspicious events without being tied to the vendor roadmap and timeline
• Focus on core business algorithms and innovation, while the platform took care of fault tolerance, operability, scalability, and performance

Furthermore, Capital One's implementation of Apex enabled the following:

• Parallel Model Scoring
• Dynamic Scalability based on Throughput or Latency
• Live Model Refresh, parallelized model scoring

A complete set of Capital One's goals, and the results it achieved with Apex

Additional Resources

• Video: Capital One presents Toppling the Mainframe with Apache Apex  - https://www.youtube.com/watch?v=98EW5NGM3u0
• Slides: Capital One presents Next Gen Decisions in Less Than 2ms with Apache Apex - http://www.slideshare.net/ApacheApex/capital-ones-next-generation-decision-in-less-than-2-ms