High-level data pipeline of the prototype

The goal of this chapter is to develop a prototype of a system that will predict the short-term change of Bitcoin price, using historical data to train the algorithm, and real-time data to predict and select algorithms that perform better. In the scope of this project, there is no attempt to predict the actual price in dollars, but only whether it would increase or not. This is because Bitcoin price, to some extent, is not actually about price but about market expectations. This can be seen as patterns in a trader's behavior, which, on a higher level, is represented by previous price itself.

Of course, there is an objective price associated with Bitcoin; miners are willing to sell Bitcoins for profit. So the base price can be estimated by knowing all bills that all miners have to pay for the Bitcoins they mine, but that is outside the scope of this project.

From this perspective, rather than trying to predict the price in dollars, it might make sense to look for trends of the price rising, dropping, or staying the same, and act accordingly. The second goal is to build a tool for experiments that allows us to try different approaches to predicting prices and evaluate it on real-life data easily. The code has to be flexible, robust, and easily extensible.

Therefore, in summary, there are three main components of the system:

• Scala script for preprocessing of historical data into the required format
• Scala app to train the ML model
• Scala web service to predict future prices