Lottery Prediction - Part 1/3 (System)

Implementing MLOps Level 0

Here's how I built MLOps cycle for predicting lottery winning numbers.
The winning numbers are predicted by the best model that is deployed after each lottery draw. The model of the highest winning probability, not the prize money expectation, was selected to be deployed.


title


Since the lottery is drawn weekly (or every several days), the data inflow is sparse. This gave me some reasons to build the system in MLOps level 0.


  - No need to implement CI
   * Data format is fixed.
   * The rule of the entire process can be fixed.
  - No need to implement CD
   * Replacement of the best model file is just simply deployment.
  - No risk about concept drift
   * Each lottery draw is independent.
   * No monitoring process is needed.


Therefore, the building the architecture could be simply separated into the steps below


  - Crawling the winning numbers after every draw
  - Upload the last draw winning numbers to draw history DB
  - Split the data into trainset and testset
  - Hyperparameter tuning
  - Select and save the best model, which provides the highest winning probability on the testset
  - Predict future winning numbers under the best model and the recent data
  - Update and display the winning probability of the best model


These steps were implemented using multiple python scripts and crontab job scheduling in Ubuntu.
I first implemented the web application in AWS EC2. AWS was a good option for the availability, I could access and work from anywhere with powerful security. However, the cloud service costed a lot to just use Ubuntu (more than $70) and it was over my budget. So I bought an Intel NUC server PC that is light, tiny, quiet, and low power consumption! and set up the same environment in ubuntu 20.04.


Once I confirmed the 'MLOps level 0' works, I've been making some automatic alert functions to get noticed on abnormal event.
  - Crawling malfunction (in the case the site for crawling can be down)
  - Absence of DB update (possibly due to connection problem to DB or to the site for crawling)
  - (To do) Alarm on bnormal prediction (e.g. low variance of the predicted numbers)