Deployment in Machine Learning Workflow

As a machine learning/deep learning researcher, we usually focus on improving the performance of the models on specific datasets. However, the story of bringing the model into real applications is far beyond that. This post provides a short description of Machine learning workflow and a part of it, Deployment.

Photo by Daan Stevens on Unsplash

1. Machine learning workflow.

Let’s have a look at machine learning workflow defined by Amazon web service (AWS) and Google cloud platform (GCP). Those are 2 examples I can find online.

Figure 1. AWS Machine learning workflow [1]

Figure 2. GCP Machine learning workflow [2]

According to the figures, we can divide the whole process into 3 main steps (corresponding to 3 colors in Figure 1).

• Step 1: Explore and Process data.
• Step 2: Modeling.
• Step 3: Deployment.

As the name suggests, we first need to collect data and make it ready for building models. Then we can try different models, tune hyperparameters, etc until we are satisfied with the results. After that, we are able to deploy the models somewhere so that they can make inference for new samples. However, the process does not end here, it is an iterative process. We have to keep track of how the models is performing, collect more data, improve models and redeploy newer versions of them.

From the perspective of DevOps, step 1 and 2 can be regarded as development, and step 3 is operation.

When doing research in machine learning, we would like to process data and develop models that outperform other state-of-the-art models on some specific datasets. In other words, we stop after step 2. However, we pay less attention to where the data comes from or how we can deliver our models to a broader number of users. This post is only about deployment, and the step of fetching data will be written some days when I have more experience.

2. Deployment.

Deployment is the method by which you integrate a machine learning model into an existing production environment to make practical business decisions based on data [3]. Here is an example of machine learning deployment.

Figure 3. An example of deployment.

In Figure 3, users can initiate requests and send data to the application (web or mobile application). It then communicates with a machine learning model via an endpoint to send data to the model and receive the predictions. Now, the application can forward the predictions to the users.

The main purpose of deployment is to make the model available for use by web or software application. However, besides, there are some sort of things we need to take care of.

• Predictions: This is the main purpose of deploying our models. There are 2 type of predictions we can make: on-demand and batch prediction. On-demand prediction requires our model to return results as soon as possible. Meanwhile, batch prediction can be executed at any time, usually runs periodically for large amount of data.
• Versioning: We should have a way to control the versions of our models.
• Monitoring: We should keep track of how our models are performing, when the models start performing worse, or is there any bugs, etc.
• Updating: We continuously improve our models, so we want to update new version of our models when they outperform older ones.
• Routing: Sometimes, we can have multiple types or replications of models handling same requests. Thus, we need to route the requests for purposes like A/B testing or load-balancing.

3. Cloud services for deployment.

At the moment, there are a number of machine learning cloud platforms which remedy us from deployment workload. We can focus on data science part, while infrastructure and orchestration will be taken care by the platform. As shown in Figure 2, every parts of deployment can be conducted on GCP platform. That is an important benefit that we can consider when developing our models.

I can list here some cloud services which I want to try in the future.

• Amazon SageMaker.
• GCP AI platform.
• Microsoft Azure AI.
• IBM data platform.

References

1. Machine Learning with Amazon SageMaker: https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-mlconcepts.html
2. GCP Machine Learning workflow: https://cloud.google.com/ai-platform/docs/ml-solutions-overview
3. Machine Learning Model Deployment: https://www.datarobot.com/wiki/machine-learning-model-deployment/