Data management

Datasets

A dataset is all of the data used to create a machine learning model. This includes both data used for training and data used to verify the model’s proper functioning.

Microsoft and Google offer an abstraction layer that allows you to create datasets from your data. The main advantage over directly using data stored in a csv file or a database is that you can easily version the training data.

Data analysis & cleaning

Data analysis and cleaning are long and tedious steps that data scientists have to undertake. This task is usually performed with code on a notebook.

Amazon offers a feature called “SageMaker Data Wrangler” that handles this task without code. Microsoft also offers it via the “Azure Machine Learning Designer”. The advantages are:

• Empower the business line to work with the data itself, since it has the most knowledge about the data
• Become more efficient

Below are captures from the Amazon and Microsoft studios allowing data cleansing without code:

For the analysis part, Microsoft can display statistics on the data (missing values, minimum, maximum, etc.), while Amazon also lets you create visualisations (histograms, scatter plots, etc.).

Summary of the three data management solutions

Model training

Auto-ML 

All three cloud solutions have an auto-ML feature, accessible from a graphical interface or via code.

AutoML involves automatically generating a machine learning algorithm. It is responsible for:

• Testing different types of feature engineering
• Testing different algorithms
• Selecting the best algorithm

This means that someone who doesn’t know anything about machine learning can create their own models independently: they just need to indicate the data to use, the column to use as a label and launch the training.

Microsoft Azure allows you to view the models tested and the conversions performed on the data automatically in the form of graphs. Note that for the uninitiated these graphs aren’t very meaningful.

Meanwhile, Amazon Vertex doesn’t offer an explanation if the purely graphical solution is used (called SageMaker Canvas). However, if you use AutoML via code (a solution called SageMaker Autopilot), a notebook is generated describing in text form the tests performed:

For Google, the information on the tested models is available in logs in JSON format, which isn’t necessarily easy to read:

The last point to note is that Google’s solution seems to be a little light on feature engineering automation compared to its competitors.

Below is a comparative table of our opinion on these three AutoML solutions:

In conclusion, although the AutoML feature may seem magical, it isn’t without risk. It isn’t advisable to put models into production if you aren’t proficient in them. How can you debug if you don’t know how it works? How can you analyse it? On the other hand, this feature is very interesting as a first step for data scientists. With a few clicks or lines of code, they can get an initial overview of which models work best.

Training “manual” models

“Manual” means that you can train your own models without using AutoML.

The three solutions offer more or less the same thing, namely:

• A notebook environment for testing, data mining, small training sessions, etc.
• Libraries (Python, Node.js, etc.) to interact with the cloud environment (creation of datasets, models, endpoints, etc.)
• The ability to choose the type of machine instance on which to run the training. This last point this is where cloud solutions’ real power lies: in a few clicks or a few lines of code, you can run training on any type of machine (choose the number of CPUs, memory, number of GPUs)
• The ability to test your code locally rather than directly in the cloud, so you don’t have to start a cloud instance whenever you do any testing
• A registry model to store and version your models

In this “manual training” part, Microsoft provides the ability to generate models through the designer’s graphical interface, therefore without doing any code. Machine learning skills are required to use this feature. In addition, the designer isn’t very intuitive, so there is limited benefit.

However, Microsoft’s strength is the model comparison part, as the graphical interface is very clear and easy to use.

With Amazon’s solution you can also compare models, although less intuitively.

However, Amazon offers a whole list of indicators and rules to make it easier to debug a model.

With Google you can use TensorBoard. However, there’s no particular benefit in using the cloud version compared to the version that can be installed locally.

Below is a summary of the model training features.

Ethical Artificial Intelligence

Ethical AI is about ensuring that the models deployed are respectful, transparent and fair. Going into more detail, this includes:

• Not reproducing societal biases (discrimination regarding a given criterion)
• Being able to understand why the algorithm made a particular decision
• Respecting individuals’ privacy
• Respecting individuals’ freedom
• Respecting the environment

Regarding the explainability aspect of the models, the three vendors propose the use of “SHAP values”. Microsoft offers LIME and PFI, while Google adds XRAI and Integrated Gradient.

Microsoft provides a very good graphical interface to compare the model’s behaviour for different types of population.

Amazon offers a basic interface, while Google doesn’t offer one.

Regarding “model impartiality”, Azure provides a few indicators (it uses the Fairlearn package), Amazon offers around 20, with a detailed explanation of each, and Google none.

Below is a summary of the ethical AI features.

Deployment of models

Once the model has been trained, the solutions offer to deploy it in a few clicks or via a few lines of code. Deploying a model means providing an endpoint (a REST API) that takes as input the data on which to make the prediction, and returns the model prediction.

In deployment you can add processing upstream or downstream of the prediction via specific code.

All three cloud providers offer predictions called:

• “Online”: predictions are made in real time (synchronous queries); predictions can be run on a single data item or several
• “Batch”:  predictions are made asynchronously, then the results are saved but not directly returned to the user; this is generally used when predictions are being made on large volumes of data

Amazon SageMaker offers two other types of deployment:

• “Asynchronous predictions”:  This type of prediction can be used when you want to make predictions in near real time but don’t want to block the user during the prediction. Compared to batch prediction, the user (or client) can be automatically notified and the result automatically retrieved.
• “Serverless predictions”: this type of deployment makes it possible to completely offload the infrastructure sizing and scaling policy. The infrastructure is automatically sized according to the load at a given moment. The important point when using this type of deployment is that the service must tolerate cold starts, as the service isn’t available until the first instance starts up.

Monitoring of models in production

All three solutions store the predictions made by the models and the data that led to those predictions.

Microsoft uses the Azure Application Insights solution to store this information in logs. Queries can be run in these logs using the Kusto language (KQL). However, these logs aren’t easy to utilise; below is an example of what can be obtained:

With Google the predictions are stored in its BigQuery data warehouse. If you aren’t an expert in BigQuery queries, the results aren’t easy to utilise either.

Lastly, Amazon saves the predictions in files on S3.

All three solutions also offer a drift analysis feature, “drift” being the change in the distribution of data over time. It’s important to keep an eye on this because if the data changes, the model will no longer perform properly, meaning it will need to be retrained.

You can also send automatic alerts if there are significant changes in distributions, i.e. drift.

One point for consideration is that in some cases the calculation of the drift indicators isn’t explained, so it’s difficult to define a threshold to automate the alerts.

Specific AI services

The three cloud providers supply Artificial Intelligence solutions for specific use cases. These are use cases that are often complex to develop yourself (image classification, chatbot, etc.).

You simply give the service your own labelled data, then an algorithm is automatically trained on it and, if the results are convincing, it can be deployed.

The table below lists the different use cases covered by each vendor, including the name of the solution.

Pricing information

Prices are broken down into several items: the instances on which studios/notebooks run, the instances on which models are trained, the instances on which models are deployed, the storage, the network, the load balancers, etc.

This means it’s very complicated to estimate the cost of using these solutions for your own needs. In the table below we compare the price of the calculation instances, which are the main cost items; the prices are for the Eastern region of the US: