The Hard Part of Machine Learning in Production

How to automate machine learning is something that drives me day in and day out.

What you do in development or education is, that you create a model and fit it to the data. Then that model is basically done forever.

Where I’m coming from, the IoT world, the problem is that machines are very different. They behave very different and experience wear.

Machines have certain processes that decrease the actual health of the machine. Machine wear is a huge issue. Models that that are built on top of a good machine don’t work forever.

When the Machine wears out, the models need to be adjusted. They need to be maintained, retrained.

That is where stream and batch processing are coming to play. In the batch processing layer you are creating the models, because you have all the data available for training.

In the stream in processing layer you are using the created models, you are applying them to new data.

The idea that you need to incorporate is this is a constant a constant cycle. Training, applying, re-training, pushing into production and applying.

What you don’t want to do is, you don’t want to do this manually. You need to figure out a process of automatic retraining and automatic pushing to into production of models.

In the retraining phase the system automatically evaluates the training. If the model no longer fits it works as long as it needs to create a good model.

After the evaluation of the model is complete and it’s good, the model gets pushed into production. Into the stream processing.

Automatic re-training and re-deploying is a very big issue, a very big problem for a lot of companies. Because most existing platforms don’t have this capability (I actually haven't seen one until now).

You can create models and then use them in production. But this loop is almost nowhere to be seen.

It is a very big issue that needs to be solved. If you want to do machine learning in production you can start with manual interaction of the training, but at some point you need to automate everything.

To train a model you are manipulating input parameters of the models.

Take deep learning for instance. To train you are manipulating for instance:

How many layers do you use. The depth of the layers, which means how many neurons you have in a layer. What activation function you use, how long are you training and so on.

You also need to keep track of what data you used to train which model.

All those parameters need to be manipulated automatically, models trained and tested.

To do all that, you basically need a database that keeps track of those variables.

How to automate this, for me, is like the big secret. I am still working on figureing it out.

Did you already have the problem of automatic re-training and deploying of models as well?

Were you able to use a cloud platform like Google, AWS or Azure?

It would be really awesome if you share your experience :)

Related posts:

Confused? Intrigued? Read on for a guide on the Roaming Designer with examples from the Chingu Developer Network (CDN) project. A Roaming Designer is someone that will be matched with one or more…

Queria pode dizer claramente que perdi minha forças, perdi minha paciência. Queria que você soubesse com toda certeza que eu desisti de nós dois, desisti de você. Queria gritar tudo que me fez não…

La climatisation est nécessaire à la vie, surtout en ce moment, où des vagues de chaleur s’abattent sur le pays. Rien n’est plus agréable que de s’échapper de la canicule de l’après-midi en entrant…