Jupyter notebooks and their derivatives, including JupyterLab, Google Colab, Hex and DeepNote are great for developing and training machine learning models. Their interactivity is ideal for the experimental nature of ML work. Their visual nature makes it easy to visualize model training results. And the fact that model building projects are self-contained means the code can stand alone in a notebook, independent of large codebases.
Notebooks And IDEs Are A False Dichotomy
Many computer science students get started with interactive programming environments like notebooks. They are good learning tools because they let students run and re-run individual lines of code, similar to a REPL. They improve on REPLs with their point-and-click interfaces and visual outputs that help students experiment and debug.
As students progress, they typically move on to directly editing entire source code files in text editors like vi and emacs, or IDEs like VSCode and IntelliJ. This is a necessary transition when working on larger, more complex projects. In these projects, many thousands of lines of code have to work in concert. In fact, understanding the interactions is the most important part, so testing individual lines is less useful. Moreover, the entire large project may have to run in perpetuity in a production environment, so developers need to replicate that environment when developing.
If students do not have experience with machine learning in an operational setting, they may misunderstand why machine learning engineers use notebooks. Because engineering students “graduated” from notebooks when they were no longer beginners, they may assume the machine learning engineers are beginners themselves. This is untrue: ML engineers are using the best tool for the job.
Modelbit’s Experience
Modelbit’s most active users range from data scientists who mostly train small and medium sized models; to ML engineers who train and deploy large, custom neural nets and vision models; to CTOs overseeing the architecture of large-scale machine learning systems. A great pleasure of building Modelbit has been working directly with CTOs as they architect their next-generation machine learning systems.
Every single one of these users uses notebooks to develop and train their ML models.
The reasons these very experienced ML architects use notebooks are simple: The interactivity makes it easier to experiment with models; the visual nature makes it easy to understand the model outputs; and they don’t need the training code itself to interact with large-scale systems.
Interactivity For Machine Learning
In a traditional software system, the entire code needs to run, top to bottom, to create the right state and generate the right outputs. If you need to get new outputs, you need to kill the entire program, make your modifications, and re-run it from the beginning.
When training a machine learning model, the first 80% of the code will be (1) initializing the necessary Python environment (e.g. the correct versions of ML frameworks, any specific model code, any checkpoint files, etc.) and (2) data engineering and prep for the training dataset. The first step can be quite time-consuming for large model frameworks like PyTorch. The second step can be even more time-consuming when training on very large datasets.
After this step is completed, you’ll want to train your model and evaluate it. If it’s not to your liking, you’ll want to change some things (e.g. hyperparameters, feature selection, model configuration, etc.) and try again. A professional ML developer will do this “try again” step over and over again.
In order to complete their work quickly, they’ll want to make this step as quick as possible so they can efficiently try many different combinations. Re-running all the code that sets up the environment and organizes the training data is not efficient! A notebook will retain the memory state from the first several cells while you edit and re-run the cell containing the training step as often as you like. This is ideal for rapid experimentation.
Visualizing Machine Learning Outputs
Machine learning models are very complex under the hood, which means that simple metrics are sometimes not enough to fully understand what the model is doing or how it’s succeeding. Graphs can be very helpful in understanding, for example, performance vs. a loss function. And for computer vision models, nothing beats an intuitive visual output of the model’s understanding of the data.
In a traditional codebase, additional code would have to be written to generate these visualizations. Perhaps outputting image files, or running an X-windowing server, or generating web pages. In a notebook, outputting a simple image representation will cause the notebook to render the image right inline! Ditto for a matplotlib graph. This is incredibly convenient for intuitively understanding your models and rapidly iterating on them.
Separability From Large Codebases
The third reason ML engineers use notebooks is because they can. Even if the model itself will be integrated into your product, the model training code will not. Model training code does not need to run alongside the rest of your application, and does not need to be tested as part of your application. For that reason, it’s okay for the training code to be in a standalone notebook, and not in a source code file alongside your application code. In fact, since it’s not part of your deployed application, it probably shouldn’t be there.
Don’t Deploy Your Notebook
While the training code isn’t going into your application, the model artifact (typically one or more very large Python objects) and inference code (typically a few hundred lines of Python code) absolutely is.
Since models are usually developed in notebooks, including and especially by the most experienced ML engineers, a question arises of what to do when it’s time to deploy the model for inference.
Too often, the model artifact and inference code are manually ported to the application code. This can mean manually pickling the model and hosting it somewhere like S3, while rewriting the model code in the application codebase (usually not Python). This is a brittle, manual process that bypasses traditional testing and CI/CD. It is decidedly not what the best ML engineers do.
Another too-common hack is to “deploy” the notebook itself. This usually means queueing up the notebook to run, top to bottom, on a daily schedule. The notebook will load up the latest data from some datastore, run a batch of inferences, and write the inferences back to the datastore. This also bypasses testing and CI/CD, and furthermore is incredibly brittle. The best ML engineers don’t do this either.
What is required is a system for deploying the finished model and inference code – including its specific Python environment, but not including ancillary training code – into a version-controlled, tested, continuously integrated production environment. Since the production environment will be isolated – remember, it needs its own specific Python environment – an API is put in front of it.
This is the problem that Modelbit solves. The best ML engineers in the world use the best tools – notebooks – to develop and train their models. Then, they use Modelbit to deploy them in a scalable, battle-tested, production-ready way.
