# Deploy to a Docker container In many machine learning applications, it can be useful to deploy a model as a standalone Docker container that can return predictions. This tutorial shows how to build a Docker container with an mlpack model serving predictions. Here the model returns predictions in a very primitive way: directly as input from a terminal, but it would be straightforward to adapt the container to provide a full REST API or similar. mlpack applications inside of Docker containers can be built in a way that the resulting container is extremely small---sometimes even less than 1 MB! *See also*: - [Installing mlpack](install.md) - [Compile an mlpack program](compile.md) - [Deploying mlpack on Windows](deploy_windows.md) - [Setting up an mlpack cross-compilation environment](../embedded/supported_boards.md) ## General workflow To make a Docker container that serves predictions, we must first train a model. Therefore, our workflow to build this container will be: * [Decide on the problem to solve](#problem-statement) * [Write a program to train the model](#model-training-program) * [Write a program to make predictions with the trained model](#prediction-program) * [Build the Docker container with the prediction program](#building-the-container) * [Run the Docker container](#run-the-container) ## Problem statement For this simple example, we will solve a problem from the cybersecurity world: [DGA detection](https://en.wikipedia.org/wiki/Domain_generation_algorithm). The example here is based on (and heavily uses the code from) mlpack's [DGA detection LSTM example](https://github.com/mlpack/examples/tree/master/cpp/lstm/dga_detection). Malware authors often write malware that communicates with a centralized command-and-control server. This can allow the malware to update itself, or to receive commands from an operator (e.g., 'start Bitcoin mining', or 'lock system and display ransomware message'). The malware author cannot hardcode a domain name into their malware, because this would be easily blocked by any antivirus software. So, instead, a *domain generation algorithm* is used to generate a series of domain names. The malware will try to contact a server at each of these domain names. The individual DGA domain names can look very random; for instance, some DGA domains generated by the `matsnu` malware family are: * `brothernerveplacebringconsult.com` * `screencatchdishtellproposed.com` * `balladoptwelladdinfluence.com` * `capitalhuntdealsmokeboxclue.com` Other malware families may generate very random looking names, like `i828ywu0ywqs.net` or similar. Since the set of domain names that can be generated by a DGA is huge, blocking individual domain names is not a realistic mitigation strategy. However, we can use machine learning techniques to detect DGA-generated domain names with a high level of accuracy! This strategy has been shown to be effective in [some previous work](https://www.arxiv.org/pdf/1611.00791). Adapting that approach for simplicity, we will train simple recurrent neural networks with LSTMs to detect benign domains and DGA domains, and then the Docker container application will read domain names as input and output a score indicating the likelihood that the domain name was generated by a DGA. ## Model training program Training a model can be done as a separate standalone program; since our goal is just to provide a container that produces predictions, the program in the container does not need to support training. The C++ code for training a DGA detector is available as the standalone program [`lstm_dga_detection_train.cpp`](https://github.com/mlpack/examples/blob/master/cpp/lstm/dga_detection/lstm_dga_detection_train.cpp) We can compile it with a call to `g++`, following the instructions from [the compilation guide](compile.md): ```sh g++ -O3 -o lstm_dga_detection_train lstm_dga_detection_train.cpp -fopenmp -larmadillo ``` Some modification of the command above may be necessary if mlpack is installed on your system in a nonstandard location, or if you are not using the Armadillo wrapper. See [Configuring mlpack with compile-time definitions](compile.md#configuring-mlpack-with-compile-time-definitions) and [Linking without the Armadillo wrapper](compile.md#linking-without-the-armadillo-wrapper) for more details. Once the program is compiled, we can train on a [dataset of domain names](https://datasets.mlpack.org/dga_domains.csv.gz). The commands below will download the prepared dataset from the mlpack website, and then run the training process. ```sh wget https://datasets.mlpack.org/dga_domains.csv.gz gunzip dga_domains.csv.gz ./lstm_dga_detection_train dga_domains.csv ``` The training process may take a while, but when it is finished, performance statistics about the models will be printed, and the model will be saved to `lstm_dga_detector.bin`. The models should achieve 99%+ accuracy on the held-out test data. ## Prediction program The examples repository also provides the standalone prediction program [`lstm_dga_detection_predict.cpp`](https://github.com/mlpack/examples/blob/master/cpp/lstm/dga_detection/lstm_dga_detection_predict.cpp). We can also compile this with a call to `g++`, following the instructions from [the compilation guide](compile.md): ```sh g++ -O3 -o lstm_dga_detection_predict lstm_dga_detection_predict.cpp -fopenmp -larmadillo -static ``` As with the training program [above](#model-training-program), some modification of the compilation command may be necessary depending on your configuration. We also specified the `-static` option here, so that the produced program is statically linked. This will help us deploy into a Docker container, since we can just run the program directly and do not need to ensure that supporting libraries are available. Note that you will need a statically-compiled version of Armadillo available to link against, or instead define the compiler option `-DARMA_DONT_USE_WRAPPER` and link with static OpenBLAS using `-lopenblas`. The prediction program reads from stdin, and once a domain is entered, a prediction is computed and the word `malicious` or `benign` is emitted, along with a score indicating the model's "confidence". A sample transcript of the program is below: ```text $ ./lstm_dga_detection_predict lstm_dga_detector_benign.bin lstm_dga_detector_malicious.bin www.mlpack.org benign (score 44.5417) mdfvkejbqoxg.ru malicious (score 22.2786) arma.sourceforge.net benign (score 19.4883) 11b5n854ublnv152.net malicious (score 7.89951) ``` ## Building the container Building a Docker container that runs `lstm_dga_detection_predict` is very simple; we only need to put the prediction program and model in the container. In fact, to save additional time, we can use a [distroless](https://github.com/GoogleContainerTools/distroless) container. This code can be used as the `Dockerfile`: ```dockerfile FROM gcr.io/distroless/static-debian12 ADD lstm_dga_detection_predict . ADD lstm_dga_detector_benign.bin . ADD lstm_dga_detector_malicious.bin . ENTRYPOINT ["./lstm_dga_detection_predict", \ "lstm_dga_detector_benign.bin", \ "lstm_dga_detector_malicious.bin"] ``` Building the container is simple (and nearly instantaneous): ```sh docker build -t lstm_dga_detector . ``` And once it is built, it is easy to see that the container is relatively small: ```text $ docker images | grep -B 1 lstm_dga_detector REPOSITORY TAG IMAGE ID CREATED SIZE lstm_dga_detector latest 9fc931a2cdae 35 seconds ago 33.7MB ``` However, we haven't even tried to optimize for size---see the [Reducing the size of the container further](#reducing-the-size-of-the-container-further) section ## Run the container The container can now be deployed or run in any standard Docker environment (including on Kubernetes, although that seems like overkill for such a simple example). Running the container locally (and interacting with the prediction service) is a simple command: ```sh docker run --rm -it lstm_dga_detector ``` Once the container has started, simply type a domain name, hit enter, and a prediction (plus score) will be printed. To build the example into a more complex application, you can use `lstm_dga_detection_predict.cpp` as a starting point. ## Reducing the size of the container further Although 33.7 MB for a container is already orders of magnitude smaller than an equivalent unoptimized Python container, it is readily possible via compilation options and a few other tricks to get the size to be significantly smaller. The vast majority of the size of the container is just the size of the compiled `lstm_dga_detection_predict`: ```text $ ls -lh -rwxrwxr-x 1 ryan ryan 31M Mar 6 20:46 lstm_dga_detection_predict -rw-rw-r-- 1 ryan ryan 80K Mar 5 19:10 lstm_dga_detector_benign.bin -rw-rw-r-- 1 ryan ryan 80K Mar 5 19:10 lstm_dga_detector_malicious.bin ``` 31 MB for a compiled program is quite large! But, as it turns out, most of this is due to dependencies. We can see this by running a command that does not perform linking, like this: ```sh g++ -O3 -c -o lstm_dga_detection_predict.o lstm_dga_detection_predict.cpp ``` This compiled (but not linked) object file is much smaller: ```text $ ls -lh lstm_dga_detection_predict.o -rw-rw-r-- 1 ryan ryan 1.6M Mar 6 20:48 lstm_dga_detection_predict.o ``` That implies that our primary size issue is not with our own code, but instead with our dependencies. Specifically, on most systems, OpenBLAS is compiled to support any architecture and this results in the statically-linked OpenBLAS library being *quite* large. On a Debian system: ```text $ cd /usr/lib/x86_64-linux-gnu/openblas-pthread/ $ ls -lh libopenblasp-r0.3.28.a -rw-r--r-- 1 root root 61M Nov 20 05:52 libopenblasp-r0.3.28.a ``` 61 MB is very large! Although not all of OpenBLAS is used by our DGA detection program, a significant portion is, and this is the primary culprit for our large program size. Two alternatives to reduce this size are: #### Use reference BLAS and LAPACK instead The reference BLAS and LAPACK implementations are often slower, but much smaller in size. * On a Debian or Ubuntu system, install `libblas-dev` and `liblapack-dev` instead of `libopenblas-dev`. * This step alone reduces the size of the statically-linked `lstm_dga_detection_predict` to ***3.4 MB***, but the use of reference BLAS and LAPACK is likely to be slower than OpenBLAS. #### Compile OpenBLAS manually Another option is to compile OpenBLAS manually [from source](https://github.com/OpenMathLib/OpenBLAS) to reduce its size. OpenBLAS is compiled with a `make` command; options for this can be found in the [OpenBLAS manual](http://www.openmathlib.org/OpenBLAS/docs/build_system/#important-variables). As an example (***note:*** do not use this command directly on your system, see the description of the options and decide which is right for your system), the following command produces an OpenBLAS library that is only 16MB. ```sh make \ TARGET=NEHALEM \ DYNAMIC_ARCH=0 \ COMMON_OPTS="-Os -ffunction-sections -fdata-sections" \ NO_SHARED=1 \ BUILD_DOUBLE=0 \ BUILD_COMPLEX=0 \ BUILD_COMPLEX16=0 \ BUILD_BFLOAT16=0 ``` Looking at each option: * `TARGET=NEHALEM` and `DYNAMIC_ARCH=0` specifies that this version of OpenBLAS can ***only*** be run on [Nehalem](https://en.wikipedia.org/wiki/Nehalem_(microarchitecture)) or newer Intel processors. - ***This causes the code to be significantly less portable.*** - However, the `DYNAMIC_ARCH=0` option *significantly* reduces the size of OpenBLAS and therefore downstream code too by compiling only for the processor of interest. - For other `TARGET` options, see `TargetList.txt` in the OpenBLAS source code. * `COMMON_OPTS="-Os -ffunction-sections -fdata-sections"` specifies that the compiler should aim to keep the compiled code as small as possible, and include section information for later stripping and further size minimization of code. * `NO_SHARED=1` specifies that only the static version of OpenBLAS (`libopenblas.a`) should be built. * `BUILD_DOUBLE=0 BUILD_COMPLEX=0 BUILD_COMPLEX16=0 BUILD_BFLOAT16=0` disables all OpenBLAS functions for data types we are not using in our program. - The `lstm_dga_detection_predict.cpp` code only uses `arma::fmat` (e.g. matrices with `float`), so we can omit the other functions. With this stripped-down version of OpenBLAS, the size of `lstm_dga_detection_predict` with no further compilation modifications is reduced to ***3.5 MB***. ***Note:*** when compiling `lstm_dga_detection_predict`, linking against the hand-compiled OpenBLAS will require specifying the `-L/path/to/openblas/` option so that the linker finds the manually-compiled OpenBLAS version. ### Compilation options for further size reduction We have gone from 31 MB to roughly 3 MB just by replacing our OpenBLAS implementation with either reference BLAS/LAPACK or a hand-compiled version. However, we can specify some additional compiler options to reduce the size further. Assuming that we have placed the manually-compiled `libopenblas.a` in the same directory as `lstm_dga_detection_predict.cpp`, we can compile with the following command to reduce size even further: ```sh g++ -o lstm_dga_detection_predict lstm_dga_detection_predict.cpp \ -Os \ -DNDEBUG \ -DARMA_DONT_USE_WRAPPER \ -ffunction-sections -fdata-sections \ -static \ -L. -lopenblas \ -Wl,--gc-sections \ -Wl,--strip-all ``` Looking at each option: * `-Os` tells the compiler to optimize each function to have a small size. * `-DNDEBUG` removes any debugging symbols or code paths. * `-DARMA_DONT_USE_WRAPPER` is an Armadillo directive that tells Armadillo not to link against the Armadillo runtime library but instead directly against OpenBLAS. - This is why we use `-lopenblas` later instead of `-larmadillo`. * `-ffunction-sections -fdata-sections` are used so that the linker can later remove unused sections from the code. * `-Wl,--gc-sections -Wl,--strip-all` tells the linker to strip all unused code from the final program. * Note that we have *omitted* `-fopenmp`; this will cause compiled code to be somewhat smaller, but it will execute serially. Given that our prediction program only predicts a single domain at a time, this is acceptable. This shaves off more than 1 MB: ```text $ ls -lh lstm_dga_detection_predict -rwxrwxr-x 1 ryan ryan 1.9M Mar 6 21:37 lstm_dga_detection_predict ``` That is a size reduction of basically ***16x*** entirely obtained through compiler options. When the Docker container is recompiled with the new version of `lstm_dga_detection_predict`, the size is significantly improved: ```text $ docker images | grep -B 1 lstm REPOSITORY TAG IMAGE ID CREATED SIZE lstm_dga_detector latest 0be5a677fd9c 6 seconds ago 4.09MB ``` But there is still room for additional size reduction, either through further compiler options or more intrusive code modifications. Likely the largest contributor to code size after the optimizations above are serialization and the standard libraries: * Avoiding the use of `data::Load()` (and thus the [Cereal](https://uscilab.github.io/cereal/) serialization library) by directly saving the weights of the RNN using Armadillo's built-in save functionality would be effective. * Replacing the standard `libc` and `libstdc++` implementation with lightweight replacements, like [`musl`](https://musl.libc.org/) and others. These further optimizations, however, are beyond the scope of this tutorial.