How to Work with Servers to Deploy Machine Learning Apps?

Daniel Morales Perez

Now that we know how to create our models in Google Colab, and then we use that model to build an API using Flask, it’s time to deploy that project to a server. We certainly have multiple deployment options, but we want to use one that is quite easy to config and use: Digital Ocean.

Content of the blog

What is Digital Ocean?
Create Digital Ocean Account and Droplet
Setting Root and Deploy users for Ubuntu
Installing all necessary packages
What is uWSGI?
Configuring WSGI
Configuring NGINX
Adding a Domain
Adding SSL Certificates
Adding Github to make the workflow

Step #1 - What is Digital Ocean?

Digital Ocean proves to be the simplest cloud computing platform for business-critical infrastructure requirements and meets the level of ease one is looking for to cope with the data handling needs. The robust infrastructure provides businesses to build web apps or API backends, deploy container-based apps, and leverage the services of this IaaS platform to reduce the unproductive workload.

It’s best to figure out what you are going to do with your project first, consider what your needs are, and then evaluate it accordingly. One of the things that made DigitalOcean be the best to choose is they have thousands of easy to follow guides covering almost everything for any type of application/software. These fantastic guides will help you get started with DigitalOcean easy than ever. Here are some things I’ve noticed:

Set up is pretty straight forward, as well as Digital Ocean has a nicer UX for launching instances (or droplets as they call them). Also, have an API that you can configure to add/launch/delete/destroy your projects/help scale.

Performance/value does not have a clear cut answer of which is better for your project. Digital Ocean simplifies pricing by bulking it all together, but if you need additional storage or extra ram you need to snapshot and restore to a larger instance. This means even if your application uses 5GB of storage and 1GB of ram, you’re paying for 30GB of storage. In this situation, I would likely still go with Digital Ocean because even though you are overpaying for storage, the instance is substantially cheaper.

However, if you are building an application that is storage heavy that says, uses 1GB of RAM and requires 300GB of storage you probably don’t want to pay for Digital Ocean’s largest instance that has 12 Cores and 32GB of Ram just to get the storage. In this case, Amazon would be cheaper and more cost-effective.

I/O performance is another factor. Read/Write heavy applications fly on SSD hosting which is expensive traditionally, but Digital Ocean offers it extremely cheap. Conversely, AWS does offer SSD storage, but it’s prohibitively expensive.

Backups on Digital Ocean are pretty cheap as well. They charge only 20% of your droplet bills for a daily backup. I found it very handy.

I guess the conclusion is if money isn’t an issue, go with AWS or Azure. If it is, go with Digital Ocean. Both are high-quality, and both scale well.

Step #2 - Create Digital Ocean Account and Droplet

It's time for us to create our server. Open up DigitalOcean and go to the Create Droplet page.

Choose your operating system

We want to use Ubuntu 22.04 for our server's operating system. It's an long-term support (LTS) release which means it will receive security updates for several more years than normal. This is crucial for production.

Choose Ubuntu 22.04 from the dropdown in the Choose an image section.

Choose your size

Go ahead and select the server size under Choose a size based upon what you feel comfortable with.

Not sure what size to use? Just start with a 2GB RAM server. You can always resize the server to a larger size later without losing any data if you want. That's the nice part of these "virtual" servers. They're not real servers, so you easily add more RAM or CPUs to your server at any time.

Choose your region

Next we'll choose the server region. This is the datacenter our server lives in. Choose one that's closest to where your users (or you) will be.

Optional settings

You can enable a few other options if you choose:

Private Networking - Useful if you want to have a separate database server, etc or talk to other servers in the datacenter.
IPv6 - Enable this to give your server an IPv6 address. Can't hurt.
Monitoring - This is useful to get some rough metrics of server usage.
Backups - Backups up your entire server to an image that you can restore from. These usually don't run very often, so you can skip these if you want. Hourly database backups are more useful.

Step #3 - Setting Root and Deploy users for Ubuntu

Now, click "Create" to create your server. It will take about 60 seconds for DigitalOcean to create your server.

Once that's complete, check your email for the password to the new server.

You can login to your server as root by running the following command, just change 1.2.3.4 to your server's public IP address:

ssh [email protected]

Creating a Deploy user

To run software on our server, we're going to create a deploy user. The deploy user has limited permissions in production to help prevent anyone from getting full control of our server if a hacker broke into our server.

While logged in as root on the server, we can run the following commands to create the deploy user and add them to the sudo group.

[email protected]
adduser deploy
adduser deploy sudo
exit

Next let's add our SSH key to the server to make it faster to login. We're using a tool called ssh-copy-id for this.

If you're on a Mac, you may need to install ssh-copy-id with homebrew first: brew install ssh-copy-id

ssh-copy-id [email protected]
ssh-copy-id [email protected]

Now you can login as either root or deploy without having to type in a password!

For the rest of this tutorial, we want to be logged in as deploy to setup everything. Let's SSH in as deploy now and we shouldn't be prompted for a password this time.

ssh [email protected]

Step #4 - Installing all necessary packages

Our first step will be to install all of the pieces that we need from the repositories. We will install pip, the Python package manager, in order to install and manage our Python components. We will also get the Python development files needed to build uWSGI and we’ll install Nginx now as well.

We need to update the local package index and then install the packages. The packages you need depend on whether your project uses Python 2 or Python 3.

If you are using Python 2, type:

sudo apt-get update
sudo apt-get install python-pip python-dev nginx

If, instead, you are using Python 3, type:

sudo apt-get update
sudo apt-get install python3-pip python3-dev nginx

Create a Python Virtual Environment

Next, we’ll set up a virtual environment in order to isolate our Flask application from the other Python files on the system.

Start by installing the virtualenv package using pip.

If you are using Python 2, type:

sudo pip install virtualenv

If you are using Python 3, type:

sudo pip3 install virtualenv

Now, we can make a parent directory for our Flask project. Move into the directory after you create it:

mkdir ~/myproject
cd ~/myproject

We can create a virtual environment to store our Flask project’s Python requirements by typing:

virtualenv myprojectenv

This will install a local copy of Python and pip into a directory called myprojectenv within your project directory.

Before we install applications within the virtual environment, we need to activate it. You can do so by typing:

source myprojectenv/bin/activate

Your prompt will change to indicate that you are now operating within the virtual environment. It will look something like this (myprojectenv)user@host:~/myproject$.

Step #5 - What is uWSGI?

Before we jump in, we should address some confusing terminology associated with the interrelated concepts we will be dealing with. These three separate terms that appear interchangeable, but actually have distinct meanings:

WSGI: A Python spec that defines a standard interface for communication between an application or framework and an application/web server. This was created in order to simplify and standardize communication between these components for consistency and interchangeability. This basically defines an API interface that can be used over other protocols.
uWSGI: An application server container that aims to provide a full stack for developing and deploying web applications and services. The main component is an application server that can handle apps of different languages. It communicates with the application using the methods defined by the WSGI spec, and with other web servers over a variety of other protocols. This is the piece that translates requests from a conventional web server into a format that the application can process.
uwsgi: A fast, binary protocol implemented by the uWSGI server to communicate with a more full-featured web server. This is a wire protocol, not a transport protocol. It is the preferred way to speak to web servers that are proxying requests to uWSGI.

WSGI Application Requirements

The WSGI spec defines the interface between the web server and application portions of the stack. In this context, “web server” refers to the uWSGI server, which is responsible for translating client requests to the application using the WSGI spec. This simplifies communication and creates loosely coupled components so that you can easily swap out either side without much trouble.

The web server (uWSGI) must have the ability to send requests to the application by triggering a defined “callable”. The callable is simply an entry point into the application where the web server can call a function with some parameters. The expected parameters are a dictionary of environmental variables and a callable provided by the web server (uWSGI) component.

In response, the application returns an iterable that will be used to generate the body of the client response. It will also call the web server component callable that it received as a parameter. The first parameter when triggering the web server callable will be the HTTP status code and the second will be a list of tuples, each of which define a response header and value to send back to the client.

With the “web server” component of this interaction provided by uWSGI in this instance, we will only need to make sure our applications have the qualities described above. We will also set up Nginx to handle actual client requests and proxy them to the uWSGI server.

Step #6 - Configuring WSGI

Now that you are in your virtual environment, we can install Flask and uWSGI and get started on designing our application:

Install Flask and uWSGI

We can use the local instance of pip to install Flask and uWSGI. Type the following commands to get these two components:

Note: Regardless of which version of Python you are using, when the virtual environment is activated, you should use the pip command (not pip3).

pip install uwsgi flask

Create a Sample App

Now that we have Flask available, we can create a simple application. Flask is a micro-framework. It does not include many of the tools that more full-featured frameworks might, and exists mainly as a module that you can import into your projects to assist you in initializing a web application.

While your application might be more complex, we’ll create our Flask app in a single file, which we will call myproject.py:

nano ~/myproject/myproject.py

Within this file, we’ll place our application code. Basically, we need to import Flask and instantiate a Flask object. We can use this to define the functions that should be run when a specific route is requested:

~/myproject/myproject.py
from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "<h1 style='color:blue'>Hello There!</h1>"

if __name__ == "__main__":
    app.run(host='0.0.0.0')

This basically defines what content to present when the root domain is accessed. Save and close the file when you’re finished.

If you followed the initial server setup guide, you should have a UFW firewall enabled. In order to test our application, we need to allow access to port 5000.

Open up port 5000 by typing:

sudo ufw allow 5000

Now, you can test your Flask app by typing:

python myproject.py

Visit your server’s domain name or IP address followed by :5000 in your web browser:

http://server_domain_or_IP:5000

You should see something like this:

When you are finished, hit CTRL-C in your terminal window a few times to stop the Flask development server.

Create the WSGI Entry Point

Next, we’ll create a file that will serve as the entry point for our application. This will tell our uWSGI server how to interact with the application.

We will call the file wsgi.py:

nano ~/myproject/wsgi.py

The file is incredibly simple, we can simply import the Flask instance from our application and then run it:

~/myproject/wsgi.py
from myproject import app

if __name__ == "__main__":
    app.run()

Save and close the file when you are finished.

Configure uWSGI

Our application is now written and our entry point established. We can now move on to uWSGI.

Testing uWSGI Serving

The first thing we will do is test to make sure that uWSGI can serve our application.

We can do this by simply passing it the name of our entry point. This is constructed by the name of the module (minus the .py extension, as usual) plus the name of the callable within the application. In our case, this would be wsgi:app.

We’ll also specify the socket so that it will be started on a publicly available interface and the protocol so that it will use HTTP instead of the uwsgi binary protocol. We’ll use the same port number that we opened earlier:

uwsgi --socket 0.0.0.0:5000 --protocol=http -w wsgi:app

Visit your server’s domain name or IP address with :5000 appended to the end in your web browser again:

http://server_domain_or_IP:5000

You should see your application’s output again:

When you have confirmed that it’s functioning properly, press CTRL-C in your terminal window.

We’re now done with our virtual environment, so we can deactivate it:

deactivate

Any Python commands will now use the system’s Python environment again.

Creating a uWSGI Configuration File

We have tested that uWSGI is able to serve our application, but we want something more robust for long-term usage. We can create a uWSGI configuration file with the options we want.

Let’s place that in our project directory and call it myproject.ini:

nano ~/myproject/myproject.ini

Inside, we will start off with the [uwsgi] header so that uWSGI knows to apply the settings. We’ll specify the module by referring to our wsgi.py file, minus the extension, and that the callable within the file is called “app”:

~/myproject/myproject.ini
[uwsgi]
module = wsgi:app

Next, we’ll tell uWSGI to start up in master mode and spawn five worker processes to serve actual requests:

~/myproject/myproject.ini
[uwsgi]
module = wsgi:app

master = true
processes = 5

When we were testing, we exposed uWSGI on a network port. However, we’re going to be using Nginx to handle actual client connections, which will then pass requests to uWSGI. Since these components are operating on the same computer, a Unix socket is preferred because it is more secure and faster. We’ll call the socket myproject.sock and place it in this directory.

We’ll also have to change the permissions on the socket. We’ll be giving the Nginx group ownership of the uWSGI process later on, so we need to make sure the group owner of the socket can read information from it and write to it. We will also clean up the socket when the process stops by adding the “vacuum” option:

~/myproject/myproject.ini
[uwsgi]
module = wsgi:app

master = true
processes = 5

socket = myproject.sock
chmod-socket = 660
vacuum = true

The last thing we need to do is set the die-on-term option. This can help ensure that the init system and uWSGI have the same assumptions about what each process signal means. Setting this aligns the two system components, implementing the expected behavior:

~/myproject/myproject.ini
[uwsgi]
module = wsgi:app

master = true
processes = 5

socket = myproject.sock
chmod-socket = 660
vacuum = true

die-on-term = true

You may have noticed that we did not specify a protocol like we did from the command line. That is because by default, uWSGI speaks using the uwsgi protocol, a fast binary protocol designed to communicate with other servers. Nginx can speak this protocol natively, so it’s better to use this than to force communication by HTTP.

When you are finished, save and close the file.

Create a systemd Unit File

The next piece we need to take care of is the systemd service unit file. Creating a systemd unit file will allow Ubuntu’s init system to automatically start uWSGI and serve our Flask application whenever the server boots.

Create a unit file ending in .service within the /etc/systemd/system directory to begin:

sudo nano /etc/systemd/system/myproject.service

Inside, we’ll start with the [Unit] section, which is used to specify metadata and dependencies. We’ll put a description of our service here and tell the init system to only start this after the networking target has been reached:

/etc/systemd/system/myproject.service
[Unit]
Description=uWSGI instance to serve myproject
After=network.target

Next, we’ll open up the [Service] section. We’ll specify the user and group that we want the process to run under. We will give our regular user account ownership of the process since it owns all of the relevant files. We’ll give group ownership to the www-data group so that Nginx can communicate easily with the uWSGI processes.

We’ll then map out the working directory and set the PATH environmental variable so that the init system knows where our the executables for the process are located (within our virtual environment). We’ll then specify the commanded to start the service. Systemd requires that we give the full path to the uWSGI executable, which is installed within our virtual environment. We will pass the name of the .ini configuration file we created in our project directory:

/etc/systemd/system/myproject.service
[Unit]
Description=uWSGI instance to serve myproject
After=network.target

[Service]
User=sammy
Group=www-data
WorkingDirectory=/home/sammy/myproject
Environment="PATH=/home/sammy/myproject/myprojectenv/bin"
ExecStart=/home/sammy/myproject/myprojectenv/bin/uwsgi --ini myproject.ini

Finally, we’ll add an [Install] section. This will tell systemd what to link this service to if we enable it to start at boot. We want this service to start when the regular multi-user system is up and running:

/etc/systemd/system/myproject.service
[Unit]
Description=uWSGI instance to serve myproject
After=network.target

[Service]
User=sammy
Group=www-data
WorkingDirectory=/home/sammy/myproject
Environment="PATH=/home/sammy/myproject/myprojectenv/bin"
ExecStart=/home/sammy/myproject/myprojectenv/bin/uwsgi --ini myproject.ini

[Install]
WantedBy=multi-user.target

With that, our systemd service file is complete. Save and close it now.

We can now start the uWSGI service we created and enable it so that it starts at boot:

sudo systemctl start myproject
sudo systemctl enable myproject

Step #7 - Configuring NGINX

Our uWSGI application server should now be up and running, waiting for requests on the socket file in the project directory. We need to configure Nginx to pass web requests to that socket using the uwsgi protocol.

Begin by creating a new server block configuration file in Nginx’s sites-available directory. We’ll simply call this myproject to keep in line with the rest of the guide:

sudo nano /etc/nginx/sites-available/myproject

Open up a server block and tell Nginx to listen on the default port 80. We also need to tell it to use this block for requests for our server’s domain name or IP address:

/etc/nginx/sites-available/myproject
server {
    listen 80;
    server_name server_domain_or_IP;
}

The only other thing that we need to add is a location block that matches every request. Within this block, we’ll include the uwsgi_params file that specifies some general uWSGI parameters that need to be set. We’ll then pass the requests to the socket we defined using the uwsgi_pass directive:

/etc/nginx/sites-available/myproject
server {
    listen 80;
    server_name server_domain_or_IP;

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/home/sammy/myproject/myproject.sock;
    }
}

That’s actually all we need to serve our application. Save and close the file when you’re finished.

To enable the Nginx server block configuration we’ve just created, link the file to the sites-enabled directory:

sudo ln -s /etc/nginx/sites-available/myproject /etc/nginx/sites-enabled

With the file in that directory, we can test for syntax errors by typing:

sudo nginx -t

If this returns without indicating any issues, we can restart the Nginx process to read the our new config:

sudo systemctl restart nginx

The last thing we need to do is adjust our firewall again. We no longer need access through port 5000, so we can remove that rule. We can then allow access to the Nginx server:

sudo ufw delete allow 5000
sudo ufw allow 'Nginx Full'

You should now be able to go to your server’s domain name or IP address in your web browser:

http://server_domain_or_IP

You should see your application output:

Step #8 - Adding a Domain

Now we want to give our users a domain to go over a browser and test our amazing machine learning model, so we want a domain name. The first step is to buy a domain in the multiple vendors out there. The one I recommend is NameCheap

Inside NameCheap

In the Nameservers section of the resulting screen, select Custom DNS from the dropdown menu and enter the following nameservers:

ns1.digitalocean.com
ns2.digitalocean.com
ns3.digitalocean.com

In Digital Ocean
Add domain inside Digital Ocean under Networking

An A record with example.com pointing to your server’s public IP address.

An A record with www.example.com pointing to your server’s public IP address

That's it

Step #9 - Adding SSL Certificates

Let’s Encrypt is a Certificate Authority (CA) that provides an easy way to obtain and install free TLS/SSL certificates, thereby enabling encrypted HTTPS on web servers. It simplifies the process by providing a software client, Certbot, that attempts to automate most (if not all) of the required steps. Currently, the entire process of obtaining and installing a certificate is fully automated on both Apache and Nginx.

Inside the Digital ocean server type the following

sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-nginx
export LC_ALL=C
python3 -m pip install cffi

Here server_name needs to be my new domain, not a domain, because this is just an Endpoint

sudo vim /etc/nginx/sites-available/myproject

Change the following line with your domain

## file
. . .
server_name example.com www.example.com;
. . .

Exit and now type the following

## ufw need to be 'Nginx Full', not 'Nginx HTTP'
sudo ufw allow 'Nginx Full'
sudo ufw delete allow 'Nginx HTTP'

## install certificate
sudo certbot --nginx -d example.com -d www.example.com
select option 2

#### INIT output IMPORTANT RENEW CERTICATE!!!!!!!
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/myprojectname/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/myprojectname/privkey.pem
   Your cert will expire on 2021-02-14. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"

Now, this certificate has 3 months expiration. If you want to renew you have to follow this command

## renew
sudo certbot renew
## response: Congratulations, all renewals succeeded. The following certs have been renewed:

## test renew
sudo certbot renew --dry-run

Step #10 - Adding Github to make the workflow

The final step we need is to install Git inside the server, and with that we can keep the normal workflow to deploy to production each new change we need.

Initial setup

## using github in server to fetch changes from there
## access to the projectname
git init


# Generate SSH key using 
ssh-keygen -t rsa -b 4096 -C "your email"
Enter file in which (just type enter)
enter github passw

#Copy the output of next command to your clipboard
cat /home/deploy/.ssh/id_rsa.pub

# Paste the above copied output to the form at https://github.com/settings/ssh/new
# and saveit

## git commands
git remote add origin [email protected]:user_name/projectname.git
git remote show origin
## SSH keys on guthub working now

git config --global user.email "[email protected]"
git config --global user.name "Your Name"
git add .
git commit -m "Ticket #1 - Commit From Server"
git push origin master

Daily workflow
Now, when we have new changes to deploy, we just need to follow these commands:

# in local
git add .
git commit -m "Ticket #1 - Commit From Server"
git push origin master

# in server
ssh deploy@IPADDRESS
cd myproject

## Force git pull origin master
git fetch origin master
git reset --hard origin/master
sudo systemctl restart endpoints

## other options
git pull origin master
sudo systemctl restart endpoints

Awesome, we have now a Machine Learning project running in a real world production environment!

Hope you learned a lot!