开源软件名称（OpenSource Name）：

开源软件地址(OpenSource Url)：

开源编程语言(OpenSource Language)：

开源软件介绍(OpenSource Introduction)：

• Slides available here.
• Original article posted here.
• Source code here.
• Inspired from my course: Learn Kubernetes by building 10 projects

Introduction

In this workshop, we're going to:

• Deploy Kubernetes services and an Ambassador API gateway.
• Examine the difference between Kubernetes proxies and service mesh like Istio.
• Access the Kubernetes API from the outside and from a Pod.
• Understand what API to choose.
• See how Service Accounts and RBAC works
• Discover some security pitfalls when building Docker images and many interesting things.
• Other things :-)

We will start by developing then deploying a simple Python application (a Flask API that returns the list of trending repositories by programming language).

Development Environment

We are going to use Python 3.6.7

We are using Ubuntu 18.04 that comes with Python 3.6 by default. You should be able to invoke it with the command python3. (Ubuntu 17.10 and above also come with Python 3.6.7)

If you use Ubuntu 16.10 and 17.04, you should be able to install it with the following commands:

sudo apt-get update
sudo apt-get install python3.6

If you are using Ubuntu 14.04 or 16.04, you need to get Python 3 from a Personal Package Archive (PPA):

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.6

For the other operating systems, visit this guide, follow the instructions and install Python3.

Now install PIP, the package manager:

sudo apt-get install python3-pip

Follow this by the installation of Virtualenvwrapper, which is a virtual environment manager:

sudo pip3 install virtualenvwrapper

Create a folder for your virtualenvs (I use ~/dev/PYTHON_ENVS) and set it as WORKON_HOME:

mkdir  ~/dev/PYTHON_ENVS
export WORKON_HOME=~/dev/PYTHON_ENVS

In order to source the environment details when the user login, add the following lines to ~/.bashrc:

source "/usr/local/bin/virtualenvwrapper.sh"
export WORKON_HOME="~/dev/PYTHON_ENVS"

Make sure to adapt the WORKON_HOME to your real WORKON_HOME. Now we need to create then activate the new environment:

mkvirtualenv --python=/usr/bin/python3 trendinggitrepositories
workon trendinggitrepositories

Let's create the application directories:

mkdir trendinggitrepositories
cd trendinggitrepositories
mkdir api
cd api

Once the virtual environment is activated, we can install Flask:

pip install flask

Developing a Trending Git Repositories API (Flask)

Inside the API folder api, create a file called app.py and add the following code:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return "Hello, World!"

if __name__ == '__main__':
    app.run(debug=True)

This will return a hello world message when a user requests the "/" route.

Now run it using: python app.py and you will see a similar output to the following one:

* Serving Flask app "api" (lazy loading)
* Environment: production
  WARNING: This is a development server. Do not use it in a production deployment.
  Use a production WSGI server instead.
* Debug mode: on
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 465-052-587

We now need to install PyGithub since we need it to communicate with Github API v3.

pip install PyGithub

Go to Github and create a new app. We will need the application "Client ID" and "Client Secret":

from github import Github
g = Github("xxxxxxxxxxxxx", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")

This is how the mini API looks like:

from flask import Flask, jsonify, abort
import urllib.request, json
from flask import request

app = Flask(__name__)

from github import Github
g = Github("xxxxxx", "xxxxxxxxxxxxx")

@app.route('/')
def get_repos():
    r = []

    try:
        args = request.args
        n = int(args['n'])
    except (ValueError, LookupError) as e:
        abort(jsonify(error="No integer provided for argument 'n' in the URL"))

    repositories = g.search_repositories(query='language:python')[:n]

    for repo in repositories:
        with urllib.request.urlopen(repo.url) as url:
            data = json.loads(url.read().decode())
        r.append(data)

    return jsonify({'repos':r })

if __name__ == '__main__':
    app.run(debug=True)

Let's hide the Github token and secret as well as other variables in the environment.

from flask import Flask, jsonify, abort, request
import urllib.request, json, os
from github import Github

app = Flask(__name__)

CLIENT_ID = os.environ['CLIENT_ID']
CLIENT_SECRET = os.environ['CLIENT_SECRET']
DEBUG = os.environ['DEBUG']

g = Github(CLIENT_ID, CLIENT_SECRET)


@app.route('/')
def get_repos():
    r = []

    try:
        args = request.args
        n = int(args['n'])
    except (ValueError, LookupError) as e:
        abort(jsonify(error="No integer provided for argument 'n' in the URL"))

    repositories = g.search_repositories(query='language:python')[:n]

    for repo in repositories:
        with urllib.request.urlopen(repo.url) as url:
            data = json.loads(url.read().decode())
        r.append(data)

    return jsonify({'repos':r })

if __name__ == '__main__':
    app.run(debug=DEBUG)

The code above will return the top "n" repositories using Python as a programming language. We can use other languages too:

from flask import Flask, jsonify, abort, request
import urllib.request, json, os
from github import Github

app = Flask(__name__)

CLIENT_ID = os.environ['CLIENT_ID']
CLIENT_SECRET = os.environ['CLIENT_SECRET']
DEBUG = os.environ['DEBUG']

g = Github(CLIENT_ID, CLIENT_SECRET)


@app.route('/')
def get_repos():
    r = []

    try:
        args = request.args
        n = int(args['n'])
        l = args['l']
    except (ValueError, LookupError) as e:
        abort(jsonify(error="Please provide 'n' and 'l' parameters"))

    repositories = g.search_repositories(query='language:' + l)[:n]


    try:
        for repo in repositories:
            with urllib.request.urlopen(repo.url) as url:
                data = json.loads(url.read().decode())
            r.append(data)
        return jsonify({
            'repos':r,
            'status': 'ok'
            })
    except IndexError as e:
        return jsonify({
            'repos':r,
            'status': 'ko'
            })

if __name__ == '__main__':
    app.run(debug=DEBUG)

In a .env file, add the variables you want to use:

CLIENT_ID="xxxxx"
CLIENT_SECRET="xxxxxx"
ENV="dev"
DEBUG="True"

Before running the Flask application, you need to source these variables:

source .env

Now, you can go to http://0.0.0.0:5000/?n=1&l=python to get the trendiest Python repository or http://0.0.0.0:5000/?n=1&l=c for C programming language. Here is a list of other programming languages you can test your code with:

C++
Assembly
Objective
Makefile
Shell
Perl
Python
Roff
Yacc
Lex
Awk
UnrealScript
Gherkin
M4
Clojure
XS
Perl
sed

The list is long, but our mini API is working fine. Now, let's freeze the dependencies:

pip freeze > requirements.txt

Before running the API on Kubernetes, let's create a Dockerfile. This is a typical Dockerfile for a Python app:

FROM python:3
ENV PYTHONUNBUFFERED 1
RUN mkdir /app
WORKDIR /app
COPY requirements.txt /app
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
COPY . /app
EXPOSE 5000
CMD [ "python", "app.py" ]

Now you can build it:

docker build --no-cache -t tgr .

Then run it:

docker rm -f tgr
docker run -it  --name tgr -p 5000:5000 -e CLIENT_ID="xxxxxxx" -e CLIENT_SECRET="xxxxxxxxxxxxxxx" -e DEBUG="True" tgr

Let's include some other variables as environment variables:

from flask import Flask, jsonify, abort, request
import urllib.request, json, os
from github import Github

app = Flask(__name__)

CLIENT_ID = os.environ['CLIENT_ID']
CLIENT_SECRET = os.environ['CLIENT_SECRET']
DEBUG = os.environ['DEBUG']
HOST = os.environ['HOST']
PORT = os.environ['PORT']

g = Github(CLIENT_ID, CLIENT_SECRET)


@app.route('/')
def get_repos():
    r = []

    try:
        args = request.args
        n = int(args['n'])
        l = args['l']
    except (ValueError, LookupError) as e:
        abort(jsonify(error="Please provide 'n' and 'l' parameters"))

    repositories = g.search_repositories(query='language:' + l)[:n]


    try:
        for repo in repositories:
            with urllib.request.urlopen(repo.url) as url:
                data = json.loads(url.read().decode())
            r.append(data)
        return jsonify({
            'repos':r,
            'status': 'ok'
            })
    except IndexError as e:
        return jsonify({
            'repos':r,
            'status': 'ko'
            })

if __name__ == '__main__':
    app.run(debug=DEBUG, host=HOST, port=PORT)

For security reasons, let's change the user inside the container from root to a user with less rights that we create:

FROM python:3
ENV PYTHONUNBUFFERED 1
RUN adduser pyuser

RUN mkdir /app
WORKDIR /app
COPY requirements.txt /app
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
COPY . .
RUN chmod +x app.py

RUN chown -R pyuser:pyuser /app
USER pyuser


EXPOSE 5000
CMD ["python","./app.py"]

Now if we want to run the container, we need to add many environment variables to the docker run command. An easier solution is using --env-file with Docker run:

docker run -it --env-file .env my_container

Our .env file looks like the following one:

CLIENT_ID="xxxx"
CLIENT_SECRET="xxxx"
ENV="dev"
DEBUG="True"
HOST="0.0.0.0"
PORT=5000

After this modification, rebuild the image docker build -t tgr . and run it using:

docker rm -f tgr;
docker run -it  --name tgr -p 5000:5000 --env-file .env  tgr

Our application runs using python app.py which is the webserver that ships with Flask and it's great for development and local execution of your program, however, it's not designed to run in a production mode, whether it's a monolithic app or a microservice.

A production server typically receives abuse from spammers, script kiddies, and should be able to handle high traffic. In our case, a good solution is using a WSGI HTTP server like Gunicorn (or uWsgi).

First, let's install gunicorn with the following command: pip install gunicorn. This will require us to update our requirements.txt with pip freeze > requirements.txt

This is why we are going to change our Docker file:

FROM python:3
ENV PYTHONUNBUFFERED 1
RUN adduser pyuser

RUN mkdir /app
WORKDIR /app
COPY requirements.txt /app
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
COPY . .
RUN chmod +x app.py

RUN chown -R pyuser:pyuser /app
USER pyuser


EXPOSE 5000
CMD ["gunicorn", "app:app", "-b", "0.0.0.0:5000"]

In order to optimize the Wsgi server, we need to set the number of its workers and threads to:

workers = multiprocessing.cpu_count() * 2 + 1
threads = 2 * multiprocessing.cpu_count()

This is why we are going to create another Python configuration file (config.py):

import multiprocessing
workers = multiprocessing.cpu_count() * 2 + 1
threads = 2 * multiprocessing.cpu_count()

In the same file, we are going to include other configurations of Gunicorn:

from os import environ as env
bind = env.get("HOST","0.0.0.0") +":"+ env.get("PORT", 5000)

This is the final config.py file:

import multiprocessing
workers = multiprocessing.cpu_count() * 2 + 1
threads = 2 * multiprocessing.cpu_count()

from os import environ as env
bind = env.get("HOST","0.0.0.0") +":"+ env.get("PORT", 5000)

In consequence, we should adapt the Dockerfile to the new Gunicorn configuration by changing the last line to :

CMD ["gunicorn", "app:app", "--config=config.py"]

Now, build docker build -t tgr . and run docker run -it --env-file .env -p 5000:5000 tgr.

Pushing the Image to a Remote Registry

A Docker registry is a storage and distribution system for named Docker images.

The images we built are stored in our local environment and can only be used if you deploy locally. However, if you choose to deploy a Kubernetes cluster in a cloud or any different environment, these images will be not found. This is why we need to push the build images to a remote registry.

Think of container registries as a git system for Docker images.

There are plenty of containers registries:

• Dockerhub
• Amazon Elastic Registry (ECR)
• Azure Container Registry (ACR)
• Google Container Registry (GCR)
• CoreOS Quay

You can also host your private container registry that supports OAuth, LDAP and Active Directory authentication using the registry provided by Docker:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

More about self-hosting a registry can be found in the official Docker documentation.

We are going to use Dockerhub; this is why you need to create an account on hub.docker.com.

Now, using Docker CLI, login:

docker login

Now rebuild the image using the new tag:

 docker build -t <username>/<image_name>:<tag_version> .

Example:

docker build -t eon01/tgr:1 .

Finally, push the image:

docker push eon01/tgr:1

A Security Notice

Many of the publicly (and even private Docker images) seems to be secure, but it's not the case. When we built our image, we told Docker to copy all the images from the application folder to the image and we push it to an external public registry.

COPY . .        

Or

ADD . .

The above commands will even copy the .env file containing our secrets.

A good solution is to tell Docker to ignore these files during the build using a .dockerignore file:

**.git
**.gitignore
**README.md
**env.*
**Dockerfile*
**docker-compose*
**.env

At this stage, you should remove any image that you pushed to a distant registry, reset the Github tokens, build the new image without any cache:

docker build -t eon01/tgr:1 . --no-cache

Push it again:

docker push eon01/tgr:1

Installing Minikube

One of the fastest ways to try Kubernetes is using Minkube, which will create a virtual machine for you and deploy a ready-to-use Kubernetes cluster.

Before you begin the installation, you need to make sure that your laptop supports virtualization:

If your using Linux, run the following command and make sure that the output is not empty:

grep -E --color 'vmx|svm' /proc/cpuinfo

Mac users should execute:

sysctl -a | grep -E --color 'machdep.cpu.features|VMX'

If you see VMX in the output, the VT-x feature is enabled in your machine.

Windows users should use systeminfo and you should see the following output:

Hyper-V Requirements:     VM Monitor Mode Extensions: Yes
                          Virtualization Enabled In Firmware: Yes
                          Second Level Address Translation: Yes
                          Data Execution Prevention Available: Yes

If everything is okay, you need to install a hypervisor. You have a list of possibilities here:

• KVM
• VirtualBox
• HyperKit
• VMware Fusion
• Hyper-V

Some of these hypervisors are only compatible with some OSs like Hyper-V (formerly known as Windows Server Virtualization) for windows.

VirtualBox is however cross-platform, and this is why we are going to use it here. Make sure to follow the instructions to install it.

Now, install Minikube.

Linux systems:

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube
sudo install minikube /usr/local/bin

MacOs:

brew cask install minikube
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 && chmod +x minikube
sudo mv minikube /usr/local/bin

Windows:

Use Chocolatey as an administrator:

choco install minikube

Or use the installer binary.

Minikube does not support all Kubernetes features (like load balancing for example), however, you can find the most important features there:

Minikube supports the following Kubernetes features:

• DNS
• NodePorts
• ConfigMaps and Secrets
• Dashboards
• Container Runtime: Docker, rkt, CRI-O, and containerd
• Enabling CNI (Container Network Interface)
• Ingress

You can also add different addons like:

• addon-manager
• dashboard
• default-storageclass
• efk
• freshpod
• gvisor
• heapster
• ingress
• logviewer
• metrics-server
• nvidia-driver-installer
• nvidia-gpu-device-plugin
• registry
• registry-creds
• storage-provisioner
• storage-provisioner-gluster

If you run minikube start a cluster called minikube will be created; however, you have other choices rather than just creating a regular Minikube cluster. In this example, we are going to create a cluster called "workshop", enable a UI to browse the API and activate tailing logs:

minikube start -p workshop --extra-config=apiserver.enable-swagger-ui=true --alsologtostderr

You have plenty of other options to start a Minikube cluster; you can, for instance, choose the Kubernetes version and the VM driver:

minikube start --kubernetes-version="v1.12.0" --vm-driver="virtualbox"  

Start the new cluster:

minikube start -p workshop --extra-config=apiserver.enable-swagger-ui=true --alsologtostderr

You can get detailed information about the cluster using:

kubectl cluster-info

If you didn't install kubectl, follow the official instructions.

You can open the dashboard using minikube -p workshop dashboard

Deploying to Kubernetes

We have three main ways to deploy our container to Kubernetes and scale it to N replica.

The first one is the original form of replication in Kubernetes, and it's called Replication Controller.

Even if Replica Sets replace it, it's still used in some codes.

This is a typical example:

apiVersion: v1
kind: ReplicationController
metadata:
  name: app
spec:
  replicas: 3
  selector:
    app: app
  template:
    metadata:
      name: app
      labels:
        app: app
    spec:
      containers:
      - name: tgr
        image: reg/app:v1
        ports:
        - containerPort: 80

We can also use Replica Sets, another way to deploy an app and replicate it:

apiVersion: extensions/v1beta1
 kind: ReplicaSet
 metadata:
   name: app
 spec:
   replicas: 3
   selector:
     matchLabels:
       app: app
   template:
     metadata:
       labels:
         app: app
         environment: dev
     spec:
       containers:
       - name: app
         image: reg/app:v1
         ports:
         - containerPort: 80

Replica Set and Replication Controller do almost the same thing.

They ensure that you have a specified number of pod replicas running at any given time in your cluster.

There are however, some differences.

As you may notice, we are using matchLabels instead of label.

Replica Set use Set-Based selectors while replication controllers use Equity-Based selectors.

Selectors match Kubernetes objects (like pods) using the constraints of the specified label, and we are going to see an example in a Deployment specification file.

Label selectors with equality-based requirements use three operators:=,== and !=.

environment = production
tier != frontend
app == my_app (similar to app = my_app)

In the last example, we used this notation:

 ...
 spec:
   replicas: 3
   selector:
     mat