Setting up CI/CD for Docker and Kubernetes Using Drone

I have been a Travis CI user. However, Travis has gotten less reliable for me lately. On top of that, I have qualms about how the acquisition of Travis by Idera, and the subsequent layoffs, were handled. Travis is also a square peg in the octagonal hole of my Kubernetes environment. It is a hosted, external service. Everything else I use to develop my applications is hosted inside of my cluster. My Docker registry and my gitops operator run in my cluster. My databases are in my cluster. My storage provider and object store run in cluster. My apps run in cluster. Why would I run my CI/CD service outside of the cluster?

I came across Drone in my research about alternatives. Drone is a fully container-native, container-loving CI solution. It’s Docker all the way down. Since my application is already “Dockerified”, my hosting environment is all Docker all day, and my deployments are already in the form of a Docker push, why not do CI/CD in Docker as well?

Drone is basically a small framework for running CI jobs made of docker containers. You build your pipeline as a series of steps, each of which is a docker base image, some configuration, and your test commands. Drone also has all the standard integrations you would expect for a CI service— it talks to GitHub, GitLab, Bitbucket, and more.

To use Drone, you’ll have to embrace the Docker way. Let go of your test scripts that are building everything from a ubuntu image or a language version manager. Let go of the “special case magic” way DBs and other supporting services are handled in other CI platforms. In exchange, you’ll find that Drone will let you use any language, any tools, and any languages, so long as they have a Docker image.

Drone Setup on Kubernetes

I followed this official (but experimental) guide for setting up Drone on Kubernetes.

Here’s the configuration I used for Kubernetes:

---
apiVersion: v1
kind: Namespace
metadata:
  name: drone
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  labels:
    app: drone
  name: drone
  namespace: drone
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: drone
      name: drone
      namespace: drone
    spec:
      containers:
      - name: drone
        image: drone/drone:1.0.0
        env:
        - name: DRONE_KUBERNETES_ENABLED
          value: "true"
        - name: DRONE_KUBERNETES_NAMESPACE
          value: "drone"
        - name: DRONE_GITHUB_SERVER
          value: "https://github.com"
        - name: DRONE_GITHUB_CLIENT_ID
          value: "REDACTED"
        - name: DRONE_GITHUB_CLIENT_SECRET
          value: "REDACTED"
        - name: DRONE_RPC_SECRET
          value: "REDACTED"
        - name: DRONE_SERVER_HOST
          value: "REDACTED.example.com"
        - name: DRONE_SERVER_PROTO
          value: "https"
        - name: DRONE_USER_FILTER
          value: "prehnRA"
        - name: DRONE_USER_CREATE
          value: username:prehnRA,admin:true
        - name: DRONE_DATABASE_DRIVER
          value: postgres
        - name: DRONE_DATABASE_DATASOURCE
          valueFrom:
            secretKeyRef:
              name: drone-postgres-url
              key: url
        ports:
          - containerPort: 80
          - containerPort: 443
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: drone
  name: drone
  namespace: drone
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: drone
  type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/tls-acme: "true" # enable certificates
    certmanager.k8s.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: "nginx"
  labels:
    app: drone
  name: drone
  namespace: drone
spec:
  rules:
  - host: REDACTED.example.com
    http:
      paths:
      - backend:
          serviceName: drone
          servicePort: 80
        path: /
  tls: # specify domains to fetch certificates for
  - hosts:
    - REDACTED.example.com
    secretName: drone-tls
---
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
  name: drone-postgres
  namespace: drone
spec:
  version: "10.2-v1"
  storageType: Durable
  storage:
    storageClassName: "rook-block"
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 256Mi
  terminationPolicy: DoNotTerminate

Note that I use kubedb in my cluster, so I am able to request a new Postgres database through Kubernetes configuration YAML. If you don’t use kubedb or similar, you’ll have to provide a db to Drone differently. By default, Drone uses a sqlite3 database, but this isn’t much good in Kubernetes by default, because if you Drone pods get restarted, you will lose your configuration and job history.

I give the Postgres url to Drone via a Kubernetes secret called drone-postgres-url in the url key. This secret must be in the drone namespace. Here’s how you can do that:

echo -n 'YOUR_POSTGRES_URL' > ./url
kubectl create secret generic drone-postgres-url -n drone --from-file=./url

Testing and Deploying An App

The application I wanted to test and deploy through Drone is an Elixir app that is released by semantic-release as a Docker image. I’ve written previously on the Revelry blog about how to do that.

You configure Drone using a .drone.yml file. The main YAML object defined in that file is a “pipeline”, which defines a series of steps and services which Drone will use to run your tests and deploy your app.

A basic pipeline for my app might look like this:

---
kind: pipeline
name: default

steps:
- name: backend
  image: elixir:1.8.0-alpine
  commands:
  - mix local.hex --force && mix local.rebar --force
  - export MIX_ENV=test
  - mix do deps.get, deps.compile, compile, phx.digest, ecto.create, ecto.migrate, test

There’s a problem: my tests won’t pass without a working database, and we don’t have one yet. In Drone, the way to get a supporting container for something like a database or a cache is via a service. Services are also just a Docker container which runs with a certain configuration. Drone will run them before your steps. Other parts of your pipeline can communicate with services over a network (services are given a hostname that matches their service name) or via a shared volume.

Here’s what the same pipeline looks like with a mariadb service:

---
kind: pipeline
name: default

steps:
- name: backend
  image: elixir:1.8.0-alpine
  commands:
  - mix local.hex --force && mix local.rebar --force
  - export MIX_ENV=test
  - mix do deps.get, deps.compile, compile, phx.digest, ecto.create, ecto.migrate, test

services:
- name: cms-database
  image: mariadb
  ports:
  - 3306
  environment:
    MYSQL_DATABASE: "cms_test"
    MYSQL_USER: "REDACTED"
    MYSQL_PASSWORD: "REDACTED"
    MYSQL_RANDOM_ROOT_PASSWORD: "yes"

I also configured my test suite to use a mariadb database at the host cms-database, port 3306, with the given username and password.

My tests pass!

Next, I need to add my deployment step. As I mentioned before, this app deploys as a docker image, via semantic-release and semantic-release-docker. In order to do that, I need to use a “docker in docker” image— which is just what it says, a Docker image containing the Docker daemon and Docker CLI.

It’s actually best to use two of these. Drone (and Docker) prefer if any long running services, such as the Docker daemon, run in their own containers. In my experience, trying to run the Docker daemon in the background of a container that is also doing other commands overcomplicates things. Running the Docker daemon in an isolated container works better, because Docker has provided out of the box initialization scripts for that scenario.

Here’s the pipeline with both “dind” parts added:

---
kind: pipeline
name: default

steps:
- name: backend
  image: elixir:1.8.0-alpine
  commands:
  - mix local.hex --force && mix local.rebar --force
  - export MIX_ENV=test
  - mix do deps.get, deps.compile, compile, phx.digest, ecto.create, ecto.migrate, test
- name: deployment
  image: docker:dind
  volumes:
  - name: dockersock
    path: /var/run
  environment:
    GH_TOKEN:
      from_secret: gh-token
    DOCKER_USERNAME: ci
    DOCKER_PASSWORD:
      from_secret: docker-password
  commands:
  - apk add nodejs nodejs-npm openssl git
  - npm install -g npm
  - (cd assets; npm install; npm run deploy)
  - npm install
  - docker build --network=host . -t REDACTED.example.com/my_repo/my_app
  - npx semantic-release

services:
- name: cms-database
  image: mariadb
  ports:
  - 3306
  environment:
    MYSQL_DATABASE: "cms_test"
    MYSQL_USER: "REDACTED"
    MYSQL_PASSWORD: "REDACTED"
    MYSQL_RANDOM_ROOT_PASSWORD: "yes"
- name: docker
  image: docker:dind
  privileged: true
  network: host
  hostNetwork: true
  mtu: 1200
  volumes:
  - name: dockersock
    path: /var/run

volumes:
- name: dockersock
  temp: {}

That’s a lot of new pieces. What’s going on? Well, we’ve added a docker:dind step to the pipeline. This will run our Docker commands, for building and pushing the Docker image. We also need npm and NodeJS, because semantic-release is a Node package. We install npm and update it to the latest version for good measure. Then (cd assets; npm install; npm run deploy) builds my assets for production. npm install installs semantic-release and the various plugins I use (see my article from the Revelry blog). I build the Docker image. Then, I run semantic-release. Since I’m using semantic-release-docker, it will login to my Docker repo and push the image.

In order to actually build the image, we need a running Docker daemon. That’s where the second dind comes in. This one is a service. It runs in privileged mode (required for the dind daemon), which means my Drone project must be flagged as “trusted.” We use a shared volume to allow the Docker CLI running in my deployment step to communicate with the Docker daemon running as a service.

I need to provide some credentials for GitHub and for my Docker registry. I do that by exposing them as environment variables. Since I don’t want to check in a sensitive credential to git, I use Drone secrets for passwords and tokens. You can set the value of the secret in the project settings UI.

When you put it all together, Drone will do the following sequence:

  • Start a mariadb database and expose it on a network
  • Start a docker and expose it on a shared volume
  • Prepare and run my test suite in an Elixir docker image
  • If my tests pass, move on to a deployment phase:

    • Build my assets
    • Build a Docker image containing my latest code
    • Tag the image based on the proper semantic version
    • Push that tagged version and latest to my Docker repo
    • Also, tag that release in GitHub

Other Drone Features

I had a lightbulb moment while I was working with Drone. I had been wondering things like “I wonder which DBs Drone supports” or “I wonder which languages Drone supports.” Then I realized: Drone supports everything that has a Docker image. To me, this makes it a more flexible and powerful tool than a CI architected like Travis or Codeship. In those CIs, you either have to wait for official support for your language (or service), or you have to hack together a working test script from a base intended for a different language or DB. The way Drone does this means that Drone immediately has a “feature list” longer than I can write here.

Beyond this, there are other Drone features worth mentioning:

  • Drone supports Agents, which allow you to scale up your CI environment to handle more simultaneous builds. They are small daemons which you deploy to as many servers as you want. Each agent receives orders from the central Drone server which dispatches builds.
  • Drone supports multiple secrets plugins. In my example, I used secrets stored in Drone’s backing DB, but Drone also supports Vault and Kubernetes secrets, among others.
  • In addition to GitHub, Drone supports GitLab, Gitea, gogs, and Bitbucket.
  • drone exec is a cool feature on the Drone CLI which lets you run a build locally on your laptop, using the same configuration.

The only real issue that I hit along the way is that I could not get Drone to accept an encrypted secret from the .drone.yml file. Supposedly, Drone supports encrypting secrets via the CLI and including the encrypted version in your .drone.yml. I could not get that to work. I had to use the database secrets method instead.

The Future of Drone

After experimenting with Drone, I think we’re going to see it get a lot of traction. It’s such a powerful tool, and it so well leverages the Docker ecosystem, that I can’t help but think that it is the future of CI.

Categories:


This Weekend I Read… (2019-04-14)

Amazon Workers Have a Hellish Job

With her job at Amazon, she hoped she could work and pursue an education at the same time. For years, the 27-year-old English major had taken other short-term warehouse jobs—mostly for retail companies, including the shoe store Zumiez. 

More than two years later, injuries to her shoulder, neck, and wrist sustained during her time at Amazon—lifting up to 100 items an hour, moving them to conveyor belts, and then hauling them into trailers—have made it nearly impossible for her to type without the aid of voice dictation software.

Between 2015 and 2018, OSHA reported 41 “severe” injuries resulting in hospitalization, including six amputations and 15 fractures, associated with Amazon delivery or fulfillment jobs. 

Amazon workers are receiving severe, life-changing injuries on the job, and Amazon is covering it up using a system of in-house “clinics”, complicit company-mandated doctors, and missing OSHA filings.

Who doesn’t think Amazon workers suffering through this should have a union? Well, Amazon for one. The company is using old-school union-busting tactics to single out and remove pro-union workers.

Meritocracy is Still Fake

Belief in meritocracy is a core part of our modern ideology, particularly in the tech industry. Unfortunately, meritocracy is a false idea that does not exist. Worse, research shows that believing in meritocracy makes you more selfish, less self-critical, and more prone to acting in discriminatory ways.

Although widely held, the belief that merit rather than luck determines success or failure in the world is demonstrably false. This is not least because merit itself is, in large part, the result of luck. Talent and the capacity for determined effort, sometimes called “grit,” depend a great deal on one’s genetic endowments and upbringing.

This is to say nothing of the fortuitous circumstances that figure into every success story. In his book Success and Luck, the U.S. economist Robert Frank recounts the long-shots and coincidences that led to Bill Gates’s stellar rise as Microsoft’s founder, as well as to Frank’s own success as an academic. Luck intervenes by granting people merit, and again by furnishing circumstances in which merit can translate into success. This is not to deny the industry and talent of successful people. However, it does demonstrate that the link between merit and outcome is tenuous and indirect at best.

According to Frank, this is especially true where the success in question is great, and where the context in which it is achieved is competitive. There are certainly programmers nearly as skilful as Gates who nonetheless failed to become the richest person on Earth. In competitive contexts, many have merit, but few succeed. What separates the two is luck.

We Get to Decide What Comes Next

We’re living in interesting times. I think there’s pressure building up on one of those socio-political-historical fault lines. We might live to see humanity evolve into its next political and economic model. This can be daunting, but it should also be exciting. After all, if we play our cards right, we can determine what this new model will be.

While I don’t agree with the authors entirely, this article gives much food for thought on this idea. It is likely that what comes next won’t be any of the old models, and won’t be accurately predicted by any futurist.

Where their argument falters is that they seem to assume that which of these models will emerge is a function of which one best addresses the challenges of the modern era, climate change, and the pressures of automation. That’s not how economies change. Economies change as a function of who holds economic power, those people’s interests, and how people generally relate to economic activity. Without radical economic democracy which places power in many hands, whatever comes next will only serve the few who currently hold power.

Though I do have to give a special shout-out for introducing me to the term “doughnut economics.”

Categories:


This Weekend I Read… (2019-03-25)

Here are some interesting (and sometimes scary) things I read this weekend:

Facebook Content Reviewers Have A Hellish Job

Content reviewers at Facebook are constantly subjected to horrifying videos of violence, racism, and conspiracy theories, and they aren’t being given the support they need. Some might wave this off as “the nature of the job,” but I think like anyone in a hazardous job they should be given the right support structure, safety equipment, and hazard pay. Instead, the moderators (who are outside contractors) are paid far less than the average Facebook employee, and are subjected to a high pressure environment where they watch 2,400 traumatizing videos in an 8 hour shift (4 per minute) and managers time their bathroom breaks.

The Real Reason for the 40 Hour Work Week

David Cain, writing at his blog Raptitude (“Getting better at being human”), puts forward an interesting theory about why we still have the 40 hour work week even though the average office worker is productive only 3 hours a day and productivity has been steadily increasing in the years since the 40 hour work week was won. His theory: it isn’t about labor supply, but instead it is about the demands for goods and services. Tired, time-constrained workers want more creature comforts, and buy more convenience items (fast food). They also prefer hobbies which take less time and energy, but more money (e.g. TV, movies, fast fashion) over hobbies which are cheap or free, but time consuming (e.g. reading, gardening, DIY crafting). His anecdotal observations which are woven throughout— such as developing a habit for expensive takeaway coffee after getting a new high-stress job— jive with my experience as well.

The Life Changing Magic Manga of Tidying Up

Did you know that there is a graphic novel version of Marie Kondo’s The Life Changing Magic of Tidying Up? It’s a fast read (about 180 pages, mostly pictures) and it is overflowing with charm and wholesome energy. I think it could serve either as a good introduction to the KonMari technique or as a quick refresher.

Categories: