♻️ Make netbox-worker it's own container

One container should ideally have one responsibility [1]. Therefore I implemented the netbox-worker to start in it's own container. This is possible, because netbox and the worker communicate via redis anyway. They still use the same image underneath, just the "command" they execute while starting different. Or in other words: I see no reason to introduce supervisord, when we already have docker-compose which can take care of running multiple processes. Also, here's another benefit: Now it's possible to view the logs of the webhook worker independently of the other netbox logs (and vice-versa). Other changes in this commit: * I don't see a reason to put a password for Redis in the docker-compose setup, so I removed it. * Slightly changed the nginx config, so that the nginx startup command becomes simpler and any error should be visible in the docker log. * Some housekeeping in the `Dockerfile`. * Added some troubleshooting advice regarding webhooks to the README. I'd like to thank Brady (@bdlamprecht [2]) here who did the harder work of figuring out what's even required to have webhooks working. [3] [1] https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#decouple-applications [2] https://github.com/bdlamprecht [3] https://github.com/ninech/netbox-docker/pull/90
2018-08-13 14:04:09 -07:00 · 2018-08-13 14:04:09 -07:00 · 013f81b791
commit 013f81b791
parent b8885e4b79
7 changed files with 129 additions and 96 deletions
--- a/10
+++ b/10
@ -6,15 +6,14 @@ RUN apk add --no-cache \
      ca-certificates \
      cyrus-sasl-dev \
      graphviz \
-      ttf-ubuntu-font-family \
      jpeg-dev \
      libffi-dev \
      libxml2-dev \
      libxslt-dev \
      openldap-dev \
      postgresql-dev \
-      wget \
-      supervisor
+      ttf-ubuntu-font-family \
+      wget

 RUN pip install \
 # gunicorn is used for launching netbox
@ -48,15 +47,12 @@ COPY docker/docker-entrypoint.sh docker-entrypoint.sh
 COPY startup_scripts/ /opt/netbox/startup_scripts/
 COPY initializers/ /opt/netbox/initializers/
 COPY configuration/configuration.py /etc/netbox/config/configuration.py
-COPY configuration/supervisord.conf /etc/supervisord.conf

 WORKDIR /opt/netbox/netbox

 ENTRYPOINT [ "/opt/netbox/docker-entrypoint.sh" ]

-VOLUME ["/etc/netbox-nginx/"]
-
-CMD ["supervisord", "-c /etc/supervisord.conf"]
+CMD ["gunicorn", "-c /etc/netbox/config/gunicorn_config.py", "netbox.wsgi"]

 LABEL SRC_URL="$URL"

--- a/README.md
+++ b/README.md
@ -54,7 +54,8 @@ To ensure this, compare the output of `docker --version` and `docker-compose --v

 ## Configuration

-You can configure the app using environment variables. These are defined in `netbox.env`.
+You can configure the app using environment variables.
+These are defined in `netbox.env`.
 Read [Environment Variables in Compose][compose-env] to understand about the various possibilities to overwrite these variables.
 (The easiest solution being simply adjusting that file.)

@ -75,6 +76,7 @@ You should therefore adjust the configuration for production setups, at least th
 * `EMAIL_*`: Use your own mailserver.
 * `MAX_PAGE_SIZE`: Use the recommended default of 1000.
 * `SUPERUSER_*`: Only define those variables during the initial setup, and drop them once the DB is set up.
+* `REDIS_*`: Use a persistent redis.

 ### Running on Docker Swarm / Kubernetes / OpenShift

@ -95,6 +97,7 @@ If a secret is defined by an environment variable and in the respective file at
 * `SECRET_KEY`: `/run/secrets/secret_key`
 * `EMAIL_PASSWORD`: `/run/secrets/email_password`
 * `NAPALM_PASSWORD`: `/run/secrets/napalm_password`
+* `REDIS_PASSWORD`: `/run/secrets/redis_password`

 Please also consider [the advice about running NetBox in production](#production) above!

@ -257,26 +260,71 @@ This usually happens when the `ALLOWED_HOSTS` variable is not set correctly.

 ### How to upgrade

-> How do I update to a newer version?
+> How do I update to a newer version of netbox?

 It should be sufficient to pull the latest image from Docker Hub, stopping the container and starting it up again:

 ```bash
 docker-compose pull netbox
-docker-compose stop netbox
-docker-compose rm -f netbox
-docker-compose up -d netbox
+docker-compose stop netbox netbox-worker
+docker-compose rm -f netbox netbox-worker
+docker-compose up -d netbox netbox-worker
 ```

+### Webhooks don't work
+
+First make sure that the webhooks feature is enabled in your Netbox configuration and that a redis host is defined.
+Check `netbox.env` if the following variables are defined:
+
+```
+WEBHOOKS_ENABLED=true
+REDIS_HOST=redis
+```
+
+Then make sure that the `redis` container and at least one `netbox-worker` are running.
+
+```
+$ docker-compose ps
+
+Name                           Command               State                Ports             
+--------------------------------------------------------------------------------------------------------
+netbox-docker_netbox-worker_1   /opt/netbox/docker-entrypo ...   Up                                     
+netbox-docker_netbox_1          /opt/netbox/docker-entrypo ...   Up                                     
+netbox-docker_nginx_1           nginx -c /etc/netbox-nginx ...   Up      80/tcp, 0.0.0.0:32776->8080/tcp
+netbox-docker_postgres_1        docker-entrypoint.sh postgres    Up      5432/tcp
+netbox-docker_redis_1           docker-entrypoint.sh redis ...   Up      6379/tcp
+```
+
+If `redis` and the `netbox-worker` are not available, make sure you have updated your `docker-compose.yml` file!
+
+Everything's up and running? Then check the log of the `netbox-worker` and/or `redis`:
+
+```bash
+docker-compose logs -f netbox-worker
+docker-compose logs -f redis
+```
+
+Still no clue? You can connect to the `redis` container and have it report any command that is currently executed on the server:
+
+```bash
+docker-compose run --rm -T redis redis-cli -h redis monitor
+
+# Hit CTRL-C a few times to leave
+```
+
+If you don't see anything happening after you triggered a webhook, double-check the configuration of the `netbox` and the `netbox-worker` containers and also check the configuration of your webhook in the admin interface of Netbox.
+
 ### Breaking Changes

-From time to time it might become necessary to re-order the structure of the container.
-Things like the `docker-compose.yml` file or your Kubernets or OpenShift configurations have to be adjusted as a consequence.
+From time to time it might become necessary to re-engineer the structure of this setup.
+Things like the `docker-compose.yml` file or your Kubernetes or OpenShift configurations have to be adjusted as a consequence.
 Since April 2018 each image built from this repo contains a `NETBOX_DOCKER_PROJECT_VERSION` label.
 You can check the label of your local image by running `docker inspect ninech/netbox:v2.3.1 --format "{{json .ContainerConfig.Labels}}"`.
+Compare the version with the list below to check whether a breaking change was introduced with that version.

-The following is a list of breaking changes:
+The following is a list of breaking changes of the `netbox-docker` project:

+* 0.4.0: In order to use Netbox webhooks you need to add Redis and a netbox-worker to your docker-compose.yml.
 * 0.3.0: Field `filterable: <boolean` was replaced with field `filter_logic: loose/exact/disabled`. It will default to `CF_FILTER_LOOSE=loose` when not defined.
 * 0.2.0: Re-organized paths: `/etc/netbox -> /etc/netbox/config` and `/etc/reports -> /etc/netbox/reports`. Fixes [#54](https://github.com/ninech/netbox-docker/issues/54).
 * 0.1.0: Introduction of the `NETBOX_DOCKER_PROJECT_VERSION`. (Not a breaking change per se.)
@ -304,9 +352,15 @@ You can use the following ENV variables to customize the build:
           Default: https://github.com/${SRC_REPO}/netbox/archive/$BRANCH.tar.gz
 ```

+### Publishing Docker Images
+
+New Docker Images are built and published every 24h by using travis:
+
+[![Build Status](https://travis-ci.org/ninech/netbox-docker.svg?branch=master)][travis]
+
 ## Tests

-To run the test coming with NetBox, use the `docker-compose.yml` file as such:
+To run the tests coming with NetBox, use the `docker-compose.yml` file as such:

 ```
 $ docker-compose run netbox ./manage.py test
--- a/configuration/supervisord.conf
+++ b/configuration/supervisord.conf
@ -1,16 +0,0 @@
-[supervisord]
-nodaemon=true
-
-[supervisorctl]
-
-[program:netbox]
-command = gunicorn -c /etc/netbox/config/gunicorn_config.py netbox.wsgi
-directory = /opt/netbox/netbox/
-stdout_logfile=/dev/stdout
-stdout_logfile_maxbytes=0
-user = nobody
-
-[program:netbox-rqworker]
-command = python3 /opt/netbox/netbox/manage.py rqworker
-directory = /opt/netbox/netbox/
-user = nobody
--- a/docker-compose.yml
+++ b/docker-compose.yml
@ -1,64 +1,63 @@
 version: '3'
 services:
-    netbox:
-        build:
-          context: .
-          args:
-            - BRANCH=${VERSION-master}
-        image: ninech/netbox:${VERSION-latest}
-        depends_on:
-        - postgres
-        - redis
-        env_file: netbox.env
-        volumes:
-        - ./startup_scripts:/opt/netbox/startup_scripts:ro
-        - ./initializers:/opt/netbox/initializers:ro
-        - ./configuration:/etc/netbox/config:ro
-        - netbox-nginx-config:/etc/netbox-nginx/
-        - netbox-static-files:/opt/netbox/netbox/static
-        - netbox-media-files:/opt/netbox/netbox/media
-        - netbox-report-files:/etc/netbox/reports:ro
-    nginx:
-        image: nginx:1.13-alpine
-        command: nginx -g 'daemon off;' -c /etc/netbox-nginx/nginx.conf
-        depends_on:
-        - netbox
-        ports:
-        - 8080
-        volumes:
-        - netbox-static-files:/opt/netbox/netbox/static:ro
-        - netbox-nginx-config:/etc/netbox-nginx/:ro
-    postgres:
-        image: postgres:10.4-alpine
-        env_file: postgres.env
-        volumes:
-        - netbox-postgres-data:/var/lib/postgresql/data
-    redis:
-        image: redis:4-alpine
-        environment:
-          REDIS_PASS_FILE: /run/secrets/redis-pass
-        command: [
-          "sh", "-c",
-          '
-          docker-entrypoint.sh
-          --appendonly yes
-          --requirepass "$$(cat $$REDIS_PASS_FILE)"
-          '
-        ]
-        volumes:
-        - ./redis-pass:/run/secrets/redis-pass
-        - netbox-redis-data:/data
-
+  netbox: &netbox
+    build:
+      context: .
+      args:
+        - BRANCH=${VERSION-master}
+    image: ninech/netbox:${VERSION-latest}
+    depends_on:
+    - postgres
+    - redis
+    - netbox-worker
+    env_file: netbox.env
+    volumes:
+    - ./startup_scripts:/opt/netbox/startup_scripts:ro
+    - ./initializers:/opt/netbox/initializers:ro
+    - ./configuration:/etc/netbox/config:ro
+    - netbox-nginx-config:/etc/netbox-nginx/
+    - netbox-static-files:/opt/netbox/netbox/static
+    - netbox-media-files:/opt/netbox/netbox/media
+    - netbox-report-files:/etc/netbox/reports:ro
+  netbox-worker:
+    <<: *netbox
+    depends_on:
+    - redis
+    entrypoint:
+    - python3
+    - /opt/netbox/netbox/manage.py
+    command:
+    - rqworker
+  nginx:
+    image: nginx:1.13-alpine
+    command: nginx -c /etc/netbox-nginx/nginx.conf
+    depends_on:
+    - netbox
+    ports:
+    - 8080
+    volumes:
+    - netbox-static-files:/opt/netbox/netbox/static:ro
+    - netbox-nginx-config:/etc/netbox-nginx/:ro
+  postgres:
+    image: postgres:10.4-alpine
+    env_file: postgres.env
+    volumes:
+    - netbox-postgres-data:/var/lib/postgresql/data
+  redis:
+    image: redis:4-alpine
+    command: redis-server --appendonly yes
+    volumes:
+    - netbox-redis-data:/data
 volumes:
-    netbox-static-files:
-        driver: local
-    netbox-nginx-config:
-        driver: local
-    netbox-media-files:
-        driver: local
-    netbox-report-files:
-        driver: local
-    netbox-postgres-data:
-        driver: local
-    netbox-redis-data:
-        driver: local
+  netbox-static-files:
+    driver: local
+  netbox-nginx-config:
+    driver: local
+  netbox-media-files:
+    driver: local
+  netbox-report-files:
+    driver: local
+  netbox-postgres-data:
+    driver: local
+  netbox-redis-data:
+    driver: local
--- a/docker/nginx.conf
+++ b/docker/nginx.conf
@ -1,5 +1,8 @@
+daemon off;
 worker_processes 1;

+error_log /dev/stderr info;
+
 events {
    worker_connections 1024;
 }
@ -16,7 +19,6 @@ http {

    server {
        listen      8080;
-        server_name localhost;
        access_log  off;

        location /static/ {
--- a/netbox.env
+++ b/netbox.env
@ -14,7 +14,6 @@ NAPALM_PASSWORD=
 NAPALM_TIMEOUT=10
 MAX_PAGE_SIZE=0
 REDIS_HOST=redis
-REDIS_PASSWORD=J5brHrAXFLQSif0K
 SECRET_KEY=r8OwDznj!!dci#P9ghmRfdu1Ysxm0AiPeDCQhKE+N_rClfWNj
 SUPERUSER_NAME=admin
 SUPERUSER_EMAIL=admin@example.com
--- a/1
+++ b/1
@ -1 +0,0 @@
-J5brHrAXFLQSif0K