LiveLab: Netbox VLAN deployment to Cisco devices

Welcome to my latest blog post where I’ll be taking you through an exciting demonstration of integrating NetBox with custom middleware to automate VLAN deployments on Cisco devices. This exploration is designed to showcase the core functionalities of this powerful tool combination in a real-world scenario.

As a network management solution, NetBox excels in documenting and organizing network infrastructures. By coupling it with middleware, I enhance its capabilities to not only manage but also automatically deploy network configurations across various devices.

In this demo, the middleware acts as a vital link that connects to NetBox. It retrieves a list of devices along with their designated configurations and then launches tasks to roll out VLAN settings to Cisco devices. This automation reduces the need for manual intervention, diminishes the chances for errors, and streamlines the operations across the network.

Please note, this demonstration is designed to show the basic functionality of integrating NetBox with middleware for VLAN deployment. The setup I’m discussing isn’t ready for a production environment but serves as a conceptual model to illustrate how automation can be effectively applied in network management.

Whether you’re a seasoned network engineer or an IT professional curious about the potential of middleware in enhancing network management tools, this guide aims to provide a foundational understanding of how to build and leverage these technologies. Let’s dive into this integration and explore the steps involved in automating VLAN deployments.

At a glance
#

graph LR; A[Admin]-->B B[Netbox]-->C[Frontend] subgraph VLAN deploy middleware C-->D[(Queue)] E[Worker]-->D end E-->F[Devices]

The environment is fully operated within Docker. The middleware frontend is integrated as a hook in NetBox. This frontend generates a task in the job queue, and a worker picks up this task, loads the device list for the specified site, and initiates a subtask for each device to perform create, update, or delete actions on the switch.

Now let’s look at the individual components.

Netbox
#

Setup
#

Install latest Netbox like described on netbox-docker project.

git clone -b release https://github.com/netbox-community/netbox-docker.git
cd netbox-docker
tee docker-compose.override.yml <<EOF
services:
  netbox:
    ports:
      - 8000:8080
EOF
docker compose pull
docker compose up -d

Create admin user
#

To create the first admin user run this command:

docker compose exec netbox /opt/netbox/netbox/manage.py createsuperuser

Prepare the environment
#

General

Create a Site: Navigate to Organization > Sites and add a new site, such as “Eberfing”
Create a Manufacturer: Go to Devices > Manufacturers to create a new manufacturer entry, for example, “Cisco”
Create a Device Type: Access Devices > Device Type and add a device type like “Generic Switch”
Create a Device Role: Visit Devices > Device Roles to establish a new device role, such as “Access Switch”
Register a New Device: Under Devices > Devices, add a new device (e.g., “testswitch”) and specify details like site (“Eberfing”), device type (“generic_switch”), and role (“access-switch”)

Create a Virtual Interface: Select the newly created device, go to + Add Components, and add a new ‘Interface Management’ type marked as ‘Virtual’.
Assign an IP to the Virtual Interface: Click on +, select ‘IP Address’, and enter “10.1.1.1/24” (including the CIDR notation). Ensure to mark this IP as the primary address.

Token for backend

Generate token: Navigate to Admin > API Tokens and create new readonly token, the token must be added to .env file

Webhook config

Webhook target: Navigate to Operations > Webhooks create new, name the hook “VLAN Middleware API”, enter the URL to the API eg.“http://10.1.1.1:5004/vlan_webhook”, method “POST”, HTTP content type “application/json” and enter a Secret like “supersecret”. The secret must also be configured in the .env file.
Event rule: Navigate to Operations > Event Rules create new, name the event “VLAN HOOK”, select Object types “IPAM > VLAN”, select the events “Creations”, “Updates” and “Deletions”, select Action type “Webhook” finally select the created Webhook “VLAN Middleware API”

Middleware VLAN deployment
#

General
#

.env-file Modify the env file to your predefined token and secret, set the cisco ssh credentials.

# Worker queue
APP_REDIS_URL=redis://cache:6379
APP_QUEUE_NAME=vlan_deployment

# User credentials for SSH access to Cisco devices
APP_CISCO_USER=automation
APP_CISCO_PASS=supersecret

# Netbox Webhook Secret
APP_NETBOX_SECRET=supersecret

# Netbox System
APP_NETBOX_URL=http://10.1.1.1:8000
APP_NETBOX_TOKEN=f69161ae5476f7048e8b216a867a4d5b52ee2eb5

# MAC HACK
OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

If you are running the demo on MAC M series OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES is required.

docker-comopse.yaml

---
version: "3.8"

services:
  vlan_middleware_frontend:
    image: vlan_middleware_frontend:1.4
    build:
      dockerfile: Dockerfile.frontend
    ports:
      - "5004:5001"
    env_file:
      - ./.env
    depends_on:
      - cache
      
  vlan_middleware_backend:
    image: vlan_middleware_backend:1.4
    build:
      dockerfile: Dockerfile.backend
    env_file:
      - ./.env
    depends_on:
      - cache

  cache:
    image: redis:7.2.4
    restart: always
    ports:
      - '6379:6379'
    command: redis-server --save 20 1 --loglevel warning
    volumes:
      - cache:/data


volumes:
  private-volume:
  cache:

Dockerfile.backend

Docker container will be created if you launch docker compose.

FROM python:3.11-slim-buster
MAINTAINER Maximilian Thoma
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY ./start_worker.py .
COPY ./vlan_deploy ./vlan_deploy
CMD [ "python3", "start_worker.py"]

Dockerfile.frontend

Docker container will be created if you launch docker compose.

FROM python:3.11-slim-buster
MAINTAINER Maximilian Thoma
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY ./start_api.py .
COPY ./vlan_deploy ./vlan_deploy
EXPOSE 5001
CMD ["gunicorn", "--bind", "0.0.0.0:5001", "start_api:app"]

Frontend
#

Code is only showed parcially, for full code see github repo below.

vlan_deploy/app.py

from apiflask import APIFlask, abort
from flask import request
from werkzeug.middleware.proxy_fix import ProxyFix
from rq import Queue
from rq.job import Job
from .worker import connection
from .auth import check_netbox_auth
from .tasks import task_enqueue_vlan_create, task_enqueue_vlan_delete
from dynaconf import FlaskDynaconf

# Initialize APIFlask
app = APIFlask(__name__, "Middleware VLAN Deployment", version="1.0", docs_path="/")
app.config['DESCRIPTION'] = "LiveLAB: VLAN deployment on Cisco devices"

FlaskDynaconf(app, env_var_prefix="APP")

# Proxy fix if used behind reverse proxy
app.wsgi_app = ProxyFix(
    app.wsgi_app, x_for=1, x_proto=1, x_host=1, x_prefix=1
)

# Load config from env variables which prefixed with APP_
app.config.from_prefixed_env(prefix="APP")

q = Queue(connection=connection, name=app.config.QUEUE_NAME)

@app.get("/hello")
def index():
    return {"message": "Hello World", "queue": app.config.QUEUE_NAME}, 200


@app.post("/vlan_webhook")
@check_netbox_auth
def vlan_webhook():
    json = request.json

    if "updated" in json['event'] or "created" in json['event']:
        try:
            vlan_id = json['data']['vid']
            vlan_name = json['data']['name']
            vlan_site = json['data']['site']['slug']

            job = Job.create(
                func=task_enqueue_vlan_create,
                args=(vlan_id, vlan_name, vlan_site), connection=connection
            )

            q.enqueue_job(job)
        except Exception as e:
            print(e)
            return abort(500, message="Failed to create job")

    if "deleted" in json['event']:
        try:
            vlan_id = json['data']['vid']
            vlan_site = json['data']['site']['slug']

            job = Job.create(
                func=task_enqueue_vlan_delete,
                args=(vlan_id, vlan_site), connection=connection
            )

            q.enqueue_job(job)
        except Exception as e:
            print(e)
            return abort(500, message="Failed to create job")

    return {"state": "ok"}

vlan_deploy/tasks.py

You can flag in the demo a device with no_vlan_deployment, in this case the device will be skipped.

import os
from rq import Queue
from rq.job import Job
from .worker import connection
from .backend import get_devices_by_site, create_or_update_vlan_on_device, delete_vlan_on_device

q = Queue(connection=connection, name=os.getenv('APP_QUEUE_NAME'))

def check_for_vlan_deployment_tag(data):
    """
    Check for VLAN Deployment Tag
    """
    data = dict(data)
    if 'tags' in data and isinstance(data['tags'], list):
        for tag in data['tags']:
            if tag.get('slug') == 'no_vlan_deployment':
                return True
    return False

def task_enqueue_vlan_create(vlan_id, vlan_name, vlan_site):
    """
    This method enqueues a job to create a VLAN on all devices in a given site.
    """
    devices = get_devices_by_site(vlan_site)

    for device in devices:
        if device.primary_ip:
            if check_for_vlan_deployment_tag(device) is False:
                ip = str(device.primary_ip).split("/")[0]

                # enqueue job for device
                job = Job.create(
                    func=task_device_vlan_create,
                    args=(ip, vlan_id, vlan_name),
                    connection=connection
                )

                q.enqueue_job(job)


def task_enqueue_vlan_delete(vlan_id, vlan_site):
    """
    Enqueues a job to delete a VLAN on all devices in the specified site.
    """
    devices = get_devices_by_site(vlan_site)

    for device in devices:
        if device.primary_ip:
            if check_for_vlan_deployment_tag(device) is False:
                ip = str(device.primary_ip).split("/")[0]

                # enqueue job for device
                job = Job.create(
                    func=task_device_vlan_delete,
                    args=(ip, vlan_id),
                    connection=connection
                )

                q.enqueue_job(job)


def task_device_vlan_create(ip, vlan_id, vlan_name):
    """
    Create a VLAN on a device.
    """
    create_or_update_vlan_on_device(ip, vlan_id, vlan_name)

def task_device_vlan_delete(ip, vlan_id):
    """
    This method is used to delete a VLAN on a specific device. 
    """
    delete_vlan_on_device(ip, vlan_id)

Backend (Worker)
#

The workers care about to handle the tasks in the job queue.

start_worker.py

The start_worker.py will be executed in the worker docker container.

from rq import Worker, Queue, Connection
from vlan_deploy.worker import connection, listen


if __name__ == '__main__':
    with Connection(connection):
        worker = Worker(list(map(Queue, listen)))
        worker.work()

vlan_deploy/worker.py

import os
import redis
from rq import Worker, Queue, Connection

# Queues to listen to
listen = [os.getenv('APP_QUEUE_NAME', 'default')]

# URL to redis server
redis_url = os.getenv('APP_REDIS_URL', 'redis://localhost:6379')

# create redis connection
connection = redis.from_url(redis_url)

if __name__ == '__main__':
    # start the worker if directly called
    with Connection(connection):
        worker = Worker(list(map(Queue, listen)))
        worker.work()

vlan_deploy/backend.py

In the backend.py is the implementation to create/update/delete VLANs on the Cisco devices. The delete function has a static list for protected_vlans like the management vlan.

import os
import pynetbox
from netmiko import ConnectHandler


def get_devices_by_site(site):
    netbox = pynetbox.api(os.getenv('APP_NETBOX_URL'), token=os.getenv('APP_NETBOX_TOKEN'))
    devices = netbox.dcim.devices.filter(site=site, manufacturer='cisco')
    return devices

def create_or_update_vlan_on_device(ip, vlan_id, vlan_name):
    device = {
        'device_type': 'cisco_ios',
        'ip': ip,
        'username': os.getenv('APP_CISCO_USER'),
        'password': os.getenv('APP_CISCO_PASS'),
    }

    with ConnectHandler(**device) as net_connect:
        config_commands = [
            f'vlan {vlan_id}',
            f'name {vlan_name}',
            'exit',
            'wr mem'
        ]
        net_connect.send_config_set(config_commands)
        print(f"VLAN {vlan_id} ({vlan_name}) created/updated successfully on device {ip}")


def delete_vlan_on_device(ip, vlan_id):
    protected_vlans = [1, 999]

    if vlan_id in protected_vlans:
        print(f"VLAN {vlan_id} is protected on device {ip}")
        return

    if vlan_id not in protected_vlans:

        device = {
            'device_type': 'cisco_ios',
            'ip': ip,
            'username': os.getenv('APP_CISCO_USER'),
            'password': os.getenv('APP_CISCO_PASS'),
        }

        with ConnectHandler(**device) as net_connect:
            config_commands = [
                f'no vlan {vlan_id}',
                'exit',
                'wr mem'
            ]
            net_connect.send_config_set(config_commands)
            print(f"VLAN {vlan_id} deleted successfully on device {ip}")

Demo run …
#

Launch the docker environment
#

docker-compose up

The docker containers will be build during launch of the environment, at the end you should see that all containers are operational.

 ✔ Container middleware-cache-1                     Created                                                                                                                                                                                   0.0s 
 ✔ Container middleware-vlan_middleware_frontend-1  Recreated                                                                                                                                                                                 0.1s 
 ✔ Container middleware-vlan_middleware_backend-1   Recreated                                                                                                                                                                                 0.1s 
Attaching to middleware-cache-1, middleware-vlan_middleware_backend-1, middleware-vlan_middleware_frontend-1
middleware-vlan_middleware_frontend-1  | [2024-06-13 17:13:49 +0000] [1] [INFO] Starting gunicorn 22.0.0
middleware-vlan_middleware_frontend-1  | [2024-06-13 17:13:49 +0000] [1] [INFO] Listening at: http://0.0.0.0:5001 (1)
middleware-vlan_middleware_frontend-1  | [2024-06-13 17:13:49 +0000] [1] [INFO] Using worker: sync
middleware-vlan_middleware_frontend-1  | [2024-06-13 17:13:49 +0000] [7] [INFO] Booting worker with pid: 7
middleware-vlan_middleware_backend-1   | 17:13:54 Worker rq:worker:603209bd9b06453da05ff7cc0a88d639 started with PID 1, version 1.16.2
middleware-vlan_middleware_backend-1   | 17:13:54 Subscribing to channel rq:pubsub:603209bd9b06453da05ff7cc0a88d639
middleware-vlan_middleware_backend-1   | 17:13:54 *** Listening on vlan_deployment...
middleware-vlan_middleware_backend-1   | 17:13:54 Cleaning registries for queue: vlan_deployment

Create/Update VLAN
#

The create and update task works on the same way.

Create a VLAN in Netbox, select Site “Eberfing”

on container log:

middleware-vlan_middleware_backend-1   | 17:17:56 vlan_deployment: vlan_deploy.tasks.task_enqueue_vlan_create(11, 'VLAN0011', 'eberfing') (b40d139f-2d06-4115-aa7b-b58e96e8fc2f)
middleware-vlan_middleware_backend-1   | 17:17:57 vlan_deployment: Job OK (b40d139f-2d06-4115-aa7b-b58e96e8fc2f)
middleware-vlan_middleware_backend-1   | 17:17:57 Result is kept for 500 seconds
middleware-vlan_middleware_backend-1   | 17:17:57 vlan_deployment: vlan_deploy.tasks.task_device_vlan_create('10.1.1.4', 11, 'VLAN0011') (c77eaddb-a832-458f-bd8b-1c0b37857c45)
middleware-vlan_middleware_backend-1   | VLAN 11 (VLAN0011) created/updated successfully on device 10.1.1.4
middleware-vlan_middleware_backend-1   | 17:17:59 vlan_deployment: Job OK (c77eaddb-a832-458f-bd8b-1c0b37857c45)
middleware-vlan_middleware_backend-1   | 17:17:59 Result is kept for 500 seconds

on Cisco device:

mtlab01#show vlan

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Gi0/1, Gi0/2, Gi0/3, Gi0/4, Gi0/5, Gi0/6, Gi0/7, Gi0/8, Gi0/9, Gi0/10
10   VLAN0010                         active
11   VLAN0011                         active
999  Management                       active
1002 fddi-default                     act/unsup
1003 token-ring-default               act/unsup
1004 fddinet-default                  act/unsup
1005 trnet-default                    act/unsup

Delete VLAN
#

Select a VLAN to delete eg. VLAN 10

on container log:

middleware-vlan_middleware_backend-1   | 17:35:30 vlan_deployment: vlan_deploy.tasks.task_enqueue_vlan_delete(10, 'eberfing') (e2c69c9f-97f1-456f-b1d5-a93c619d4a39)
middleware-vlan_middleware_backend-1   | 17:35:30 vlan_deployment: Job OK (e2c69c9f-97f1-456f-b1d5-a93c619d4a39)
middleware-vlan_middleware_backend-1   | 17:35:30 Result is kept for 500 seconds
middleware-vlan_middleware_backend-1   | 17:35:30 Cleaning registries for queue: vlan_deployment
middleware-vlan_middleware_backend-1   | 17:35:30 vlan_deployment: vlan_deploy.tasks.task_device_vlan_delete('10.1.1.4', 10) (a02add47-0d05-49a0-b53f-3ff1811ff512)
middleware-vlan_middleware_backend-1   | VLAN 10 deleted successfully on device 10.1.1.4
middleware-vlan_middleware_backend-1   | 17:35:33 vlan_deployment: Job OK (a02add47-0d05-49a0-b53f-3ff1811ff512)
middleware-vlan_middleware_backend-1   | 17:35:33 Result is kept for 500 seconds

on Cisco device:

mtlab01#show vlan

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Gi0/1, Gi0/2, Gi0/3, Gi0/4, Gi0/5, Gi0/6, Gi0/7, Gi0/8, Gi0/9, Gi0/10
11   VLAN0011                         active
999  Management                       active
1002 fddi-default                     act/unsup
1003 token-ring-default               act/unsup
1004 fddinet-default                  act/unsup
1005 trnet-default                    act/unsup

Works :-)