Boosting NetBox Performance: A Guide to Scaling Workers with Docker and Native Installations

Table of Contents

NetBox is an essential tool for network documentation and IP address management, but as your infrastructure grows, so do the demands placed on it. By default, NetBox runs with a single worker to manage background tasks like executing scripts and processing webhooks. In smaller environments, this works just fine, but in larger setups, that one worker can quickly become overwhelmed. The solution? Spawn more workers to handle the load.

In this guide, I’ll walk you through how to scale NetBox workers using both Docker and native installations. Adding more workers will allow for parallel task processing, leading to faster job completion and improved performance.

Scaling Workers with Docker 🐳
#

If you’re running NetBox in Docker, you can easily scale the number of workers using Docker’s built-in orchestration features. By adjusting the replicas parameter, you can quickly increase the number of worker instances and let Docker handle task distribution.

What’s Happening with Docker Replicas?
#

Docker’s replicas feature is part of its service scaling functionality. By setting the number of replicas, you instruct Docker to create and manage multiple instances of the same service. This is especially useful in a containerized environment because Docker takes care of distributing the workload among all the running containers. In the case of NetBox, more worker containers mean faster task processing, as each worker can independently handle a task.

How to Add More Workers in Docker
#

Edit docker-compose.override.yml:
To scale your workers in Docker, you need to modify your docker-compose.override.yml file. This file lets you override the default settings in your docker-compose.yml. Here’s a sample configuration to spawn five worker instances:
```
services:
  netbox:
    ports:
      - 8000:8080
  netbox-worker:
    deploy:
      replicas: 5
```
This setup tells Docker to launch five replicas of the netbox-worker service. Each replica runs as a separate worker, allowing NetBox to process tasks in parallel.
Deploy the changes:
After modifying the configuration, apply the changes with the following command:
```
docker-compose up -d
```
This will start five worker instances, distributing background tasks across them to improve system responsiveness.
Verify the workers:
To confirm that the workers are running, go to the NetBox admin interface and navigate to Admin > System > Background Tasks. You should see multiple workers listed, indicating that they’re ready to handle concurrent tasks.

Scaling Workers on Native Installations 💻
#

For those running NetBox via a native installation—whether on physical hardware, a virtual machine, or a cloud instance—you can scale workers using systemd templates. This method lets you manage multiple instances of a service without duplicating configuration files.

Understanding systemd Templates
#

Systemd templates offer a flexible way to manage multiple instances of a service without the need for duplicate service files. By using a templated service file (like netbox-rq@.service), you can instantiate multiple services with a single configuration file, assigning each a unique identifier. This makes scaling services like NetBox workers simple and efficient.

How to Add More Workers on a Native Installation
#

Create a systemd template:
Start by copying the existing netbox-rq.service file to create a templated version that can handle multiple worker instances:
```
cp /etc/systemd/system/netbox-rq.service /etc/systemd/system/netbox-rq@.service
```
The @ symbol in the service name allows systemd to treat this file as a template, so you can instantiate multiple workers (e.g., netbox-rq@1.service, netbox-rq@2.service, etc.).
Reload systemd:
After creating the template, reload systemd to recognize the new configuration:
```
systemctl daemon-reload
```
Stop the old single worker:
If you’re moving from a single worker setup to multiple workers, you’ll need to stop and disable the existing worker:
```
systemctl stop netbox-rq
systemctl disable netbox-rq
```
Start the new workers:
To start five workers, use the following command to enable and start the worker instances:
```
systemctl enable --now netbox-rq@{1..5}.service
```
This command will spawn five worker instances, each identified by a unique number (1 to 5), allowing them to run concurrently and process tasks in parallel.
Verify the workers:
To confirm that the workers are running, go to the NetBox admin interface and navigate to Admin > System > Background Tasks. You should see multiple workers listed, indicating that they’re ready to handle concurrent tasks.
Restart workers after an upgrade:
When upgrading NetBox, remember to restart all worker instances to ensure they’re running the latest version. Here’s how you can restart all five workers:
```
systemctl restart netbox-rq@{1..5}.service
```

Conclusion
#

In larger NetBox environments, a single worker may not be enough to handle all the background tasks efficiently. Scaling workers allows NetBox to process tasks in parallel, significantly reducing the time it takes to complete jobs like executing scripts or processing webhooks.

Whether you’re using Docker or a native installation, scaling NetBox workers is a relatively straightforward process. Docker’s replicas feature allows for quick scaling with minimal configuration, while systemd templates on a native installation give you flexibility and control over how many workers you run.

By following these steps, you’ll ensure that your NetBox instance remains responsive and capable of handling the demands of your growing infrastructure. Happy scaling! 😊

Logging in Docker & Python: Guide to JSON logging with Graylog

1 July 2024·1443 words·7 mins