In the ever-evolving landscape of network management and automation, the role of Network DevOps has become increasingly pivotal. Combining the principles of development and operations, Network DevOps aims to streamline and automate network infrastructure processes, enhancing efficiency, reliability, and scalability. Python, with its extensive array of powerful libraries, provides the perfect toolkit for Network DevOps practitioners.
In this article, I will delve into my favorite libraries, showcasing their advantages and providing simple examples to demonstrate why they are indispensable for Network DevOps professionals.
Requests#
Documentation: Requests documentation
The requests library is one of the most popular for making HTTP requests in Python. It simplifies communication
with web APIs through an intuitive and user-friendly interface.
Advantages:
- Easy to use
- Supports HTTP methods like GET, POST, PUT, DELETE, etc.
- Built-in support for SSL/TLS connections
- Handles cookies and sessions
Examples#
Simple GET request#
import requests
response = requests.get('https://api.m.lab/data')
print(response.json())
Requests has a built-in converter from json to python list/dictionaries.
Simple POST request#
import requests
data = {'foo': 'bar'}
response = requests.post('https://api.m.lab/data', json=data)
print(response.status_code)
APIFlask: Modern API development#
Documentation: APIFlask documentation
APIFlask is a lightweight library for creating RESTful APIs, building on Flask and offering enhanced features like authentication and autocreation of API documentation (openapi doc).
Advantages:
- Simple and quick setup
- Easy integration with Flask ecosystem
- Supports OpenAPI specifications for documentation
- Definition of schemas incl. data validation
Examples#
Basic API endpoint#
from apiflask import APIFlask, Schema
from apiflask.fields import String
app = APIFlask(__name__)
class ItemInSchema(Schema):
name = String(required=True)
class ItemOutSchema(Schema):
name = String()
@app.post('/items')
@app.input(ItemInSchema)
@app.output(ItemOutSchema)
def create_item(data):
return data
if __name__ == '__main__':
app.run()
Complex endpoint with multiple methods#
from apiflask import APIFlask
app = APIFlask(__name__)
items = {}
@app.get('/items/<int:item_id>')
def get_item(item_id):
return items.get(item_id, {"error": "Item not found"}), 200 if item_id in items else 404
@app.post('/items')
def add_item(data):
item_id = len(items) + 1
items[item_id] = data
return {"id": item_id}, 201
@app.delete('/items/<int:item_id>')
def delete_item(item_id):
if item_id in items:
del items[item_id]
return {}, 204
return {"error": "Item not found"}, 404
if __name__ == '__main__':
app.run()
Other examples#
Flask: Web Framework#
Documentation: Flask documentation / Jinja template system documentation
Flask is a micro web framework that provides the essentials for web development with flexibility and simplicity.
I use Flask for small web interfaces because its seamless integration of authentication, template engine, and plugins allows for rapid development and quick project initiation.
Advantages:
- Minimalist and easy to get started
- Extensible with numerous plugins and extensions
- Suitable for small to medium applications -> Ideal for microservices
- Extreme flexible template system (Jinja)
- Large community
Examples#
Basic Flask app#
from flask import Flask, jsonify
app = Flask(__name__)
@app.route('/api/status')
def status():
return jsonify({"status": "Running"})
if __name__ == '__main__':
app.run()
Template rendering#
Template:
<html>
<body>
<h1>{{ msg }}</h1>
</body>
</html>
App:
from flask import Flask, render_template
app = Flask(__name__)
@app.route('/')
def home():
msg = "Hello world"
return render_template('index.html', msg=msg)
if __name__ == '__main__':
app.run()
Other examples#
- FLASK with LDAP Authentication against Active Directory and group authorization for specific pages
- LANbugs - Flask articles (german)
Loguru#
Documentation: loguru documentation
Loguru is a modern logging library that aims to make logging in Python simple and powerful.
Advantages:
- Easy setup and configuration
- Supports asynchronous logging
- Provides useful logging features like automatic rotation and serialization
- Drop-in replacement for standard logging library
- Decorator for capture exceptions
Examples#
Basic logging#
from loguru import logger
logger.info("This is an info message")
Logging with rotation#
from loguru import logger
logger.add("file.log", rotation="500 MB")
logger.warning("This is is a warning message")
Capture exceptions#
from loguru import logger
@logger.catch
def main():
x = 0
y = 0
z = x / y
return z
if __name__ == "__main__":
main()
Loguru prints all variable values during runtime in the exception catch, which is a significant aid in debugging.
2024-06-12 21:16:27.924 | ERROR | __main__:<module>:13 - An error has been caught in function '<module>', process 'MainProcess' (74660), thread 'MainThread' (8582908928):
Traceback (most recent call last):
> File "/Users/netdevops/_DEV/log/test.py", line 13, in <module>
main()
└ <function main at 0x1022459e0>
File "/Users/netdevops/_DEV/log/test.py", line 8, in main
z = x / y
│ └ 0
└ 0
ZeroDivisionError: division by zero
CiscoConfParse: Parsing cisco configurations#
Documentation: ciscoconfparse documentation
ciscoconfparse helps to parse, query and modify Cisco IOS-syle configurations.
Advantages:
- Simplifies handling of large configuration files
- Allows complex queries on configuration data
- Facilitates automated changes to configurations
Examples#
Basic config parsing#
from ciscoconfparse import CiscoConfParse
config = """
interface GigabitEthernet0/1
description Link to core switch
ip address 10.0.0.1 255.255.255.0
!
interface GigabitEthernet0/2
description Link to access switch
ip address 10.0.1.1 255.255.255.0
"""
parse = CiscoConfParse(config.splitlines())
interfaces = parse.find_objects(r"^interface")
for intf in interfaces:
print(intf.text)
Find interfaces with IP addresses#
from ciscoconfparse import CiscoConfParse
config = """
interface GigabitEthernet0/1
description Link to core switch
ip address 10.0.0.1 255.255.255.0
!
interface GigabitEthernet0/2
description Link to access switch
ip address 10.0.1.1 255.255.255.0
"""
parse = CiscoConfParse(config.splitlines())
interfaces_with_ip = parse.find_objects_w_child(r"^interface", r"ip address")
for intf in interfaces_with_ip:
print(intf.text)
See also#
Dynaconf: Dynamic configuration management#
Documentation: dynaconf documentation
dynaconf is a configuration management library that allows dynamic and flexible configuration.
Advantages:
- Supports multiple file formats (JSON, YAML, TOML, etc.)
- Environment variable integration
- Layered configurations for development, testing, and production
- Direct import to flask / apiflask possible
Examples#
Basic config loading#
from dynaconf import Dynaconf
settings = Dynaconf(
settings_files=['settings.toml']
)
print(settings.DATABASE_URL)
Environment specific config#
# settings.toml
[production]
DEBUG = false
DATABASE_URL = "sqllite:///prod.sqlite"
[development]
DEBUG = true
DATABASE_URL = "sqllite:///dev.sqlite"
from dynaconf import Dynaconf
settings = Dynaconf(
settings_files=['settings.toml'],
environments=True,
default_env="development",
env="production"
)
print(settings.DATABASE_URL)
If you execute the file per default development is used.
# Fix
export ENV_FOR_DYNACONF=production
# per call
ENV_FOR_DYNACONF=production python script.py
Environment variables can be easy injected to docker containers :-)
PyMongo: Seamless MongoDB integration#
Documentation: pymongo documentation
PyMongo is the official MongoDB driver for Python, providing seamless integration with MongoDB databases.
Advantages:
- Easy to use and integrate with MongoDB
- Supports complex queries and aggregation
- Efficient handling of large datasets
Examples#
Look to other article which cares about pymongo Working with MongoDB and Python
RQ: Simple job queues#
Documentation: rq documentation
RQ (Redis Queue) is a simple library for creating background jobs, such as sending emails or processing data.
Advantages:
- Easy to set up and use
- Integrates with Redis
- Supports job scheduling and result tracking
Examples#
Basic job queue#
tasks.py - Contains all background tasks functions
def background_task(x, y):
return x + y
Start in same share worker node
rq worker --with-scheduler
I will publish a demo with docker workers and will link this here.
Execute test_rq.py to enqueue job
from redis import Redis
from rq import Queue
from tasks import background_task
import time
redis_conn = Redis()
q = Queue(connection=redis_conn)
job = q.enqueue(background_task, 2, 3)
time.sleep(2)
print(job.result) # Output: 5
Result:
22:01:09 Worker rq:worker:4f2bae663f824a0da5bdeec49b6734fb started with PID 76396, version 1.16.2
22:01:09 Subscribing to channel rq:pubsub:4f2bae663f824a0da5bdeec49b6734fb
22:01:09 *** Listening on default...
22:01:09 Cleaning registries for queue: default
22:01:15 default: tasks.background_task(2, 3) (cf74bf6c-3b3f-4953-9edb-7c9e21f9f013)
22:01:15 default: Job OK (cf74bf6c-3b3f-4953-9edb-7c9e21f9f013)
22:01:15 Result is kept for 500 seconds
Netmiko: Simplified network device communication#
Documentation: netmiko documentation
Netmiko is a library built on top of Paramiko, designed to simplify the communication with network devices.
Advantages:
- Supports a wide range of network devices
- Simplifies sending commands and parsing output
- Easy to use and well-documented
Examples#
Basic device connection and command execution#
from netmiko import ConnectHandler
device = {
'device_type': 'cisco_ios',
'host': 'hostname',
'username': 'user',
'password': 'passwd',
}
connection = ConnectHandler(**device)
output = connection.send_command('show ip int brief')
print(output)
connection.disconnect()
Paramiko: SSH2 protocol for network devices#
Documentation: paramiko documentation
Paramiko provides an easy way to connect to remote devices using the SSH2 protocol, making it ideal for network automation.
Advantages:
- Securely connect to remote devices
- Execute remote commands and transfer files
- Support for both client and server functionality
Examples#
Basic SSH connection and command execution#
import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('localhost', username='netdevops', password='password',look_for_keys=False, allow_agent=False)
stdin, stdout, stderr = ssh.exec_command('ls -l')
print(stdout.read().decode())
ssh.close()
SFTP file transfer#
import paramiko
transport = paramiko.Transport(('localhost', 22))
transport.connect(username='netdevops', password='password')
sftp = paramiko.SFTPClient.from_transport(transport)
sftp.put('./file.txt', '/Users/netdevops/file.txt')
sftp.get('/Users/netdevops/file.txt', './file2.txt')
sftp.close()
transport.close()
Ansible: Powerful automation framework#
Documentation: ansible documentation
Ansible is a powerful automation framework that uses simple, human-readable YAML files to manage and configure systems.
Advantages:
- Agentless architecture
- Extensive module library for various tasks
- Strong community support
Examples#
Look at the various articles:
Pandas: Powerful data manipulation#
Documentation: pandas documentation
Pandas is a powerful data manipulation library that provides data structures and functions needed to manipulate structured data seamlessly.
Advantages:
- Efficient handling of large datasets
- Easy-to-use data structures like DataFrame and Series
- Comprehensive functions for data manipulation and analysis
- Easy export to CSV, Excel etc. for reports
Examples#
Basic dataframe creation and export to excel#
Note: For export to excel you need additional library openpyxl.
import pandas as pd
data = {
'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35]
}
df = pd.DataFrame(data)
print(df)
df.to_excel("output.xlsx")
NTC Templates: TextFSM parsing#
ntc_templates is a powerful library that addresses this challenge by providing a collection of TextFSM templates for parsing the output of network device commands. Developed by the Network to Code (NTC) community, these templates convert raw command outputs into structured data, enabling easier data manipulation and automation.
Advantages:
- Converts diverse command outputs into a standardized structured format.
- Simplifies the parsing process with pre-defined templates
- Easily integrates with other network automation tools
- Regularly updated and maintained by the Network to Code community
- Own templates also possible
Examples#
Basic parsing with NTC templates#
import textfsm
from ntc_templates.parse import parse_output
# Example raw output from a network device command
raw_output = """
Interface IP-Address OK? Method Status Protocol
Ethernet0 192.168.0.1 YES manual up up
Ethernet1 192.168.0.2 YES manual up up
"""
# Parse the raw output using the appropriate TextFSM template
parsed_output = parse_output(platform="cisco_ios", command="show ip interface brief", data=raw_output)
# Print the structured output
print(parsed_output)
# Output: [{'Interface': 'Ethernet0', 'IP-Address': '192.168.0.1', 'OK?': 'YES', 'Method': 'manual', 'Status': 'up', 'Protocol': 'up'}, {'Interface': 'Ethernet1', 'IP-Address': '192.168.0.2', 'OK?': 'YES', 'Method': 'manual', 'Status': 'up', 'Protocol': 'up'}]
Other examples#
Look on my article Ansible: Parse Cisco commando outputs with TEXTFSM | German version