Building an Automated Unified Monitoring Stack: Integrating LibreNMS, SNMP, Tailscale, UniFi & UISP APIs

Executive Summary

Observability in a multi-site network requires more than simple ping checks. To maintain high availability, detect performance degradation, and monitor system health across multiple locations (Sopor Site and Mircea Eliade Site), we built a centralized, automated monitoring hub powered by LibreNMS.

This architecture unifies SNMP polling, Tailscale mesh network routing, UniFi Site Manager & UISP EdgeMAX APIs, Out-of-Band HPE iLO Processors, and HTTP Web Service health probes into a single glass dashboard.

1. System Architecture & Component Overview

The central monitoring stack is hosted on nas3 (192.168.1.50) at the Sopor Site using containerized Docker services.

2. Component Integration Deep-Dive

2.1 Centralized Docker Stack Deployment on nas3

LibreNMS is deployed on nas3 using Docker Compose. The environment consists of three co-located containers:

• librenms: Core web UI and poller engine.
• librenms_db: Optimized MariaDB database storage.
• librenms_redis: In-memory cache for distributed poller job queues.

# Docker service verification on nas3
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

Access URL: http://192.168.1.50:8000

2.2 SNMP Polling Infrastructure (net-snmp & HPE iLO)

SNMP polling provides telemetry (CPU, RAM, drive usage, temperature sensors, network interface throughput) for Linux hosts and out-of-band management cards.

A. Linux Server SNMP Configuration (nas1, nas2, nas3)

Each host runs net-snmp configured with a unified read-only community string (voina.org):

/etc/snmp/snmpd.conf:

com2sec readonly default voina.org
group MyROGroup v2c readonly
view all included .1 80
access MyROGroup "" any noauth exact all none none
syslocation Mircea Eliade Site / Sopor Site
syscontact gvoina@voina.org

B. Out-of-Band HPE iLO Hardware Integration

HPE iLO management processors (nas1-ilo, nas2-ilo, nas3-ilo) monitor hardware health (power supplies, fan speeds, drive arrays, thermal sensors) independently of the OS.

We programmed the iLO firmware directly from local Linux command line using ipmitool:

# Set iLO SNMP read community string to voina.org via IPMI driver interface
sudo modprobe ipmi_devintf
sudo ipmitool lan set 2 snmp voina.org

iLO Inventory Table:

2.3 Tailscale Mesh Overlay Integration

By combining LibreNMS with Tailscale, we avoided opening SNMP UDP port 161 or SSH management ports to the public internet.

1. Cross-Site Routing: LibreNMS on nas3 (192.168.1.50) polls nas1-ilo (192.168.6.2) directly across the encrypted Tailscale WireGuard subnet router tunnel.
2. Dynamic IP Independence: Even if nas1 changes its public ISP IP, the Tailscale overlay maintains an unbroken connection.

2.4 API Integration: UniFi Site Manager & UISP EdgeMAX

To gain visibility into wireless access points, network switches, and edge routers, LibreNMS was connected to controller APIs:

1. UniFi Site Manager API Integration:
   • Integrates connected access points, client counts, and VLAN traffic statistics.
   • Discovers downstream switches automatically via LLDP/CDP topology mapping.
2. UISP (Ubiquiti Infrastructure) API Integration:
   • Tracks EdgeRouter-4 (EdgeRouter-4-Sopor / 192.168.100.2) links, packet loss, and CPU load.

2.5 HTTP Web Services Monitoring

LibreNMS Services Module is configured to execute active HTTP health probes against critical web endpoints:

1. nas1 WordPress Blog: Probes http://blog.voina.org to confirm HTTP 200 OK and response latency.
2. Home Assistant Portal: Monitors smart home control interface availability.
3. Orange Fiber Gateway (Sopor): Probes http://192.168.100.1 (Huawei ONT management portal) to ensure WAN internet gateway health.

3. Automated Operations: Weekly Kernel Update & Backup Protection

Monitoring works hand-in-hand with system automation. To ensure servers remain secure without causing service disruptions, an automated weekly kernel reboot check was deployed across nas1, nas2, and nas3.

The Challenge: Preventing Reboots During Active Backups

On nas1, an automatic reboot during a daily MySQL dump or LTO-5 tape backup could cause severe data corruption.

The Solution: Process-Aware Reboot Script (auto_reboot_if_needed.sh)

Every Sunday at 04:00 AM (0 4 * * 0), cron triggers the following script on nas1:

#!/bin/bash
# Weekly Kernel Security Update & Reboot Check (nas1)

# 1. Safety check: Ensure no backup job is running on nas1
if pgrep -f "backup" >/dev/null 2>&1 || pgrep -f "backupTape" >/dev/null 2>&1 || pgrep -f "tar" >/dev/null 2>&1 || pgrep -f "mysqldump" >/dev/null 2>&1 || pgrep -f "mkltfs" >/dev/null 2>&1 || [ -f /tmp/backup.lock ]; then
    logger -t AUTO_REBOOT "Security update check: Backup job is currently RUNNING on nas1! Aborting reboot to protect backup integrity."
    exit 0
fi

# 2. Check running kernel vs latest installed rpm kernel
RUNNING_KERNEL=$(uname -r)
LATEST_KERNEL=$(rpm -q kernel-core --queryformat '%{VERSION}-%{RELEASE}.%{ARCH}\n' 2>/dev/null | sort -V | tail -n1)

if [ -n "$LATEST_KERNEL" ] && [ "$RUNNING_KERNEL" != "$LATEST_KERNEL" ]; then
    logger -t AUTO_REBOOT "New kernel security update ($LATEST_KERNEL) detected on nas1. No backup is running. Rebooting..."
    sync; sync
    /sbin/reboot
else
    logger -t AUTO_REBOOT "nas1 running latest kernel $RUNNING_KERNEL. No reboot needed."
fi

4. Key Takeaways & Results

• Complete Single-Pane Observability: All servers, switches, out-of-band iLO processors, and web applications are visible in a single dashboard at http://192.168.1.50:8000.
• Zero Public Port Exposure: All cross-site telemetry passes securely through Tailscale WireGuard tunnels.
• Safe Automated Maintenance: Automated kernel updates keep systems patched while intelligent process locks guarantee zero backup corruption.