Achieving Clarity – Proactive FusionPBX Monitoring for a Growing SMB

“Callspire Solutions,” is a small-to-medium-sized business. It offers consultancy services and implements FusionPBX for its flexibility and cost-effectiveness. They began to encounter irregular call quality as their team and call volume increased, necessitating a strong monitoring plan to preserve efficiency and professionalism. This case study discusses the deployment of proactive monitoring for Callspire’s FusionPBX configuration.

The Challenge: Scaling Pains and Elusive Quality Issues Callspire’s initial 

The deployment of FusionPBX, which was operating on a small virtual server, went perfectly. But as they expanded the number of extensions and concurrent calls, they encountered:

• Reports of choppy audio and one-way voice.
• Occasionally, softphone errors occur, making it unable to make or receive calls.
• Challenging to determine if problems were carrier-specific, network-dependent, or system-related. 
• The requirement for a method to track system resources and the functionality of FreeSWITCH, FusionPBX’s main engine.

The Investigation: Combining System-Level, FreeSWITCH, and Network Monitoring

Callspire’s IT administrator, Sarah, took an extensive approach:

1. System Resource Monitoring:

• Standard Linux Tools: Sarah used tools like vmstat (virtual memory statistics), iostat (CPU and I/O statistics), and htop (for real-time process and resource viewing) to develop basic server monitoring. She created basic cron jobs to record important metrics in order to identify trends.
• Disk Space: Disk space should be checked frequently because full disks can interfere with FreeSWITCH and log rotation.

1. FreeSWITCH & FusionPBX Layer:
   • fs_cli (FreeSWITCH Command Line Interface): This became Sarah’s primary tool for real-time diagnostics.
     • sofia status profile <profile_name>: To check SIP profile status, registration counts, and gateway states.
     • show calls: To view active calls and their states.
     • show channels: To inspect active channels and their details.
     • fsctl loglevel <level>: To minimize disk space, log verbosity should be dynamically increased while troubleshooting and then reduced afterward.
   • FusionPBX Logs: Accessed via Advanced > Log Viewer in the FusionPBX GUI, and directly on the server (/var/log/freeswitch/freeswitch.log). Sarah looked for error messages, SIP error codes (e.g., 4xx, 5xx responses), and unusual activity.
   • sngrep: This SIP message visualizer that ran on a terminal was really helpful. Sarah was able to record and examine SIP dialogue in real time by using sngrep on the FusionPBX server. This allowed her to see problems such as incorrect SDP negotiation, NAT issues, or carrier errors.
   • Call Detail Records (CDRs): Through its GUI (Apps > Call Detail Records), FusionPBX offers CDRs. These were used by Sarah to monitor call success rates, identify numbers that regularly failed, and link particular call records to user complaints.
2. Network and Quality of Service (QoS) Monitoring:
   • Packet Capture (tcpdump): For particularly tricky audio issues, Sarah used tcpdump to capture RTP streams (e.g., tcpdump -i eth0 -w capture.pcap udp portrange 16384-32768) and then analyzed them in Wireshark on her local machine, looking for jitter, packet loss, and out-of-order packets.
   • QoS Implementation: Verified that their network router was configured with QoS rules to prioritize VoIP traffic (SIP and RTP) over less time-sensitive data.
   • Bandwidth Monitoring: used the reporting features on their firewall to make sure there was enough bandwidth set off specifically for VoIP.
3. Security Monitoring (Basic):
   • FusionPBX Operator Panel: Sarah made sure the operator panel was only accessible from reliable internal IPs and kept an eye on its logs for any unusual access attempts because she was aware of previous vulnerabilities (such as the Aon RCE disclosure).
   • Fail2Ban: Fail2Ban was put into place to prevent brute-force attacks by automatically blocking offending IP addresses and keeping an eye on FreeSWITCH logs for failed registration attempts.

The Solution & Implementation:

The investigation revealed several key areas for improvement:

• Resource Bottleneck: During peak hours, htop displayed a significant CPU load. The original VM was too small for their expanding needs. RAM and CPU cores in the VM were increased.
• Network Jitter: A significant jitter was detected during calls to a specific remote office via sngrep and tcpdump analysis. This was linked to packet reordering caused by an unmanaged switch at the remote location. The switch was replaced.
• Carrier Saturation: Late afternoon calls across one of their SIP trunks showed a greater failure rate, according to CDR data. The provider admitted local capacity problems and rerouted their traffic after being contacted with this data.
• Log Level Management: To avoid unnecessary disk use, Sarah set up a strategy to run FreeSWITCH at the default log level, raising it only during active troubleshooting.

The Outcome & Lessons Learned: By implementing these monitoring strategies and solutions, Callspire significantly improved call quality and system reliability.

• Proactive is Better than Reactive: Continuous monitoring helped catch issues before they became widespread.
• Layered Approach: Monitoring needs to cover the host system, the application (FusionPBX/FreeSWITCH), and the network.
• Leverage Open Source Tools: Tools like fs_cli, sngrep, tcpdump, and basic Linux utilities provide powerful monitoring capabilities without extra cost.
• Data-Driven Decisions: Having logs and metrics (CDR, sngrep traces) was crucial when talking to service providers.
• Security is Part of Monitoring: Keeping an eye on access logs and implementing tools like Fail2Ban is essential.

Callspire has a clear process for troubleshooting VoIP issues and currently plans frequent evaluations of monitoring dashboards. They eventually set up a basic Grafana instance using node_exporter for server metrics and investigated freeswitch_exporter for deeper insights.