Scaling with Stability – Comprehensive FreeSWITCH Monitoring for a Custom VoIP Service

Audiora Systems is a business creating a customized VoIP service for specialized markets. They used FreeSWITCH’s strength and adaptability to build its platform. Ensuring high availability, call quality, and thorough performance monitoring was crucial as they got ready to scale their services. The thorough approach to FreeSWITCH monitoring taken by Audiora Systems is reviewed in this case study.

The Challenge: Ensuring Carrier-Grade Reliability and Performance at Scale

Low latency, high call completion rates, and superior audio quality were required for the service provided by Audiora Systems. Among their challenges were:

• Monitoring a distributed FreeSWITCH configuration that includes several media servers and SIP proxies.
• Proactively identifying and resolving problems that might impact thousands of calls at once.
• Monitoring key performance indicators (KPIs), such as call setup times, network effectiveness ratios (NER), and answer-seizure ratios (ASR).
• Integrating monitoring for performance regression testing in their CI/CD pipeline.
• Managing security monitoring for a VoIP system that is visible to the public.

The Investigation: A Multi-Tiered Monitoring Ecosystem

The engineering team at Audiora Systems created an advanced monitoring stack:

1. Core FreeSWITCH Metrics:
   • fs_cli Automation: Scripts were developed to periodically query fs_cli on each FreeSWITCH instance for critical metrics:
     • status: General health, uptime, sessions per second.
     • show calls count: Current active calls.
     • show channels count: Current active channels.
     • sofia status profile <profile_name> gateway <gw_name>: Status of all SIP gateways, including registration state, failures, and latency.
     • mem_summary: To track memory usage by FreeSWITCH.
   • Event Socket Layer (ESL): To receive events in real-time, a custom application subscribed to the FreeSWITCH ESL. For the purpose of calculating KPIs (ASR, NER, and call duration), they were able to:
     • Track call setup, progress, and teardown events.
     • Instantaneously identify specific error events or channel failures.
     • Enter information into a time-series database.
2. System & Network Level Monitoring:
   • Prometheus & Grafana: This turned into the basis of their alerting and metrics visualization. 
     • node_exporter: Deployed on all FreeSWITCH servers to collect system-level metrics (CPU, RAM, disk I/O, network stats).
     • freeswitch_exporter: used to import metrics unique to FreeSWITCH directly into Prometheus.
     • Custom Exporters: Created for metrics, like particular ESL event counts, that are not addressed by standard exporters.
   • SIP Proxy Monitoring (Kamailio): Using Kamailio‘s internal statistics and kamailio-prometheus-exporter, they tracked Kamailio’s performance (transaction counts, error rates, and memory usage) while they used it as a proxy and for SIP load balancing. 
   • Network Telemetry: Employed network monitoring tools (such as flow data analysis and SNMP for switches/routers) for keeping an eye on jitter, link saturation, and node latency throughout their network segments. 
   • sngrep and tcpdump: Support engineers use this tool to thoroughly troubleshoot specific call issues; it is frequently triggered by alerts from the Prometheus/Grafana stack.
3. Call Quality & Performance Testing:
   • SIPp: SIPp was integrated into their testing framework to replicate different call loads and scenarios (such as high CPS calls and registration storms) against staging environments and sporadically against production during periods of low traffic. This aided in locating performance regressions and bottlenecks.
   • Active Probing: deployed probes that periodically placed simulated calls across their whole platform to measure feature functionality, audio quality (where feasible using scoring algorithms similar to MOS), and call setup time.
4. Log Aggregation & Analysis:
   • ELK Stack (Elasticsearch, Logstash, Kibana): Centralized FreeSWITCH logs (as well as logs from other parts, such as Kamailio and their application servers). This made it possible to: 
     • Look for particular error messages or call identifiers in all logs.
     • To visualize log trends, such as error rates by FreeSWITCH instance, dashboards can be created in Kibana.
     • Configuring notifications according to log trends (such as an abrupt increase in SIP 401 Unauthorized responses).
5. Security Monitoring:
   • Fail2Ban & IP Tables: To block malicious IPs, Fail2Ban was aggressively used in addition to custom FreeSWITCH filters.
   • Intrusion Detection System (IDS): To monitor known VoIP attack signatures in network traffic, an IDS (such as Suricata) was deployed.
   • Regular Audits: FreeSWITCH configurations (dialplan, ACLs, and SIP profiles) are frequently tested for security best practices.

The Solution & Implementation:

Audiora Systems’s monitoring strategy was an ongoing evolution:

• Iterative Dashboarding: Grafana dashboards were continuously improved in response to new metrics that were determined to be significant and operational requirements.
• Alerting Policies: Prometheus’s Alertmanager was used to create comprehensive alerting policies with various severity levels and notification channels (PagerDuty for critical alerts, and Slack for low priority). By including links to pertinent dashboards or troubleshooting runbooks, alerts were made actionable.
• Capacity Planning: To ensure they could proactively scale resources, trend analysis, and capacity planning were conducted using historical data from Prometheus.
• Root Cause Analysis (RCA) Process: The abundance of monitoring data helped conduct comprehensive RCAs after incidents happened, which resulted in preventative actions. For example, ESL event data linked to a particular media server that had high packet loss on its network interface following one instance of deteriorated audio, necessitating a hardware replacement.

The Outcome & Lessons Learned: With great stability and performance, Audiora Systems successfully launched and expanded its VoIP service.

• Visibility at Scale: For large FreeSWITCH deployments, a thorough, multi-layered monitoring stack is a must.
• Automation is crucial: Managing complex systems requires automating log analysis, alerting, and metric collection.
• Real-time Event Processing: By utilizing the ESL, one can gain a profound understanding of FreeSWITCH’s behavior in real-time.
• Proactive Performance Testing: SIPp and other tools assist in locating limitations and regressions before they affect customers.
• Security as an Essential Component of Monitoring: To safeguard a public VoIP platform, ongoing security monitoring and response are essential.
• Invest in Centralized Logging: For operational intelligence and troubleshooting, an ELK stack (or something comparable) is essential.

Because of their dedication to careful monitoring, Audiora Systems was able to provide dependable service, promptly resolve problems, and expand their FreeSWITCH infrastructure with assurance.