Like any other technology, mission-critical communication networks experience issues and failures. In these scenarios, Motorola Solutions Technical Support Technologists are standing by to help. I am part of this group. When customers call in, my goal is to serve as their liaison and partner in getting their communications network restored and operational.
While network outages are worrisome, most systems are redundant or fault-tolerant, so operations may be impaired but still functional. My role is to help facilitate support to get the system operating at a target performance level. This includes working with onsite teams to dissect problems in order to provide a quick resolution.
I also provide guidance via email or phone to help our customers’ internal IT staff resolve system incidents. This can include analyzing logs, providing troubleshooting tips and remotely accessing their system to resolve incidents.
If additional resources are required to investigate and restore service, the incident is escalated to our engineering group or a local service team.
Resolution in Action
We recently had a customer report intermittently bad or missing audio on a console at their public safety answer point (PSAP). This problem began on a Friday afternoon. Due to the critical nature of the incident, we decided to escalate the problem to engineering in order to restore service as soon as possible.
My colleagues and I worked with engineering throughout the night. It was important to take a systematic approach to troubleshooting because the reported symptoms didn’t identify what was causing the issue.
First, we compared the running software with the current version to make sure there weren’t any Field Service Bulletins relating to the incident. These bulletins provide proactive alerts for software and hardware incidents and instructions on how to resolve them.
Next, we worked with the customer and their PSAP staff to perform testing and collect data in order to help identify exactly when the issue occurred.
We then pulled data from the entire transport system, analyzed it and highlighted any problems identified, such as a misconfiguration. The analysis showed that the customer was running older software and that some of their switches were not configured correctly, but this was not causing the problem.
To get to the root cause, we took a core router out of service. This action improved console performance, but not to an acceptable level. We kept the router powered down and continued troubleshooting.
Finally, we identified a problem with a console gateway. Taking that equipment out of service stopped the problem. Because the system is redundant, this did not cause an outage. Within 48 hours, the replacement parts were on site and the issue was fixed.
Summary
This particular case was complicated because the customer experienced two unforeseen, simultaneous failures – a core router and console gateway.
A PSAP system is complex, with lots of moving parts. Because our team received the necessary data and the determined the exact dates and times that the issue occurred, we were able to analyze and isolate the faulty hardware that was creating the problem. This helped us get to the root cause and resolve the issue.
This is the kind of scenario that makes my job rewarding. As part of the Motorola Solutions team, I can count on having the resources to do whatever it takes to solve our customers’ most complex problems.
One failure in your mission-critical communication network can have a systematic impact that compromises operations,security and safety.
