It’s Time to Upgrade Legacy Alarms:

Capital dollars are any telco’s most prized resource, but there is another resource that is increasingly scarce within telecommunications companies large and small: technician man-hours.

When industry leaders convene at conferences, the challenge of retaining high-caliber field technical talent is often a discussion topic. Beset by this issue, the industry has taken steps to reimagine the role of the field tech and to design new recruitment techniques to ensure a steady supply of qualified and competent technicians. Yet even with these efforts field teams continue to shrink. After years of constant winnowing of the workforce, executives want (and shareholders demand) those workforces to remain lean. Accomplishing this lean efficiency will require an investment in technology to supplement the man-hours that continue to disappear every year.

A streamlined and efficient workforce depends on reimagining the diagnostic apparatus technicians and managers use to understand the health of the outside plant. The alarms and alerts that pulse data from the field to guide operations are aging in the same way the workforce is.

An example of collected data.

To maximize workforce efficiency, telecommunications companies must automate the way they gather site-level insights. This will mean opening new pathways to gathering and analyzing site data, making response systems and procedures more flexible and accommodating for modern technology.

Site Information Is Not Enough
Earlier this year, a typical seasonal storm knocked out commercial power to regions of a carrier’s network. More than a dozen sites lost commercial power. And several of these sites support critical E911 service.

Most telecommunications providers are prepared for this. There are alarms that watch commercial power and trigger alerts at the Network Operations Center (NOC). However, for a host of reasons, the legacy systems did not work in this situation. And had the local CO/OSP manager been relying solely on the alarms delivered to the NOC, he would only have known about power outages at fewer than a quarter of the sites; the majority would have bled battery power silently into the night.

Luckily, this telco had invested in a DC power management system that could more accurately detect power outages and provide more insight about the situation at each site. Alerts did not just come in to the NOC, but via email alerts to key personnel who were able to respond more rapidly than their legacy system allowed. Since the technicians were also notified along with the NOC, response time was reduced and potential outages were prevented.

This storm is a dramatic example of the shortcomings of basic alarms, and how they can cost a company money and man-hours while also costing them network reliability. This scenario isn’t the only reason the legacy alarm and alert apparatus needs an upgrade. Efficiency is lost constantly through more mundane situations that exert a slow burn of resources. For routine issues, like service outages and power failures, many companies have little more than a basic environmental alarm. Such alarms are hardly capable of identifying a failed battery from a downed piece of networking equipment.

When an alarm’s root cause is opaque, who should be dispatched and with what equipment? Simple alarms cannot answer these questions, and instead initiate a process where time and resources are wasted diagnosing the issue.

This is where remote management and monitoring comes in handy. By utilizing next-generation equipment to remotely manage site assets, diagnostic work can be streamlined, and sometimes automated completely. By monitoring the batteries and DC power, managers can quickly know the difference between a commercial power failure and an outage of networking equipment. Then, the right technician can be dispatched to solve the problem.

Truck Rolls are expensive and time consuming.

Connect, Analyze, and Sound the Alarm
Gathering and analyzing data often starts with connecting the remote site to a Cloud-based application for data processing. Outside plant sites have come to harbor many methods of remote communication, often without standardization across a single carrier’s network. Thus, remote hardware must be flexible to use Ethernet, integrate with DSLAM backhaul channels, DSL modems, or even dial-up modems connected to POTS lines. Hard contact alarms are always a great level of redundancy.

Connectivity at the site to enable data collection is the first step, but where there is data there must be analytics. Cloud-based analytics software will have algorithms hunting for patterns in the data, which can yield a wealth of information to guide and optimize maintenance. This could mean predicting battery failure, spotting rectifier issues, identifying faulty commercial power feeds, and many other possibilities. Connecting equipment performance data to the Cloud unlocks the potential of Big Data and machine learning to find issues that hordes of technicians could never hope to identify.

Interestingly, when enabling remote monitoring and data analytics, companies may experience more alarms than before. This is a good thing. They will be able to do more to improve uptime and operational efficiency — if they restrain the impulse to respond to every alarm with a truck roll.

The ideal situation for remote monitoring should include 2 things:

First, data should be available to the managers and technicians who may need access via Cloud-based software. This seems straightforward, but reliable software is surprisingly uncommon, and the capabilities are often limited. Techs and managers should be able to access the site data from anywhere and on any device.

Second, alarms need to be prioritized based on criticality, and the important ones need flexible paths to reach the technicians and managers who care. For some carriers, this will require NOC integration, while others will opt for email or smartphone alerts. The best practice would be some combination of the two so that issues are well documented and friction is removed from the response process.

This is a critical point. Carriers like to hold on to their old ways with fervor, and centralizing alarms at a NOC is no exception to this. There have been great gains in efficiency through such centralization, but there are still more gains to be achieved through empowerment of technicians.

With access to data and analytics at their fingertips, technicians no longer have to operate like reactive robots responding to every alarm with a truck roll. They can become remote operators who take their time to carefully evaluate which scenarios require hands-on attention. Rather than immediately visiting a site with an alarm, technicians should be empowered to diagnose, and possibly even correct, the alarm remotely.

Even with the most advanced analytic capabilities that allow for remote triaging of site issues, technicians will take time to adjust to these new tools. It might be challenging to train technicians to respond to an increase in alarms with fewer site visits, but it is the only path forward to sustainable productivity growth.

Despite the difficulty that this change will encounter, the man-hours that can be saved through this methodology are immense. Most telcos execute preventive maintenance visits every 6 to 18 months, and if issues are found, a return trip is almost always necessary. As mentioned earlier, basic alarms may also require follow up visits once the root cause is diagnosed.

Based on our observations working with carriers large and small across the country, we estimate most sites see approximately 3.5 visits per year. With some 350,000 to 500,000 remote wireline sites (and growing) in the US, the number of visits is staggering (Source: Millennium Marketing). Carriers implementing remote management solutions are finding that they can cut the number of visits per site in half, and possibly further over the long run as predictive analytics and the way they allocate technician time improve. Hundreds of thousands of truck rolls could be eliminated simply by improving the way carriers gather alarms and alerts from the field.

Make Those Man-Hours Count
Big Data, Industrial Internet of Things, Machine Learning — these are buzzwords that outside plant professionals do not use with the same over-indulgence as other technologists. But these are now maturing technologies that are earning their place in the toolbox of network professionals who are responsible for maintaining the nation’s communications networks.

As the industry grapples with the slow attrition of technicians, these tools are not luxuries, but are necessities. Contrary to what many may believe, the technology to monitor in detail every aspect of sites is improving rapidly and coming down in cost. The payback can often be quantified, and may be surprisingly short. How many free man-hours can you generate simply by upgrading your outdated alarm system?