site author: Anthony Wheeler email: email@example.com
Having worked for AT&T from 1990 to 1997, I gained considerable insight into the nature of that monopoly, and witnessed a large corporation struggle to reform into a competitive entity. The experience was instructive, and worthy of detailed consideration.
Classic AT&T was formed in 1984 after divestiture, and ended in 2005 when it was purchased by SBC. SBC subsequently adopted the AT&T logo after subsuming the former behemoth.
Prior to 1984, most telecommunications in the US was a monopoly generally referred to as ‘Ma Bell’. In most cases, for most citizens, nobody else could provide telephone service in any form, up to and including inside wire and the actual telephone. This state of affairs began circa 1913 after an agreement was made between AT&T and the US Federal Government, one that allowed AT&T to operate more or less as a monopoly until divesture in 1984. Justification for this monopoly was provided by AT&T’s executive leadership:
…one policy, one system [AT&T's] and universal service, no collection of separate companies could give the public the service that [the] Bell... system could give.
Theodore Newton Vail, President, AT&T, 1907
It is important to note that the only way a commercial entity can operate for any length of time as a monopoly, let alone for almost seven decades, is through the complicity of the government, and actual regulations that prohibit additional private entities to enter the same market and complete.
In 1984, divestiture broke up AT&T into eight distinct entities: AT&T the long distance company, having retained Bell Labs and their traditional equipment manufacturer (later to break off and become Lucent in the 90’s); and seven Regional Bell Operating Companies, or RBOCs:
After 1984, The RBOCs retained the local monopoly while AT&T became a competitive long distance company. Their largest rivals were Sprint and MCI.
With divestiture, the government created two artificial industries, local and long distance. The country was divided into arbitrary LATAs (Local Access Transport Area). ‘Local’ service, by definition, was a call originating and terminating in the same LATA. For this class of calls, the local telephone company (usually an RBOC, although the country was full of tiny independents as well) provided all network, billing, maintenance and provisioning.
‘Long distance’ service was any call that originated in one LATA and terminated in another. This was the case even when the two LATAs resided within the same local company’s network. In all cases, the local company was required to hand off the call to the customer’s chosen long distance provider via a network connection. The long distance carrier (usually AT&T, Sprint or MCI) would provide the network between the two LATAs, and then hand off the call to the receiving local provider. The originating and terminating portions of the connection were called ‘access’. The long distance carrier paid both the originating and terminating provider access fees, rated by the minute. These costs, of course, were passed along to the customer, each call generating originating and terminating access costs, along with the cost of maintaining the long distance network. Post 1984, the major negotiating issue between the ROBCs and AT&T was the cost of access.
For those who lived in the 50’s, 60’s and 70’s, they will recall how expensive long distance calls could be. For example (from the internet):
In 1997, a fifteen minute call would cost $3.75. Over the course of a month, long distance bills for individual consumers could easily reach $200-$300. A large portion of this cost can be attributed to access charges that all long distance carriers had to pay.
Today, the distinction between local and long distance hardly exists, and few people pay per-minute charges for voice calls. As a distinct industry, long distance no longer exists in any meaningful form. More on why below.
I joined AT&T in 1990, and worked for several years in Network Services Division. At that time, AT&T was consolidating seven regional operation centers into two, one in Denver and one in Conyers, Georgia. Each center had Network Management, provisioning and maintenance responsibility across half the country.
Sprint dropped their famous pin in 1989, creating a sense of desperate urgency in AT&T. Ironically, Sprint’s claim of providing superior service through all digital networks rested on the back of AT&T’s finest technology, as AT&T was wholesaling network bandwidth to Sprint, with Sprint requiring it be entirely digital. At that time, a good portion of the AT&T network was still analog, negatively impacting voice quality.
Everything was rushed in AT&T to digitalize the entire network and reduce operating cost. When I arrived in Denver, the new operations center wasn’t yet operational, so we crammed into an equipment building that housed a 4ESS switch (the mainstay of the AT&T network), the network management center (NOC), and all the technicians that provided maintenance and provisioning support across the entire western US. The earthquake in San Francisco in 1989 (the one that happened during the World Series) exacerbated the consolidation, as the Western Operations Center in Oakland was damaged, and instead of doing repairs, AT&T simply moved their operations and people directly to Denver. It was a chaotic mess, at best.
It’s important to note here that such drastic measures never would have occurred in the old AT&T (prior to 1984), where prices were regulated and assured (on the high side), employment was permanent, and customers captive. (A bumper sticker from that time read: ‘Ma Bell – We don’t care because we don’t have to’.)
In the industry, advances in technology were very slow, with some central offices using the same step switches that were first implemented over fifty years before. The 4ESS (#4 Electronic Switching System), the primary toll switch (long distance), was first deployed in 1976, and while the central processor was upgraded every couple of years, the basic structure remained intact into the 2000’s. In most cases, the switch took an entire floor of a large equipment building. During my time, there were about 125 AT&T 4ESSs operational in the US. To jump ahead, the capacity of those large switches can be accommodated today in one or two racks in a modern data center.
While I had several assignments in those first five years, my primary focus was Nodals Provisioning. This service entailed a direct-connect T1 between an AT&T 4ESS and a customer PBX (Private Branch Exchange – essentially a local switch for that company). At one time I supervised agents performing routing and trunking translations in the 4ESS. These tranlations (essentially programing) provided the switch and the network with the information and structure necessary to originate and complete voice calls to that company’s PBX. The T1 had a measly 1.544MBs of capacity with 24 dedicated voice channels. In contrast, most broadband services today provide a minimum of ten times that much bandwidth. As I sit here, I am enjoying a total of 45MBs on my cable internet, roughly the same as a T3, a private network capacity unheard of in the 90’s.
Later, I had national process responsibility for the entire service, including the engineering, design and implementation of the T1 facility between the AT&T switch and the customer premises. This entailed developing a detailed design for the T1 in AT&T’s network, including DACs (digital cross connect) assignments, T3 assignments (every T3 could accommodate 28 T1s), switch terminations, and local POP (Point of Presence) interface. Because the local company had a monopoly on access, AT&T had to issue an ASR (Access Service Request) for the tail circuit, a T1 that spanned the distance between the POP (where AT&T and the local company co-located) and the customer premises.
Had the status quo in the industry maintained, this type of service would likely still be common today. But in 1996, the Telecom Act was passed by Congress, allowing the possibility to compete for local services. This proved to be the death knell for AT&T.
Prior to 1996, and in anticipation of competing in the local market, AT&T established another business unit with the intent of entering the local market when it became legal for them to do so. I joined that new business unit in 1995, moving to headquarters in New Jersey, providing me a front row seat to the ensuing industrial failure.
Over many decades, AT&T, as an industrial organization, had grown fat, top-heavy, slow, complacent and inefficient. When the competitive gloves came off in 1984, AT&T struggled to make the adjustment necessary to succeed: many reorganizations, massive lay-offs, new systems developed with the intent to automate –all conducted within a corporate culture unfamiliar with the creativity and leanness necessary to implement complex and challenging solutions. For instance, the primary provisioning platform was designed to perform routing and trunking automatically, thereby eliminating the need for human intervention. Unfortunately, the system was developed and implemented without real-world understanding of the process (a common failure in major system development) and actually required more staff-hours to perform the function. The agents I supervised had to interface with a system that wasn’t designed for human interface. As a result, the system was difficult to learn, exceedingly slow, and prone to putting customers out of service. Given the reduced staff based on false assumptions of increased productivity, the agents had difficulty keeping up with the workload, and due dates were rarely met.
In addition, most of the technicians and clerical staff were unionized, making it extremely difficult to address performance or attitude issues, contributing to continuing poor performance. Seniority ruled in every way, among the union workforce. During layoffs, the younger employees went first, with no regard to skills, attitude, performance or value, thus bleeding the company of potential energy, creativity and leadership. Many of the technicians who remained were bitter and unhappy, and generally unmotivated. They had been moved out of their home switching center, across the country to Denver or Conyers to sit at a desk in front of a computer and work troubles. Prior to being moved, the technicians knew their central office front and back. They maintained manual tub files (rows of 3X5 cards) with records of cross-connects and port assignments. When they received a network order, they had intimate knowledge of their equipment and frames, and ensured the records remained accurate. This was crucial when shooting trouble, as improper information could easily lead to mistakes, ones that could impact network performance. For example, in the early 90s AT&T suffered a major outage at Thomas Street in NYC. From Wikipedia:
On September 17, 1991, management failure, power equipment failure, and human error combined to completely disable AT&T's central office switch at 33 Thomas. As a result, over 5 million calls were blocked, and the FAA private lines were also interrupted, disrupting air traffic control to 398 airports serving most of the northeastern United States. Because the building was designed to be self-sufficient, AT&T had a load shedding agreement with the electric utility, where they would voluntarily switch from utility power to on-site generators on request. This was a routine procedure that had been performed successfully in the past, but on this occasion, it went wrong. After switching power sources, standard procedure was to check all the equipment power supplies, known as DC plants, for problems. But due to scheduled training, the check was not performed, and one plant went on battery backup. The alarms were not detected until it was too late to maintain uninterrupted power.
I recall walking into the NOC (Network Operations Center) that day and seeing the board lit up in red, so many calls were blocked. All of the Thomas Street technicians were out at the time, ironically in a training session about the new power alarms. This event added to the frustration of career technicians, certain it never would have happened in the old days.
In addition to monitoring distant alarms, the technicians in the new operations centers had to rely on accurate network databases, some of them so old nobody new how to program them (TIRKS, for instance - Trunks Integrated Record Keeping System, an operations support system from Telcordia Technologies, originally developed by the Bell System during the late 1970s). These databases were notoriously inaccurate, requiring significant rework and/or double work to implement new service or new equipment. For example, at another telephone company I worked for, I knew engineers who would physically check every cross connect and port assignment to ensure they were available before issuing the build order to the field. This entailed driving from one office to another, significantly increasing the time it took to engineer the installation of new equipment. They couldn’t trust the inventory databases, and it took less time to do the manual checks than rework the design when work orders were inevitably rejected by the field technicians when they found one of the assignments already in use.
At AT&T, the operational support system architecture was incredibly complex, given that new systems and databases were piggybacked on existing ones, until the multitude of interface and protocols grew to unbelievably complex proportions.
Even with the unhealthy corporate environment, the constant uncertainty, and high stress-levels, the older technicians couldn’t leave: they were bound with golden hand-cuffs, retirement benefits just a few years away, with no way to make their union-inflated wages elsewhere. This contributed to a frustrated and defensive technical work force, a significant impediment to corporate success.
These corporate conditions were a direct and historical result of the bloated company AT&T had become. In a Genuinely Free Society, no such private entity would have arisen. Without the monopoly supported by the protective shield of government regulation, the telecommunications industry would have evolved far more dynamically and effectively. The beginning of a more dynamic and creative telecom industry can be seen to emerge after the passing of the 1996 telecom act.
But not in AT&T. As part of AT&T’s newest business unit, one charged with entering the local telecom market, I witnessed first hand the ineffectiveness and inability of that organization to create a new business. Beginning with management’s debilitating hubris, including a tremendous underestimation of the complexities of local telephone service, along with a multiple layer management structure the size and shape of a wedding cake, covered with the frosting of political infighting and age-old bureaucratic traditions, the division shuffled and shivered for over two years without deploying anything. Speaking with senior executives, I learned that they expected to easily penetrate the market and overwhelm anyone else in it. They assumed that because they had been the same company a mere twelve years earlier, and retained Bell Labs and the equipment manufacturer, they had the organizational and technical wherewithal to enter the market with relative ease. How difficult could it be?
For the first year I served on various product teams to develop offerings for the new division. The magnitude of our collective ignorance was (in hindsight) staggering. We knew nothing of E911, or directory services, or voicemail, or call waiting, or how a local central office was wired. I learned about ‘main distribution frames’ for the first time, along with everyone else in the room. My responsibility was to design provisioning and maintenance processes for local services. After a few months learning what I could, I locked myself in my basement for a week and developed the process documents. The results were included in AT&T’s ‘Model Contract’, a document provided to the different regions that were negotiating with the RBOCs. I was amused to find that my process documentation ended up in MCI’s model contract in precisely the same form, including my unique document identifiers. Years later when I worked in a LEC (Local Exchange Carrier) I discovered how ridiculously awful my processes documents were: at AT&T, I didn’t know a damn thing about local telephone service.
The new law required the RBOCs to resell portions of their local networks to other providers (known as CLECs – Competitive Local Exchange Carriers). These were called ‘unbundled network elements,’ and included local loops (the copper wire between the local serving office and the residence) and switch ports. The RBOC was required to allow other carriers to co-locate in their equipment buildings, and provide those competitive carriers with access to the RBOCs switch and local loops. Another option was to resell the entire service by paying the RBOC a wholesale rate and then charging the end user something higher. Ordering any of these options required a new form, called an LSR (Local Service Request). I drafted the first version of this form while still in New Jersey.
One of the product teams I supported was Centrex. This service was provided by creating a virtual PBX in the local switch. Instead of installing a physical PBX on site, Centrex allowed the customer to enjoy all of the features and advantages of a PBX without the attendant costs. The local switch was partitioned, and each station at the customer site was programed with the numbers and features the customer wanted. For a large Centrex customer, there might be tens of thousands of variables during provisioning, and in the end, we could not figure out how to get existing AT&T ordering systems to accept all the necessary data, let alone downstream inventory databases and billing systems. After turning in the requirements document, some six months in the making, it was estimated that it would take another two years to deploy the initial version of the systems needed to support the deployment of Centrex.
We worked hard in the product teams to develop the marketing, process and system requirements for an array of local services. All too often, after developing an intricate solution of one sort or another, management at higher levels, people without the context or technical understanding, changed something, forcing the team to rework the entire solution. Given that we were generally incompetent to begin with, we suffered additional incompetence that made our effort that much more useless.
After a year of working on the product teams, I transferred to one of the regions, where I supported the AT&T team (working alongside MCI and Sprint) in an effort to get the RBOC to support our entry into their business. A year later, late in 1997, we hadn’t issued any LSRs in the region, or acquired one new local customer, despite full support from the regulatory bodies. Just meeting after meeting after meeting. The RBOC did a nice job delaying the process, although it wasn’t that difficult, given AT&T’s approach to the negotiations.
While the RBOC could not enter the long distance business until a number of strict milestones had been met, there were no restrictions on the long distance companies (or any other company) entering the local arena. The RBOC was required by law to support these competitors by unbundling their network and/or reselling their service.
AT&T signaled their failure with the decision to base their local strategy entirely on resale. Originally, they planned to build their own network, and would use unbundled elements and resale as a stepping stone to full market entry. This was the strategy Sprint used to enter the long distance industry back in the late 80’s, and it was the only viable strategy for AT&T. Simply reselling local service was a non-starter, as they would forever be beholden to the local provider for network quality, maintenance and provisioning, with very small financial margins. Once this decision was made, I made mine, and left AT&T shortly thereafter, certain that it was only a matter of time before the former commercial giant would cease to exist.
In January of 1998, AT&T purchased TCG for $11.3 billion to serve as their local company, and closed their local division with a total loss of over $2 billion. Ironically, when SBC purchased AT&T some seven years later, they only paid a little over $16 billion for the entire company.
The RBOCs always had a tremendous advantage over the long distance carriers. They owned all the local access, including copper and fiber. Inter-city infrastructure of the long distance carriers was always easier and cheaper to develop, whereas the local infrastructure the more costly and difficult to replicate. Once gaining permission, the RBOCs could provide long distance services much cheaper than the incumbents. Even so, the old RBOCs, and the companies derived from them, face many of the same cultural, technical and organizational challenges as AT&T. They have more time, however, and began with a solid customer and network base, providing genuine opportunities. The major players (SBC and Verizon, for instance) have invested heavily in the growing cell phone business, and compete with cable companies using fiber networks in the broadband market.
This case study provides several key points related to government intervention, and the ensuing consequences:
The point which is so important is the basic fact that it is impossible for any man to survey more than a limited field, to be aware of the urgency of more than a limited number of needs. Whether his interests center round his own physical needs, or whether he takes a warm interest in the welfare of every human being he knows, the ends about which he can be concerned will always be only an infinitesimal fraction of the needs of all men.
This is the fundamental fact on which the whole philosophy of individualism is based. It does not assume, as is often asserted, that man is egoistic or selfish or ought to be. It merely starts from the indisputable fact that the limits of our powers of imagination make it impossible to include in our scale of values more than a sector of the needs of the whole society, and that, since, strictly speaking, scales of value can exist only in individual minds, nothing but partial scales of values exist—scales which are inevitably difference and often inconsistent with each other.
Friedrich Hayek, The Road to Serfdom
Laissez faire does not mean: Let soulless mechanical forces operate. It means: Let each individual choose how he wants to cooperate in the social division of labor; let the consumers determine what the entrepreneurs should produce.
Ludwig von Mises, Human Action
AT&T is not alone among the vanquished. Consider the following former corporate giants that no longer exist (although not necessarily due to government intervention):
(the examples cited above excerpted from http://www.nbcnews.com/id/41027460/ns/business-us_business/t/most-memorable-companies-vanished/#.WRUDIzt6pHg)
Pan Am, Borders, Circuit City, Polaroid, Block Buster Video, and Tower Records also no longer exist. Joseph Schumpeter called such corporate failure “creative destruction”. This in stark contrast to government institutions that continue to exist and operate regardless of poor efficiency, inferior value, lack of customer/client approval, old technology or lost purpose. The United States Post Office serves as a good example, as a commercial entity with ever increasing prices (cost of stamps) that requires annual subsidies from taxpayers to remain in existence. Another good example would be public schools, including universities, that are so ineffective that many taxpayers opt to send their children to private institutions despite the fact they are already paying, through taxes, for the local school or state university.
AT&T is just one example of how government intervention can distort an industry, and hinder the development of preferable alternatives. The government-supported monopoly perpetuated high telephone costs to consumers, and retarded technological innovation in the industry. The world would have been better off had the government just left well enough alone.