That time I fat-fingered an IP address and brought the Network down
Distinguished Systems Engineer (DSE)
Technology, Innovation, Thought Leadership, Enterprise Networking, SDN
As a child growing up on a farm, my father always used to tell me that "if it ain't broke, don't fix it". In the old days, there were no changing things if it was in satisfactory state, right?
Many years later, as a network engineer responsible for a 40,000 user network, I learned my lesson all too well when I "tweaked" the routing protocol one afternoon and fat-fingered an IP address connecting to the regional network gateway, cutting off the entire campus for 15 long minutes. My father’s words were very quickly reinforced that afternoon.
Like me, I’m sure most of network engineers have a story of a time when a simple change had wider ramifications, so the guiding motto has always been “better safe than sorry".
The problem with this approach in the modern days is that demand for network changes are increasing: more devices, more traffic and the need for a more effective security plan. Plus, the repercussions today can be far more noticeable than the time when I made a typing mistake, given the astonishing number of people and devices connected to the network.
Just last weekend, one of the largest airports in the world was down for over an hour causing the check-in service to stop and planes to stay on the ground. The first news reported this was due to a power outage, but later on it became public that the reason was in fact the network. Known upgrades were overdue and the old way of managing the network suddenly backfired.
The other problem with the set and forget approach is that while static means stability, it also means rigidity - and these are not the times for that. Quite the opposite, really. Today, businesses are laser focused on delivering real-time customer experience and innovation, asking of IT and the network what they never asked before: implement changes, and implement them in minutes or hours versus weeks as it’s still normal today.
Luckily, the approach to networking is evolving to become more dynamic and the fear of bringing it down at the minimal change no longer holds true. Not if you’re on the right network, that is.
Cisco has reimagined the network to be much different from as we know it today. Automated instead of manual. Manageable in seconds instead of months. Informed by its data instead of unware. This is no small quest and the implications to both network engineers and business leaders are pretty significant, starting with the speed of change offered by automation.
Automation to the rescue
Automation and controllers are the main drivers for this new approach to network operations and change. The great thing about it is the inherent benefit of determinism. If I make an automated change 100 times, I would expect the same result. If I was to manually type a message 100 times, I might not be so lucky.
We can think of automation as a "trust me" model. We trust the machine to implement our intent in the appropriate way. For example, a simple statement like “WebEx application is business relevant” or “PCI devices cannot be directly accessed by contractors” can be translated and rendered into network configuration commands.
A lot of the early focus has been on pure automation tools, but more recently “trust me” has been augmented to "show me", or assurance. Combined with automation, assurance allows the verification of network changes. It is the automated policy control that makes sure the network is doing what was programmed to do, proactively highlighting deviations. Together, automation and assurance are a powerful combination that reduce the risk associated with more frequent rate and volume of network changes.
As companies incorporate automation into their networks, the way they plan for it should change. With a traditional network, you would design it up front for the next 4 to 8 years as you knew you would not make major changes over that period of time.
With automation, the network design and planning become a lot more dynamic, allowing a base set of capabilities to be deployed and then modified over time. Again, this reduces the risk of change as it means that smaller steps can be taken and validated before moving to the next step. The other benefit is in the significant time savings in rolling out change, both in terms of time and risk.
As shown in the report “Cisco SD-Access: Enterprise Networking Made Fast and Flexible”, creating network segmentation takes 15 minutes with automation versus 7 days with a traditional manual approach.
Acquiring new skills
This new way of networking challenges everything we ever learned about networking, requiring a new set of skills and capabilities.
Let’s take learning to ride a bicycle as an example. Start in a small constrained space, with a very simple use case (straight line) and on a mountain bike (large tyres and low seat). Over time, increase the sophistication of the motor pattern, for example by changing to a road bike or increasing the distance.
Same concepts apply to network engineers as they adopt automation, because in this new way of networking nothing is set in stone for the next 8 years and progressive advancements are possible. Now you can measure success step-by-step, iterate quickly, and work your way up until full automation.
In working with companies everyday, I can tell this transition is about the augmentation of networking skills and a different way of thinking, which takes time and effort to change. The great part is that once you get the hang of it, you’ll evolve really fast and never forget those skills. Just like riding a bicycle!