How to End the Shadow IT vs. Legacy IT Civil War

By brad.schulteis -

Amazon Web Services is an amazing technological enabler, and can be found at the core of many of the most prolific technologies we now use every day.

The platform has also functioned as a catalyst, breaking through traditional IT roadblocks and in many ways assisting the adoption of DevOps methodologies. The cloud has been transformative for IT, and AWS, with its rapid pace of innovation and widespread adoption, has facilitated much of that change.

For example, with AWS there is no longer a lengthy capital expenditure or physical asset procurement process to acquire IT infrastructure, so it is no longer fulfilled as the result of long workflow of requests ending with the “provisioning team." As this shift continues, the entire role of “enterprise IT” has changed.

 

These ongoing changes have sparked something of a civil war between legacy enterprise IT and emerging DevOps practitioners who want to use AWS, often doing so without the knowledge of enterprise IT. It's  become known as “Shadow IT,” or the use of unapproved technologies within a company.

It’s not that existing IT leadership is inherently against AWS, the cloud in general or any of its promises, such as rapid innovation. In fact, in most organizations, IT leadership is charged with fostering such innovation. The struggle is about loss of control and visibility — and that may be the biggest hurdle to enterprise adoption of AWS.

Enterprise IT vs. Shadow IT

Frequently, enterprise IT leadership has spent years building careers around IT service management processes and frameworks. They understand the perils that come from technology failures, and have established careful, complex regimes to drastically reduce the possibility of failures — and minimize negative effects when they occur.

Those who end up operating in the Shadows are often from a younger generation, raised in the era of self-service immediacy. They are often unwilling to accept that anything technology-related should take more than a few minutes or be artificially rationed by anyone else. They've likely successfully used AWS to “get it done,” while breaking through the mantra of “this is how we've always done it.”

Unfortunately, this can mean trampling over many of the very IT processes put in place to protect the organization — processes often created as a result of painful past experiences, or for a very real need, with legal and fiscal underpinnings. And while bypassing them is met in some quarters with praise, doing so can also result in mistakes, breaches and broken legal obligations.

The role of the CIO

CIOs didn’t join the C-Suite 30 years ago simply to deliver technology. They were invited to become C-level leaders because of the changing dynamics of the business in general, and their role was essential in delivering information and value to the rest of the organization.

That remains more true today than ever. The CIO has a fiduciary and legal responsibility to the organization, and in executing this responsibility, the CIO must create policy, process and yes, paperwork, in order to to deliver on these requirements.

The CIO’s role often comes with the unenviable job of having to account for every dollar spent on IT. Shadow IT has become a serious problem for organizations that require this type of fiscal governance.

I have heard story after story from CIOs being approached by their CFOs, inquiring about an enormous AWS bill. If this happened just a year or two ago, the CIO was often dumbfounded. “We don’t use AWS!" he would likely protest. "It’s never been authorized by IT. We don’t have a way to protect our customers’ data and there is no fiscal governance.”

The CFO, looking over the thousands of dollars owed to AWS, generally has had one simple reply: “Fix it.”

It is the CIO left holding the bag for all things technology when things go badly, and in many organizations, the CIO is legally culpable for all IT decisions. Think about any major data breach you’ve read about over the past few years — the CIO is often the first person representing the breach to the media, and likely the first person held accountable for any negative outcomes.

https://youtu.be/17BjtTb47RM

Worst case scenario

Picture this: a customer sues your organization over a data breach, because its data was on your system — a system that was (unbeknownst to the CIO) running in  AWS, and so almost immediately identified by one of the millions of bad actors actively scanning AWS for known vulnerabilities.

Since it was never properly protected, it was basically wide open to the world. In performing the post mortem, it was uncovered that while enterprise IT originally denied developers the ability to install the requested software and open the requested ports due to unmitigated vulnerabilities, Shadow IT took over.

The developers opened an AWS account, installed vulnerable software themselves, and moved the customer’s data without authorization. The customer was happy — right up until that unmitigated vulnerability was exploited, and all of its proprietary information was exfiltrated and shared with its competitors. The CIO is either culpable for the technology decision to use the unmitigated platform in an insecure fashion, or guilty of gross negligence for his complete lack of knowledge of the true situation. Pick your poison.

Exacerbating the civil war

Fast forward 12 to 18 months, and imagine that enterprise IT has now authorized the use of AWS, but in doing so, one of two things has likely happened. They either threw up their arms in defeat, and simply allowed the unrestricted and uncontrolled use of AWS, or they have placed onerous restrictions on its use, almost entirely eliminating its utility, further exacerbating the civil war.

Again, IT is either being blamed by the rest of the organization for legitimately failing to provide any oversight or control, or being blamed for stifling innovation and inhibiting progress.

Neither response is done out of malice or dereliction of duty, but both outcomes are effects of the very same cause: AWS is complicated and very different than legacy IT — so different that enterprise IT  doesn’t know how to best marry governance and control with self-service, rapid elasticity and autonomous freedom.

Enterprise IT organizations have been feverishly attempting to make AWS “enterprise ready” for years, in ways that just don’t work in the cloud. Often they try to take the same approaches as with on-premises IT infrastructure— sometimes resulting in almost laughable outcomes. Request workflows with approvals measured in weeks, EC2 instances with so many agents and custom configurations that the bootstrap process takes hours or cutting a ticket to “the guy” that manually installs the app.

The primary tenets of cloud computing include self-service, on demand and rapid elasticity. Stifling these invalidates the premise — and the promise — of cloud computing. So for a CIO, how does this jibe with the duty to protect the organization and provide fiscal governance of IT dollars?

Modification and notification are key

This is the incredibly tenuous line that CIOs in the age of cloud must walk. They are still charged with protecting the information of the organizations they serve. They must continue to provide oversight and control of IT while simultaneously relinquishing control. And, they must act in the financial best interests of the organization, which means imparting fiscal governance over the IT budget while once-fixed expenses become variable and are now often directly controlled by the lowest levels of the organization.

Balancing all these competing requirements in the cloud, and more specifically in AWS, is very difficult at scale without severely limiting its power. But finding this happy medium is essential to successfully running AWS in an IT enterprise and ending this civil war.

Don’t abandon legacy governance processes; modify them to account for and embrace the changes AWS brings. For instance, where a legacy budget process likely inserted itself before new infrastructure was procured, modify this process to detect the change as it happens and trigger notifications. This way, it's possible to learn about budgetary impacts in real time without inhibiting rapid innovation.

For costly changes, contact the responsible party immediately. Track less costly changes over time, and review them periodically with the responsible team. Similar process changes can occur on the security side. With AWS, you can monitor the entire account for security relevant changes in real time, even denying certain requests across the board or preventing specific changes based on environment. With security, it's probably safest to be a little more restrictive, but automation techniques can be used to limit the additional burden.

An example of this in practice might mean setting an update policy such that nobody has the necessary permissions to modify a running production stack, but anyone can submit the requested changes through an integration with your IT service management tool.

When an approval authority clicks “approve," an API request with the right permissions is automatically completed. This ensures visibility of changes, documents all changes and enforces proper separation of duties.

Technology leaders must find new ways like these to instill best practices while freeing their organizations from the burden of trying to manage it themselves.

Rackspace combines the right mix of automation and people to deploy and manage your AWS environments more efficiently, freeing your organization to focus on its core mission. We give you on-demand access to the tools, automation, best practices and hundreds of AWS experts when you need it. We take a security first approach, ensuring that you are building the most secure environments from the start, and providing continuous monitoring to identify potential trouble across all of your AWS accounts. We provide monthly account reviews, to identify cost and performance optimization opportunities. We can even provide 24x7 operational support for your entire AWS environment.

Visit http://www.rackspace.com/aws for more information about Fanatical Support for AWS and how it can help your business. And download our free white paper Best Practices for Fanatical Support for AWS, which offers further detail regarding implementation options for AWS, including identity management, auditing and billing.