Research shows that a typical Fortune 1000 company that uses data 10 percent more effectively can generate $2 billion in additional revenue. In the consumer space, retailers can increase margins by 60 percent. Every enterprise must pay attention to Big Data.

What is Big Data?

Big Data is the collection and storage of massive amounts of data. IDC defines Big Data as projects collecting 100 terabytes of data, comprising two or more data formats. Big Data also refers to datasets so large that typical database software tools cannot properly capture, store, manage and analyze it. According to Garnter, there are three dimensions that create challenges and opportunities around Big Data: volume (amount of data), velocity (speed of data in and out) and variety (range of data types and sources).

Overall, Big Data is the continuous stream of digital information that arrives through many channels, including email, social media, mobile devices, system logs, customer or B2B transactions, and much more.

Big Data delivers a host of benefits, including productivity, higher revenues, better decision making and innovation. But the amount of data is growing at the astounding rate of 40 percent to 60 percent a year, and this flood of digital information can oftentimes be too big for an enterprise to handle with its current solutions. This presents some new challenges.

The Challenges

To generate useful insights, data must be acquired, stored and analyzed in thoughtful and cost-effective ways. However, there are challenges, including:

• Existing infrastructure can’t handle Big Data: Because Big Data is less structured, traditional databases and analytics platforms cannot handle it. In fact, Big Data is unexplored terrain that will stretch and strain any company’s IT infrastructure.
• Higher CapEx: Trying to handle Big Data without rethinking an existing on-premise infrastructure can result in the need to buy additional drives and servers to accommodate growing data, which can lead to sky-rocketing capital expenses.
• Higher costs for non-strategic resources: The more servers and storage drives brought in-house, the more IT personnel is needed to tend to them.
• Vendor lock-in that limits choices: When you’re locked into one vendor, you have to keep buying from them, even if you never see an acceptable ROI from their wares. You’re caught in a vicious cycle.

The Solution: Move Big Data to the Cloud…

One way to overcome these challenges and get more value out of Big Data is by moving it to the cloud. The cost-effective storage and pay-as-you-go model makes the cloud well-suited for Big Data. For example, consider a company that needs more computing resources to handle bursts of Big Data analytics on an unpredictable schedule. The cloud can provide a simple and affordable way to spin up as many extra servers as needed, release them as soon as the job is complete, and pay only for the time they were actually used.

Most CFOs consider the utility billing for cloud computing, paid out of operating budgets or OpEx, to be better for the corporate balance sheet. This means the cloud can provide the best of both worlds: better services plus a stronger bottom line.

According to Forrester researcher Holger Kisker using Big Data in the cloud makes sense for three key reasons:

1. Big Data requires a spectrum of advanced technologies, skills, and investments. Do you really want this all in-house?
2. Big Data includes huge amounts of external data. Does it make sense to move and manage all this data behind your firewall?
3. Big Data needs a lot of data services. Why not focus on the value of your analysis, instead of simply managing your data?

For all these reasons, moving your Big Data projects to the cloud makes a lot of sense.

For more information on Big Data in the cloud, please check out the whitepaper “Turning Big Data Into Big Dollars.” And check out a recap of the Enterprise Cloud Forum webinar on how to turn big data into big dollars.

Or perhaps you’ve forwarded corporate email to a personal account to read it on the go since your company doesn’t provide you a corporate device? And in extreme cases, you might have grabbed your credit card and ordered cloud services from Cloud Service Provider to work on a small project, but bypassing the necessary approvals from IT or the business.

If you’ve done any of these things, you’re participating in a new trend known as Shadow IT. The rise and evolution of cloud services, and the ease with which they can be purchased with little more than the virtual swipe of a credit card, is fueling this Shadow IT trend, in which IT systems and software – including cloud services — are built and used within enterprises without the approval of IT or management.

Shadow IT has become a quick and easy way for end users to improve their agility and innovation. But along with the list of pros comes a list of cons as well.

Shadow IT could pose a security, concern creating security holes and potentially leading to damaging a company’s reputation and business if incorrectly executed. It could also expose company data to third party cloud service providers and become a compliance issue if it is not done properly and without contractual due diligence.

But if done correctly and properly executed, Shadow IT can spur innovation. Rather than standing in the way of employees purchasing and using the tools to do their jobs better, CIOs should embrace and encourage it. Much like the bring your own device (BYOD) trend, Shadow IT will persist and IT departments will be bypassed. CIOs who take notice and make it work will create a new level of freedom.

It makes perfect sense why Shadow IT, BYOD and other employee-driven technology trends are taking hold so tightly in the enterprise: Traditional IT organizations are taking longer to fulfill business needs because they are understaffed and spend the vast majority – some say 80 percent – of their time supporting day-to-day issues and fighting fires. At the same time, some IT pros are focused so heavily on technology that they can lose sight of business requirements.

At the same time, employees are becoming more tech savvy and innovative, and the prices of some popular cloud services – Dropbox, Google Disk, iCloud, YouSendIt and many others –are dropping or, in many cases, they’re free.

Consider this example: At a major financial institution, end-users became frustrated with central IT’s inability to respond to business needs. As a workaround, the employees took matters into their own hands. The result was more agility for employees and the business.

As more budget is flagged for cloud-based services, Shadow IT is in a position to continue cutting costs, particularly for IT infrastructure. The corporate technology landscape is changing, and CIOs need to take notice. IT, too, has to understand that its role should be to help the business move forward and innovate, not to create roadblocks. Embracing and creating policies around Shadow IT will make it much easier to manage.

IT leadership needs to loosen its stranglehold and spend less time fighting to retain control. Instead, IT should work on self-provisioning and IT automation, tools that will help employees have easy access to the tools they need in a controlled and approved environment.

Support from IT and from the CIO, and a clearly defined IT governance framework, can eliminate many of the risks of Shadow IT, and defining cloud use policies and working closely with cloud service providers can go a long way toward maintaining that control.