The sangu’s scribes (let’s liken them to the IT department) entered and stored data about such valuables as livestock and beer, the fermenting of which the Sumerians are credited with discovering. This penchant for data led the Sumerians to another breakthrough. Around 2600 BCE they invented a mathematical table that had rows and columns for calculating. We recognize it today as the spreadsheet. And guess what they used it for? Inventory management.

I mention this only because so little seems to have changed since those days in Mesopotamia. That is, IT is still mostly beholden to our modern equivalent to the sangu. I came across a recent study by Gartner and the Financial Executives Research Foundation that claims that 42% of IT departments report to the CFO with only one-third answering directly to the king, er, CEO.

This hoary tradition of putting the CIO under the finance leader makes sense when most of what the IT department did was count things and store information about them in structured formats, especially when most of the data related to the worth of the enterprise or kingdom. Having your top finance person being responsible for all aspects of the information, including the systems used to manage the data, whether that be clay tablets or data warehouses, makes sense in that context.

But I predict that Big Data will upend that ancient organizational history because it’s not just numbers about the value of inventory or weekly cash flow. Getting insight from Big Data requires a broader understanding of the business, one that should take more CIOs out from under the CFO’s shadow, if they take the lead on applying analytics to these new data sources.

Gartner also says that in 2012 CEOs will be pushing for more analytics projects, in part because they want to exploit Big Data for growth. This could be a great year, then, for CIOs to make the case that they need to report directly to the CEO. And taking the helm on Big Data projects might be the best way to end the ancient history of IT’s slot on the corporate org chart.

But is it really “worthless”?

I vigorously disagree. Even if data is not usable to the task at hand, it may become so later to another analyst. And if it is redundant, understanding what processes are causing the repetition might lead to improved business performance. Understanding the whole of a Big Data opportunity means being able to discern what part of your data set is worthwhile and which is not. That does not mean that what you discard today will be worthless tomorrow. It depends on the questions you ask the data and how you ask them and when.

More to the point, as Hal Varian, Google’s Chief Economist observes, to truly conduct valid predictive analytics, perhaps the most important potential of Big Data, you need to start with a random data set, even if it’s a small one. However, as engineers at Google know, to get that truly random data set, that sliver of data needs to come from a massive amount of information. Without a large enough pool of data to draw from, the validity of your data set and subsequent analytics loses precision. In other words, even unused, Big Data generates the most valid data sets for modeling.

Philip Russom, TDWI’s director of data management, adds another critical aspect of Big Data. He argues that Big Data is discovery oriented, where you look for facts you never knew before. He warns if you over massage Big Data you risk eliminating outliers in the data, which might be exactly what you need to find, such as in credit card fraud.

Blithely dismissing 99.9% of Big Data as worthless shows a lack of nuance and insight when it comes to understanding the Big Data problems modern enterprises face. Seeing the world strictly through a traditional database mentality, where only the purest, cleanest, most massaged, and manageably small data set is trusted, leads to bald and false proclamations about the worth of Big Data.

Such a view does not lead to effective predictive analytics. It does not lead forward. It strikes me as purely a defensive position from those who do not have the tools to exploit the enormous worth of Big Data.

Given this rapid adoption by executives, Raskino suggested during a webinar last month that CIOs should “put some cool app on the CEO’s iPad” to show off the innovation and capabilities of his or her IT department. Few apps, in my mind, would be more effective than one built around analytics, particularly one that showed how applying data to business issues can foster growth because, as the Gartner survey showed, it’s the top concern for CEOs across the globe.

Although there are some very interesting analytics apps on the iTunes App Store for iPad, as you can imagine, most are for entry-level analytics businesses. I don’t think CEOs at large organizations would be that impressed with most of them. (Plus, the App Store’s predilection toward consumers means that a search for “Analytics” in the Business section of Apple’s online marketplace yields numerous curious results, such as “PiPuzzle: Valentine’s Day Edition,” “Circus Challenge,” and “Palmistry—destiny in your hand,” among others. And, while palm reading could help some CIOs in reading their CEO’s views on IT, it’s probably not the best app with which to start.)

However, the Gartner survey does suggest what kind of app CEOs might appreciate for their tablet: a customer-centric one. According to the Gartner research, nearly 60% of CEOs see increasing their investment in their CRM infrastructure, more than any other IT area. The second item on a CEO’s IT wish list for 2012: data-driven management.

Could you ask for a better combination for making a CEO happy? Customers are the foundation for any business growth. CRM systems are rich with customer data about your business. Creating an app that delivers relevant, decision-focused data about your customers, then, should be a high priority for any IT team that has a CEO carrying an iPad.

However, CERN says that its tests produce vastly greater amounts of data than gets studied. Some experiments can create up to one petabyte of data per second. Lucky for the data storage managers, not to mention the tired-eye physicists, on average all but 200 Mbytes per second of that petabyte are deemed “uninteresting data” and discarded by the system.

It’s Big Data quantities like we see at the LHC that helped prompt the U.S. government late last month to announce $200 million dollars in research and development funds specifically for scientists confronting the data deluge. According to a statement from the White House, the funds are necessary to develop the technologies capable of “managing, analyzing, visualizing, and extracting useful information from large and diverse data sets.” And it’s hoped that this investment in Big Data management “will accelerate scientific discovery and lead to new fields of inquiry that would otherwise not be possible.”

I am optimistic that those of us on the technology side will be up to the task of handling the data needs of science. But even I was daunted by recent news of the Square Kilometer Array (SKA). Headquartered in Manchester, UK, with a target completion date in 2024, the proposed 20-nation radio telescope research project dwarfs all other Big Data initiatives I’ve seen so far. This single project is currently estimated to produce one exabyte of data every day; or the equivalent of six weeks worth of the total volume of data traversing the Internet in 2011.

Large-scale, multi-nation collaborative science, such as the LHC and SKA, are relatively small in number and confront Big Data problems on a scale few of us can imagine. But they bear watching because almost all of us can learn from their Big Data solutions.

But he wrote that before Big Data blew into town. Now we have a moving picture of the wind.

Data visualization experts Fernanda Viégas and Martin Wattenberg have given us this beautiful near real-time view of wind in motion throughout the continental United States. The striking live image uses the massive National Digital Forecast Database maintained by the National Weather Service. By clicking on the map you can drill down and see the wind blowing in your area of interest; or simply become mesmerized by flowing patterns the wind makes.

In addition to being a thing of beauty itself, the visualization also offers practical insight for energy companies in deploying wind farms to produce electricity. It can also be a tool for firefighters in the American West in their annual battle with forest fires. And even farmers can use the visualization to help them locate the best places to establish effective wind breaks to fight wind erosion, a major hindrance to agricultural productivity.

Viégas and Wattenberg are hunting for similar wind databases from other regions on the globe so they can build a model for those geographies as well. If you know of any such databases, you should let them know.

The promise of Big Data is to help us see things we never could before. No where is that truer than with this visualization of wind.

But there was much more SAP brought to the competitive landscape than the fastest database available. It brought a well-conceived strategy to win customers and increase market share. Oh, and it brought money to implement the plan. Lots of money. About a half billion dollars.

Businessweek’s headline sums up our new database world nicely:

SAP Challenges Oracle With $500 Million Hana Incentive

Of course, how SAP spends the money is what will change the game. In that regard the company has devised two-pronged strategy that, first, will help CIOs decide it’s time to move off slow, legacy databases ($337 million) and, second, encourage startups to build applications on top of the world’s fastest database ($155 million).

That seems like the right balance of investments. First, there a lot of CIOs with tired, overworked databases undermining application performance. But they are always fighting for IT resources and need to pick their budget battles carefully. Merely showing CFOs how fast a HANA database infrastructure will be for a company is not always going to be persuasive. But revealing the partnership mentality SAP is bringing to the table in terms of its investment with customers will definitely help win over the keepers of the purse strings. As one observer summed up nicely: “The game plan: Push long-time database partner Oracle out of the way and take a bigger chunk of the enterprise IT spending pie. It’s unclear how this adventure will turn out, but SAP has some cojones.”

Step two is to increase the application portfolio on HANA by funding startups. With far fewer software startups by comparison to IT customers, fewer dollars are needed. But there’s another less money is needed: software developers naturally gravitate toward the best, fastest, most advanced technology. They have a built-in desire to partner with state-of-the-art vendors. A little extra venture capital money, well, that’s icing on the cake for many of them.

Another area SAP thought through carefully is pricing. The company made it simple. As Information Week points out, in a similar ERP analytics environment with Oracle you would need three licenses. “With Hana there’s just one license,” the news source says.

Finally, training will be a top priority. As Computerworld reports: “SAP is also focused on making sure developers and database administrators have access to the best training materials for HANA, as well as development and test environments at no cost….”

This is a complete strategy that over the long term threatens Oracle. That’s why Larry Ellison recently said that SAP “must be on drugs” to consider competing against his company. But as Forbes magazine countered: “Well SAP is doing some really good performance enhancing drugs. SAP is standing tall shouting that they are indeed in the database market.”
–

I know, I know. SAP is an enterprise resource planning (ERP) and business applications company. Its software runs on databases like those from Sybase and other companies. That’s true. But the fact is, SAP is also database company.

Don’t believe me? Well, in addition to having the products in its Sybase arsenal, SAP makes the fastest, most robust database available today: HANA. HANA’s in-memory, massively parallel processing architecture makes it ideally suited for the era of Big Data. And I mean really Big Data.

In a recent independent benchmark HANA raced through a 100TB test database with 100 billion records. First, HANA achieved a 20x data compression level, which was remarkable. More impressive, though, was that with no caching, indexing, or materializing of the query results, the query responses were a mere 300 to 500 milliseconds. Compare this to some Oracle documentation that has claimed it was “lightning fast” at processing 100 million records in one second. HANA, then, can run 1,000 times more data in less than one-half the time than Oracle.

Beyond benchmarks, in the real world of Wall Street, one HANA application is using Sybase CEP (Complex Event Processing) to feed more than 2.1 million updates per second into the database. In a retail environment in Japan, one customer achieved 400,000 times performance improvement over its previous database environment. Adobe uses HANA to analyze customer data in real time and T-Mobile runs three HANA databases to analyze and reduce customer churn. It’s stories like these that make HANA the fastest growing product in SAP history.

HANA portends to change not just perceptions about SAP as a database company, but real lives. Imagine, if you will, a cancer patient sitting in a doctor’s office having a sample taken to analyze her genome. With those test results, medical professionals can prepare a targeted, personally designed treatment program for the patient. The problem is that today those test results take at least one month to complete. With HANA, the potential is that the doctor will get those results during the patient’s initial office visit and can begin the treatment immediately.

HANA dramatically changes the database industry landscape. Now, according to Gartner, SAP has “put extraordinary competitive pressure on the megavendors, including IBM, Microsoft and Oracle” to follow its lead. However, they are a long, long way behind the new database technology leader.

Big Data is fast becoming SAP’s best friend because it cries out for a database like HANA. And with Big Data on everyone’s lips these days they’ll soon be talking about SAP as the database company for our Big Data era. And it won’t be shocking news to anyone.

Plus, more people inside companies have more requirements to use more data more often for their jobs. This not only adds to the massive amounts of data being retained, it puts pressure on IT departments to have more of that information fingertip-ready for users. In addition, virtually all enterprises are hanging onto their data for longer periods. Heightened governance and regulatory concerns make storing data the default policy in many organizations.

Gartner estimates large companies are seeing growth rates between 40%-60% annually. So, it doesn’t take long for those data store costs to mount quickly.

How can you manage these costs? Well, first of all, you need to have a data retirement policy. When can you safely archive and ultimately delete information not covered by regulations? Naturally, each organization will have their own requirements. And some data, even seldom accessed information, might never be removed such as CRM data on long-term customers and HR information on employees.

But other information—and there’s a lot of it—can be retired after negotiating with the information owners. And you can add leverage to the discussions by making certain that the data owners also own the costs of the data store over time. Nothing like a line item on their budget to send department heads to the delete key.

Yet, with hot data inside a database, storage costs can often be overlooked. That’s because the key metric for a production database is generally performance. Putting any constraints on performance might jeopardize a service-level agreement, so it’s a step seldom taken lightly.

With the release of Sybase Adaptive Server Enterprise 15.7, in-database compression is now an option. Needless to say, rates will vary, but we’ve seen between 40%-80% compression ratios, certainly enough to make a notable impact on storage costs.

What’s more important, I think, is that performance remains the top priority for ASE users, so the new compression capabilities does not put your SLA at risk. ASE can compress a single row so as to eliminate empty spaces in fixed-length columns; page directory and page index compression can be applied at the page or block level; and large objects (LOB) are compressed in-database. The overall effect is to actually reduce I/O time. Backup performance is also greatly improved as a result.

There’s more information here in this white paper. If you have data storage cost problems—and you do—it’s worth a moment of your time.

As a citizen of Great Britain, I do not take sides in either contest. However, I am rather partial to the data geeks who, increasingly, are helping democratize the use of analytics in daily life. By exposing the average citizen to the value of analytics, these data geeks are not just improving the public’s understanding of their world, they are fostering an interest in the field at a time when we need more analytics experts.

For example, computer scientists at the University of Illinois have created a website that uses analytics to help college basketball fans pick the most likely winners in the NCAA Men’s Basketball Tournament. At the same time, the New York Times has hired a well-known psephologist to write a regular blog about Big Data and politics. Both efforts not only raise awareness of the use of data in our lives, they underscore the accuracy analytics brings to an event before any hoops are shot or votes are cast. They show, with reliable track records, that using analytics can improve your understanding of not just the past, but of the future as well. That’s very appealing to people.

Hollywood also gave a boost to understanding analytics when it touted the sports movie Moneyball as an Oscar contender for Best Picture at this year’s Academy Awards. The movie raked in more than $100 million in ticket sales where a few million filmgoers witnessed Brad Pitt’s character discuss the importance of analytics in picking the best ballplayers for the least cash, a classic business intelligence decision process. As a Tinseltown flack might say about analytics, “You can’t buy that kind of publicity.”

More important than high-profile exposure of analytics to wider audiences is encouraging the broader use of the technology. Arguably, no one has done a better job at putting BI tools into the hands of everyday people than Google. In addition to publishing blogs and articles aimed at small business users to help them exploit their own business data, it has posted ten, simple how-to videos targeting analytics tyros. Like the tools themselves, Google offers the instructive videos for free.

I suspect that the current popular interest in analytics will continue because data increasingly has become intertwined with our modern lives. More than ever, we see ourselves and our actions in the context of data. And I hope that the hype that Hollywood movies, election cycles, and sporting events bring to the technology translates into genuine interest among younger people to study the underlying math in analytics. As we all know from the now-famous McKinsey report on Big Data, we face a shortfall of hundreds of thousands of analysts, data-savvy managers, and analytics experts in the coming years; a shortfall that could put many enterprises at a competitive disadvantage.

Without attracting more young people to the field, the competition for the few skilled practitioners available will become heavily weighted in favor of only the largest and wealthiest organizations. But if all of this recent publicity does, indeed, spur more students to major in mathematics and statistics, the building blocks of analytics, then the democratization of analytics will have been a great success at more than just the box office.

Still, it is legitimate question and one posed by Bloomberg Businessweek Research Services in a recently published report called “The Current State of Business Analytics: Where Do We Go From here?” (You can download the survey here if you have an account at The Data Warehouse Institute.) While the answer derived from their research is less emphatic than mine, it is ultimately supportive of my position.

I’m not going to review all of the information in the report. But there are a few key areas of interest relating to the question at hand. First, there is the expansion of business analytics within companies. Compared to a similar survey conducted in 2009 when 90% of $100+ million companies around the globe said they used business analytics, 97% say they do today. Further, 58% of respondents say their use of analytics increased significantly or moderately in the past 12 months.

Another point in the survey underscoring the acceleration of analytics in business is the hunt for talent. Forty-three percent of those polled have current plans to add to their analytics personnel, while another 22% acknowledge they need more people in the area, but for various reasons are not recruiting today.

That growth tells me that virtually every mid-to-large enterprise accepts the value of analytics. Perhaps the only data-driven business tool more prevalent in an organization today is the spreadsheet.

That’s because, as the report notes, “money talks.” The top three reasons companies are using analytics are to reduce costs, improve profits, and manage risk. In short, analytics are successful because they can directly help an enterprise succeed in their business.

However, the report’s authors seem to equivocate on the value of analytics in one area because the survey reveals that companies continue to rely on intuition more than they do data during the decision-making process. In its words: “While organizations recognize that business analytics provide additional insight for decision-making, survey results seem to show that analytics cannot fully replace experience and knowledge.” They put the ratio at 60/40 in favor of intuition over analytics.

That’s not a bad thing, from my perspective. And, in fact, a more nuanced survey might have discovered how tightly integrated intuition and analytics are in the most successful enterprises. By that, I mean, the best queries and models executed inside an analytics engine are those derived from business experience. Good intuition leads to savvy queries which bring the best results. In fact, the report supports my interpretation by noting that in companies “using analytics effectively,” that 60/40 ratio of intuition to analytics shifts to 53/47.

To me, the ideal balance for effective organizations is a 50/50 balance of business know-how and analytics. Maybe the next Bloomberg Businessweek poll will show that. Until then, I’ll argue that business analytics is slightly underrated.