Sybase Blogs

Big Data Will Change the (Very) Old IT Org Chart

Irfan Khan, Sybase CTO — Tue, 15 May 2012 15:11:36 EST

It’s not surprising that the first known information storage system—the clay tablet—was used for structured data. Yes, the Sumerians used their cuneiform writing to compile data on the wealth of temples around 3200 BCE. The temple’s leader held a title of sangu, which roughly translates to accountant, (in effect, the CFO). The sangu reported directly to the king, (the CEO of Sumer, if you will).

The sangu’s scribes (let’s liken them to the IT department) entered and stored data about such valuables as livestock and beer, the fermenting of which the Sumerians are credited with discovering. This penchant for data led the Sumerians to another breakthrough. Around 2600 BCE they invented a mathematical table that had rows and columns for calculating. We recognize it today as the spreadsheet. And guess what they used it for? Inventory management.

I mention this only because so little seems to have changed since those days in Mesopotamia. That is, IT is still mostly beholden to our modern equivalent to the sangu. I came across a recent study by Gartner and the Financial Executives Research Foundation that claims that 42% of IT departments report to the CFO with only one-third answering directly to the king, er, CEO.

This hoary tradition of putting the CIO under the finance leader makes sense when most of what the IT department did was count things and store information about them in structured formats, especially when most of the data related to the worth of the enterprise or kingdom. Having your top finance person being responsible for all aspects of the information, including the systems used to manage the data, whether that be clay tablets or data warehouses, makes sense in that context.

But I predict that Big Data will upend that ancient organizational history because it’s not just numbers about the value of inventory or weekly cash flow. Getting insight from Big Data requires a broader understanding of the business, one that should take more CIOs out from under the CFO’s shadow, if they take the lead on applying analytics to these new data sources.

Gartner also says that in 2012 CEOs will be pushing for more analytics projects, in part because they want to exploit Big Data for growth. This could be a great year, then, for CIOs to make the case that they need to report directly to the CEO. And taking the helm on Big Data projects might be the best way to end the ancient history of IT’s slot on the corporate org chart.

The Worth of Big Data

Irfan Khan, Sybase CTO — Tue, 08 May 2012 16:01:07 EST

You may have heard that some in the industry believe that 99.9% of Big Data is “worthless.” Indeed, I’ve written about the scientists at the Large Hadron Collider who discard the vast majority of the petabytes of data their experiments produce. And we all know about the huge swathes of our SANs wasted on unused data. Organizations confront mountains of extraneous and redundant data. That’s a well-understood problem.

But is it really “worthless”?

I vigorously disagree. Even if data is not usable to the task at hand, it may become so later to another analyst. And if it is redundant, understanding what processes are causing the repetition might lead to improved business performance. Understanding the whole of a Big Data opportunity means being able to discern what part of your data set is worthwhile and which is not. That does not mean that what you discard today will be worthless tomorrow. It depends on the questions you ask the data and how you ask them and when.

More to the point, as Hal Varian, Google’s Chief Economist observes, to truly conduct valid predictive analytics, perhaps the most important potential of Big Data, you need to start with a random data set, even if it’s a small one. However, as engineers at Google know, to get that truly random data set, that sliver of data needs to come from a massive amount of information. Without a large enough pool of data to draw from, the validity of your data set and subsequent analytics loses precision. In other words, even unused, Big Data generates the most valid data sets for modeling.

Philip Russom, TDWI’s director of data management, adds another critical aspect of Big Data. He argues that Big Data is discovery oriented, where you look for facts you never knew before. He warns if you over massage Big Data you risk eliminating outliers in the data, which might be exactly what you need to find, such as in credit card fraud.

Blithely dismissing 99.9% of Big Data as worthless shows a lack of nuance and insight when it comes to understanding the Big Data problems modern enterprises face. Seeing the world strictly through a traditional database mentality, where only the purest, cleanest, most massaged, and manageably small data set is trusted, leads to bald and false proclamations about the worth of Big Data.

Such a view does not lead to effective predictive analytics. It does not lead forward. It strikes me as purely a defensive position from those who do not have the tools to exploit the enormous worth of Big Data.

JPMorgan’s Cautionary Tale of Position Monitoring

Neil McGovern, Director of Marketing — Wed, 02 May 2012 21:14:54 EST

“JPMorgan Chase Loses $2B in Poorly Monitored Synthetic Hedge.” Headlines like this have abounded since Thursday, screaming out for a standup comedian, you know, one of the smart-mouth ones like Jon Stewart, to riff off:

“Hey, when I lose my keys, I look behind the fridge … did they try that yet?”

“Jeez … for $2B I can get them a real hedge, and I can even throw in a shrubbery!”

But this event further opens the door for critique from those of us who have been selling real-time analytics software -- and frequently discussing our key use cases on trade and position monitoring.

The bank’s strategy was “flawed, complex, poorly reviewed, poorly executed and poorly monitored,” JPMorgan CEO Jamie Dimon said in a conference call Thursday. He described the bank’s derivatives position that bet on market recovery via corporate bonds as “egregious, [and] self-inflicted,” adding, “We will admit it, we will fix it and move on.”

Now I like Jamie Dimon, and I think this probably started and escalated below his personal radar, so I don’t blame him directly. However it seems amazing to me that a bet can be so poorly designed and poorly monitored.

Ready for LEI - Industry Thinks So

Neil McGovern, Director of Marketing — Wed, 02 May 2012 21:14:14 EST

Some welcome good news for financial institutions came out in April's "Building a Data Management Foundaton" webcast. Most respondents (80%) felt that their current data management systems are in good shape for future regulatory requirements. These regulatory requirements include Legal Entity Identifier (LEI) adoption, where a plurality have either made necessary changes or have a plan to do so.

More good news are the anticipated benefits from LEI - with over 86% of the audience convinced that LEI will be a positive change within 5 years of adoption.

In March 2012, Ira Hammerman, Senior Managing Director and General Counsel of SIFMA outlined the key benefits the industry expects to see, which include more efficient data aggregation, and this improved risk modeling and management. That Mr Hammerman put data aggregation improvements at the top of his list shows that the even the senior management in the industry recognises that data management needs improvement

I'll be monitoring the rollout of LEI with our customers to see if the promised value appears, and the timescale of both adoption and value generation. Interesting times.

Analytics in Our iPad Era

Irfan Khan, Sybase CTO — Tue, 01 May 2012 15:31:10 EST

Tablets are increasingly the tech tool of choice for CEOs. According to the 2012 CEO & Senior Business Executive survey by Gartner, 41% of CEOs polled said they use an iPad. Mark Raskino, an analyst with the research firm, says he expects that percentage to be at least 50% by now. And it’s growing.

Given this rapid adoption by executives, Raskino suggested during a webinar last month that CIOs should “put some cool app on the CEO’s iPad” to show off the innovation and capabilities of his or her IT department. Few apps, in my mind, would be more effective than one built around analytics, particularly one that showed how applying data to business issues can foster growth because, as the Gartner survey showed, it’s the top concern for CEOs across the globe.

Although there are some very interesting analytics apps on the iTunes App Store for iPad, as you can imagine, most are for entry-level analytics businesses. I don’t think CEOs at large organizations would be that impressed with most of them. (Plus, the App Store’s predilection toward consumers means that a search for “Analytics” in the Business section of Apple’s online marketplace yields numerous curious results, such as “PiPuzzle: Valentine’s Day Edition,” “Circus Challenge,” and “Palmistry—destiny in your hand,” among others. And, while palm reading could help some CIOs in reading their CEO’s views on IT, it’s probably not the best app with which to start.)

However, the Gartner survey does suggest what kind of app CEOs might appreciate for their tablet: a customer-centric one. According to the Gartner research, nearly 60% of CEOs see increasing their investment in their CRM infrastructure, more than any other IT area. The second item on a CEO’s IT wish list for 2012: data-driven management.

Could you ask for a better combination for making a CEO happy? Customers are the foundation for any business growth. CRM systems are rich with customer data about your business. Creating an app that delivers relevant, decision-focused data about your customers, then, should be a high priority for any IT team that has a CEO carrying an iPad.

SAP Sybase IQ MARKETPLACE Launches!

Andrew Neugebauer, Product Manager for Sybase IQ — Fri, 27 Apr 2012 14:02:35 EST

We have launched the new Sybase IQ Marketplace – it’s a place for buyers and sellers from all over the world to discover the latest Data Warehousing, Business Intelligence and Analytics solutions integrated with SAP Sybase IQ. Also learn about the Sybase IQ Express Edition which is available for educators, DBAs and developers as a free download from the site to test drive SAP Sybase IQ and all its features.

Learn about our partner solutions that can help you to stay competitive when you need to deliver innovation and increase value to your customers. To do this, you need a partner that can help you create a new enhanced offering to expand your portfolio of products. By creating new analytics applications, or adding analytics capabilities to an existing application, you gain an efficient way to maintain a leading edge.

The Sybase IQ Marketplace allows us to showcase our best in breed partners and create a community in which to allow them to expand the market reach of their software solutions. So, please visit the new Sybase IQ Marketplace at www.sybase.com/iq-marketplace and check out what new innovations are available! Check out Anayltics Solutions, Data Warehouse Infrastructure and Business Intelligence Tools partners and check back periodically as we add more partners.

Why Partner with Sybase IQ?

For over 15 years Sybase has pioneered column-based databases for analytics with Sybase IQ. Today Sybase IQ is an enterprise-class Big Data Analytics platform. Sybase IQ delivers the performance partners require, in a form factor that can be optimized for deployment with their analytics solution. Sybase IQ can scale to solve the toughest analytic problems while slashing maintenance costs for you and your customer. It’s that simple.

The Sybase partner program is open to ISV and OEM Application Vendors. Click here to read more http://www.sybase.com/detail?id=1096464

Optimism and Alarm as Events Continue to Shake Market Confidence

Derek Klobucher, Financial Services Editor — Thu, 26 Apr 2012 06:00:51 EST

“The financial service industry has an image problem,” market information and strategies magazine Financial Advisor wrote after last week’s SIFMA Private Client Conference. “Earning back trust was a common theme during the one-day conference.”

There was optimism as well, with talk of nobility in financial services, increasing the “standard of care” and “stewardship.” But many acknowledged the damage done by market manipulation, bad actors and poor performance, with one speaker venturing that Generation X is completely soured on investing.

After tales of Goldman Sachs managing directors calling clients “muppets,” can you blame Generation X? Even recent activity would validate its concerns.

Shelved Confidence

A U.S. federal court recently fined Amsterdam-based high-frequency trading firm Optiver $14 million in relation to multiple attempts to manipulate oil prices with the disruptive “banging the close” trading technique out of its Chicago office in 2007. Three Optiver employees received two- to eight-year bans from trading commodities.

So it is no surprise that more than half of Americans do not trust bankers and brokers to do what is best for the economy. We could justifiably be surprised that the number is only as high as 54 percent. Fidelity Investments’ Kathleen Murphy shared that figure from a CNN poll at the SIFMA Conference, adding that the number is up from 30 percent in 1990.

But this isn’t just a matter of holding culprits accountable. It’s also about holding authorities accountable for punishing wrongdoers -- and not-exactly-wrongdoers.

Market Capital Punishment

That initiative suffered a setback recently when U.K. regulators with the Financial Services Authority failed to fine former UBS executive John Pottage £100,000 for not acting fast enough to fix compliance issues within his arm of the firm. The unprecedented attempt to punish senior management for poor oversight could have been a watershed for restoring confidence in the financial services industry.

Questions aside about who defines “fast enough” and whether or not it is reasonable to make an example of someone in the name of restoring confidence, this case telegraphs regulatory intentions. Authorities will surely learn from this defeat, and make similar attempts with different tactics in the future.

And there is something to be said for visible effort.

Louder than Words

“High-profile crackdowns on insider trading seen in the U.S. and U.K. in the wake of the financial crisis have not been mirrored in Japan,” The Financial Times noted Sunday. “Examples of punishment for market abuses are rare.”

Financial services firms could put their clients before themselves, one speaker at the SIFMA conference suggested, even taking on the role of “a good parent” or caring physician.

But just one thing will improve confidence, according to Fidelity’s Murphy.

“Not by asking for it, not by advertising,” she said. “We need to earn their trust by actions.”

Scientists Push Big Data Limits

Irfan Khan, Sybase CTO — Tue, 24 Apr 2012 16:59:57 EST

Physicists studying the results of tests at the Large Hadron Collider (LHC) at CERN, the Swiss nuclear research laboratory outside Geneva, have a lot of data to ponder. In fact, they have nearly 50% more than they had originally estimated. Initially, the LHC was expected to generate 15 petabytes of usable data each year. Recent reports have raised that number to more than 22 petabytes annually.

However, CERN says that its tests produce vastly greater amounts of data than gets studied. Some experiments can create up to one petabyte of data per second. Lucky for the data storage managers, not to mention the tired-eye physicists, on average all but 200 Mbytes per second of that petabyte are deemed “uninteresting data” and discarded by the system.

It’s Big Data quantities like we see at the LHC that helped prompt the U.S. government late last month to announce $200 million dollars in research and development funds specifically for scientists confronting the data deluge. According to a statement from the White House, the funds are necessary to develop the technologies capable of “managing, analyzing, visualizing, and extracting useful information from large and diverse data sets.” And it’s hoped that this investment in Big Data management “will accelerate scientific discovery and lead to new fields of inquiry that would otherwise not be possible.”

I am optimistic that those of us on the technology side will be up to the task of handling the data needs of science. But even I was daunted by recent news of the Square Kilometer Array (SKA). Headquartered in Manchester, UK, with a target completion date in 2024, the proposed 20-nation radio telescope research project dwarfs all other Big Data initiatives I’ve seen so far. This single project is currently estimated to produce one exabyte of data every day; or the equivalent of six weeks worth of the total volume of data traversing the Internet in 2011.

Large-scale, multi-nation collaborative science, such as the LHC and SKA, are relatively small in number and confront Big Data problems on a scale few of us can imagine. But they bear watching because almost all of us can learn from their Big Data solutions.

A Windstorm of Big Data

Irfan Khan, Sybase CTO — Tue, 17 Apr 2012 14:06:41 EST

In his fascinating and comprehensive 1984 study Heaven’s Breath: A Natural History of the Wind, Lyall Watson observed, “Wind is invisible.” And, as a result, “There are no photographs of the wind.”

But he wrote that before Big Data blew into town. Now we have a moving picture of the wind.

Data visualization experts Fernanda Viégas and Martin Wattenberg have given us this beautiful near real-time view of wind in motion throughout the continental United States. The striking live image uses the massive National Digital Forecast Database maintained by the National Weather Service. By clicking on the map you can drill down and see the wind blowing in your area of interest; or simply become mesmerized by flowing patterns the wind makes.

In addition to being a thing of beauty itself, the visualization also offers practical insight for energy companies in deploying wind farms to produce electricity. It can also be a tool for firefighters in the American West in their annual battle with forest fires. And even farmers can use the visualization to help them locate the best places to establish effective wind breaks to fight wind erosion, a major hindrance to agricultural productivity.

Viégas and Wattenberg are hunting for similar wind databases from other regions on the globe so they can build a model for those geographies as well. If you know of any such databases, you should let them know.

The promise of Big Data is to help us see things we never could before. No where is that truer than with this visualization of wind.

Evolving the Perfect Stress Test

Derek Klobucher, Financial Services Editor — Tue, 17 Apr 2012 06:15:33 EST

Citigroup needed something positive. It reported increased revenue and profit during its Q1 earnings Monday, which is fortunate, given that “the global banking group could use some good news to counter the disappointment after it recently failed the Fed’s stress test,” Forbes noted last week.

Stress testing, or the U.S. Federal Reserve’s Comprehensive Capital Analysis and Review (CCAR), is a rigorous examination that delves into the health of banks via 25 crisis-simulating variables. Such variables include 13 percent unemployment and a 50 percent drop in equity prices. The test is meant to inspire confidence that -- unlike in 2008 -- banks today can weather harrowing storms.

But what happens when there’s no confidence in the test?

Survival of the Fit Test

A combined 96 percent of North American financial services professionals were “not at all confident” or only “somewhat confident” that stress testing addresses all of the important risks to the banking system, according to a survey Sybase released Tuesday. This was remarkably consistent with results abroad (where other testing regimes reign), such as 93 percent in EMEA and 86 percent in APAC.

The cynic in me, which is coincidentally 86 percent of me, wondered: Are these people really dismayed with stress testing as it is? Or are they dismayed with stress testing as a reality that the Fed imposed on them last month?

Despite their afore registered lack of confidence in the system, almost half of the North American respondents (46 percent) indicated that stress testing should take place at least once every six months. More impressive, 68 percent of those surveyed in EMEA and 60 percent in APAC also preferred semi-annual testing. The figure was 84 percent in the U.K.

The six-month survey sampled 185 senior managers and C-suite risk management professionals from major capital markets firms in North America, EMEA and APAC on the latest challenges facing the global capital markets industry.

Natural Correction

One big takeaway is something we’ve known all along: The market likes confidence. Another big takeaway is that the market practitioners surveyed want these tests to boost their confidence.

Stress testing will do so by identifying the weak links in the banking industry’s chain.

“These scenarios ... mirror the situation witnessed during the global economic crisis of 2008, so aren’t completely implausible,” Boston-based market watcher Trefis noted last month. “So even if you’re not an investor in bank stocks, these tests can tell you which banks are best-positioned to withstand a potential economic downturn.”

It will take a while for the Fed to work out the bugs, as it would with any new, complex system that follows an international crisis. Feedback from annual tests -- or twice the feedback from semi-annual tests -- will help authorities in the U.S. and around the globe hone their examinations, perpetuating the toughest traits and weeding out the irrelevant.

In my bovine-themed post from earlier this year, I asked whom new regulations and their subsequent tests should try to please. It must be everyone trying to invest with confidence, from the tech-savvy firm to the day trader to the middle-class home buyer.

We can all be thankful for the recovering economy that Citi’s CEO credited with helping the New York-based holding improve its Q1 performance. But we must prepare for the worst in the meantime.

And finding the right way to prepare will require some evolution.