Sybase Analytics

Hello, my name is Mike Upchurch and I’m the COO of Fuzzy Logix. Our company was founded based on 12 years of research and development related to creating analytics and optimizing their performance. When we started, our goals were to develop a library of analytical models that had very high performance and which could be deployed pervasively.

To meet our performance goals, our team of engineers wrote the models in C/C++, designed them to take advantage of parallelism and to leverage other performance enhancing techniques. Once we had a highly tuned set of models, we found another challenge. We noticed that the performance of analytics was hamstrung due to the time required to move large datasets. As we researched the solution, our CEO, Partha Sen, had a revelation. His question to our engineers was “Why move the data to the analytics, if we can move the analytics to the data?” And thus begun our quest to convert our large library of data mining and predictive models so that they would run beside the data source; in-database. Effectively, we perform the analytics on the data as it leaves the database and we only move the results to the presentation layer. In terms of speed improvement, we see a 10X to 100X (or more) gain in performance. In fact, the larger the data, the greater the speed advantage. Since the amount of data being created and stored is growing daily, this is one of the best ways for companies to meet the challenges of deriving insight from their ever expanding databases without seeing a corresponding (and linear) growth in processing time.

Once we attained our speed goal, we set out to find a way to make analytics pervasive. There were a number of reasons based on years of observation and experience. For example, we noticed that most analysis was being performed by a small group of people with specialized skills. Not only did they need to understand quantitative models, but they had to learn special languages and complex analytics platforms. We also noticed that statisticians and quantitative teams spent a lot of time running and re-running models on behalf of their business colleagues. Most of the people requesting the analysis could understand and leverage the results of the models, but didn’t have the skills or access to the expensive tools required to run them. This time our question was “How could we create models that could be run by any device, presentation layer or development tool and push the models out into all areas of an organization?” And the answer was to use the most common data access language ever invented; SQL. To use our solutions, you simply write SQL statements and pass parameters.

The result of combining performance and pervasiveness was that we saw an exponential increase how companies were leveraging analytics. Statisticians no longer had to own all modeling tasks, but could instead build models and give access to end-users via standard reporting tools such as MicroStrategy and Excel. Now users could run their data through the models by simply selecting items from a drop down list or off a menu. Because almost any tool can call SQL, we were able to embedded analytics into business processes and operations. That meant that large numbers of people, in many different areas, could make decisions based on the output of analytic models. The quantitative teams saw other benefits. Not only did they have more time to work on custom models, but they also gained a library of building blocks that could be combined to create higher order models. Since they were able to run the models so effectively on entire datasets, we saw a great leap in their efficiency.

Since we now we had a set of analytical solutions that delivered incredible performance and were easy to deploy, the next question was “So how do we leverage these advantages to create value?” And the answer will be covered in our next entry; where I’ll highlight some examples of how companies have found success using in-database analytics and Sybase IQ.