How to query Big Data quickly: Stream those queries!

Over time, we’re generating massive amounts of new data, thanks to a growing number of connected devices and services. As this Big Data gets bigger, it becomes a challenge to gain insights through traditional database queries: More data to sift through means more lag time between a query and actionable results. Parallel processing is one way to solve this problem, but streaming queries can help.

Damian Black of SQLstream at Structure:Data 2012

(c) 2012 Pinar Ozger. [email protected]

Damian Black, CEO, SQLstream explained how at the GigaOm Structure:Data conference on Thursday. “The key to getting massive scale parallelism is data flow execution,” Black said. “In the relational world we can use streams of queries with ordered data in the context of other records in the stream before it and after it, as well as compared to other streams.”
Using the gaming world as an example, Black likened this to a continuously updated real-time leaderboard. “By streaming big data, analyzing and processing millions of data bits and reacting to the output,” a massive amount of data can be queried in a far shorter time period.
But this approach isn’t unique to online gaming, Black suggested. Any data that fits what he called “S3 data” can take advantage of this approach; specifically data from sensors, systems and services.
That suggests a broad range of uses, especially as more connected consumer devices gain the use of sensors. And the number of services is expanding too; Black noted that streaming queries are ideally suited to services such as text messaging, Twitter and even real-time pricing information. The days of instant price changes based on true supply and demand may be a SQL query away.
Watch the livestream of Structure:Data here.

Watch live streaming video from gigaombigdata at