Why Amazon built its data stream processing engine Kinesis

Turns out, Amazon(s AMZN) built its real-time data streaming processing service Kinesis because it was trying to solve some of its own internal data streaming issues. Ryan Waite, AWS Data Services general manager, said at Gigaom’s Structure Data conference in New York on Thursday that as AWS’s own data sets got bigger and bigger, the company needed a new way to process the data in real time around important applications like figuring out how much computing customers used and should be charged for.

Kinesis, which became available in December, enables Amazon and its customers to capture a stream of bits of data, store the data across multiple data centers and make it available for processing in just seconds. Real-time processing is becoming increasingly attractive to internet companies as they look for ways to move beyond batch-processing workloads.

Waite said some of Kinesis’ first customers are in the gaming industry and in online ad technology. Game company Supercell has been using Kinesis to do in-game data processing to tweak games and make them better. The applications are all about taking advantage of the timeliness of the data rather than waiting minutes or hours to be able to analyze it.

[protected-iframe id=”91c9ae3bd5fb72117bec5c4bbc784c66-14960843-34305775″ info=”http://new.livestream.com/accounts/74987/events/2795568/videos/45672406/player?autoPlay=false&height=360&mute=false&width=640″ width=”640″ height=”360″ frameborder=”0″ scrolling=”no”]


Structure Data Ticker

Photo by Jakub Mosur