DemandTec Explains How It Keeps Big Data on Target

DemandTec, a retail-forecasting software provider, has convinced Target Corp. (s tgt) — an existing customer — to hand over even more of its shopping data in order to better set prices and forecast demand. Target is following a growing trend in retail that involves using more granular customer data to predict demand, thanks to advances in software and technology. But stores still need companies like DemandTec (s DMAN) to help them turn the plentiful straw of digital data into predictive gold, a service that becomes more challenging as more data is introduced.

DemandTec has helped Target gather retail information at the store level for years, tracking how many cartons of Tropicana orange juice were sold in a single store, across a city or even statewide. It established causal relationships between that data and ads and promotions to forecast demand and the impact of pricing changes on buying habits. But now Target is using DemandTec’s Shopper Insights software to go another level deeper, tracking not just the items, but instead an entire basket of goods and associated demographic information. Now, for example, Target will know that a person who bought Tropicana also bought a brand-name cereal and a lawn chair.

Derek Smith, VP, Retail Industry Marketing with DemandTec, says that adding the data associated with tracking an entire basket of purchases caused an “exponential increase in the information” the company processed. To prepare, DemandTec purchased data-warehousing hardware from Netezza (s nz). But the hardware infrastructure for storing and processing this data isn’t as interesting as the statistics and knowledge that DemandTec says they have to apply in order to make real conclusions for their customers. It’s not enough to have a firehose, they have to figure out which streams affect purchase decisions and which are meaningless.

That job will soon get harder. Smith says the next level of data that retailers will want to incorporate will be from social-media sites and personalized ad campaigns delivered via mobile phones, as well as more data streams covering phenomena that have a proven impact on retail spending such as weather information. Unlike weather data, much of the social information is unstructured, which means DemandTec will look for partners to provide the structure DemandTec needs, or the company will have to do it themselves. Already companies like Microsoft (s msft) and Infochimps are looking at ways to provide both meaning and a market for data (GigaOM Pro sub req’d).

The ability to store and process terabytes of information in order for Target to tweak its prices by 3 cents and drive a 5-percent increase in purchases by its most profitable customers is now here. When it comes to the future, DemandTec is looking for services that help it get a handle on the massive amounts of unstructured data out there, such as which of the 65 million tweets a day has a direct relationship to what a person will buy. Sounds like a golden opportunity.