Nielsen to measure Twitter chatter about TV shows

http://www.nytimes.com/2013/10/07/business/media/nielsen-to-measure-twitter-chatter-about-tv.html

Execs are talking about measuring tweet volume and the reach of those tweets, but isn’t the real value in figuring out what people think? It’s not worth touting that 200,000 people tweeted and 4 million people saw those tweets if the overall sentiment is that the show sucks. But given the history of shows such as “Arrested Development,” 20,000 of the right people tweeting about how great something is might be worth noting even if ratings aren’t high.

If machines can perform sentiment analysis accurately, how will that change business?

Recent research at Stanford (see Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank, Richard Socher, et al) seems to have advanced sentiment analysis of text to a new level of about 85.7% accuracy for full sentences, if the authors are to be believed. Imagine that the state of the art might soon be approaching 100% accuracy: what would be the impact on business?

Marketing has moved dramatically in recent years away from surveys and focus groups, toward social listening: monitoring what people in social networks are saying about product, brands, and activities related to them. Marketing staff are vacuuming up comments about products, campaigns, and promotions, and trying to respond in real-time, as well as logging all the data for later analysis.

If machinery can do all or part of this, then an increasing part of the jobs of marketing, customer support, and other functions (like HR) might be accomplished by machinery geared to sentiment analysis, and AI-based approaches to taking next steps based on individual and aggregated sentiment.

It may also be that a large part of what middle managers do involves sentiment analysis. Consider tools like 15five or Tiny Pulse (see TINYpulse is a small and simple anonymous feedback tool), which are intended to be a quick technique to get a handle on staff sentiment. But if semantic analysis and AI can compbine to do all or part of that job, managers might be able to spend their time doing other things, or managing a larger number of people.

This is perhaps a codicil to the piece I wrote earlier today, How many of today’s jobs could be computerized? A whole lot. This may be part of the wave of technological advancement that the researchers I reported on in that piece were talking about.

Twitter can help you beat the spread on NFL bets, kind of

New research out of Carnegie Mellon University shows that analyzing fans’ tweets can help gamblers make better bets on NFL games. Sometimes. Their technique wasn’t very effective at picking winners or betting the over/under, but it was 55 percent accurate on bets against the spread (and then only during the middle of the season). I doubt anyone will undertake this effort themselves for such a slight edge, but there might be a business here if someone can figure out a consistently accurate model.

Little Brother is watching

There has always been an aspect of worker surveillance in the workplace, even leaving aside industries with open conflict between labor and management, like dock workers, teamsters, and agricultural laborers. The rise of Taylorism in the early 20th century, when the stopwatch became a symbol of efficiency at all costs has slowly ebbed, and much of the work of today professional knowledge workers goes unclocked. But just because we don’t actually punch in when we enter and exit the office — and where we might be working remotely and hardly coming to the office at all — doesn’t mean we aren’t being watched. In fact, the technologies that we have today are much more capable of discovering resentful workers than industrial era foremen ever could.

IBM recently announced a new security tool that crunches big data in novel ways to find clues of ‘disgruntled employees’  that might pose a threat:

Joel Schectman, IBM Security Tool Can Flag ‘Disgruntled Employees’

The new tool, called IBM Security Intelligence with Big Data, is designed to crunch decades worth of emails, financial transactions and website traffic, to detect patterns of security threats and fraud. Beyond its more conventional threat prevention applications, the new platform, based on Hadoop, a framework that processes data-intensive queries across clusters of computers, will allow CIOs to conduct sentiment analysis on employee emails to determine which employees are likely to leak company data, Mr. Bird said. That capability will look at the difference between how an employee talks about work with a colleague and how that employee discusses work on public social media platforms, flagging workers who may be nursing grudges and are more likely to divulge company information. “By analyzing email you can say this guy is a disgruntled employee and the chance that he would be leaking data would be greater,” Mr. Bird said of IBM’s new tool.

For example, a company could analyze employee emails that express a positive sentiment to a manager at work, but detect “when he’s talking to a peer or someone outside the company, the sentiment comes out a little different,” Mr. Bird said.  Such a pattern, combined with other factors, could cause an employee to be flagged for more investigation by an IT team. Sentiment analysis works by parsing patterns in words and phrases that signify whether the intent behind a message is likely positive, negative or neutral.

I once worked at a firm where — a few years before I worked there — a disgruntled and mentally unstable employee had watch through an insurance company’s data center with a large homemade electromagnet, and he degaussed a bunch of irretrievable data. I’m sure the company would have liked to have headed off that episode with a tool like IBM’s.

Sentiment analysis is a common capability of today’s social media analytic tools, used to track what is being said about a company’s services and products. And of course, these tool can — and will be — used as in the IBM tool example, to track what employees are saying in their off hours.

Many companies go to great lengths to tell workers what they can and cannot say in their corporate and extracurricular social media activities. But it is very easy for companies to go to far. For example, it is a commonplace for companies to state that employees must not say anything disparaging about the business online, or to discuss internal business issues in public forums like Facebook or Twitter. However, this is illegal, and in many cases will lead to a bad outcome for the company, perhaps just a public outcry, like the recent Applebee case — a waitress was fired after posting a picture of a customer’s receipt, a minister who complained about the 18% tip added to the check for large parties (below).

Applebee Receipt

Applebee Receipt

But companies can get into real difficulties if they fire workers who are engaged in what the National Labor Relations Board considers ‘concerted speech’: that is, discussions between workers about pay, working conditions, safety concerns, changes in workplace policies, and other related topics. And stating that workers have to sign an agreement that curtails such rights or face termination is also illegal.

At the core, it comes down to the rights of workers to self-organize to improve their working conditions, and to meet publicly to do so. In this regard, public forums like Facebook are as protected as a public square, the union hall, or the cafeteria. Companies cannot decide unilaterally that the operations of the business are confidential, and cannot be spoken of.

However, this does not mean that workers can share trade secrets, disclose confidential information, or grips that their supervisor change their work schedule. But it does mean that workers can complain if the company makes workplace changes that raise safety concerns.

A good example of a company overstepping is that of Royal Ahold in 2012:

Workers Win Battle Over Employer Crackdowns on Social Media – Bruce Vail via AlterNet

Leaders of the United Food & Commercial Workers (UFCW) union and the Teamsters have successfully backed down a large multinational conglomerate that attempted to impose such restrictions on more than 100,000 workers across the New England and Mid-Atlantic regions, union officials said.  Complaints to the National Labor Relations Board (NLRB) have resulted in the New York-based unit of the company withdrawing the disputed policy, and a settlement of similar complaints is imminent in the Baltimore area, they said.

The fight erupted late last year when supermarket chains owned by the Dutch retailing conglomerate Royal Ahold began demanding that employees sign a “Social Policy Guidelines” document that warned of dire consequences if workers used social media outlets like Facebook and Twitter to communicate too freely about their jobs. The grocery chains—Stop & Shop in New England/New York, Giant Food in the Mid-Atlantic, Martin’s Food Markets in Virginia, and a separate home delivery service called Peapod—threatened disciplinary action, including possible dismissal, if employees refused to sign the document or violated any of the guidelines.

 […]
In any event, a settlement of the charges in NLRB’s Baltimore region appears to be imminent, according to NLRB spokeswoman Shelly Skinner. Documents have been circulated among all the parties to the complaints, Skinner said, and the NLRB is taking the position that the language of the Giant policy is overly broad. The labor agency also sees merit in the charge that the policy could chill the exercise of the employees’ protected rights, she said. Armstrong added that his understanding of the settlement is that Giant will no longer threaten dismissal for employees who refuse to sign the policy document.

For UFCW, this victory is part of a larger struggle taking place in the realm of social media, according to Amber Sparks, director of new media at the union’s international headquarters in Washington, D.C. The union is using social media, especially Facebook, as a way to connect workers with each other and their union, she said. These efforts are provoking reactions from employers like Giant who see Facebook campaigns for fair labor contracts, or new organizing initiatives, as a threat, she said.

As a result, companies should be very careful in the writing and administration of social media policies, and instead of simply doing what social media gurus recommend, they should consult with a good lawyer well-versed in labor law.

My recommendation is to listen to employees public discourse for sentiment analysis on an aggregated basis, to take the temperature of the company. But I do not recommend tracking employees, looking for gripers. Obviously, in many regulated industries communications between staff and customers must be monitored. But attempting to track the political opinions of employees for the sake of firing those that don’t match your company’s political positions is odious, even if it might turn out to be legal., depending on the state you are in. It might be attractive to use a power just because you have it, but that doesn’t mean you should.

My hope is that Little Brother won’t be watching, even if he can.

Why the trick to analyzing Twitter data is more data

Although Twitter is pushing itself as a platform to gauge public opinion around popular events — including the upcoming presidential election — not everyone is buying the hype. Stats about sentiment and tweet velocity are certainly interesting, but man cannot live on tweet data alone.

Big data as a tool for detecting (and punishing?) bullies

A group of researchers has developed a machine learning model that can detect tweets relating to bullying, and even identify bullies, victims and witnesses. Next, it wants to add sentiment analysis to determine individuals’ emotional states. But if they see trouble, how do they intervene?

5 sites that crunch data to help you predict the president

Big data and data science have already proven their worth in the worlds of online advertising and marketing, and now they’re being turned to elections. Here are five sites to follow if you want to impress your peers with data-driven insights on who’ll win in November.

Infographic: IBM says go away (for Memorial Day)

Consumers are feeling pretty perky about things going into Memorial Day, according to new IBM sentiment analysis. That means stores and hotels might have reason to smile over the holiday weekend, as people seem to want to travel — and shop — over the long weekend.