If your perception of sentiment analysis was shaped by Twitter-sentiment toys, it’s time for a relook. These “toys” simplistically score tweets positive/negative/neutral based solely on keyword presence without regard for context. Semantically rooted sentiment technologies do better by getting at contextual word sense and by discerning sentiment at “feature” level, and they handle more than just social-media analyses. Online/on-social measurement and engagement are important, but businesses interact with customers and the market and collect feedback via many channels, for instance, contact centers, e-mail, and surveys. Ability to handle these diverse sources, and to integrate with enterprise systems that capture customer transactions and profiles, is an essential ingredient of enterprise-scale sentiment analysis.
How does this bit of theory play out in practice, among people engaged in real-world customer relationship management (CRM), marketing and market research, and business intelligence (BI)? I polled three industry figures to find out: Banafsheh Ghassemi, VP, Marketing – eCRM & Customer Experience (CE) at The American Red Cross; Marshall Toplansky, president of “mass opinion business intelligence” vendor WiseWindow; and Next-Generation Market Research guru Tom Anderson, who heads Anderson Analytics. I posed them three questions, exploring the path to enterprise-scale sentiment analysis.
My first question gets at a basic question, essentially, is sentiment analysis worthwhile? The responses address ways sentiment analysis complements established methods and channels, in time frame and as a cross-check.
Seth> What has your sentiment-analysis experience been like? Have you gained new customer or market insights, and have you been able to do anything new, anything you couldn’t have done without sentiment analysis?
Marshall> “The big revelation to us has been the volatile nature of both consumer sentiment and the business metrics they are indicators of. When you use traditional marketing research to understand sentiment, you are dealing with long time frames. Research does a good job of identifying long-term trends. But, people are living increasingly in the short-term. They capitalize on market moves quickly and have a set of short-term tactics. Mass consumer sentiment from online sources, unlike marketing research, has been able to identify these tactics and measure results in the kind of fast timeframes contemporary businesses require. You could never have even designed a survey in the time it takes to get in and out of moves indicated by sentiment.”
Banafsheh> “[Let me tell you about our] in-kind donation scenario. In recent years, due to harder economic times, people have shown more interest in contributing through unsolicited in-kind donations rather than money. Due to cost structures associated with storage, transportation, and delivery of such contributions we are unable to accept such donations. We have seen some negative emotional reaction to this on our social networks. However, when we look for context in other channels where the same interest is voiced, such as calls to our public inquiry line, we have seen that we could do better to proactively communicate and educate the public in why we don’t accept these donations. The proactive awareness could very well minimize, if not eliminate the negative sentiments that come with a preconceived expectation.”
Tom> ”There’s no doubt that sentiment analysis has been useful on several projects and that we have gained market insights thanks to being able to segment and code data based on various sentiment approaches. Overall usefulness really depends on the data though, which is different on a a case by case basis.
”Most of our data contains a lot of fields other than unstructured data, so we’ve been lucky to have a lot of options from the beginning. Sentiment becomes a lot more important when you are looking at data that is less rich, like Twitter which more or less is just 140 characters with a data stamp.”
On to Q2, how sentiment relates to other data elements. My one-sentence summary of the three answers is, There’s clear correlation, but you don’t want to make too much of it. But read the responses for yourself.
Seth> How are your organization and clients matching online or social sentiment or enterprise feedback with information from other data sources?
Banafsheh> “I wouldn’t say we ‘match’ the information, but look for context in our other more structured (e.g., CRM) and richer data sources (e.g., email which is extremely rich). [The in-kind donation scenario illustrates this.]
Marshall> “We have seen a good deal of client interest in correlating online consumer sentiment to a number of important business metrics. For instance, in one case, correlation analysis found that changes in online sentiment relating to product problems was found to be a leading indicator of call center volume. In another, changes in weekly sentiment related to product quality were found to lead weekly stock prices. And, in another, sentiment related to a leading musical group was a strong indicator or changes in sales of music.
“To our minds, this is not surprising. It seems obvious to us (with plenty of hindsight) that when you have hundreds of thousands of people expressing their preferences, their actions will tend to follow.”
Tom> “It depends on the project, there have been many so we’ve probably done just about everything at least once. We’ve merged LinkedIn data with survey data, CRM data with social media and survey data, and call center data with matching operator notes with field technicians notes, to name just a few.
“I think though that there is a misconception out there among some that it’s a good idea to pipe all text data into one source and make comparisons across data. Having worked with quantitative data for many years I can tell you that often times those cross comparisons sound better in theory than in reality. I think it’s far more important to think about the various resources you have, identify the most important ones and look at them individually. After that is done, you will have a better understanding of what can be gained by merging the data.”
My last question was intended to be practical and forward looking –
Seth> Do you have any guidance for folks who are new to sentiment analysis or who have been using sentiment technologies only for social-media analysis?
Banafsheh> “First, I would say using sentiment technologies with social data is valuable in any Voice of the Customer (VOC) program toolkit to the extent that it is not the only data source used. Otherwise, the insights produced will be very narrow and limited insights as it would be if any one of the other data source was used. If you examine each one of your touchpoints and channels where free format VOC is captured, such as surveys, call-center notes, customer e-mails, etc., in isolation, you will very likely find sentiment patterns in each that is more slanted in one direction than another. For example, you may find your surveys show more positive sentiment, your letters may show more neutral, and your Twitter feed more negative, and so forth. So if you only took one source you will draw incomplete conclusions.
“Second, these tools are still fairly challenged when it comes to social ‘parlance,’ the abbreviations, excessive snark, emoticons and so forth. So be aware of those limitations and recognize that the outputs will still require human intervention and validation of the results. Recently we compared coding using SPSS’ text analytics with the coding manually provided by Red Cross volunteers and found poor agreement. The text analysis software coded 26% of the positive comments as positive. The software was unable to assign a sentiment to more than half. The 21% of positive comments coded as negative by the SPSS software were generally related to the use of words describing the seriousness of the hurricane or the extent of the damage. Examples ‘hit very hard by Irene,’ ‘Many blood drives were forced to close,’ or ‘Instead of whining about the dud, how about joining the Red Cross.’ There was modestly more success with negative comments, where 53% were coded correctly by the software.”
Marshall> “My advice is about WHO should be adopting the use of sentiment technologies. Give it to the people that run real metrics in the company. If you let market research people control this tool, you will move too slowly. If you give it to the social media engagement group, you will get no strategic value from it. Give it to the operations people, who have to create better forecasts and design real-time key performance indicators for the business. This is where the real strategic and tactical value lies for sentiment analysis.”
Tom> “In regard to sentiment generally, the tendency is to compare machine coded sentiment to human coding. The problem is most of us are so far removed from human coding that we forget just how inaccurate it can be. We also seem to forget that a lot of data really is neither negative nor positive.
“Personally I don’t like to compare human coding to machine coding, we find that we can do so much more with machine coded data. If companies are looking at text analytics and sentiment analysis as just a way to cut or eliminate human coding costs, they’re not understanding the true benefits.
I’ll split out part of Tom Anderson’s answer, regarding social media analysis, which characterizes as a “pet peeve.” Tom’s thoughts on this particular topic –
Tom> “Everyone is talking about social media analysis. In reality though 99% of what’s being called social media analysis is just Twitter or Twitter plus blog data. This represents only about 10% of the population, and for those of us who blog or are on Twitter, we know just how ‘special’ this population and messages we propagate are. A lot of it is definitely rather promotional in nature.
“I’m not saying Twitter and blog analysis has no value. It’s good to understand what drives online buzz and consequently some of the Search Engine Optimization (SEO) efforts. However, until Facebook tears down their walled garden (I think it will happen soon), we’re not seeing what most regular people are saying. Until then more focused research among brand enthusiasts on special discussion boards will probably remain most useful. Sadly, relatively little of this is actually being done. Those projects we’ve done in that area have been rather successful, and sentiment analysis certainly was a critical component in all of them.”
I had posed my questions to Marshall Toplansky, Banafsheh Ghassemi, and Tom Anderson in connection to the November 9, 2011 Sentiment Analysis Symposium. Banafsheh was a panelist, Marshall gave a lightning talk and his company was a sponsor, and Tom would’ve attended if he hadn’t had a schedule conflict. The conference went really well. See for yourself: Videos are online at sentimentsymposium.com/SS2011w/presentations.html. And please do revisit the symposium site in mid-January, when information on our May symposium, the 3rd New York symposium, should be online.
The Prediction/Feeling/Wish categorization is conceptually quite interesting. It reflects the richness of human emotion, that we may have a variety of different purposes and meanings when we express seemingly similar feelings, attitudes, and opinions. That you hope Manchester United will beat Arsenal doesn’t mean you think a draw won’t be the outcome.
