 

The Challenge
NLP technology is undoubtedly changing the way businesses interact with consumers. However, although AI models are becoming smarter at listening, understanding, and reacting, they still struggle with understanding sentiment – that is, the emotions, mood, or beliefs behind a person’s words.

Take this customer review of the (very expensive) Samsung’s UN85S9 HDTV for example:

“My wife and I bought this after selling our daughter Amanda into white slavery. We actually got a refurbished. It’s missing the remote, but oh well — for $10K off I can afford a universal, right? The picture is amazing. I’ve never seen the world with such clarity. Amanda, if you’re reading this, hang in there. honey! We’ll see you in a year!”

The sarcasm in this review is evident to any human reader, however with words like “amazing” and “clarity,” it could be easy for an algorithm to mistake this as an authentically positive review.

Companies are working hard to create models that don’t make mistakes like these, building models that are trained to look beyond superficial understandings of what people are saying to examine the nuances of how it’s being said, in both short- and long-form writing. That’s why they are investing in sentiment analysis (also known as opinion mining), a sub-field of Natural Language Processing (NLP) that tries to identify the opinion behind any piece of text.

Our client is one of these visionary companies. Their goal was to create a sophisticated sentiment model that would deliver broad-level, composite scores (positive, negative, and neutral) for long-form content, while simultaneously sifting through individual paragraphs, sentences, and words to extract granular insights.

 

The Solution
Step 1
Document segmentation
To create the model, our client provided us with more than 100,000 documents, ranging from short paragraphs of reviews from their e-commerce site to full-length 1,500-word articles published online. We began by analyzing those documents and developing an optimal segmentation methodology. On average, we cut each document into 4 distinct pieces, though the variance was wide-ranging. The longest document had 84 unique segments.

 

Step 2
Sentiment annotation
Our crowd tagged the sentiment of each individual segment, while also providing high-level sentiment scores for each document as a whole. During that process, we ran a wide range of automated gatekeeping procedures to monitor their quality of work in real-time. In the end, we sourced half a million annotations on the original 100,000 documents.

 
The Results
The high-quality data we provided ensured our client’s model was able to accurately detect the sentiment behind text 97.3% of the time.

Training Sentiment Analysis Models