The National Coffee Association estimates that more than 150 million Americans drink coffee every day, many of them at one of the 24,000+ coffee shops in the country. According to Yelp, there are 1,576 places to get coffee in Austin alone. Which raises the question: what’s the best coffee shop in Austin?
Eater, Thrillist, Foursquare, and TimeOut Austin, and Yelp all have their lists. But the methods for creating these lists all come down to subjectivity or some opaque algorithm. For example, take a look at the top 5 listings from Yelp:
Four out of the 5 have equivalent ratings of 4.5. Mozart’s even breaks the top 5 with an average rating of just 4.0. In fact, when you zoom out a little, 46 out of the top 100 Yelp coffee shops have the exact same 4.5 rating. Clearly, star ratings are limited and don’t tell the whole story, especially when you can only rate a business on Yelp in increments of 0.5.
The next thing you know, you’re sifting through hundreds of reviews to find out which coffee shops are really the best. But there are so many! Have you read The Paradox of Choice by Barry Shwartz? Too many options can stress you out. So we decided to enlist the help of one of our new favorites, VADER, discovered by our new digital marketing analyst, Syed.
Darth Vader? No, VADER.
VADER (Valence Aware Dictionary and Sentiment Reasoner) is a machine learning tool we’ve been using recently for a number of clients to conduct sentiment analysis. What’s sentiment analysis? Well, technically, it’s a way of turning qualitative data (reviews) into quantitative data (sentiment scores) to measure the extent to which a reviewer is writing from a positive or negative emotional state, as well as the extent to which the words they use are positive or negative.
In layman's terms: it’s a way to tell how positive or negative a comment is, and how passionate that person is.
The exciting thing is that VADER learned to judge sentiment from ALL OF TWITTER as well as a large number of human raters. This helps eliminate bias by relying on “the wisdom of the crowd”: collective opinion is more trustworthy than individual opinion. In fact, in a large study, VADER not only out-performed 11 other models, but it even beat individual human raters. Oh, and VADER can be deployed to analyze massive amounts of data really quickly.
How we used VADER to find the best coffee shops in Austin
To start, we zeroed in on the top 100 coffee shops according to Yelp. Then, we mined the reviews for every single one of those shops and collected all 20,000+ of them in a single sheet. After looking at the data, we decided to eliminate any coffee shop with less than 50 reviews in order to keep the sample sizes reasonable (I know, RIP new locations, but c’est la vie; it just wasn’t fair to pit 1,000+ reviews against 4). That left 69 shops.
Next, we VADER-ized those puppies! Every review was analyzed and assigned sentiment scores. Then, every shop was given average sentiment scores based on its reviews. The outputs of this technique are 1) a measure of sentiment intensity of the comment with a range from -100% to +100%; and 2) a percentage breakdown of the actual sentiment based on the words used (positive, neutral, or negative). Here are the results rated by most positive intensity and then by most positive words:
As you can see, not all reviews are created equal. For example, Lucky Lab Coffee received the most INTENSELY POSITIVE reviews, whereas Mañana received the highest proportion of reviews who used positive words. For a simplified example, “I LOVE THIS PLACE!!” scores high on positive intensity, because VADER takes into account things like capitalization and punctuation. However, only 25% of the words in the review are strictly positive (“love”). On the other hand, “Love Mañana — the best!” is less intense, but has a higher proportion of positive words (“love” and “best”).
This made us wonder: is there a way to combine the two to find the shop with the highest number of positively RAVE reviews?
Introducing the Positive Passion Index (PPI) and the definitive list of the best coffee shops in Austin
By multiplying the average positive intensity by the average percent positive words, we developed a metric we’re calling the “Positive Passion Index” (PPI). The PPI—on a scale of 0 to 100 with 100 being the best—gives us a holistic idea of the true sentiment behind a review, giving equal weight to the words a person is using AND how passionate they are about what they’re saying. Using PPI, we can give our definitive list of the best coffee shops in Austin:
We love geeking out on data, design, and more
If you look up our top 5 PPI list on Yelp, every one of those shops has an average rating of 4.5. Sentiment analysis allows us to go beyond crude approximations and turn valuable qualitative data into quantitative observations that couldn’t be done manually at scale.
We’re only just beginning to discover all of the applications for this technique, from social media management to message testing to product research and reviews, but we’re thrilled by the insights.
Of course, data isn’t everything. Associate Creative Director, Marshall Walker Lee, would like to share his design sensibilities and coffee love with you via his list of the top 5 coffee shops in Austin: Wright Bros Brew & Brew, Flat Track, Seventh Flag, Mañana, and Caffé Medici.
Want to learn more about this project or have an idea about sentiment analysis applied to your work? Contact us.