ML Insights

Insights Into A Movie Review

Our Sentiment Analysis Model has achieved an accuracy of 89.3% in classifying reviews from the Internet Movie Database (IMDB). That’s good, amazing – the reviews are often tricky to read. But 1 review of 10 is still misclassified. Why is that?

In the series ML Insights we investigate misclassifications in more detail. We take wrongly classified examples and ask:

  • Why did the machine misclassify something?
  • Why does a human being understand it correctly?

As a first example, we take a false negative classified review of a movie with Madonna. But let us first take a quick look at the model used.

The Model

The model consists of a 100-dimensional Glove Embedding layer, a hidden LSTM layer with 256 units and an output layer with sigmoid activation. It has been trained and tested with 50’000 reviews of the IMDB Dataset.

Training on an 8-core desktop computer with Tensorflow as backend took about 5 hours. You can find the model in our portfolio for download.

Test Review Number 81

Why is this review of a Madonna movie classified so clearly false negative (0.03)?

0.03 So, Madonna isn’t Meryl Streep. Still, this is one of her first films and a comedy at that. Give her a break! Sure, the movie is mediocre at best and pales in comparison to its earlier counterpart w/ Katherine Hepburn, Bringing Up Baby. For what it is, though(a piece of fluff), it’s quite a bit of fun to watch. I’ve yet to hear anyone that slams Madonna’s acting skills back it up w/ evidence or even adjectives other than “awful”, “bad”, or other such vague descriptive words. If you wanna see bad acting or justify the argument that singers should stick to singing, how about Whitney Houston?? She’s had the most undeserved commercial success of any actress in history and couldn’t act her way out of a hatbox. The American public obviously cannot discern the difference between a credible performance in a movie and star power. I think Madonna has always been at least credible in her movies. Get real people. Madonna-bashing is so 90’s.

If we look at the classifications of the individual sentences, then the review seems rather positive at first glance. At second glance we see that sentence 6 is strongly negative (0.06).

0.71 So, Madonna isn’t Meryl Streep
0.86 Still, this is one of her first films and a comedy at that
0.82 Give her a break
0.30 Sure, the movie is mediocre at best and pales in comparison to its earlier counterpart w/ Katherine Hepburn, Bringing Up Baby
0.48 For what it is, though(a piece of fluff), it’s quite a bit of fun to watch
0.06 I’ve yet to hear anyone that slams Madonna’s acting skills back it up w/ evidence or even adjectives other than “awful”, “bad”, or other such vague descriptive words
0.28 If you wanna see bad acting or justify the argument that singers should stick to singing, how about Whitney Houston?
0.52 She’s had the most undeserved commercial success of any actress in history and couldn’t act her way out of a hatbox
0.51 The American public obviously cannot discern the difference between a credible performance in a movie and star power
0.57 I think Madonna has always been at least credible in her movies
0.83 Get real people
0.74 Madonna-bashing is so 90’s.

Quotes

Do the quotes “awful” and “bad” in sentence 6 deteriorate the classification? They are not really referring to the movie. Let’s replace them with “wonderful” and “good”. The classification of the sentence improves as expected (0.50).

0.50 I’ve yet to hear anyone that slams Madonna’s acting skills back it up w/ evidence or even adjectives other than “wonderful”, “good”, or other such vague descriptive words

The classification of the whole review, however, improves only little (0.05). The two quotes do not have much influence on the classification of the entire review (which is certainly correct).

0.05 So, Madonna isn’t Meryl Streep. Still, this is one… Madonna-bashing is so 90’s.

A quick test shows that the model correctly classifies a simple review with quotes. Although we have two quotes with negative adjectives in the first review, it is correctly classified positively. And vice versa, the second review with two positive adjectives is correctly classified negative.

0.65 Some say the movie is ‘awful’ or ‘bad’, I say it’s great.
0.39 Some say the movie is ‘wonderful’ or ‘good’, I say it’s awful.

Core Message

Let’s now focus on the core messages of the review. On those statements that really refer to the film. Are they classified positively? The film is “mediocre at best”, “a piece of fluff” and “quite a bit of fun to watch”. So let us focus on sentences 3 and 4. We see that the model also classifies these sentences negatively.

0.30 Sure, the movie is mediocre at best and pales in comparison to its earlier counterpart w/ Katherine Hepburn, Bringing Up Baby
0.48 For what it is, though(a piece of fluff), it’s quite a bit of fun to watch
0.16 Sure, the movie is mediocre at best and pales in comparison to its earlier counterpart w/ Katherine Hepburn, Bringing Up Baby. For what it is, though(a piece of fluff), it’s quite a bit of fun to watch

In order to understand why a human being classifies the film positively, we must condense the statement even further. What does the reviewer want to say? Why does he/she ultimately classify the movie positively, even though it is “mediocre” and “a piece of fluff”? Simply because “it’s quite a bit of fun to watch”!

0.93 It’s quite a bit of fun to watch.

The model classifies this condensed statement clearly positive. Fun! That’s the key to understand this review. Although the movie is “mediocre at best” it is “fun”. And that counts (for humans). In the midst of all the negative statements, the model has some problems weighting this.

Based on the review however, the question remains whether the film should be classified “positive”.

Signup for Our Newsletter

About the author: Thomas studied computer linguistics and philosophy and graduated with a PhD in computer science. He has worked as a consultant for natural language processing and application development for major Swiss banks. Thomas is founder of ipublia. He lives with his family in Zürich.

Leave a Reply