Yelp Dataset Challenge: Review Rating Prediction
On Yelp restaurant reviews, TF-IDF unigrams+bigrams with logistic regression reached 64% validation accuracy; test accuracy was 54% with RMSE 0.92.
Key Findings
Methodology
The paper formulates 1–5 star prediction as five-class classification on the Yelp Dataset Challenge 2014 restaurant subset. Reviews are lowercased and stripped of stop words and punctuation, then represented by TF-IDF unigrams, unigrams+bigrams, unigrams+bigrams+trigrams, or Latent Semantic Indexing. Each representation is paired with Logistic Regression, Multinomial Naive Bayes, Perceptron, and Linear SVC, yielding 16 systems.
Key Results
- With unigrams, Logistic Regression achieved the best validation result: RMSE 0.85 and accuracy 58%, compared with Linear SVC at 0.87 and 57%, Naive Bayes at 0.96 and 52%, and Perceptron at 1.25 and 43%.
- Adding bigrams improved every classifier. Using the top 10,000 features, Logistic Regression reached RMSE 0.78 and 64% accuracy, while Linear SVC reached 0.81 and 63%. This is 44 percentage points above the five-class random baseline of 20%.
- Trigrams produced essentially no further gain: the best result remained 0.78 RMSE and 64% accuracy. On the held-out test set, Logistic Regression obtained RMSE 0.92 and 54% accuracy, whereas Linear SVC obtained 1.05 and 56%, exposing a validation–generalization gap.
Significance
The study demonstrates that star ratings can be inferred from free-form text at useful, though imperfect, accuracy using transparent linear models. This matters for platforms where users submit text without a rating, and for review ranking or recommendation pipelines. Scientifically, the work clarifies that phrase-level information is more valuable than isolated words, while also showing that rating prediction is affected by class imbalance, heterogeneous user rationales, and overfitting. It therefore provides a reproducible benchmark rather than claiming a complete solution.
Technical Contribution
The principal contribution is a controlled comparison of four feature representations and four classifiers. Feature dimensionality is explicitly reported: 171,846 unigrams, 7,612,422 unigram+bigrams, and 31,677,669 features after adding trigrams. LSI applies SVD, M=U·S·V^T, to the word-review matrix; the singular-value curve levels near 200 topics. Linear SVC uses three-fold cross-validation for C, with C=1.0 selected consistently. The work is an engineering and empirical contribution, not a new learning theory.
Novelty
The paper’s novelty lies in evaluating all 16 representation–classifier combinations for individual Yelp restaurant review ratings while distinguishing this task from business-average rating prediction. Unlike bag-of-opinions or metadata-rich approaches, it deliberately focuses on semantic analysis of review text alone. Its value is a clear, reproducible baseline showing exactly how local phrase features, dimensionality, and linear learners interact on a large real-world corpus.
Limitations
- The study covers restaurants only, so vocabulary, discourse style, and rating behavior may not transfer to hotels, shopping, health, or other Yelp categories.
- The data are skewed: roughly 66% of restaurant reviews receive four or five stars. Moreover, test accuracy fell below validation accuracy, suggesting insufficient regularization and possible selection overfitting.
- The largest representation contains about 32 million features; Python experiments required 36–48 hours per plot and substantial memory.
Future Work
The authors propose POS tagging, spelling correction, and adjective–noun or noun–noun extraction to create more meaningful and efficient n-grams. They also suggest extending LSI beyond 200 dimensions and applying SVD to other n-grams or selected syntactic constructs. Further directions include ordinal logistic regression, nonlinear-kernel SVC, feature mappings, stronger regularization, and broader evaluation across categories and domains.
AI Executive Summary
Online reviews combine a rich explanation with a compressed 1–5 star judgment. Recovering the judgment from text is harder than ordinary sentiment classification: two people can assign two stars for entirely different reasons, while restaurant reviews are heavily concentrated at the high end. As a result, a model must identify not only positive and negative language, but also which aspects mattered to the reviewer.
Nabiha Asghar evaluates this problem on Yelp Dataset Challenge 2014. The source contains 42,153 businesses and 1,125,458 reviews; the study retains 14,403 restaurants and 706,646 restaurant reviews. After lowercasing and removing stop words and punctuation, the author compares TF-IDF unigrams, unigram+bigrams, unigram+bigrams+trigrams, and LSI topic vectors. These are paired with Logistic Regression, Multinomial Naive Bayes, Perceptron, and Linear SVC, producing 16 models. An 80/20 split and three-fold cross-validation are used.
The strongest validation system is Logistic Regression over the top 10,000 unigram+bigrams: RMSE 0.78 and accuracy 64%, versus 63% and 0.81 for Linear SVC. Unigrams alone reach 58% accuracy, while trigrams add almost nothing because exact three-word sequences rarely recur. LSI’s singular values flatten near 200 topics, but its classifiers are weaker. On the held-out test set, Logistic Regression records 54% accuracy and RMSE 0.92; Linear SVC records 56% and 1.05. The paper’s lasting contribution is therefore a transparent, reproducible baseline and a practical lesson: phrase modeling helps, but validation gains do not guarantee robust deployment.
Deep Analysis
Background
Review sites such as Yelp, Amazon, TripAdvisor, and Epinions act as online word-of-mouth and influence purchases, sales, and revenue. Prior work includes Leung et al.’s relative-frequency sentiment dictionary, Qu et al.’s bag-of-opinions, and Ganu et al.’s sentence-level sentiment modeling. Fan and Khademi predicted business-average Yelp ratings using regression and engineered features. This paper instead predicts the rating attached to an individual review using text alone.
Core Problem
Given S={(r_i,s_i)}, where r_i is a review and s_i∈{1,2,3,4,5}, the goal is to learn a mapping from text to the author’s star class. The task is difficult because identical ratings may reflect different aspects, negation or sarcasm can invert word polarity, and approximately 66% of restaurant reviews are four or five stars. Accuracy can therefore conceal minority-class failures.
Innovation
- �� Treats review rating prediction explicitly as five-class classification rather than regression.
- �� Performs a complete 4×4 comparison of representations and learners.
- �� Quantifies scalability: 171,846 unigram features, 7,612,422 unigram+bigrams, and 31,677,669 features with trigrams.
- �� Tests LSI/SVD topic compression and identifies an approximate 200-topic singular-value elbow.
Methodology
- �� Input: business.json and review.json from Yelp; retain restaurant businesses and their reviews.
- �� Preprocessing: lowercase text and remove stop words and punctuation.
- �� Features: construct a word-review matrix and apply TF-IDF; for LSI, decompose M as U·S·V^T and use V’s topic representation.
- �� Learners: Logistic Regression estimates P(s|r); Multinomial Naive Bayes assumes conditional feature independence; Perceptron trains for 50 iterations; Linear SVC maximizes margin with C selected by three-fold validation.
- �� Output and evaluation: predict one of five stars using accuracy and RMSE.
Experiments
The dataset spans Phoenix, Las Vegas, Madison, Waterloo, and Edinburgh. The full corpus has 42,153 businesses and 1,125,458 reviews; the restaurant subset has 14,403 businesses and 706,646 reviews. Eighty percent is used for training and 20% for testing, with three-fold cross-validation inside training. Feature-count sweeps emphasize the top 10,000 TF-IDF features. Linear SVC uses tolerance 0.001; C=1.0 is selected in every reported case.
Results
For unigrams, Logistic Regression is best at RMSE 0.85 and 58% accuracy. Adding bigrams improves it to 0.78 and 64%, with Linear SVC at 0.81 and 63%. Trigrams do not improve performance because their exact sequences are rarely shared across reviews. LSI’s singular values flatten near 200 topics, yet validation accuracy remains only around 0.43 for the weaker settings. On test data, Logistic Regression gives 54% accuracy and 0.92 RMSE; Linear SVC gives 56% and 1.05.
Applications
The approach can assign provisional stars to text-only reviews, support review ranking, and provide a feature for recommendation or summarization systems. Deployment requires category-specific training, calibration, monitoring of high-star imbalance, and sparse linear infrastructure. Because test accuracy is moderate, predicted stars should complement rather than replace explicit user ratings.
Limitations & Outlook
The models rely on lexical and local phrase evidence, so they struggle with sarcasm, long-distance negation, aspect trade-offs, and reviewer-specific standards. Trigram and LSI experiments are computationally constrained; up to roughly 32 million features caused high memory use and 36–48 hours per plot. The validation–test discrepancy indicates that regularization and selection were inadequate. Ordinal objectives, stratified external testing, and domain transfer are necessary next steps.
Plain Language Accessible to non-experts
Imagine a restaurant review as a customer’s order ticket arriving at a kitchen that must guess the number of stars. First, the kitchen cuts the ticket into ingredients: individual words such as “fresh,” then neighboring pairs such as “slow service,” and finally longer three-word pieces. A weighing system gives little importance to words printed on almost every ticket, such as “food,” and more importance to unusual clues that distinguish one ticket from another.
Four judges then compete. One judge estimates how likely each star rating is; another treats every clue as separate evidence; a third repeatedly corrects mistakes; the fourth tries to draw the safest dividing lines between ratings. Combining four ways of preparing the ticket with four judges creates 16 contestants.
The winner on practice data used words plus two-word phrases and Logistic Regression: it was correct 64% of the time, compared with 20% for random guessing among five stars. Three-word phrases added almost nothing because customers rarely repeat the same exact wording. On new test tickets, however, performance fell to 54% for Logistic Regression. The lesson is simple: a recipe that performs well in the kitchen may still need testing with different customers.
ELI14 Explained like you're 14
Suppose you are playing a game where a friend posts only a restaurant comment and you must guess the star rating. “The pizza was amazing, but we waited forever” might mean three stars. Another player could also give three stars because parking was terrible. Same score, totally different reason—so this is not just a simple happy-versus-angry game!
Researchers used hundreds of thousands of Yelp restaurant reviews. They removed distracting marks, looked at single words, and then looked at word pairs such as “slow service.” They also tried three-word chunks and a method that groups words appearing in similar situations. Four computer players competed: one calculated probabilities, one made a simplifying independence assumption, one kept fixing wrong guesses, and one searched for a wide safety gap between classes.
The best practice setup combined single words and pairs with Logistic Regression. It guessed correctly 64% of the time, while random guessing among five ratings would get only 20%. Single words alone reached 58%. Three-word chunks did not help much because two strangers rarely write the exact same three-word phrase. Pretty clever, right?
But the saved test reviews told a more realistic story. Logistic Regression fell to 54%, while Linear SVC got 56%. That is like winning practice rounds but losing points on the real exam because you memorized the practice questions. Future systems should understand sarcasm, “not good,” different restaurant types, and the fact that one star is closer to two stars than to five. Then the guesses could be both smarter and fairer!
Glossary
TF-IDF (term frequency–inverse document frequency)
A weighting scheme that downweights words appearing everywhere and emphasizes words distinctive to a document. Technically, it combines within-document frequency with inverse corpus frequency.
Used to weight unigram, bigram, and trigram matrices.
Unigram
A feature consisting of one token, forming a bag-of-words representation. It captures lexical presence but not word order.
The corpus yields 171,846 unigram features.
Bigram
A feature made from two consecutive tokens. It preserves limited local context, such as modifiers and short negation patterns.
Adding bigrams produces the best reported validation model.
LSI (Latent Semantic Indexing)
A dimensionality-reduction method that maps words and documents into latent topic directions. It is implemented through matrix factorization and can connect related contexts beyond exact word matching.
The paper computes M=U·S·V^T and examines roughly 200 topics.
Linear SVC
A linear support-vector classifier that seeks a large-margin decision boundary while allowing errors controlled by C. It is efficient for sparse, high-dimensional text.
Three-fold validation repeatedly selects C=1.0.
RMSE
The square root of mean squared prediction error. Because errors are squared, predictions far from the true star rating receive extra penalty.
Reported alongside classification accuracy.
Open Questions Unanswered questions from this research
- 1 Bag-of-words features do not reliably resolve sarcasm, negation, or competing aspects. Aspect-aware encoders, ordinal losses, and explanations linked to evidence are needed.
- 2 Generalization beyond restaurants and the five cities remains unknown. Future work needs stratified, temporal, cross-category, and external-domain evaluations.
Applications
Immediate Applications
Automatic rating completion
A review platform can apply Logistic Regression to the top 10,000 TF-IDF unigrams+bigrams when users submit text without stars. The output should be labeled as predicted, calibrated for class imbalance, and audited against human ratings before public display.
Review ranking support
Search and recommendation systems can use predicted stars as one ranking signal to surface potentially positive or negative restaurant comments. Since test accuracy is only 54% for Logistic Regression, it should be combined with explicit ratings, review counts, dates, and confidence estimates.
Long-term Vision
Explainable ordinal review intelligence
Combining ordinal Logistic Regression, aspect-level sentiment, syntactic features, and domain adaptation could produce predictions that state both likely rating and reasons. Obstacles include privacy, reviewer bias, sarcasm, calibration, and changing language; meaningful progress would require broader labeled benchmarks.
Abstract
Review websites, such as TripAdvisor and Yelp, allow users to post online reviews for various businesses, products and services, and have been recently shown to have a significant influence on consumer shopping behaviour. An online review typically consists of free-form text and a star rating out of 5. The problem of predicting a user's star rating for a product, given the user's text review for that product, is called Review Rating Prediction and has lately become a popular, albeit hard, problem in machine learning. In this paper, we treat Review Rating Prediction as a multi-class classification problem, and build sixteen different prediction models by combining four feature extraction methods, (i) unigrams, (ii) bigrams, (iii) trigrams and (iv) Latent Semantic Indexing, with four machine learning algorithms, (i) logistic regression, (ii) Naive Bayes classification, (iii) perceptrons, and (iv) linear Support Vector Classification. We analyse the performance of each of these sixteen models to come up with the best model for predicting the ratings from reviews. We use the dataset provided by Yelp for training and testing the models.