explainable-review-detection

Explainable detection of AI-generated product reviews

My final-year dissertation for a BSc in Computer Science at Lancaster University (2026).

I fine-tuned DistilBERT and RoBERTa to tell human-written product reviews apart from AI-generated ones, then built the parts a moderation team would actually need around them: word-level explanations with LIME, a plain-English report for each flagged review, and a triage step that sends uncertain cases to a person rather than acting on them automatically.

Read the paper (PDF) · Project page

Results

Evaluated on 8,105 held-out reviews (baseline on its own 20% split, see limitations).

Model	Accuracy	Macro F1	Human recall	AI recall
TF-IDF + logistic regression	86.44%	0.86	0.87	0.85
DistilBERT	97.91%	0.979	0.965	0.993
RoBERTa	96.87%	0.969	0.939	0.999

I chose DistilBERT, even though RoBERTa is the larger model and catches slightly more fakes. In moderation, the costly mistake is accusing a real customer. DistilBERT wrongly flags 3.5% of genuine reviews, RoBERTa 6.1% and the baseline about 13%.

The two transformers disagreed on 244 test reviews (3%). DistilBERT was right on 194 of them and RoBERTa on 50. The most common disagreement was a short, casual human review that RoBERTa flagged as AI.

Confusion matrices for DistilBERT and RoBERTa on 500 test reviews

Explaining a wrong answer

Accuracy alone doesn’t help a moderator decide whether to trust a single flag. This is a human-written review that DistilBERT flagged as AI with 98% confidence. LIME shows which words pushed it each way:

LIME explanation of a false positive

No phrase here obviously sounds like AI. Ordinary words (“my”, “year”) add up to a confident but wrong verdict. The explanation gives the moderator something concrete to question, instead of a bare score.

ModeratorReport.py turns the prediction and the LIME words into text a non-technical reviewer can act on:

MODERATOR REPORT
Result:     AI-GENERATED | HUMAN-WRITTEN
Confidence: <model confidence>

Reason: <why, naming the words that most influenced the model>
Advice: remove / check manually (if confidence < 80%) / no action needed

TriageReport.py sorts reviews into high-risk, low-risk and ambiguous queues using a 0.70 confidence threshold. On 100 test reviews it was 95% accurate. However, three of the four false positives landed in the high-risk queue with over 89% confidence, which a threshold can’t catch. That’s why the system is designed to support moderators, not replace them.

Checking for shortcuts

Human reviews in this dataset are about 30% longer than AI ones (397 vs 305 characters). A model could score well just by measuring length. length_analysis.py tests this. The correlation between review length and predicted class is only −0.22, and correct predictions sit near 1.0 confidence while most errors fall well below.

Length analysis

The baseline’s coefficients (Top10Words.py) show a stylistic difference in the data. AI reviews lean on generic evaluative words like problem, reason and materials. Human reviews hedge and tell stories, with words like actually, instead and maybe.

Repository layout

common.py                  dataset, 70/10/20 split, checkpoint paths, inference wrapper
baseline/
  BaselineClassifier.py    TF-IDF + logistic regression
  Top10Words.py            most discriminative words per class
transformer/
  train.py                 shared fine-tuning loop (Hugging Face Trainer)
  DistilBERT.py            distilbert-base-uncased, lr 2e-5
  roBERTa.py               roberta-base, lr 1e-5
analysis/
  DatasetAnalysis.py       class balance and length distribution
  cm.py                    side-by-side confusion matrices
  compare.py               every test review where the two models disagree
  length_analysis.py       length-shortcut check
explainability/
  DistilBERTLIME.py        LIME explanation for one review (HTML)
  ContradictionLIME.py     LIME on both models for a disagreement case
  ModeratorReport.py       plain-English report
  TriageReport.py          confidence-based triage to CSV
paper/                     the dissertation

Running it

Python 3.10+. Run everything from the repository root; the dataset downloads from Hugging Face on first use.

pip install -r requirements.txt

python -m baseline.BaselineClassifier      # about a minute on a laptop CPU
python -m transformer.DistilBERT           # GPU recommended; saves ./distilbert_results/best
python -m transformer.roBERTa

python -m analysis.compare
python -m explainability.ModeratorReport "Great product, works exactly as described."

The training scripts pick CUDA, Apple MPS or CPU automatically and adjust the batch size to match. I developed locally and ran the full training on Lancaster’s HEX GPU cluster. To use checkpoints stored somewhere else, set DISTILBERT_CHECKPOINT / ROBERTA_CHECKPOINT.

Limitations

Generalisation is untested. The AI-written reviews in the dataset (Salminen et al., 2022) were generated with GPT-2, and the human reviews come from Amazon. The models will have learned GPT-2 artefacts, and I haven’t tested them on text from newer LLMs.
The baseline uses a different split. It uses scikit-learn’s 80/20 split and the transformers use a 70/10/20 split, so the test sets differ slightly. The 11-point gap is far larger than that difference could explain.
LIME is approximate. It explains a local surrogate model rather than the network itself, and its output varies a little between runs. Comparing it with SHAP or Integrated Gradients is the obvious next step.

Stack

Python · PyTorch · Hugging Face Transformers & Datasets · scikit-learn · LIME · pandas · matplotlib / seaborn