Hate Speech Classification

The problem

Binary classification on short text: does it contain hate speech. The course supplied pre-computed TF-IDF features, and a prior group's report had already flagged a suspicious number worth investigating before trusting anything: a Random Forest that scored F1 = 0.9973 on validation.

Approach

• Ran EDA before training anything: class balance, text-length distribution, and TF-IDF sparsity, because each of those decides a downstream choice (whether to stratify the split, what an honest baseline even looks like).
• Used a scree plot of PCA explained variance to choose the component count at the elbow, rather than guessing round numbers like 2000, 1000, 500 or 100 the way the prior group had.
• Noticed that Kaggle's public leaderboard score equals the fraction of the test set predicted as label 0, which means submitting an all-zero baseline reveals the class balance for free but the leaderboard score itself is not a real performance signal. Local stratified validation was trusted instead.
• Locked the model class, hyperparameters and decision threshold before ever touching the sealed holdout, then ran that holdout exactly once.

Catching an evaluation artefact before it became the headline number

A prior group's report led with F1 = 0.9973 from a Random Forest on validation. That number did not survive a single honest, sealed holdout: the trustworthy out-of-sample F1 is 0.653, and 0.9973 is treated here as an evaluation artefact rather than a real result. The more useful finding came from what actually moved the real number: on this imbalanced TF-IDF problem, calibrating the decision threshold (to tau = 0.4011) delivered the entire ~0.021 F1 lift over baseline, while a 9-hour elasticnet hyperparameter sweep on the same model produced changes indistinguishable from noise. The generalisation gap between dev cross-validation (0.657) and the sealed holdout (0.653) was 0.004, the smallest of the four models compared, which is what made the number worth trusting.

Stack

• scikit-learn
• TF-IDF
• LightGBM