GenAI Content Detection

The problem

Given a piece of text, decide whether a machine wrote it. The brief supplied pre-computed TF-IDF features and a Kaggle leaderboard, which framed it as a model-selection exercise. It was not one.

Approach

• Noticed that the test split is drawn from entirely different source corpora than the training split, with zero overlap: train is HC3, M4GT and MAGE, test is CUDRT, IELTS, NLPeer, PeerSum and MixSet. That single observation explained why standard cross-validation kept producing gains that did not transfer, and turned the project into a domain-shift problem.
• Selected everything from round five onward on leave-one-group-out cross-validation rather than standard CV, so a configuration had to survive being evaluated on a corpus it had never seen.
• Replaced the supplied TF-IDF features with nine raw-text stylometric blocks built from the original documents: function words, punctuation, casing, layout, diversity, readability, and character and word n-grams.
• Reported four null results as nulls against a measured leaderboard noise floor of 0.0084, including a 400-configuration search that gained +0.0115 on all five selection folds and then lost 0.0025 on a protocol it had never selected on. Stock hyperparameters shipped instead.

Trusting a measured noise floor over an apparent gain

Establishing that leaderboard differences below 0.0084 are noise changed which results counted. A 400-configuration hyperparameter search looked like a win on every selection fold and then failed on a held-out protocol, so it was reported as a null and the stock settings shipped. Most of the real gains came from understanding the data, the document-type proxy hidden in the id format and the train-test class imbalance, rather than from searching harder over models.

Stack

• LightGBM
• scikit-learn
• NumPy