[P] XGBoost + TF-IDF for emotion prediction — good state accuracy but struggling with intensity (need advice)

A developer's machine learning project to predict emotional states from journal entries has achieved promising classification results but reveals significant challenges in predicting emotional intensity. Using a dataset of approximately 1,200 noisy, short journal reflections, the developer employed a TF-IDF (Term Frequency-Inverse Document Frequency) vectorizer for text processing with up to 1,500 features and combined it with scaled metadata—including stress levels, energy, sleep hours, and engineered features like stress-energy interactions. An XGBoost classifier successfully predicts one of six emotional states with 66-67% accuracy, though confusion persists between semantically similar classes.

However, the model's performance on predicting the intensity of those emotions (on a 1-5 scale) remains poor. Treating intensity as a classification task yielded only 21% accuracy, and switching to an XGBoost regressor only reduced the Mean Absolute Error (MAE) to around 1.22—meaning predictions are off by more than one full intensity point on average. The developer notes that text features appear to dominate the model, with metadata and custom feature engineering providing minimal improvement, and is now seeking advice on better approaches for this ordinal prediction problem.

The core challenge lies in the dataset's inherent noise and subjectivity. With only 1,200 samples, the model must generalize from limited, personal reflections where labels for both 'state' and 'intensity' are inherently ambiguous. The developer's post has sparked discussion on potential next steps, including experimenting with different models like LightGBM, exploring alternative methods for combining text and structured data, and investigating feature engineering techniques specifically tailored for capturing emotional intensity rather than just category.