Semantic shift method uncovers community slang in fine-tuned LLMs

A new paper from researchers at an undisclosed institution introduces an unsupervised technique to detect slang, unique entities, and folklore within online communities by analyzing semantic shift in fine-tuned language models. The method works by fine-tuning a pretrained DistilRoBERTa model on text corpora from specific Reddit subreddits (r/Technology, r/Gaming, r/WorldofWarcraft). They then compute the cosine similarity between each word's embedding in the base model and the fine-tuned model. Words with the lowest similarity (bottom 10%)—indicating the highest semantic shift—are found to have unique community-specific meanings, such as insider jargon and references. Conversely, words in the top 10% of similarity carry universal semantics.

The approach, detailed in a 6-page paper on arXiv, requires no labeled data or manual annotation, making it scalable for any online community. By modeling the distribution of cosine similarities over the entire lexicon, the researchers successfully isolated terms that are significant only within the target subreddit. For example, in r/WorldofWarcraft, words like 'tank' or 'healer' shifted significantly compared to their general usage. This method could be applied to improve chatbots, recommendation systems, and moderation tools that need to understand community-specific language. The work also provides insights into how fine-tuning encodes cultural knowledge, with implications for domain adaptation and social network analysis.