| Download | - View final version: An evaluation of language models for hyperpartisan ideology detection in Persian Twitter (PDF, 634 KiB)
|
|---|
| Link | https://aclanthology.org/2024.eurali-1.8 |
|---|
| Author | Search for: Shayegan, Sahar Omidi; Search for: Nejadgholi, Isar1; Search for: Pelrine, Kellin; Search for: Yu, Hao; Search for: Levy, Sacha; Search for: Yang, Zachary; Search for: Godbout, Jean-François; Search for: Rabbany, Reihaneh |
|---|
| Affiliation | - National Research Council of Canada. Digital Technologies
|
|---|
| Format | Text, Article |
|---|
| Conference | The 2nd Workshop on Resources and Technologies for Indigenous, Endangered and Lesser-resourced Languages in Eurasia @LREC-COLING 2024 (EURALI), May 25, 2024, Torino, Italia |
|---|
| Subject | computational social science; Persian language; ideology prediction |
|---|
| Abstract | Large Language Models (LLMs) have shown significant promise in various tasks, including identifying the political beliefs of English-speaking social media users from their posts. However, assessing LLMs for this task in non-English languages remains unexplored. In this work, we ask to what extent LLMs can predict the political ideologies of users in Persian social media. To answer this question, we first acknowledge that political parties are not well-defined among Persian users, and therefore, we simplify the task to a much simpler task of hyperpartisan ideology detection. We create a new benchmark and show the potential and limitations of both open-source and commercial LLMs in classifying the hyper-partisan ideologies of users. We compare these models with smaller fine-tuned models, both on the Persian language (ParsBERT) and translated data (RoBERTa), showing that they considerably outperform generative LLMs in this task. We further demonstrate that the performance of the generative LLMs degrades when classifying users based on their tweets instead of their bios and even when tweets are added as additional information, whereas the smaller fine-tuned models are robust and achieve similar performance for all classes. This study is a first step toward political ideology detection in Persian Twitter, with implications for future research to understand the dynamics of ideologies in Persian social media. |
|---|
| Publication date | 2024-05 |
|---|
| Publisher | ELRA Language Resource Association
International Committee on Computational Linguistics (ICCL) |
|---|
| Licence | |
|---|
| In | |
|---|
| Language | English |
|---|
| Peer reviewed | Yes |
|---|
| Export citation | Export as RIS |
|---|
| Report a correction | Report a correction (opens in a new tab) |
|---|
| Record identifier | 1fffe41c-58d3-4d96-b67c-88d0fc6a8b36 |
|---|
| Record created | 2024-06-03 |
|---|
| Record modified | 2024-06-04 |
|---|
