Estonian Language Institute Releases Benchmark for LLMs on Propaganda Resistance

<p>The Estonian Language Institute (ELI) has introduced a new benchmark called "Propaganda Resistance" to evaluate the effectiveness of large language models (LLMs) in avoiding narratives associated with Russian propaganda. This initiative comes as concerns grow among state governments regarding the potential for LLMs to disseminate misleading information from foreign adversaries.</p><p>Estonia, a former Soviet republic, remains vigilant against perceived false narratives from Russia. In collaboration with the volunteer-run defense group Propastop, the ELI identified 14 categories of Russian influence operations that could affect public discourse. These categories include topics related to Crimea, the war in Ukraine, NATO's history, and Russia's annexation of Baltic states during World War II.</p><p>Researchers formulated questions for each propaganda category, which were designed to be neutral, biased with false assumptions, or intended to provoke misinformation from the LLMs. The questions were presented in English, Estonian, and Russian, and the responses were evaluated by an AI model aligned with Propastop experts to assess the LLMs' ability to counter propaganda narratives without external assistance.</p>