Politicians vs ChatGPT
A study of presuppositions in French and Italian political communication
DOI:
https://doi.org/10.62408/ai-ling.v1i1.5Keywords:
Implicit communication, Presupposition, AI-generated language, Political communication, French, ItalianAbstract
This paper aims to provide a comparison between texts produced by French and Italian politicians on polarizing issues, such as immigration and the European Union, and their chatbot counterparts created with ChatGPT 3.5. In this study, we focus on implicit communication, in particular on presuppositions and their functions in discourse, which have been considered in the literature as a potential linguistic feature of manipulation. This study also aims to contribute to the emerging literature on the pragmatic competences of Large Language Models. Our results show that, on average, ChatGPT-generated texts contain more questionable presuppositions than the politicians’ texts. Furthermore, most presuppositions in the former texts show a different distribution and different discourse functions compared to the latter. This may be due to several factors inherent in the ChatGPT architecture, such as a tendency to be verbose and repetitive in longer texts, as exemplified by the occurrence of political slogans mainly formed by change-of-state verbs as presupposition triggers (e.g., dobbiamo costruire il nostro futuro, ‘we must build our future’).
References
Agresti, Alan. 2019. An Introduction to Categorical Data Analysis (3rd edition). Hoboken, NJ: John Wiley & Sons.
Antoun, Wissam & Mouilleron, Virginie & Sago, Benoît & Seddah, Djamé. 2023. Towards a robust detection of Language Model-Generated text: Is ChatGPT that easy to detect? In Actes de CORIA-TALN 2023. Actes de la 30e Conférence sur le Traitement Automatique des Langues Naturelles (TALN), volume 1: travaux de recherche originaux, 14−27. ATALA.
Artstein, Ron & Poesio, Massimo. 2008. Inter-Coder agreement for Computational Linguistics. Computational Linguistics 34(4). 555−596. http://doi.org/10.1162/coli.07-034-R2
Barattieri di San Pietro, Chiara & Frau, Federico & Mangiaterra, Veronica & Bambini, Valentina. 2023. The pragmatic profile of ChatGPT: Assessing the communicative skills of a conversational agent. In Bambini, Valentina & Barattieri di San Pietro, Chiara (eds), Multidisciplinary perspectives on ChatGPT and other Artificial Intelligence Models / Prospettive multidisciplinari su ChatGPT e altri modelli di intelligenza artificiale, Sistemi Intelligenti 35 (2), 379−400. https://doi.org/10.1422/108136
Bai, Hui & Voelkel, Jan & Eichstaedt, Johannes C. & Willer, Robb. 2023. Artificial intelligence can persuade humans on political issues. OSF Preprints. https://doi.org/10.31219/osf.io/stakv
Bender, Emily M. & Gebru, Timnit & McMillan-Major, Angelina & Shmitchell, Shmargaret. 2021. On the dangers of stochastic parrots: Can Language Models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ‘21). Association for Computing Machinery, 610−623. https://doi.org/10.1145/3442188.3445922
Brocca, Nicola & Garassino, Davide & Masia, Viviana. 2016. Politici in rete o nella rete dei politici? L’implicito nella comunicazione politica italiana in Twitter. Philologie im Netz, PhiN-Beiheft 11, 66−79. http://web.fu-berlin.de/phin/beiheft11/b11t06.pdf (last accessed on 26.05.2024)
Brown, Tom B. & Mann, Benjamin & Ryder, Nick & Subbiah, Melanie & Kaplan, Jared & Dhariwal, Prafulla & Neelakantan, Arvind & Shyam, Pranav & Sastry, Girish & Askell, Amanda et al. 2020. Language Models are Few-Shot learners. ArXiv. https://doi.org/10.48550/arXiv.2005.14165
Buchanan, Ben & Lohn, Andrew & Musser, Micah & Sedova, Katerina. 2021. Truth, Lies, and Automation. How Language Models Could Change Disinformation. Georgetown: Center for Security and Emerging Technology. https://cset.georgetown.edu/publication/truth-lies-and-automation (last accessed on 26.05.2024). https://doi.org/10.51593/2021CA003
Burtell, Matthew & Woodside, Thomas. 2023. Artificial influence: An Analysis of AI-Driven persuasion. ArXiv. https://doi.org/10.48550/arXiv.2303.08721
Cai, Zhenguang G. & Duan, Xufeng & Haslett, David A. & Wang, Shuqi & Pickering, Martin J. 2023. Does ChatGPT resemble humans in language use? ArXiv. https://doi.org/10.48550/arXiv.2303.08014
Chaka, Chaka. (2023). Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools. Journal of Applied Learning and Teaching 6(2). https://doi.org/10.37074/jalt.2023.6.2.12
Cominetti, Federica & Gregori, Lorenzo & Lombardi Vallauri, Edoardo & Panunzi, Alessandro. 2022. IMPAQTS: un corpus di discorsi politici italiani annotato per impliciti linguistici. In Cresti, Emanuela & Moneglia, Massimo (eds), Corpora e Studi Linguistici. Atti del LIV Congresso Internazionale di Studi della Società di Linguistica Italiana, 151−164. Milano: Officinaventuno. https://doi.org/10.17469/O2106SLI000009
De Cesare, Anna-Maria. 2023. Assessing the quality of ChatGPT’s generated output in light of human-written texts. A corpus study based on textual parameters. CHIMERA: Romance Corpora and Linguistic Studies 10. 179–210. https://revistas.uam.es/chimera/article/view/17979 (last accessed on 26.05.2024)
Domaneschi, Filippo & Canal, Paolo & Masia, Viviana & Lombardi Vallauri, Edoardo & Bambini, Valentina. 2018. N400 and P600 modulation in presupposition accommodation: The effect of different trigger types. Journal of Neurolinguistics 45. 13–35. https://doi.org/10.1016/j.jneuroling.2017.08.002
Downs, Anthony. 1957. An economic theory of political action in democracy. Journal of Political Economy 65(2). 135−150. https://doi.org/10.1086/257897
Ekin, Sabit. 2023. Prompt Engineering for ChatGPT: A Quick Guide to Techniques, Tips, and Best Practices. TechRxiv. May 04, 2023. https://doi.org/10.36227/techrxiv.22683919.v2
Field, Andy & Miles, Jeremy & Field, Zoë. 2012. Discovering Statistics Using R. London: Sage.
Gamer, Matthias & Lemon, Jim. 2019. Irr: Various Coefficients of Interrater Reliability and Agreement (v. 0.84.1) [Computer software]. https://CRAN.R-project.org/package=irr (last accessed on 06.06.2024)
Garassino, Davide, & Brocca, Nicola & Masia, Viviana. 2022. Is implicit communication quantifiable? A corpus-based analysis of British and Italian political tweets. Journal of Pragmatics 194. 9–22. https://doi.org/10.1016/j.pragma.2022.03.024
Garassino, Davide & Masia, Viviana & Brocca, Nicola. 2019. Tweet as you speak. The role of implicit strategies and pragmatic functions in political communication: Data from a diamesic comparison. Rassegna Italiana di Linguistica Applicata 2/3. 187–208. https://doi.org/10.5167/uzh-220284
Goldstein, Josh A. & Chao, Jason & Grossman, Shelby & Stamos, Alex & Tomz, Michael. 2023. Can AI write persuasive propaganda? SoCArXiv. https://doi.org/10.31235/osf.io/fp87b
Goldstein, Josh A. & Sastry, Girish & Musser, Micah & Gentzel, Matthew, & Sedova, Katerina. 2023. Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations. Stanford Internet Observatory, OpenAI, and Georgetown University’s Center for Security and Emerging Technology. https://fsi.stanford.edu/publication/generative-language-models-and-automated-influence-operations-emerging-threats-and (last accessed on 03.05.2024)
Grisot, Cristina. 2017. A quantitative approach to conceptual, procedural and pragmatic meaning: evidence from inter-annotator agreement. Journal of Pragmatics 111. 245−262. https://doi.org/10.1016/j.pragma.2017.06.020
Hoek, Jet & Scholman, Merel C.J. 2017. Evaluating discourse annotation: some recent insights and new approaches. In Proceedings of the 13th Joint ISO-ACL Workshop on Interoperable Semantic Annotation (Isa-13), 1−13. https://aclanthology.org/W17-7401/ (last accessed on 18.01.2024)
Hoek, Jet & Scholman, Merel C.J. & Sanders, Ted J.M. 2021. Is there less annotator agreement when the discourse relation is underspecified?. In Proceedings of the First Workshop on Integrating Perspectives on Discourse Annotation, 1–6. Association for Computational Linguistics. http://doi.org/10.22028/D291-38635
Kiparsky, Carol & Kiparsky, Paul. 1971. Fact. In Steinberg, Danny D. & Jakobovitz, Leon A. (eds), Semantics: An Interdisciplinary Reading, 345−369. Cambridge: Cambridge University Press.
Kreps, Sarah R. & McCain, Miles & Brundage, Miles. 2022. All the news that’s fit to fabricate: AI-Generated text as a tool of media misinformation. Journal of Experimental Political Science 9(1). 104−117. https://doi.org/10.1017/XPS.2020.37
Landis, J. Richard & Koch, Gary G. 1977. The measurement of observer agreement for categorical Data. Biometrics 33(1). 159−174. https://doi.org/10.2307/2529310
Levshina, Natalia. 2015. How to do Linguistics with R. Data exploration and Statistical Analysis. Amsterdam: John Benjamins. https://doi.org/10.1075/z.195
Loftus, Elizabeth F. 1975. Leading questions and the eyewitness report. Cognitive Psychology 7(4). 560−572. https://doi.org/10.1016/0010-0285(75)90023-7
Langford, Jennifer & Holmes, Virginia M. 1979. Syntactic presupposition in sentence comprehension. Cognition 7(4). 363–383. https://doi.org/10.1016/0010-0277(79)90022-2
Lombardi Vallauri, Edoardo. 2009. La struttura informativa. Forma e funzione negli enunciati linguistici. Roma: Carocci. http://hdl.handle.net/11590/179329
Lombardi Vallauri, Edoardo. 2016. The “exaptation” of linguistic implicit strategies. Springer Plus 5, 1106. https://doi.org/10.1186/s40064-016-2788-y
Lombardi Vallauri, Edoardo. 2019. La lingua disonesta. Bologna: il Mulino.
Lombardi Vallauri, Edoardo. 2021. Manipulative shallow processing induced by Presuppositions and Topics: Theoretical perspectives and experimental evidence. Frontiers in Communication 6. https://doi.org/10.3389/fcomm.2021.610807
Lombardi Vallauri, Edoardo & Masia, Viviana. Implicitness impact: measuring texts. Journal of Pragmatics 61, 161–184. https://doi.org/10.1016/j.pragma.2013.09.010
Loewen, Shawn & Plonsky, Luke. 2016. An A-Z of Applied Linguistics Research Methods. London: Palgrave.
Masia, Viviana & Canal, Paolo & Ricci, Irene & Lombardi Vallauri, Edoardo & Bambini, Valentina. 2017. Presupposition of new information as a pragmatic garden path: Evidence from Event-Related Brain Potentials. Journal of Neurolinguistics 42. 31–48. https://doi.org/10.1016/j.jneuroling.2016.11.005
Masia, Viviana & Garassino, Davide & Brocca, Nicola & de Saussure, Louis. 2023. Recalling presupposed information. Evidence from the online processing of presuppositions in political tweets. Pragmatics & Cognition 30:1. 92–119. https://doi.org/10.1075/pc.22011.mas
Mercier, Hugo & Sperber, Dan. 2017. The Enigma of Reason. Boston, MA: Harvard University Press.
OpenAI. 2021. ChatGPT (Version GPT-3.5) [Computer software]. https://openai.com/ (last accessed on 06.05.2023).
Pratim Ray, Partha. 2023. ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems 3. 121−154. https://doi.org/10.1016/j.iotcps.2023.04.003
Qiu, Zhuang & Duan, Xufeng & Cai, Zhenguang G. 2023. Pragmatic Implicature Processing in ChatGPT. PsyArXiv. https://osf.io/preprints/psyarxiv/qtbh9 (last accessed on 24.05.2024). https://doi.org/10.31234/osf.io/qtbh9
R Core Team. 2022. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna (Version 4.2.2) [Computer software]. https://www.R-project.org/ (last accessed on 06.06.2024)
Reboul, Anne. 2011. A Relevance-Theoretic Account of the Evolution of Implicit Communication. Studies in Pragmatics 13. 1−19. http://pragmatics.gr.jp/content/files/SIP_013/SIP_13_Reboul.pdf (last accessed on 26.05.2024).
Reboul, Anne. 2017. Communication and Cognition in the Evolution of Language. Oxford: Oxford University Press. https://doi.org/10.1093/acprof:oso/9780198747314.001.0001
Sbisà, Marina. 2007. Detto non detto. Le forme della comunicazione implicita. Roma-Bari: Editori Laterza.
Schwarz, Florian. 2015. Presuppositions vs. asserted content in online processing. In Schwarz, Florian (ed.), Experimental Perspectives on Presupposition: Studies in Theoretical Psycholinguistics, 89−108. Dordrecht: Springer. https://doi.org/10.1007/978-3-319-07980-6
Sperber, Dan & Clément, Fabrice & Heintz, Cristophe & Mascaro, Olivier & Mercier, Hugo & Origgi, Gloria & Wilson, Deirdre. 2010. Epistemic vigilance. Mind & Language 25(4). 359–393. https://doi.org/10.1111/j.1468-0017.2010.01394.x
Spooren, Wilbert & Degand, Liesbeth. 2010. Coding coherence relations: Reliability and validity. Corpus Linguistics and Linguistic Theory 6(2). 241−266. https://doi.org/10.1515/cllt.2010.009
Stalnaker, Robert C. 1973. Pragmatic Presuppositions. In Munitz, Milton Karl & Unger, Peter K. (eds), Semantics and Philosophy, 471−482. New York: New York University Press.
Stalnaker, Robert C. 2002. Common Ground. Linguistics and Philosophy 25. 701−721. https://doi.org/10.1023/A:1020867916902
Vach, Werner & Gerke, Oke. 2023. Gwet’s AC1 is not a substitute for Cohen’s kappa. A comparison of basic properties. MethodsX 10, 102212. https://doi.org/10.1016/j.mex.2023.102212
Venables, William N. & Ripley, Brian D. 2002. Modern Applied Statistics with S (3rd edition). New York: Springer.
White, Jules & Fu, Quchen & Sandborn, Michael & Olea, Carlos & Gilbert, Henry & Elnashar, Ashraf & Spencer-Smith, Jesse & Schmidt, Douglas C. 2023. A Prompt Pattern catalog to enhance Prompt Engineering with ChatGPT. ArXiv, https://doi.org/10.48550/arXiv.2302.11382
Wickham, Hadley. 2016. ggplot2: Elegant Graphics for Data Analysis. New York: Springer.
Zou, Andy & Wang, Zifan & Carlini, Nicholas & Nasr, Milad & Kolter, J. Zico & Fredrikson, Matt. 2023. Universal and transferable adversarial attacks on aligned Language Models. ArXiv, https://doi.org/10.48550/arXiv.2307.15043
Zeileis, Achim & Meyer, David & Hornik, Kurt. 2007. Residual-based shadings for visualizing (conditional) independence. Journal of Computational and Graphical Statistics 16 (3). 507−525. https://doi.org/10.1198/106186007X237856
Downloads
Published
Versions
- 2024-07-08 (2)
- 2024-07-04 (1)
How to Cite
Issue
Section
License
Copyright (c) 2024 Davide Garassino, Viviana Masia, Brocca Nicola, Alice Delorme Benites
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
