Evaluating Language Dependency in Large Language Models: A Study on Programming Queries in English and Spanish Academic Article in Scopus uri icon

abstract

  • As the integration of Artificial Intelligence tools, such as large language models (LLMs), into computing education increases, understanding their impact on students¿ learning becomes crucial. According to recent research, LLMs perform well when processing input in the English language. Still, they struggle when processing input in other languages or inputs containing non-English syntax or symbols, such as different languages and programming queries. Therefore, this study evaluates whether programming queries, particularly code generation queries in Spanish, a widely spoken language other than English, present challenges similar to those in code generation tasks compared to English queries. By doing this, this study aims to identify accuracy differences in the code generated by LLMs (Codex and Copilot) for English and Spanish input on a set of programming problems sourced from LeetCode. The study compares the performance of LLMs on three complexity levels of tasks, including basic, medium, and advanced code generation tasks. The results show that both Codex and Copilot show a significant decline in accuracy for Spanish as compared to English, particularly as task complexity increases from basic to advanced level. The Codex shows a significant decline in accuracy for Spanish inputs (85%) compared to English (92%). Similarly, Copilot shows a significant increase in accuracy for English inputs (93%) compared to Spanish (87%), with higher error rates across syntax, runtime, and logical errors in both. By comparing the results across multiple languages, the findings show that LLMs perform better on English-language inputs for code generation. Additionally, it demonstrated that Copilot also has superior adaptability and reliability in handling multilingual programming tasks compared to Codex. These results serve as a foundation and further emphasize the need for improvement in multilingual capabilities, as well as the language-dependent limitations of LLMs. © 2025 Latin American and Caribbean Consortium of Engineering Institutions. All rights reserved.

publication date

  • January 1, 2025