abstract
- This study introduces an innovative framework integrating Small Language Models (SLMs) with concept-based analysis to evaluate medical translation accuracy. Using medical concept embeddings (Cui2vec) and Unified Medical Language System (UMLS) identifiers, we assessed English-to-Spanish translations beyond traditional metrics. Our comparison of similarity measures revealed that cosine similarity and Pearson correlation best preserved semantic accuracy. Domain-specific models (Meditron, Meerkat) demonstrated superior semantic consistency compared to general-purpose models like Llama-2. These findings establish SLMs as efficient alternatives to Large Language Models for medical translation evaluation, particularly beneficial for low-resource languages. By focusing on conceptual meaning preservation rather than lexical equivalence, this approach enhances cross-lingual medical communication while maintaining clinical precision.