The COVID-19 pandemic disrupted education, especially in STEM, causing losses in foundational math that are observable until now. Challenges such as reduced teacher guidance affected students' preparedness for higher education. The proliferation of educational tools designed to improve students' knowledge also began, including IA-powered ones. Studies highlight the need for innovative strategies that integrate these tools, especially for first-year STEM students. This study explores AI tools to improve STEM students' understanding of mathematics topics. The primary aim was to determine whether AI interventions could improve mathematical performance and serve as adequate educational resources, using a validated test. A mixed-method cohort design was used. Quantitative data were obtained from a 10-item math test administered to first-year engineering students, and psychometric analyses of reliability, difficulty, and discrimination were conducted. Afterwards, AI tools (ChatGPT, Jungle, and ChatPDF) were used in specific courses. Intervention 1 was in the first semester ("Electricity and Magnetism"), and Intervention 2 was in the second ("Modeling Engineering and Science"). Pre- and post-tests measured learning gains, and surveys captured student perceptions. Reliability analysis indicated moderate internal consistency; some items showed low discriminative ability, suggesting the need for revision. After the AI intervention, post-intervention means scores increased relative to the pre-intervention exam, and score variability decreased in both interventions, with larger gains in Intervention 2. Intervention 1 showed gender differences, favoring males, but Intervention 2 did not. Differences in test scores suggest that the interventions were supportive in improving math performance. ChatGPT was used by most students; Jungle and ChatPDF were less widely used, mainly due to limited awareness. Findings suggest that AI tools can effectively support mathematics learning when well integrated and when students are familiar with them. Using the procedure outlined in this work, students' gains in mathematics learning can be validated and quantitatively evaluated after using these tools.
