AuthorDriver’s stress detection is a critical research area to help reduce the likelihood of traffic accidents and driver’s health complexities due to prolonged stress. Previous work in this area is heavily based on traditional machine learning models to classify the driver’s stress levels using handcrafted features extraction techniques. Extracting the best features using these approaches is always a challenging task. Recently, deep learning techniques have emerged for constructing reliable features automatically and classifying the classes with high accuracy. However, large deep learning models face gradient exploding or vanishing problems. Moreover, acquiring a large dataset for training an entire network from scratch is also a challenging task. This paper is based on the deep transfer learning technique to avoid these problems and to reduce computational cost and time. Seven models are proposed for real-world driver’s stress levels detection using Electrocardiogram (ECG) signals. Different Convolutional Neural Network (CNN)-based pre-trained networks are used to classify the driver’s three stress levels. The time-frequency ECG components for the three stress levels are obtained as scalogram images using a normalized Continuous Wavelet Transform (CWT) filter bank and Morse wavelet. Results show that Model 5 based on Xception outperforms the GoogLeNet, DarkNet-53, ResNet-101, InceptionResNetV2, DenseNet-201, and InceptionV3 based models by 11.32%, 11.32%, 9.45%, 7.54%, 5.66%, and 1.88% respectively and achieves 98.11% overall validation accuracy. Ranking estimation using fuzzy logic approach shows that Xception based Model 5 achieves the highest rank for driver’s high and medium stress levels, while DenseNet-201 based Model 4 achieves the highest rank for low-stress level detection among the other models.