In today’s leaf disease detection, the accuracy of recognition has never been of such importance as it is now. In this aspect, leaf disease recognition method based on machine learning relies heavily on the size of the region of interest and the dispersion of lesions. Professional instrument for leaf disease detection remains a challenging task in accuracy and convenience. A new lightweight model based on advanced residual network and attention mechanism for extracting more accurate region of interest and the lesion, SE-VRNet, was proposed. The proposed SE-VRNet incorporated deep variant residual network (VRNet) and a squeeze-and-excitation (SE) module with attention mechanism, in order to solve the problem that the feature extraction was difficult due to the dispersed location of the leaf disease. The accuracy of top-1 and top-3 obtained by the model SE-VRNet on NewData is 99.73% and 99.98%, respectively, and the accuracy of top-1 and top-3 obtained by the model on SelfData is 95.71% and 99.89%, respectively. The experimental results on the datasets of PlantVillage, OriData, NewData and SelfData were better than other state-of-the-art methods, demonstrating the effectiveness and feasibility of the proposed SE-VRNet in identifying leaf diseases with mobile devices.