Cross-Age Face Recognition (CAFR) remains one of the most challenging tasks in the field of face recognition, as the aging process significantly affects the facial appearance. Another limitation is the insufficient special datasets that cover a wide range of ages. The key to tackle this problem is to separate the variations caused by aging process from facial features and obtain stable person-specific features. Specifically, we proposed a novel end-to-end CNN method called Age Adversarial Convolutional Neural Network (AA-CNN) with parallel network architecture. By adversarial training in Age Discrimination Network (ADN), the features extracted by AA-CNN are invariant to age variation, while remaining identity discriminative via joint training in Identity Recognition Network (IRN). Furthermore, we adopt a pyramid architecture of feature fusion to assist the ADN in adversarial training to obtain effective age-elated information. The training datasets of AA-CNN are labeled by identity or age, and there is no need to search for the datasets with both identity and age labels. Extensive experiments have been conducted on the challenging aging face datasets, including FG-NET dataset, MORPH Album 2 dataset, Cross-Age Celebrity dataset, and Cross-Age LFW dataset, which demonstrates the superiority and effectiveness of the AA-CNN model.