The cerebral substratum of age-related cognitive decline was evaluated in an elderly-cohort followed for 12 years (n=306). Participants, free of dementia, received neuropsychological assessments every two years and an MRI exam at baseline and four years later. Cognitive decline was evaluated on two broadly used tests to detect dementia: the Free and Cued Selective Reminding Test (FCSRT), a verbal episodic memory task, and the Isaacs Set Test (IST), a semantic fluency task. Using voxel-based approach, the relationship between cognitive decline with 1/ baseline grey matter volumes and 2/ grey matter volume loss between the two scans was explored. Baseline volumes analysis revealed that FCSRT and IST declines were both associated with lower volumes of the medial temporal region. Volumes loss analysis confirmed that both declines are related to medial temporal lobe atrophy and revealed that FCSRT decline was specifically associated with atrophy of the posterior cingulate cortex whereas IST decline was specifically related to temporal pole atrophy. These results suggest that cognitive decline across aging is firstly related to structural modifications of the medial temporal lobe, followed by an atrophy in the posterior midline structures for episodic memory and an atrophy of the temporal pole for semantic fluency.