Investigation of relationships between age-related changes in regional brain volumes and changes in domain-specific cognition could provide insights into the neural underpinnings of individual differences in cognitive aging. Domain-specific cognition (memory, verbal fluency, visuospatial ability) and tests of executive function and attention (Trail-Making Test Part A and B) and 47 brain volumes of interest (VOIs) were assessed in 836 Baltimore Longitudinal Study of Aging participants with mean follow-up of 4.1 years (maximum 23.1 years). To examine the correlation between changes in domain-specific cognition and changes in brain volumes, we used bivariate linear mixed effects models with unstructured variance-covariance structure to estimate longitudinal trajectories for each variable of interest and correlations among the random effects of these measures. Higher annual rates of memory decline were associated with greater volume loss in 14 VOIs primarily within the temporal and occipital lobes. Verbal fluency decline was associated with greater ventricular enlargement and volume loss in 24 VOIs within the frontal, temporal, and parietal lobes. Decline in visuospatial ability was associated with volume loss in 3 temporal and parietal VOIs. Declines on the attentional test were associated with volume loss in 4 VOIs located within temporal and parietal lobes. Greater declines on the executive function test were associated with greater ventricular enlargement and volume loss in 10 frontal, parietal, and temporal VOIs. Our findings highlight domain-specific patterns of regional brain atrophy that may contribute to individual differences in cognitive aging.