The large-scale evolution of the SARS-CoV-2 virus has been marked by rapid turnover of genetic clades. New variants show intrinsic changes, notably increased transmissibility, as well as antigenic changes that reduce the cross-immunity induced by previous infections or vaccinations. Here we establish a fitness model for SARS-CoV-2 that integrates antigenic and intrinsic selection. The model builds on tracking of time-resolved sequence data, epidemiological records, and cross-immunity data of viral variants. We show that antigenic selection, including contributions of vaccinations and previous infections, had a sizeable impact on the recent evolution of SARS-CoV-2 and emerges as the dominant evolutionary force in the transition from a pandemic to an endemic state. The fitness model can serve the continued surveillance of SARS-CoV-2 evolution in two ways. First, it successfully predicts the short-term evolution of circulating strains, allowing an early identification of new variants that are likely to displace the previously prevalent variant. Second, it maps time-dependent windows of selection that predict the likely antigenic profile of emerging variants.