Alterations in brain size and organization represent some of the most distinctive changes in the emergence of our species. Yet, there is limited understanding of how genetic factors contributed to altered neuroanatomy during human evolution. Here, we analyze neuroimaging and genetic data from up to 30,000 people in the UK Biobank and integrate with genomic annotations for different aspects of human evolution, including those based on ancient DNA and comparative genomics. We show that previously reported signals of recent polygenic selection for cortical anatomy are not replicable in a more ancestrally homogeneous sample. We then investigate relationships between evolutionary annotations and common genetic variants shaping cortical surface area and white-matter connectivity for each hemisphere. Our analyses identify single-nucleotide polymorphism heritability enrichment in human-gained regulatory elements that are active in early brain development, affecting surface areas of several parts of the cortex, including left-hemispheric speech-associated regions. We also detect heritability depletion in genomic regions with Neanderthal ancestry for connectivity of the uncinate fasciculus; this is a white-matter tract involved in memory, language, and socioemotional processing with relevance to neuropsychiatric disorders. Finally, we show that common genetic loci associated with left-hemispheric pars triangularis surface area overlap with a human-gained enhancer and affect regulation of ZIC4, a gene implicated in neurogenesis. This work demonstrates how genomic investigations of present-day neuroanatomical variation can help shed light on the complexities of our evolutionary past.