The human cerebral hemispheres show a left–right asymmetrical torque pattern, which has been claimed to be absent in chimpanzees. The functional significance and developmental mechanisms are unknown. Here, we carried out the largest-ever analysis of global brain shape asymmetry in magnetic resonance imaging data. Three population datasets were used, UK Biobank (N = 39 678), Human Connectome Project (N = 1113), and BIL&GIN (N = 453). At the population level, there was an anterior and dorsal skew of the right hemisphere, relative to the left. Both skews were associated independently with handedness, and various regional gray and white matter metrics oppositely in the two hemispheres, as well as other variables related to cognitive functions, sociodemographic factors, and physical and mental health. The two skews showed single nucleotide polymorphisms-based heritabilities of 4–13%, but also substantial polygenicity in causal mixture model analysis, and no individually significant loci were found in genome-wide association studies for either skew. There was evidence for a significant genetic correlation between horizontal brain skew and autism, which requires future replication. These results provide the first large-scale description of population-average brain skews and their inter-individual variations, their replicable associations with handedness, and insights into biological and other factors which associate with human brain asymmetry.