White matter tracts form the structural basis of large-scale functional networks in the human brain. We applied brain-wide tractography to diffusion images from 30,810 adult participants (UK Biobank), and found significant heritability for 90 regional connectivity measures and 851 fiber tracts. Multivariate genome-wide association analyses identified 355 independently associated lead SNPs across the genome, of which 77% were not previously associated with human brain metrics. Enrichment analyses implicated neurodevelopmental processes including neurogenesis, neural differentiation, neural migration, neural projection guidance, and axon development, as well as prenatal brain expression especially in stem cells, astrocytes, microglia and neurons. We used the multivariate association profiles of lead SNPs to identify 26 genomic loci implicated in structural connectivity between core regions of the left-hemisphere language network, and also identified 6 loci associated with hemispheric left-right asymmetry of structural connectivity. Polygenic scores for schizophrenia, bipolar disorder, autism spectrum disorder, attention-deficit hyperactivity disorder, reading ability, left-handedness, Alzheimer’s disease, amyotrophic lateral sclerosis, and epilepsy showed significant multivariate associations with structural connectivity, each implicating specific sets of brain regions with trait-relevant functional profiles. This large-scale mapping study revealed common genetic contributions to the human brain's structural connectome in the general adult population, and its links with polygenic disposition to brain disorders and behavioral traits.