Studies of the relative sizes of brain components in mammals suggest that areas responsible for sensory processing, including visual processing, are correlated with aspects of ecology, especially activity pattern. Some studies suggest that primate orbit convergence and binocular vision are correlated with the overall size of the brain as well as components of the visual pathway, such as the lateral geniculate nucleus. However, the question remains whether components of the visual pathway are correlated with orbit convergence and binocular visual field overlap in nonprimate mammals. Here, we examine the relationship between orbit convergence and the volumes of components of the visual pathway (optic tract, dorsal lateral geniculate nucleus and primary visual cortex). Data on orbit orientation are combined with those on overall brain volume as well as brain component volumes in a taxonomically diverse sample of mammals. Our results demonstrate that nonprimate mammals scale isometrically for component volumes along the visual pathway, whereas primates display negatively allometric relationships. However, only among primates is higher orbit convergence correlated with volumetrically larger lateral geniculate nuclei and visual cortices. Diurnal primates exhibit statistically larger visual pathway components when compared to nocturnal primates. Nonprimate mammals do not display activity pattern differences with the single exception of optic tract sizes. We conclude that binocular vision was a much stronger factor in the evolution of the visual system in primates than in other mammals.