We present results from a systematic Chandra and HST study of the X-ray source and globular cluster populations of the nearby elliptical galaxy NGC4261. This galaxy shows a distinctly asymmetric distribution of X-ray sources which does not follow the smooth distribution of its optical light. The HST data show that a similar distribution is followed by its globular cluster population. The deep Chandra observations show that the same asymmetry holds down to X-ray sources with luminosities of $sim5 imes10^{37}~ m{erg/s}$. A comparison between the Chandra sources and the globular clusters shows that a large fraction of the X-ray sources are associated with globular clusters. We discuss these results in the context of the field vs. globular cluster formation scenarios of low-mass X-ray binaries, and the merger history of elliptical galaxies.