Observations from ROSAT and eROSITA have argued that Milky Way (MW), Andromeda, and lower-mass galaxies exhibit extended soft X-ray (\(1\,\)keV) diffuse halos out to radii \(R\gtrsim 100\,\)kpc in the circumgalactic medium (CGM). If interpreted as thermal emission from hot gas, the surprisingly shallow surface brightness profiles \(S_{X} \propto R^{-1}\) of this emission are difficult to explain, and contradict other observations. We show that such halos instead arise from inverse Compton (IC) scattering of CMB photons with GeV cosmic ray (CR) electrons. GeV electrons have long (\(\sim\),Gyr) lifetimes and escape the galaxy, forming a shallow extended radial profile out to \(R\gtrsim 100\,\)kpc, where IC off the CMB dominates their losses and should produce soft, thermal-like X-ray spectra peaked at \(\sim 1\,\)keV. The observed keV halo luminosities and brightness profiles agree well with those expected for CRs observed in the local interstellar medium (LISM) escaping the galaxy, with energetics consistent with known CRs from SNe and/or AGN, around galaxies with stellar masses \(M_{\ast} \lesssim 2\times 10^{11}\,M_{\odot}\). At higher masses observed X-ray luminosities are larger than predicted from IC and should be dominated by hot gas. In the MW+M31, the same models of escaping CRs reproduce {\em Fermi} \(\gamma\)-ray observations if we assume an LISM-like proton-to-electron ratio and CR-pressure-dominated halo. In all other halos, the associated non-thermal radio and \(\gamma\)-ray brightness is far below detectable limits. If we have indeed detected the expected IC X-ray halos, the observations provide qualitatively new and stringent constraints on the properties of the CGM and CR physics: the observed X-ray brightness {\em directly} traces the CR lepton energy density \(e_{\rm cr,\,\ell}\) in the CGM (without any degenerate parameters). The implied \(e_{\rm cr,\,\ell}\) agrees well with LISM values at radii \(R\lesssim 10\,\)kpc, while following the profile predicted by simple steady state models of escaping CRs at larger radii. The inferred CR pressure is a major part of the total pressure budget in the CGM of Milky Way-mass galaxies, suggesting that models of thermally dominated halos at Milky Way mass may need to be revised. The measurement of X-ray surface brightness and total luminosity allows one to further determine the effective CGM diffusivity/CR streaming speed at radii \(\sim 10-1000\,\)kpc. We show these also agree with LISM values at small radii but the inferred diffusivity increases significantly at larger radii, consistent with independent CGM constraints from UV absorption at \(\sim 100\,\)kpc.