© 2018 The Author(s). Neutron stars are a common compact endpoint of the life of stars. Magnetized and rotating neutron stars emit beams of radiation, which can only be seen when the observer and the beam stand aligned. Periodic recurrence of such alignment gives rise to pulsations and to the name 'pulsar' for the star itself. We present a physical model for the non-thermal emission of pulsars. With just four physical parameters, we fit the spectra of the γ- and X-ray pulsars across seven orders of magnitude in energy. We find that all detections can be encompassed in a continuous variation of the model parameters. The model explains the appearance of sub-exponential cutoffs at high energies as a natural consequence of synchro-curvature-dominated losses, unveiling that curvature-only emission plays a less significant role - if any - in the spectrum of most pulsars. The model also explains the flattening of the X-ray spectra at soft energies as a result of propagating particles being subject to synchrotron losses all along their trajectories. Using this model, we analyse how observations in γ-rays can predict the detectability of the pulsar in X-rays, and vice versa.