© 2018 American Physical Society. We present a systematic powder neutron diffraction study on Sr-based M-type hexaferrites SrScxFe12-xO19 using high-resolution and high-flux instruments. We have derived the magnetic configurations for the compounds with x=1.4, 1.6, and 1.8 at room temperature and at 2 K revealing a ferrimagnetic structure and a complex conical magnetic structure, respectively, where the latter is decomposed into the axial ferrimagnetic and the helicoidally modulated component in the hexagonal a-b plane. Our temperature- and composition-dependent investigation reveals that doping nonmagnetic Sc into this system reinforces the frustration and pushes the transition between the two ordered phases toward higher temperatures. The continuous opening of the spin cone suggests a smooth transition from an easy-axis toward an easy-plane anisotropy with decreasing temperature, which can be reinforced by increasing Sc content.