Starburst galaxies and star-forming active galactic nuclei are among the candidate sources thought to contribute appreciably to the extragalactic gamma-ray and neutrino backgrounds. NGC 1068 is the brightest of the star-forming galaxies found to emit gamma-rays from 0.1 to 50 GeV. Precise measurements of the high-energy spectrum are crucial to study the particle accelerators and probe the dominant emission mechanisms. We have carried out 125 hr of observations of NGC 1068 with the MAGIC telescopes in order to search for gamma-ray emission in the very-high-energy band. We did not detect significant gamma-ray emission, and set upper limits at the 95% confidence level to the gamma-ray flux above 200 GeV f < 5.1 × 10-13 cm-2 s-1. This limit improves previous constraints by about an order of magnitude and allows us to put tight constraints on the theoretical models for the gamma-ray emission. By combining the MAGIC observations with the Fermi-LAT spectrum we limit the parameter space (spectral slope, maximum energy) of the cosmic ray protons predicted by hadronuclear models for the gamma-ray emission, while we find that a model postulating leptonic emission from a semi-relativistic jet is fully consistent with the limits. We provide predictions for IceCube detection of the neutrino signal foreseen in the hadronic scenario. We predict a maximal IceCube neutrino event rate of 0.07 yr-1.