Monitoring of airborne pollen concentrations provides an important source of information for the globally increasing number of hay fever patients. Airborne pollen is traditionally counted under the microscope, but with the latest developments in image recognition methods, automating this process has become feasible. A challenge that persists, however, is that many pollen grains cannot be distinguished beyond the genus or family level using a microscope. Here, we assess the use of Convolutional Neural Networks (CNNs) to increase taxonomic accuracy for airborne pollen. As a case study we use the nettle family (Urticaceae), which contains two main genera (Urtica and Parietaria) common in European landscapes which pollen cannot be separated by trained specialists. While pollen from Urtica species has very low allergenic relevance, pollen from several species of Parietaria is severely allergenic. We collect pollen from both fresh as well as from herbarium specimens and use these without the often used acetolysis step to train the CNN model. The models show that unacetolyzed Urticaceae pollen grains can be distinguished with > 98% accuracy. We then apply our model on before unseen Urticaceae pollen collected from aerobiological samples and show that the genera can be confidently distinguished, despite the more challenging input images that are often overlain by debris. Our method can also be applied to other pollen families in the future and will thus help to make allergenic pollen monitoring more specific.