Pulsars, which are highly magnetized, rapidly rotating neutron stars, are remnants of massive stars that have undergone supernova explosions. These objects provide essential insights into neutron star evolution and the equation of state for dense matter. However, current pulsar identification models heavily rely on multi-modal data and lose their effectiveness when one or more modalities are missing. This limitation underscores the need for a unified architecture that can handle both single- and multi-modal inputs while minimizing the occurrence of false positives. To address this challenge, we propose a novel approach, Pulsar Recognition using ResNet-18 and multi-head attention (PRMA). The core innovation of PRMA lies in the integration of Transformer encoders within the ResNet-18 backbone, allowing for multi-stage embedding of features. This architecture is specifically designed to support both single- and multi-modal data inputs. Furthermore, a data-level fusion strategy is employed to process standardized diagnostic plots that originate from multiple modalities, ensuring that critical information from various sources is effectively utilized. Experimental validation using Five-hundred-meter Aperture Spherical Telescope data demonstrates the superiority of PRMA over existing methods. In both single-modal and multi-modal settings, PRMA achieves a significantly lower false positive rate, as low as 0.25% and 0.13%, respectively. These results highlight PRMA’s potential as a powerful tool for large-scale pulsar data screening, offering an efficient and precise solution that addresses the challenges of missing modalities while maintaining high classification accuracy. The PRMA framework represents a novel paradigm in pulsar identification, enhancing the precision of pulsar detection and offering valuable insights into the dynamic nature of neutron star systems.