Solar cycle prediction is of great importance not only for modern technologies, but also for understanding the mechanisms underlying solar activity. In this paper, a novel Multimodal Time Series Prediction Network (MTSP-Net) is proposed for the forecasting of solar cycle 25 and 26 by employing the sunspot number (SSN) data obtained from WDC-SILSO, Royal Observatory of Belgium, Brussels. Our method is based on multiscale temporal convolution, incorporating modules such as periodic time encoding, phase-modulated attention and long short-term memory block, and takes multiple physical features as auxiliary inputs to enhance the prediction performance. The analysis indicates that the proposed model has the ability of capturing both short-term fluctuations and longer-term cyclical patterns of the data. The Root Mean Squared Error and Mean Absolute Error of the model are 21.04 and 18.56, respectively. Finally, solar cycle 25 that was predicted to occur in 2024 June with the peak value of 155.1 by 13 months smoothed SSN. This result is closer to the actual observed values than those obtained by other methods, which occured in 2024 October with a peak value of 160.8. For the forecasting of solar cycle 26, the peak value is predicted to be 158.45 in 2034 April, which is consistent with the results of existing methods.