This paper presents the parallelization of two widely used numerical solvers for the solution of partial differential equations on structured meshes, namely, the ADI (Alternating-Direction Implicit) solver for tridiagonal linear systems and the SIP (Strongly Implicit Procedure) solver for the penta-diagonal systems. Both solvers were parallelized using CUDA Fortran on GPGPUs (General-Purpose Graphics Processing Units). Compared to the SIP solver, the ADI solver is simple and easy to implement, although its implicitness and dependency are not as strong as the SIP solver and thus its convergence is slower. In fact, the strong dependency of the SIP solver makes it difficult for parallelization. The parallel ADI solver (P-ADI) is based on the Parallel Cyclic Reduction (PCR) algorithm, while the parallel SIP solver (P-SIP) uses the wave front method (WF) following a diagonal line calculation strategy. To map the solution schemes onto the hierarchical thread-blocks framework of CUDA on GPU, the P-ADI solver adopted the one-block-with-iterations and the multi-blocks-without-iterations methods, while the P-SIP solver used the warp-mapping strategy. Both the P-ADI and the P-SIP have been integrated into the University of Mississippi-National Center for Computational Hydroscience and Engineering’s (NCCHE) two-dimensional (2D) hydrodynamic model, CCHE2D-CUDA model. Comparisons of these two parallel solvers and their efficiency were demonstrated by examples and application. Both parallel solvers demonstrated higher efficiency than their serial counterparts on CPU (Central Processing Unit). The P-ADI solver is faster than the P-SIP solver because of its parallel-friendly PCR algorithm and the weaker dependency than that of SIP.