The GMRES algorithm and ILU preconditioners are among the most commonly used methods for solving large-scale sparse linear systems. Based on excellent parallel computing performance, GPUs have been a promising hardware architecture. We devote our efforts into the development of a GMRES solver with ILU(k) preconditioners on GPUs and achieve favorable speedup effects. Our GPU computation includes high performance algorithms such as GMRES, SPMV, RAS, nested RAS, ILU(0), ILUT, BILU(k), decoupled ILU(k) and parallel triangular solver.