The instability of nuclear matter due to particle-hole excitation modes has been studied in the framework of several relativistic mean-field (RMF) models. It is found that both the longitudinal and the transversal modes depend sensitively on the parameter sets used. The important impact of the vector and vector-scalar nonlinear terms on the stability of both modes is demonstrated. Our finding corroborates the result of previous studies, namely that certain RMF models cannot be used in high-density applications. However, we show that for certain parameter sets of RMF models this shortcoming can be alleviated by adding these nonlinear terms.