Stock market volatility has a significant impact on many economic and financial activities in the world. Forecasting stock price movement plays an important role in setting an investment strategy or determining the right timing for trading. However, stock price movements are noisy, nonlinear, and chaotic. It is difficult to forecast stock trends for improving return on investment. Here, we proposed a novel improved particle swarm optimization (IPSO) and long-short term memory (LSTM) hybrid model for stock price forecasting. An adaptive mutation factor was used as a parameter for model optimization to avoid premature convergence to a local optimum. Furthermore, we presented a nonlinear approach to improve the inertia weight of the particle swarm optimization and then used the IPSO model to optimize the hyperparameters of LSTM. The experimental results showed that the proposed model outperformed the other related baseline models: support-vector regression, LSTM and PSO-LSTM on the Australian stock market index. These results indicated the proposed model possesses high reliability and good forecasting capability.