Abstract: In this paper, the variational inequality problem is transformed as an unconstrained optimization problem through the generalized $D$-gap function. A non-monotone hybrid Newton method based on Zhang H.C.'s non-monotone line search technique is proposed for minimizing the general form of the generalized $D$-gap function. Then, the global convergence property of the algorithm is analyzed. Under some proper conditions, we prove that the algorithm is globally quadratically convergent. Moreover, we obtain a global error bound of the algorithm when the mapping $F$ is strongly monotone without Lipschitz continuous. Numerical results indicate that the new algorithm is efficient.

Key words: generalized $D$-gap function, non-monotone line search, global convergence, global error bound