The development of algorithms for optimizing the loss function of artificial neural networks is introduced is this work. The KSD (Krylov Subspace Descent) algorithm is extended to MKSD (Modified KSD) algorithm which has adaptively variable subspace dimension instead of fixed dimension. Some numerical examples of optimizing the fully connected neural network problems by MKSD, KSD and SGD (Stochastic Gradient Descent) algorithms are given. The numerical results show that the MKSD method has certain advantages over other methods.