An accurate and effective discretization of diffusion operators is very important in some practical applications such as radiation hydrodynamics. In this paper, we discuss the numerical solution of anisotropic diffusion problems on arbitrary polygonal meshes. A cell-centered finite volume scheme is constructed based on the harmonic averaging point through a certain linearity-preserving approach. The new scheme has only cell-centered unknowns, is locally conservative and has a compact stencil, which reduces to a nine-point scheme on structured quadrilateral meshes. Since the interpolation algorithm based on the harmonic averaging point is a two-stencil and positivity-preserving one, the construction of the scheme is largely simplified. Moreover, since we only use the common topology of 2D and 3D meshes, the extension of the new scheme to the 3D case is very easy and most of codes can be shared. In numerical experiments, we employ some typical distorted meshes and diffusion problems with both continuous and discontinuous coefficients to test our scheme. Numerical results show that the new scheme has a second-order accuracy on many distorted polygonal meshes.