This article is devoted to the algorithm of the forecast of stress and strain state nonho-mogeneous layered physical nonlinear rock massif. The modeling based on applying the whole package of computational methods: variational method, discrete continuation on numeric parameter method, quasi-linearization of nonlinear value boundary problem, finite difference method, Thomas algorithm and iterative process.