Nonlinear Optimization Examples

The input argument fun refers to an IML module that specifies a function that returns f, a vector of length m for least squares subroutines or a scalar for other optimization subroutines. The returned f contains the values of the objective function (or the least squares functions) at the point x. Note that for least squares problems, you must specify the number of function values, m, with the first element of the opt argument to allocate memory for the return vector. All the modules that you can specify as input arguments ("fun," "grd," "hes," "jac," "nlc," "jacnlc," and "ptit") accept only a single input argument, x, which is the parameter vector. Using the GLOBAL clause, you can provide more input arguments for these modules. Refer to the section Numerical Considerations for an example.

All the optimization algorithms assume that f is continuous inside the feasible region. For nonlinearly constrained optimization, this is also required for points outside the feasible region. Sometimes the objective function cannot be computed for all points of the specified feasible region; for example, the function specification might contain the SQRT or LOG function, which cannot be evaluated for negative arguments. You must make sure that the function and derivatives of the starting point can be evaluated. There are two ways to prevent large steps into infeasible regions of the parameter space during the optimization process:

All the optimization techniques except the NLPNMS subroutine require continuous first-order derivatives of the objective function f. The NLPTR, NLPNRA, and NLPNRR techniques also require continuous second-order derivatives. If you do not provide the derivatives with the IML modules "grd," "hes," or "jac," they are automatically approximated by finite-difference formulas. Approximating first-order derivatives by finite differences usually requires n additional calls of the function module. Approximating second-order derivatives by finite differences using only function calls can be extremely computationally expensive. Hence, if you decide to use the NLPTR, NLPNRA, or NLPNRR subroutines, you should specify at least analytical first-order derivatives. Then, approximating second-order derivatives by finite differences requires only n or additional calls of the function and gradient modules.

For all input and output arguments, the subroutines assume that

You can verify your analytical derivative specifications by computing finite-difference approximations of the derivatives of f with the NLPFDD subroutine. For most applications, the finite-difference approximations of the derivatives are very precise. Occasionally, difficult objective functions and zero x coordinates cause problems. You can use the par argument to specify the number of accurate digits in the evaluation of the objective function; this defines the step size h of the first- and second-order finite-difference formulas. See the section Finite-Difference Approximations of Derivatives.

Note: For some difficult applications, the finite-difference approximations of derivatives that are generated by default might not be precise enough to solve the optimization or least squares problem. In such cases, you might be able to specify better derivative approximations by using a better approximation formula. You can submit your own finite-difference approximations by using the IML module "grd," "hes," "jac," or "jacnlc." See Example 14.3 for an illustration.

In many applications, calculations used in the computation of f can help compute derivatives at the same point efficiently. You can save and reuse such calculations with the GLOBAL clause. As with many other optimization packages, the subroutines call the "grd," "hes," or "jac" modules only after a call of the "fun" module.

The following statements specify modules for the function, gradient, and Hessian matrix of the Rosenbrock problem:

   proc iml;
      start F_ROSEN(x);
         y1 = 10. * (x[2] - x[1] * x[1]);
         y2 = 1. - x[1];
         f  = .5 * (y1 * y1 + y2 * y2);
         return(f);
      finish F_ROSEN;

      start G_ROSEN(x);
         g = j(1,2,0.);
         g[1] = -200.*x[1]*(x[2]-x[1]*x[1]) - (1.-x[1]);
         g[2] =  100.*(x[2]-x[1]*x[1]);
         return(g);
      finish G_ROSEN;

      start H_ROSEN(x);
         h = j(2,2,0.);
         h[1,1] = -200.*(x[2] - 3.*x[1]*x[1]) + 1.;
         h[2,2] =  100.;
         h[1,2] = -200. * x[1];
         h[2,1] = h[1,2];
         return(h);
      finish H_ROSEN;

The following statements specify a module for the Rosenbrock function when considered as a least squares problem. They also specify the Jacobian matrix of the least squares functions.

   proc iml;
      start F_ROSEN(x);
         y = j(1,2,0.);
         y[1] = 10. * (x[2] - x[1] * x[1]);
         y[2] = 1. - x[1];
        return(y);
      finish F_ROSEN;

      start J_ROSEN(x);
         jac = j(2,2,0.);
         jac[1,1] = -20. * x[1]; jac[1,2] = 10.;
         jac[2,1] = -1.;         jac[2,2] =  0.;
         return(jac);
      finish J_ROSEN;