@c DO NOT EDIT!  Generated automatically by munge-texi.

@c Copyright (C) 1996-2012 John W. Eaton
@c
@c This file is part of Octave.
@c
@c Octave is free software; you can redistribute it and/or modify it
@c under the terms of the GNU General Public License as published by the
@c Free Software Foundation; either version 3 of the License, or (at
@c your option) any later version.
@c 
@c Octave is distributed in the hope that it will be useful, but WITHOUT
@c ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
@c FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
@c for more details.
@c 
@c You should have received a copy of the GNU General Public License
@c along with Octave; see the file COPYING.  If not, see
@c <http://www.gnu.org/licenses/>.

@node Optimization
@chapter Optimization

Octave comes with support for solving various kinds of optimization
problems.  Specifically Octave can solve problems in Linear Programming,
Quadratic Programming, Nonlinear Programming, and Linear Least Squares
Minimization.

@menu
* Linear Programming::       
* Quadratic Programming::       
* Nonlinear Programming::       
* Linear Least Squares::        
@end menu

@c @cindex linear programming
@cindex quadratic programming
@cindex nonlinear programming
@cindex optimization
@cindex LP
@cindex QP
@cindex NLP

@node Linear Programming
@section Linear Programming

Octave can solve Linear Programming problems using the @code{glpk}
function.  That is, Octave can solve

@tex
$$
  \min_x c^T x
$$
@end tex
@ifnottex

@example
min C'*x
@end example

@end ifnottex
subject to the linear constraints
@tex
$Ax = b$ where $x \geq 0$.
@end tex
@ifnottex
@math{A*x = b} where @math{x @geq{} 0}.
@end ifnottex

@noindent
The @code{glpk} function also supports variations of this problem.

@c glpk scripts/optimization/glpk.m
@anchor{doc-glpk}
@deftypefn {Function File} {[@var{xopt}, @var{fmin}, @var{status}, @var{extra}] =} glpk (@var{c}, @var{A}, @var{b}, @var{lb}, @var{ub}, @var{ctype}, @var{vartype}, @var{sense}, @var{param})
Solve a linear program using the GNU @sc{glpk} library.  Given three
arguments, @code{glpk} solves the following standard LP:
@tex
$$
  \min_x C^T x
$$
@end tex
@ifnottex

@example
min C'*x
@end example

@end ifnottex
subject to
@tex
$$
  Ax = b \qquad x \geq 0
$$
@end tex
@ifnottex

@example
@group
A*x  = b
  x >= 0
@end group
@end example

@end ifnottex
but may also solve problems of the form
@tex
$$
  [ \min_x | \max_x ] C^T x
$$
@end tex
@ifnottex

@example
[ min | max ] C'*x
@end example

@end ifnottex
subject to
@tex
$$
 Ax [ = | \leq | \geq ] b \qquad LB \leq x \leq UB
$$
@end tex
@ifnottex

@example
@group
A*x [ "=" | "<=" | ">=" ] b
  x >= LB
  x <= UB
@end group
@end example

@end ifnottex

Input arguments:

@table @var
@item c
A column array containing the objective function coefficients.

@item A
A matrix containing the constraints coefficients.

@item b
A column array containing the right-hand side value for each constraint
in the constraint matrix.

@item lb
An array containing the lower bound on each of the variables.  If
@var{lb} is not supplied, the default lower bound for the variables is
zero.

@item ub
An array containing the upper bound on each of the variables.  If
@var{ub} is not supplied, the default upper bound is assumed to be
infinite.

@item ctype
An array of characters containing the sense of each constraint in the
constraint matrix.  Each element of the array may be one of the
following values
@table @asis
@item "F"
A free (unbounded) constraint (the constraint is ignored).

@item "U"
An inequality constraint with an upper bound (@code{A(i,:)*x <= b(i)}).

@item "S"
An equality constraint (@code{A(i,:)*x = b(i)}).

@item "L"
An inequality with a lower bound (@code{A(i,:)*x >= b(i)}).

@item "D"
An inequality constraint with both upper and lower bounds
(@code{A(i,:)*x >= -b(i)} @emph{and} (@code{A(i,:)*x <= b(i)}).
@end table

@item vartype
A column array containing the types of the variables.
@table @asis
@item "C"
A continuous variable.

@item "I"
An integer variable.
@end table

@item sense
If @var{sense} is 1, the problem is a minimization.  If @var{sense} is
-1, the problem is a maximization.  The default value is 1.

@item param
A structure containing the following parameters used to define the
behavior of solver.  Missing elements in the structure take on default
values, so you only need to set the elements that you wish to change
from the default.

Integer parameters:

@table @code
@item msglev (@w{@code{LPX_K_MSGLEV}}, default: 1)
Level of messages output by solver routines:
@table @asis
@item 0
No output.

@item 1
Error messages only.

@item 2
Normal output.

@item 3
Full output (includes informational messages).
@end table

@item scale (@w{@code{LPX_K_SCALE}}, default: 1)
Scaling option:
@table @asis
@item 0
No scaling.

@item 1
Equilibration scaling.

@item 2
Geometric mean scaling, then equilibration scaling.
@end table

@item dual    (@w{@code{LPX_K_DUAL}}, default: 0)
Dual simplex option:
@table @asis
@item 0
Do not use the dual simplex.

@item 1
If initial basic solution is dual feasible, use the dual simplex.
@end table

@item price   (@w{@code{LPX_K_PRICE}}, default: 1)
Pricing option (for both primal and dual simplex):
@table @asis
@item 0
Textbook pricing.

@item 1
Steepest edge pricing.
@end table

@item round   (@w{@code{LPX_K_ROUND}}, default: 0)
Solution rounding option:
@table @asis
@item 0
Report all primal and dual values "as is".

@item 1
Replace tiny primal and dual values by exact zero.
@end table

@item itlim   (@w{@code{LPX_K_ITLIM}}, default: -1)
Simplex iterations limit.  If this value is positive, it is decreased by
one each time when one simplex iteration has been performed, and
reaching zero value signals the solver to stop the search.  Negative
value means no iterations limit.

@item itcnt (@w{@code{LPX_K_OUTFRQ}}, default: 200)
Output frequency, in iterations.  This parameter specifies how
frequently the solver sends information about the solution to the
standard output.

@item branch (@w{@code{LPX_K_BRANCH}}, default: 2)
Branching heuristic option (for MIP only):
@table @asis
@item 0
Branch on the first variable.

@item 1
Branch on the last variable.

@item 2
Branch using a heuristic by Driebeck and Tomlin.
@end table

@item btrack (@w{@code{LPX_K_BTRACK}}, default: 2)
Backtracking heuristic option (for MIP only):
@table @asis
@item 0
Depth first search.

@item 1
Breadth first search.

@item 2
Backtrack using the best projection heuristic.
@end table

@item presol (@w{@code{LPX_K_PRESOL}}, default: 1)
If this flag is set, the routine lpx_simplex solves the problem using
the built-in LP presolver.  Otherwise the LP presolver is not used.

@item lpsolver (default: 1)
Select which solver to use.  If the problem is a MIP problem this flag
will be ignored.
@table @asis
@item 1
Revised simplex method.

@item 2
Interior point method.
@end table

@item save (default: 0)
If this parameter is nonzero, save a copy of the problem in
CPLEX LP format to the file @file{"outpb.lp"}.  There is currently no
way to change the name of the output file.
@end table

Real parameters:

@table @code
@item relax (@w{@code{LPX_K_RELAX}}, default: 0.07)
Relaxation parameter used in the ratio test.  If it is zero, the textbook
ratio test is used.  If it is non-zero (should be positive), Harris'
two-pass ratio test is used.  In the latter case on the first pass of the
ratio test basic variables (in the case of primal simplex) or reduced
costs of non-basic variables (in the case of dual simplex) are allowed
to slightly violate their bounds, but not more than
@code{relax*tolbnd} or @code{relax*toldj (thus, @code{relax} is a
percentage of @code{tolbnd} or @code{toldj}}.

@item tolbnd (@w{@code{LPX_K_TOLBND}}, default: 10e-7)
Relative tolerance used to check if the current basic solution is primal
feasible.  It is not recommended that you change this parameter unless you
have a detailed understanding of its purpose.

@item toldj (@w{@code{LPX_K_TOLDJ}}, default: 10e-7)
Absolute tolerance used to check if the current basic solution is dual
feasible.  It is not recommended that you change this parameter unless you
have a detailed understanding of its purpose.

@item tolpiv (@w{@code{LPX_K_TOLPIV}}, default: 10e-9)
Relative tolerance used to choose eligible pivotal elements of the
simplex table.  It is not recommended that you change this parameter unless
you have a detailed understanding of its purpose.

@item objll (@w{@code{LPX_K_OBJLL}}, default: -DBL_MAX)
Lower limit of the objective function.  If on the phase II the objective
function reaches this limit and continues decreasing, the solver stops
the search.  This parameter is used in the dual simplex method only.

@item objul (@w{@code{LPX_K_OBJUL}}, default: +DBL_MAX)
Upper limit of the objective function.  If on the phase II the objective
function reaches this limit and continues increasing, the solver stops
the search.  This parameter is used in the dual simplex only.

@item tmlim (@w{@code{LPX_K_TMLIM}}, default: -1.0)
Searching time limit, in seconds.  If this value is positive, it is
decreased each time when one simplex iteration has been performed by the
amount of time spent for the iteration, and reaching zero value signals
the solver to stop the search.  Negative value means no time limit.

@item outdly (@w{@code{LPX_K_OUTDLY}}, default: 0.0)
Output delay, in seconds.  This parameter specifies how long the solver
should delay sending information about the solution to the standard
output.  Non-positive value means no delay.

@item tolint (@w{@code{LPX_K_TOLINT}}, default: 10e-5)
Relative tolerance used to check if the current basic solution is integer
feasible.  It is not recommended that you change this parameter unless
you have a detailed understanding of its purpose.

@item tolobj (@w{@code{LPX_K_TOLOBJ}}, default: 10e-7)
Relative tolerance used to check if the value of the objective function
is not better than in the best known integer feasible solution.  It is
not recommended that you change this parameter unless you have a
detailed understanding of its purpose.
@end table
@end table

Output values:

@table @var
@item xopt
The optimizer (the value of the decision variables at the optimum).

@item fopt
The optimum value of the objective function.

@item status
Status of the optimization.

Simplex Method:
@table @asis
@item 180 (@w{@code{LPX_OPT}})
Solution is optimal.

@item 181 (@w{@code{LPX_FEAS}})
Solution is feasible.

@item 182 (@w{@code{LPX_INFEAS}})
Solution is infeasible.

@item 183 (@w{@code{LPX_NOFEAS}})
Problem has no feasible solution.

@item 184 (@w{@code{LPX_UNBND}})
Problem has no unbounded solution.

@item 185 (@w{@code{LPX_UNDEF}})
Solution status is undefined.
@end table
Interior Point Method:
@table @asis
@item 150 (@w{@code{LPX_T_UNDEF}})
The interior point method is undefined.

@item 151 (@w{@code{LPX_T_OPT}})
The interior point method is optimal.
@end table
Mixed Integer Method:
@table @asis
@item 170 (@w{@code{LPX_I_UNDEF}})
The status is undefined.

@item 171 (@w{@code{LPX_I_OPT}})
The solution is integer optimal.

@item 172 (@w{@code{LPX_I_FEAS}})
Solution integer feasible but its optimality has not been proven

@item 173 (@w{@code{LPX_I_NOFEAS}})
No integer feasible solution.
@end table
@noindent
If an error occurs, @var{status} will contain one of the following
codes:

@table @asis
@item 204 (@w{@code{LPX_E_FAULT}})
Unable to start the search.

@item 205 (@w{@code{LPX_E_OBJLL}})
Objective function lower limit reached.

@item 206 (@w{@code{LPX_E_OBJUL}})
Objective function upper limit reached.

@item 207 (@w{@code{LPX_E_ITLIM}})
Iterations limit exhausted.

@item 208 (@w{@code{LPX_E_TMLIM}})
Time limit exhausted.

@item 209 (@w{@code{LPX_E_NOFEAS}})
No feasible solution.

@item 210 (@w{@code{LPX_E_INSTAB}})
Numerical instability.

@item 211 (@w{@code{LPX_E_SING}})
Problems with basis matrix.

@item 212 (@w{@code{LPX_E_NOCONV}})
No convergence (interior).

@item 213 (@w{@code{LPX_E_NOPFS}})
No primal feasible solution (LP presolver).

@item 214 (@w{@code{LPX_E_NODFS}})
No dual feasible solution (LP presolver).
@end table

@item extra
A data structure containing the following fields:
@table @code
@item lambda
Dual variables.

@item redcosts
Reduced Costs.

@item time
Time (in seconds) used for solving LP/MIP problem.

@item mem
Memory (in bytes) used for solving LP/MIP problem (this is not
available if the version of @sc{glpk} is 4.15 or later).
@end table
@end table

Example:

@example
@group
c = [10, 6, 4]';
A = [ 1, 1, 1;
     10, 4, 5;
      2, 2, 6];
b = [100, 600, 300]';
lb = [0, 0, 0]';
ub = [];
ctype = "UUU";
vartype = "CCC";
s = -1;

param.msglev = 1;
param.itlim = 100;

[xmin, fmin, status, extra] = ...
   glpk (c, A, b, lb, ub, ctype, vartype, s, param);
@end group
@end example
@end deftypefn


@node Quadratic Programming
@section Quadratic Programming

Octave can also solve Quadratic Programming problems, this is
@tex
$$
 \min_x {1 \over 2} x^T H x + x^T q
$$
@end tex
@ifnottex

@example
min 0.5 x'*H*x + x'*q
@end example

@end ifnottex
subject to
@tex
$$
 Ax = b \qquad lb \leq x \leq ub \qquad A_{lb} \leq A_{in} \leq A_{ub}
$$
@end tex
@ifnottex

@example
@group
     A*x = b
     lb <= x <= ub
     A_lb <= A_in*x <= A_ub
@end group
@end example

@end ifnottex

@c qp scripts/optimization/qp.m
@anchor{doc-qp}
@deftypefn  {Function File} {[@var{x}, @var{obj}, @var{info}, @var{lambda}] =} qp (@var{x0}, @var{H})
@deftypefnx {Function File} {[@var{x}, @var{obj}, @var{info}, @var{lambda}] =} qp (@var{x0}, @var{H}, @var{q})
@deftypefnx {Function File} {[@var{x}, @var{obj}, @var{info}, @var{lambda}] =} qp (@var{x0}, @var{H}, @var{q}, @var{A}, @var{b})
@deftypefnx {Function File} {[@var{x}, @var{obj}, @var{info}, @var{lambda}] =} qp (@var{x0}, @var{H}, @var{q}, @var{A}, @var{b}, @var{lb}, @var{ub})
@deftypefnx {Function File} {[@var{x}, @var{obj}, @var{info}, @var{lambda}] =} qp (@var{x0}, @var{H}, @var{q}, @var{A}, @var{b}, @var{lb}, @var{ub}, @var{A_lb}, @var{A_in}, @var{A_ub})
@deftypefnx {Function File} {[@var{x}, @var{obj}, @var{info}, @var{lambda}] =} qp (@dots{}, @var{options})
Solve the quadratic program
@tex
$$
 \min_x {1 \over 2} x^T H x + x^T q
$$
@end tex
@ifnottex

@example
@group
     min 0.5 x'*H*x + x'*q
      x
@end group
@end example

@end ifnottex
subject to
@tex
$$
 Ax = b \qquad lb \leq x \leq ub \qquad A_{lb} \leq A_{in} \leq A_{ub}
$$
@end tex
@ifnottex

@example
@group
     A*x = b
     lb <= x <= ub
     A_lb <= A_in*x <= A_ub
@end group
@end example

@end ifnottex
@noindent
using a null-space active-set method.

Any bound (@var{A}, @var{b}, @var{lb}, @var{ub}, @var{A_lb},
@var{A_ub}) may be set to the empty matrix (@code{[]}) if not
present.  If the initial guess is feasible the algorithm is faster.

@table @var
@item options
An optional structure containing the following
parameter(s) used to define the behavior of the solver.  Missing elements
in the structure take on default values, so you only need to set the
elements that you wish to change from the default.

@table @code
@item MaxIter (default: 200)
Maximum number of iterations.
@end table
@end table

@table @var
@item info
Structure containing run-time information about the algorithm.  The
following fields are defined:

@table @code
@item solveiter
The number of iterations required to find the solution.

@item info
An integer indicating the status of the solution.

@table @asis
@item 0
The problem is feasible and convex.  Global solution found.

@item 1
The problem is not convex.  Local solution found.

@item 2
The problem is not convex and unbounded.

@item 3
Maximum number of iterations reached.

@item 6
The problem is infeasible.
@end table
@end table
@end table
@end deftypefn


@c pqpnonneg scripts/optimization/pqpnonneg.m
@anchor{doc-pqpnonneg}
@deftypefn  {Function File} {@var{x} =} pqpnonneg (@var{c}, @var{d})
@deftypefnx {Function File} {@var{x} =} pqpnonneg (@var{c}, @var{d}, @var{x0})
@deftypefnx {Function File} {[@var{x}, @var{minval}] =} pqpnonneg (@dots{})
@deftypefnx {Function File} {[@var{x}, @var{minval}, @var{exitflag}] =} pqpnonneg (@dots{})
@deftypefnx {Function File} {[@var{x}, @var{minval}, @var{exitflag}, @var{output}] =} pqpnonneg (@dots{})
@deftypefnx {Function File} {[@var{x}, @var{minval}, @var{exitflag}, @var{output}, @var{lambda}] =} pqpnonneg (@dots{})
Minimize @code{1/2*x'*c*x + d'*x} subject to @code{@var{x} >= 0}.  @var{c}
and @var{d} must be real, and @var{c} must be symmetric and positive
definite.  @var{x0} is an optional initial guess for @var{x}.

Outputs:
@itemize @bullet
@item minval

The minimum attained model value, 1/2*xmin'*c*xmin + d'*xmin

@item exitflag

An indicator of convergence.  0 indicates that the iteration count
was exceeded, and therefore convergence was not reached; >0 indicates
that the algorithm converged.  (The algorithm is stable and will
converge given enough iterations.)

@item output

A structure with two fields:
@itemize @bullet
@item "algorithm": The algorithm used ("nnls")

@item "iterations": The number of iterations taken.
@end itemize

@item lambda

Not implemented.
@end itemize
@seealso{@ref{doc-optimset,,optimset}, @ref{doc-lsqnonneg,,lsqnonneg}, @ref{doc-qp,,qp}}
@end deftypefn


@node Nonlinear Programming
@section Nonlinear Programming

Octave can also perform general nonlinear minimization using a
successive quadratic programming solver.

@c sqp scripts/optimization/sqp.m
@anchor{doc-sqp}
@deftypefn  {Function File} {[@var{x}, @var{obj}, @var{info}, @var{iter}, @var{nf}, @var{lambda}] =} sqp (@var{x0}, @var{phi})
@deftypefnx {Function File} {[@dots{}] =} sqp (@var{x0}, @var{phi}, @var{g})
@deftypefnx {Function File} {[@dots{}] =} sqp (@var{x0}, @var{phi}, @var{g}, @var{h})
@deftypefnx {Function File} {[@dots{}] =} sqp (@var{x0}, @var{phi}, @var{g}, @var{h}, @var{lb}, @var{ub})
@deftypefnx {Function File} {[@dots{}] =} sqp (@var{x0}, @var{phi}, @var{g}, @var{h}, @var{lb}, @var{ub}, @var{maxiter})
@deftypefnx {Function File} {[@dots{}] =} sqp (@var{x0}, @var{phi}, @var{g}, @var{h}, @var{lb}, @var{ub}, @var{maxiter}, @var{tol})
Solve the nonlinear program
@tex
$$
\min_x \phi (x)
$$
@end tex
@ifnottex

@example
@group
     min phi (x)
      x
@end group
@end example

@end ifnottex
subject to
@tex
$$
 g(x) = 0 \qquad h(x) \geq 0 \qquad lb \leq x \leq ub
$$
@end tex
@ifnottex

@example
@group
     g(x)  = 0
     h(x) >= 0
     lb <= x <= ub
@end group
@end example

@end ifnottex
@noindent
using a sequential quadratic programming method.

The first argument is the initial guess for the vector @var{x0}.

The second argument is a function handle pointing to the objective
function @var{phi}.  The objective function must accept one vector
argument and return a scalar.

The second argument may also be a 2- or 3-element cell array of
function handles.  The first element should point to the objective
function, the second should point to a function that computes the
gradient of the objective function, and the third should point to a
function that computes the Hessian of the objective function.  If the
gradient function is not supplied, the gradient is computed by finite
differences.  If the Hessian function is not supplied, a BFGS update
formula is used to approximate the Hessian.

When supplied, the gradient function @code{@var{phi}@{2@}} must accept
one vector argument and return a vector.  When supplied, the Hessian
function @code{@var{phi}@{3@}} must accept one vector argument and
return a matrix.

The third and fourth arguments @var{g} and @var{h} are function
handles pointing to functions that compute the equality constraints
and the inequality constraints, respectively.  If the problem does
not have equality (or inequality) constraints, then use an empty
matrix ([]) for @var{g} (or @var{h}).  When supplied, these equality
and inequality constraint functions must accept one vector argument
and return a vector.

The third and fourth arguments may also be 2-element cell arrays of
function handles.  The first element should point to the constraint
function and the second should point to a function that computes the
gradient of the constraint function:
@tex
$$
 \Bigg( {\partial f(x) \over \partial x_1},
        {\partial f(x) \over \partial x_2}, \ldots,
        {\partial f(x) \over \partial x_N} \Bigg)^T
$$
@end tex
@ifnottex

@example
@group
                [ d f(x)   d f(x)        d f(x) ]
    transpose ( [ ------   -----   ...   ------ ] )
                [  dx_1     dx_2          dx_N  ]
@end group
@end example

@end ifnottex
The fifth and sixth arguments, @var{lb} and @var{ub}, contain lower
and upper bounds on @var{x}.  These must be consistent with the
equality and inequality constraints @var{g} and @var{h}.  If the
arguments are vectors then @var{x}(i) is bound by @var{lb}(i) and
@var{ub}(i).  A bound can also be a scalar in which case all elements
of @var{x} will share the same bound.  If only one bound (lb, ub) is
specified then the other will default to (-@var{realmax},
+@var{realmax}).

The seventh argument @var{maxiter} specifies the maximum number of
iterations.  The default value is 100.

The eighth argument @var{tol} specifies the tolerance for the
stopping criteria.  The default value is @code{sqrt(eps)}.

The value returned in @var{info} may be one of the following:

@table @asis
@item 101
The algorithm terminated normally.
Either all constraints meet the requested tolerance, or the stepsize,
@tex
$\Delta x,$
@end tex
@ifnottex
delta @var{x},
@end ifnottex
is less than @code{@var{tol} * norm (x)}.

@item 102
The BFGS update failed.

@item 103
The maximum number of iterations was reached.
@end table

An example of calling @code{sqp}:

@example
function r = g (x)
  r = [ sumsq(x)-10;
        x(2)*x(3)-5*x(4)*x(5);
        x(1)^3+x(2)^3+1 ];
endfunction

function obj = phi (x)
  obj = exp(prod(x)) - 0.5*(x(1)^3+x(2)^3+1)^2;
endfunction

x0 = [-1.8; 1.7; 1.9; -0.8; -0.8];

[x, obj, info, iter, nf, lambda] = sqp (x0, @@phi, @@g, [])

x =

  -1.71714
   1.59571
   1.82725
  -0.76364
  -0.76364

obj = 0.053950
info = 101
iter = 8
nf = 10
lambda =

  -0.0401627
   0.0379578
  -0.0052227
@end example

@seealso{@ref{doc-qp,,qp}}
@end deftypefn


@node Linear Least Squares
@section Linear Least Squares

Octave also supports linear least squares minimization.  That is,
Octave can find the parameter @math{b} such that the model
@tex
$y = xb$
@end tex
@ifnottex
@math{y = x*b}
@end ifnottex
fits data @math{(x,y)} as well as possible, assuming zero-mean
Gaussian noise.  If the noise is assumed to be isotropic the problem
can be solved using the @samp{\} or @samp{/} operators, or the @code{ols}
function.  In the general case where the noise is assumed to be anisotropic
the @code{gls} is needed.

@c ols scripts/statistics/base/ols.m
@anchor{doc-ols}
@deftypefn {Function File} {[@var{beta}, @var{sigma}, @var{r}] =} ols (@var{y}, @var{x})
Ordinary least squares estimation for the multivariate model
@tex
$y = x b + e$
with
$\bar{e} = 0$, and cov(vec($e$)) = kron ($s, I$)
@end tex
@ifnottex
@w{@math{y = x*b + e}} with
@math{mean (e) = 0} and @math{cov (vec (e)) = kron (s, I)}.
@end ifnottex
 where
@tex
$y$ is a $t \times p$ matrix, $x$ is a $t \times k$ matrix,
$b$ is a $k \times p$ matrix, and $e$ is a $t \times p$ matrix.
@end tex
@ifnottex
@math{y} is a @math{t} by @math{p} matrix, @math{x} is a @math{t} by
@math{k} matrix, @math{b} is a @math{k} by @math{p} matrix, and
@math{e} is a @math{t} by @math{p} matrix.
@end ifnottex

Each row of @var{y} and @var{x} is an observation and each column a
variable.

The return values @var{beta}, @var{sigma}, and @var{r} are defined as
follows.

@table @var
@item beta
The OLS estimator for @math{b}.
@tex
$beta$ is calculated directly via $(x^Tx)^{-1} x^T y$ if the matrix $x^Tx$ is
of full rank.
@end tex
@ifnottex
@var{beta} is calculated directly via @code{inv (x'*x) * x' * y} if the
matrix @code{x'*x} is of full rank.
@end ifnottex
Otherwise, @code{@var{beta} = pinv (@var{x}) * @var{y}} where
@code{pinv (@var{x})} denotes the pseudoinverse of @var{x}.

@item sigma
The OLS estimator for the matrix @var{s},

@example
@group
@var{sigma} = (@var{y}-@var{x}*@var{beta})'
  * (@var{y}-@var{x}*@var{beta})
  / (@var{t}-rank(@var{x}))
@end group
@end example

@item r
The matrix of OLS residuals, @code{@var{r} = @var{y} - @var{x}*@var{beta}}.
@end table
@seealso{@ref{doc-gls,,gls}, @ref{doc-pinv,,pinv}}
@end deftypefn


@c gls scripts/statistics/base/gls.m
@anchor{doc-gls}
@deftypefn {Function File} {[@var{beta}, @var{v}, @var{r}] =} gls (@var{y}, @var{x}, @var{o})
Generalized least squares estimation for the multivariate model
@tex
$y = x b + e$
with $\bar{e} = 0$ and cov(vec($e$)) = $(s^2)o$,
@end tex
@ifnottex
@w{@math{y = x*b + e}} with @math{mean (e) = 0} and
@math{cov (vec (e)) = (s^2) o},
@end ifnottex
 where
@tex
$y$ is a $t \times p$ matrix, $x$ is a $t \times k$ matrix, $b$ is a $k
\times p$ matrix, $e$ is a $t \times p$ matrix, and $o$ is a $tp \times
tp$ matrix.
@end tex
@ifnottex
@math{y} is a @math{t} by @math{p} matrix, @math{x} is a @math{t} by
@math{k} matrix, @math{b} is a @math{k} by @math{p} matrix, @math{e}
is a @math{t} by @math{p} matrix, and @math{o} is a @math{t*p} by
@math{t*p} matrix.
@end ifnottex

@noindent
Each row of @var{y} and @var{x} is an observation and each column a
variable.  The return values @var{beta}, @var{v}, and @var{r} are
defined as follows.

@table @var
@item beta
The GLS estimator for @math{b}.

@item v
The GLS estimator for @math{s^2}.

@item r
The matrix of GLS residuals, @math{r = y - x*beta}.
@end table
@seealso{@ref{doc-ols,,ols}}
@end deftypefn


@c lsqnonneg scripts/optimization/lsqnonneg.m
@anchor{doc-lsqnonneg}
@deftypefn  {Function File} {@var{x} =} lsqnonneg (@var{c}, @var{d})
@deftypefnx {Function File} {@var{x} =} lsqnonneg (@var{c}, @var{d}, @var{x0})
@deftypefnx {Function File} {[@var{x}, @var{resnorm}] =} lsqnonneg (@dots{})
@deftypefnx {Function File} {[@var{x}, @var{resnorm}, @var{residual}] =} lsqnonneg (@dots{})
@deftypefnx {Function File} {[@var{x}, @var{resnorm}, @var{residual}, @var{exitflag}] =} lsqnonneg (@dots{})
@deftypefnx {Function File} {[@var{x}, @var{resnorm}, @var{residual}, @var{exitflag}, @var{output}] =} lsqnonneg (@dots{})
@deftypefnx {Function File} {[@var{x}, @var{resnorm}, @var{residual}, @var{exitflag}, @var{output}, @var{lambda}] =} lsqnonneg (@dots{})
Minimize @code{norm (@var{c}*@var{x} - d)} subject to
@code{@var{x} >= 0}.  @var{c} and @var{d} must be real.  @var{x0} is an
optional initial guess for @var{x}.

Outputs:
@itemize @bullet
@item resnorm

The squared 2-norm of the residual: norm(@var{c}*@var{x}-@var{d})^2

@item residual

The residual: @var{d}-@var{c}*@var{x}

@item exitflag

An indicator of convergence.  0 indicates that the iteration count
was exceeded, and therefore convergence was not reached; >0 indicates
that the algorithm converged.  (The algorithm is stable and will
converge given enough iterations.)

@item output

A structure with two fields:
@itemize @bullet
@item "algorithm": The algorithm used ("nnls")

@item "iterations": The number of iterations taken.
@end itemize

@item lambda

Not implemented.
@end itemize
@seealso{@ref{doc-optimset,,optimset}, @ref{doc-pqpnonneg,,pqpnonneg}}
@end deftypefn


@c optimset scripts/optimization/optimset.m
@anchor{doc-optimset}
@deftypefn  {Function File} {} optimset ()
@deftypefnx {Function File} {} optimset (@var{par}, @var{val}, @dots{})
@deftypefnx {Function File} {} optimset (@var{old}, @var{par}, @var{val}, @dots{})
@deftypefnx {Function File} {} optimset (@var{old}, @var{new})
Create options struct for optimization functions.

Valid parameters are:
@itemize @bullet
@item AutoScaling

@item ComplexEqn

@item FinDiffType

@item FunValCheck
When enabled, display an error if the objective function returns a complex
value or NaN@.  Must be set to "on" or "off" [default].

@item GradObj
When set to "on", the function to be minimized must return a second argument
which is the gradient, or first derivative, of the function at the point
@var{x}.  If set to "off" [default], the gradient is computed via finite
differences.

@item Jacobian
When set to "on", the function to be minimized must return a second argument
which is the Jacobian, or first derivative, of the function at the point
@var{x}.  If set to "off" [default], the Jacobian is computed via finite
differences.

@item MaxFunEvals
Maximum number of function evaluations before optimization stops.
Must be a positive integer.

@item MaxIter
Maximum number of algorithm iterations before optimization stops.
Must be a positive integer.

@item OutputFcn
A user-defined function executed once per algorithm iteration.

@item TolFun
Termination criterion for the function output.  If the difference in the
calculated objective function between one algorithm iteration and the next
is less than @code{TolFun} the optimization stops.  Must be a positive
scalar.

@item TolX
Termination criterion for the function input.  If the difference in @var{x},
the current search point, between one algorithm iteration and the next is
less than @code{TolX} the optimization stops.  Must be a positive scalar.

@item TypicalX

@item Updating
@end itemize
@end deftypefn


@c optimget scripts/optimization/optimget.m
@anchor{doc-optimget}
@deftypefn  {Function File} {} optimget (@var{options}, @var{parname})
@deftypefnx {Function File} {} optimget (@var{options}, @var{parname}, @var{default})
Return a specific option from a structure created by
@code{optimset}.  If @var{parname} is not a field of the @var{options}
structure, return @var{default} if supplied, otherwise return an
empty matrix.
@end deftypefn

