The ENTROPY Procedure (Experimental)

Maximum Entropy-Based Seemingly Unrelated Regression

In a multivariate regression model, the errors in different equations might be correlated. In this case, the efficiency of the estimation can be improved by taking these cross-equation correlations into account. Seemingly unrelated regression (SUR), also called joint generalized least squares (JGLS) or Zellner estimation, is a generalization of OLS for multi-equation systems.

Like SUR in the least squares setting, the generalized maximum entropy SUR (GME-SUR) method assumes that all the regressors are independent variables and uses the correlations among the errors in different equations to improve the regression estimates. The GME-SUR method requires an initial entropy regression to compute residuals. The entropy residuals are used to estimate the cross-equation covariance matrix.

In the iterative GME-SUR (ITGME-SUR) case, the preceding process is repeated by using the residuals from the GME-SUR estimation to estimate a new cross-equation covariance matrix. ITGME-SUR method alternates between estimating the system coefficients and estimating the cross-equation covariance matrix until the estimated coefficients and covariance matrix converge.

The estimation problem becomes the generalized maximum entropy system adapted for multi-equations as follows:

$\begin{eqnarray*} \mr {maximize} & H(p,w) \: = \: -p’ \, \ln (p) \: - \: w’ \, \ln (w) \\ \mr {subject\, to} & y \: = \: X \, Z \, p \: + \: V \, w \\ & 1_{KM} \: = \: (I_{KM} \, \otimes \, 1_{L}’) \, p \\ & 1_{MT} \: = \: (I_{MT} \, \otimes \, 1_{L}’) \, w \end{eqnarray*}$

where

$\beta \: = \: Z \, p$

$\! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \ \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! Z \: = \: \left[ \begin{array}{ccccccccccccccc} z_{11}^{1} & \cdot \cdot \cdot & z_{L1}^{1} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & \ddots & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & z_{11}^{K} & \cdot \cdot \cdot & z_{L1}^{K} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & \ddots & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & z_{1M}^{1} & \cdot \cdot \cdot & z_{LM}^{1} & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \ddots & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & z_{1M}^{K} & \cdot \cdot \cdot & z_{LM}^{K} \end{array} \right]$

$p \: = \: \left[ \begin{array}{ccccccccccccccc} p_{11}^{1} & \cdot & p_{L1}^{1} & \cdot & p_{11}^{K} & \cdot & p_{L1}^{K} & \cdot & p_{1M}^{1} & \cdot & p_{LM}^{1} & \cdot & p_{1M}^{K} & \cdot & p_{LM}^{K} \end{array} \right]’$

$e \: = \: V \, w$

$\! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \ \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! V \: = \: \left[ \begin{array}{ccccccccccccccc} v_{11}^{1} & \cdot \cdot \cdot & v_{11}^{L} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & \ddots & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & v_{1T}^{1} & \cdot \cdot \cdot & v_{1T}^{L} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & \ddots & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & v_{M1}^{1} & \cdot \cdot \cdot & v_{M1}^{L} & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \ddots & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & v_{MT}^{1} & \cdot \cdot \cdot & v_{MT}^{L} \end{array} \right]$

$\! \! \! \! \! \! \! \! \! \! w \: = \: \left[ \begin{array}{ccccccccccccccc} w_{11}^{1} & \cdot & w_{11}^{L} & \cdot & w_{1T}^{1} & \cdot & w_{1T}^{L} & \cdot & w_{M1}^{1} & \cdot & w_{M1}^{L} & \cdot & w_{MT}^{1} & \cdot & w_{MT}^{L} \end{array} \right]’ \,$

y denotes the MT column vector of observations of the dependent variables; $\mb {X}$ denotes the (MT x KM ) matrix of observations for the independent variables; p denotes the LKM column vector of weights associated with the points in Z; w denotes the LMT column vector of weights associated with the points in V; $1_{L}$ , $1_{KM}$ , and $1_{MT}$ are L-, KM-, and MT-dimensional column vectors, respectively, of ones; and $\mb {I_{KM}}$ and $\mb {I_{MT}}$ are (KM x KM) and (MT x MT) dimensional identity matrices. The subscript l denotes the support point $(l=1, 2, \ldots , L)$ , k denotes the parameter $(k=1, 2, \ldots , K)$ , m denotes the equation $(m=1, 2, \ldots , M)$ , and t denotes the observation $(t=1, 2, \ldots ,T)$ .

Using this notation, the maximum entropy problem that is analogous to the OLS problem used as the initial step of the traditional SUR approach is

$\begin{eqnarray*} \mr {maximize} & H(p,w) \: = \: -p’ \, \ln (p) \: - \: w’ \, \ln (w) \\ \mr {subject\, to} & \, (y \: - \: X \, Z \, p ) \: = \sqrt {\Sigma } \: V \, w \\ & 1_{KM} \: = \: (I_{KM} \, \otimes \, 1_{L}’) \, p \\ & 1_{MT} \: = \: (I_{MT} \, \otimes \, 1_{L}’) \, w \end{eqnarray*}$

The results are GME-SUR estimates with independent errors, the analog of OLS. The covariance matrix $\hat{\Sigma }$ is computed based on the residual of the equations, $Vw=e$ . An $L’L$ factorization of the $\hat{\Sigma }$ is used to compute the square root of the matrix.

After solving this problem, these entropy-based estimates are analogous to the Aitken two-step estimator. For iterative GME-SUR, the covariance matrix of the errors is recomputed, and a new $\hat{\Sigma }$ is computed and factored. As in traditional ITSUR, this process repeats until the covariance matrix and the parameter estimates converge.

The estimation of the parameters for the normed-moment version of SUR (GME-SUR-NM) uses an identical process. The constraints for GME-SUR-NM is defined as:

$X’y \: = \: X’(\mb {S} ^{-1} {\otimes } \mb {I} )X \, Z \, p \: + \: X’(\mb {S} ^{-1} {\otimes } \mb {I} )V \, w$

The estimation of the parameters for GME-SUR-NM uses an identical process as outlined previously for GME-SUR.