The ENTROPY Procedure (Experimental)

Maximum Entropy-Based Seemingly Unrelated Regression

In a multivariate regression model, the errors in different equations might be correlated. In this case, the efficiency of the estimation can be improved by taking these cross-equation correlations into account. Seemingly unrelated regression (SUR), also called joint generalized least squares (JGLS) or Zellner estimation, is a generalization of OLS for multi-equation systems.

Like SUR in the least squares setting, the generalized maximum entropy SUR (GME-SUR) method assumes that all the regressors are independent variables and uses the correlations among the errors in different equations to improve the regression estimates. The GME-SUR method requires an initial entropy regression to compute residuals. The entropy residuals are used to estimate the cross-equation covariance matrix.

In the iterative GME-SUR (ITGME-SUR) case, the preceding process is repeated by using the residuals from the GME-SUR estimation to estimate a new cross-equation covariance matrix. ITGME-SUR method alternates between estimating the system coefficients and estimating the cross-equation covariance matrix until the estimated coefficients and covariance matrix converge.

The estimation problem becomes the generalized maximum entropy system adapted for multi-equations as follows:

\begin{eqnarray*} \mr{maximize} & H(p,w) \: = \: -p’ \, \ln (p) \: - \: w’ \, \ln (w) \\ \mr{subject\, to} & y \: = \: X \, Z \, p \: + \: V \, w \\ & 1_{KM} \: = \: (I_{KM} \, \otimes \, 1_{L}’) \, p \\ & 1_{MT} \: = \: (I_{MT} \, \otimes \, 1_{L}’) \, w \end{eqnarray*}

where

\[ \beta \: = \: Z \, p \]
\[ \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \ \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! Z \: = \: \left[ \begin{array}{ccccccccccccccc} z_{11}^{1} & \cdot \cdot \cdot & z_{L1}^{1} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & \ddots & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & z_{11}^{K} & \cdot \cdot \cdot & z_{L1}^{K} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & \ddots & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & z_{1M}^{1} & \cdot \cdot \cdot & z_{LM}^{1} & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \ddots & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & z_{1M}^{K} & \cdot \cdot \cdot & z_{LM}^{K} \end{array} \right] \]
\[ p \: = \: \left[ \begin{array}{ccccccccccccccc} p_{11}^{1} & \cdot & p_{L1}^{1} & \cdot & p_{11}^{K} & \cdot & p_{L1}^{K} & \cdot & p_{1M}^{1} & \cdot & p_{LM}^{1} & \cdot & p_{1M}^{K} & \cdot & p_{LM}^{K} \end{array} \right]’ \]
\[ e \: = \: V \, w \]
\[ \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \ \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! \! V \: = \: \left[ \begin{array}{ccccccccccccccc} v_{11}^{1} & \cdot \cdot \cdot & v_{11}^{L} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & \ddots & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & v_{1T}^{1} & \cdot \cdot \cdot & v_{1T}^{L} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & \ddots & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & v_{M1}^{1} & \cdot \cdot \cdot & v_{M1}^{L} & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \ddots & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & v_{MT}^{1} & \cdot \cdot \cdot & v_{MT}^{L} \end{array} \right] \]
\[ \! \! \! \! \! \! \! \! \! \! w \: = \: \left[ \begin{array}{ccccccccccccccc} w_{11}^{1} & \cdot & w_{11}^{L} & \cdot & w_{1T}^{1} & \cdot & w_{1T}^{L} & \cdot & w_{M1}^{1} & \cdot & w_{M1}^{L} & \cdot & w_{MT}^{1} & \cdot & w_{MT}^{L} \end{array} \right]’ \, \]

y denotes the MT column vector of observations of the dependent variables; $\mb{X}$ denotes the (MT x KM ) matrix of observations for the independent variables; p denotes the LKM column vector of weights associated with the points in Z; w denotes the LMT column vector of weights associated with the points in V; $1_{L}$, $1_{KM}$, and $1_{MT}$ are L-, KM-, and MT-dimensional column vectors, respectively, of ones; and $\mb{I_{KM}}$ and $\mb{I_{MT}}$ are (KM x KM) and (MT x MT) dimensional identity matrices. The subscript l denotes the support point $(l=1, 2, \ldots , L)$, k denotes the parameter $(k=1, 2, \ldots , K)$, m denotes the equation $(m=1, 2, \ldots , M)$, and t denotes the observation $(t=1, 2, \ldots ,T)$.

Using this notation, the maximum entropy problem that is analogous to the OLS problem used as the initial step of the traditional SUR approach is

\begin{eqnarray*} \mr{maximize} & H(p,w) \: = \: -p’ \, \ln (p) \: - \: w’ \, \ln (w) \\ \mr{subject\, to} & \, (y \: - \: X \, Z \, p ) \: = \sqrt {\Sigma } \: V \, w \\ & 1_{KM} \: = \: (I_{KM} \, \otimes \, 1_{L}’) \, p \\ & 1_{MT} \: = \: (I_{MT} \, \otimes \, 1_{L}’) \, w \end{eqnarray*}

The results are GME-SUR estimates with independent errors, the analog of OLS. The covariance matrix $\hat{\Sigma }$ is computed based on the residual of the equations, $Vw=e$. An $L’L$ factorization of the $\hat{\Sigma }$ is used to compute the square root of the matrix.

After solving this problem, these entropy-based estimates are analogous to the Aitken two-step estimator. For iterative GME-SUR, the covariance matrix of the errors is recomputed, and a new $\hat{\Sigma }$ is computed and factored. As in traditional ITSUR, this process repeats until the covariance matrix and the parameter estimates converge.

The estimation of the parameters for the normed-moment version of SUR (GME-SUR-NM) uses an identical process. The constraints for GME-SUR-NM is defined as:

\[ X’y \: = \: X’(\mb{S} ^{-1} {\otimes } \mb{I} )X \, Z \, p \: + \: X’(\mb{S} ^{-1} {\otimes } \mb{I} )V \, w \]

The estimation of the parameters for GME-SUR-NM uses an identical process as outlined previously for GME-SUR.