A small sample study of some sandwich
estimators to handle heteroscedasticity
By Viking Westman
Department of Statistics
Uppsala University
Supervisor: Ronnie Pingel
Abstract
This simulation study sets out to investigate Heteroscedasticity-Consistent Covariance Matrix Estimation using the sandwich method in relatively small sample sizes. The different estimators are evaluated on how accurately they assign confidence intervals around a fixed, true coefficient, in the presence of random sampling and both homo- and heteroscedasticity. A measure of Standard Error is also collected to further analyze the coefficients. All of the HC-estimators seemed to overadjust in most homoscedastic cases, creating intervals that way overlapped their specifications, and the standard procedure that assumes homoscedasticity produced the most consistent intervals towards said specifications. In the presence of heteroscedasticity the comparative accuracy improved for the HC-estimators and they were often better than the non-robust error estimator with the exception of estimating the intercept, which they all heavily underestimated the confidence intervals for. In turn, the constant estimator was subject to a larger mean error for said parameter - the intercept. While it is clear from previous studies that Sandwich estimation is a method that can lead to more accurate results, it was rarely much better than, and sometimes strictly worse than the non-robust, constant variance provided by the OLS-estimation. The conclusion is to stay cautious when applying HC-estimators to your model, and to test and make sure that they do in fact improve the areas where heteroscedasticity presents an issue.
Key words
1. Introduction
Linear modelling is a core concept of statistics, and is frequently applied in an array of fields. Whether working in finance, epidemiology, sports or almost anything, both linear and more complex models allow us to understand simple theoretical concepts and predict outcomes based on data inputs. In its simplest form, a linear model is a correlation-based estimation of some ‘true’, underlying relationship between variables with some disturbances, the most notable being inherent variance. The goal of a statistician is typically to filter out as many of these disturbances as possible to find a model accurately descriptive of either some reality or theoretical concept. There are varying degrees of accuracy and robustness one can reach within a linear model. Excluding the variations caused by the amount and accuracy of the data - applying or accounting for certain statistical concepts can, if not improve our model’s accuracy, account more correctly for problems within the data input to increase robustness. A positive application can be a typical correlation-based analysis, while examples of problematic concepts are things like outliers in the data or incorrect statistical assumptions. Specifically, in ordinary least squares regression, those assumptions regard the: linearity of the parameter relationship, homoscedasticity in terms of residual variance, independence of observations and normality as for the model distribution. In this paper, homoscedasticity, and specifically methods to deal with the absence of it - heteroscedasticity, are looked at.
include Variance Stabilizing Transformations (VST, example taking square roots of squared variance), Weighted Least Squares (WLS), and Robust Standard Errors (RSE), which this paper specifically will deal with. In this paper, the RSE’s are obtained by Covariance Matrix Estimation through the sandwich method, covered more in depth in the theory section. Using RSE’s has the upside of wide and more general applicability, whereas VST and WLS tend to be case specific.
2. Theory
The seminal article by White (1980) introduced the sandwich-estimators the methods for dealing with heteroscedasticity, and they were further developed with the help of Heteroscedasticity-Consistent Covariance Matrix estimation. The sandwich method (eq. 1) unsurprisingly does this by ‘sandwiching’ a function labeled the “meat” (eq. 3) between two other functions labeled “bread” (eq. 2) (Zeileis 2004, 2006) in order to obtain the variance of the sample coefficient, .︿β
BMB
S = (1)
With the components:
(X X)
B = T −1 (2)
X ΩX
M = T (3)
For the standard OLS regression, β︿is the Best Linear Unbiased Estimator (BLUE) with a variance of s2(X X)T −1, consisting of the variance and the regressor observations where the heightened T indicates a matrix transposal. This is a simplification of the sandwich-esque equation, now in full:
X X)
( T −1 XTΩX (X X)T −1 (4)
(X X)
B = T −1 (5)
X ω X
M = T i (6)
Depending on your choice of HCE, ωi takes on one of several different forms clarified below. Several papers revolve around the sandwich package and theoretical application of these HC-estimators presented by White (HC0, 1980), MacKinnon & White (HC1-3, 1985), with a later addition from Cribari-Neto (HC4, 2004), and lastly HC5, excluded on the merits of being extra computationally requiring. The package developed for R is typically used in statistical environments that deal with heteroscedasticity, autocorrelation or both. Zeileis (2004) presented the estimators as the following:
3. Simulation Setup
The theoretical model looks like this:
β β X β X u
Yi = 0 + 1 1i + 2 2i + i (13)
It consists of the previously mentioned β- andu-parameters, along with the regressors, X, and the regressand, Y. The desire was to use R and the Sandwich package to construct a test similar, but not identical, to previous studies like MacKinnon & White (1985) and Long & Ervin (2000). The “true” or “fixed” coefficient values were partly on that basis and partly for interpretability set as the following:
17 β0 = β1 = − 2
5 β2 =
These fixed -coefficients were set arbitrarily and bear no tangible impact on the study results.β
4. Results
5. Discussion
Due to the scope of this paper, coefficient values and standard deviations that lead to the presented data in the first place were not generated as an output. It would be well within reason to study that particular data in order to further analyze the effects which the sandwich method has on the actual data, and it would have been the next step taken in this paper with more time and resources at hand. That analysis could potentially have been of greater interest if empirical data was used, but as for testing the estimators, entirely simulated data does just as well. The arbitrarily chosen fixed parameters, along with the rate of heteroscedasticity, could have been assigned different values. This would yield different results at varying samples, and there is an argument to be made as to having some really large/small coefficients that could in theory act differently and provide different issues with H-consistency, but this is maybe more than anything an argument for a deeper study on this subject. Heteroscedasticity often arises in cross-sectional data. If the simulated data doesn’t emulate the tendencies of cross-sectional data, or other issues encountered in practice, the HC-estimators are potentially not evaluated in terms of applicability or usefulness. Although arbitrary parameters can present an issue of discussion - questions of why not other arbitrary numbers are used and such, they could also potentially have strength in pointing out the lack of universality provided in a solution for a problem that often, but not always, presents itself in a certain way; like with heteroscedasticity and cross-sectional data, where it’s most commonly found. It is important to note that one can test for heteroscedasticity in empirical data, in order to avoid measures of varying degrees of complication in order to fix something that isn’t necessarily present. There are several methods, including residual analysis, the Breusch-Pagan test, etc. that can help the statistician when faced with uncertainty. Further, residual plots could have been attached to help with visual interpretation.
did not for any case perform much worse than any of the HC-estimators, except in terms of error size on specific coefficients. The β︿0-coefficient for the heteroscedastic sample sizes of especially 50 and 100 was subject to relatively large underestimation, and the cause of this is unclear. β︿0 is unique in that it is the intercept, but that did not seem to have the same effect on the results of MacKinnon & White (1985), as the pattern of underestimation was not present in their results. A discovery is that HC-estimators slightly tend to overcorrect in terms of CI width. Especially the HC3 and maybe HC4 estimator gave standard deviations that include the fixed coefficient values in the 97% range, most notably the homoscedastic cases with lower sample sizes.
6. Conclusion
It seems that HC-estimators sometimes over adjust the coefficient variance leading to incorrect confidence intervals in homoscedastic data. In the presence of heteroscedasticity they didn’t necessarily outperform the OLS constant in terms of assigning proper confidence intervals, except for maybe HC3 and HC4 as the sample size increased. Curiously, the OLS method assigned the most accurate CI’s to the intercept specifically, and even more so as the sample size increased. This was seemingly done at the cost of mean error size, which in turn offset a lot of the more striking tendencies of the CI-analysis, depending on your error tolerance. Compared to most previous studies, the results in this one do not praise the HC estimators as highly. The assumption going into this, based on most previous studies, was that the HC3 and HC4 estimators would outperform the other estimators, seeing as all sample sizes were smaller than the N = 250 recommendation. This wasn’t as clear cut of a case, again considering primarily the occasional over estimation of the active parameters, and the opposite for the intercept. There are apparent and problematic patterns in the results pointing to this, and it’s hard to motivate anything other than more research on the subject. Perhaps a study with a wide spread of different data types and sample sizes could conclude when and where HC-estimation is a suitable method and when it isn’t.
References
Carroll RJ, Wang S, Simpson DG, Stromberg AJ, Ruppert D. (1998) “The Sandwich (Robust Covariance Matrix) Estimator”. Unpublished Technical Report.
Cribari-Neto F (2004). “Asymptotic Inference Under Heteroscedasticity of Unknown Form”
Computational Statistics & Data Analysis, 45, 215–233.
Long JS, Ervin LH (2000). “Using Heteroscedasticity Consistent Standard Errors in the Linear Regression Model” The American Statistician, 54, 217–224.
MacKinnon JG, White H (1985). “Some Heteroscedasticity-Consistent Covariance Matrix Estimators with Improved Finite Sample Properties” Journal of Econometrics, 29, 305–325. White H (1980). “A Heteroscedasticity-Consistent Covariance Matrix and a Direct Test for Heteroscedasticity” Econometrica, 48, 817–838.
Zeileis A (2004). “Econometric Computing with HC and HAC Covariance Matrix Estimators”
Journal of Statistical Software, 11(10), 1–17. URL: http://www.jstatsoft.org/v11/i10/
Zeileis A (2006). “Object-oriented Computation of Sandwich Estimators”
