On Comparison of Some Ridge Parameters in Ridge Regression

In this article, a new approach to obtain the ridge parameter introduces for the multiple linear regression model suffers from the problem of multicollinearity. Furthermore, we compare the proposed ridge parameter with the other well-known ridge-parameters through ridge estimators evaluated elsewhere in terms of mean squares error (MSE) criterion. Finally, a numerical example and simulation study has been conducted to illustrate the optimality of the proposed ridge parameter.


Introduction
Consider the linear regression model where Y is a n×1 random vector of response variables, X is a known n×p matrix with full column rank,  is a n×1 vector of errors with E() = 0 and Cov() = 2  I n .  is a p×1 vector of unknown regression parameters and 2  is the unknown variance parameter. For the sake of convenience, we assume that the matrix X and response variable Y are standardized in such a way that X X ' is a non-singular correlation matrix and Y X ' is the correlation between X and Y. The most common estimator for  is the ordinary least squares estimator where Z = XT , which implies that Z Z ' =  , and α =  ' T (Montgomery et al., [25]) Generally, we use the ordinary least squares technique to estimate the regression parameter α because of its simplicity and easiness of computation.
Then OLS estimator of α is given by ( Therefore, OLS estimator of β is given by OLS OLS T  . However, in some situations the regressors are nearly or perfectly linearly related, the problem of multicollinearity is said to exit and in such cases the usual inference based on such models will become erroneous. Multicollinearity also tends to produce OLS estimates that are unstable and large in absolute values. Ordinary Ridge regression (RR) introduced by Hoerl and Kennard [15] is intended to overcome the problem of multicollinearity by adding a positive constant (or ridge parameter) k , normally lies between 0 and 1, to the diagonal elements of the least square estimator. It is given as: We observe that, when k = 0 in (5), MSE of OLS estimator of α is recovered. Hence Numerous papers have been written, for estimating the  in the presence of multicollinearity. Some of the selected well known methods used for estimation are listed below.
Since ridge estimators can have a serious bias, Singh and Chaubey [30] introduced the Jackknifed ridge regression (JRR) estimator by using the jackknife procedure to reduce the bias of the generalized ridge regression (GRR) estimator. It is given as: and mean square error of Batah et al. [4] suggested a new ridge estimator namely Modified Jackknifed ridge regression (MJR) estimator by combining idea of GRR and JRR. They established the MSE superiority over both the GRR and JRR estimators. It is given as: Also, mean square error of Recently, Fallah and Salam, [7] introduce an alternative shrinkage estimator, called modified unbiased ridge (MUR) estimator. This estimator is obtained from Unbiased Ridge Regression (URR), and stated that the MUR estimator is more efficient and more reliable than OLS, RR, and URR estimators based on Matrix Mean Squared Error (MMSE). It is given as: Much of the discussions on ridge regression concern the problem of finding good empirical value of k . Many different techniques for estimating k have been proposed or suggested by various researchers. To mention a few, Hoerl and Kennard [15], Hoerl et al. [16], McDonald and Galarneau [24] , Hocking et al. [14], Lawless and Wang [20], Kibria [18], Dempster et al. [5], Gunst and Mason [11], Wichern and Churchill [33], Hemmerle and Brantle [13], Lawless [21], Golub et al. [9], Gibbons [8], Nordberg [28], Saleh and Kibria [29], Haq and Kibria [12], Singh and Tracy [31], Wencheko [32], Kibria [19], Khalaf where is variance inflation factor of j th regressor.
All these methods of estimating ridge parameter are used in Section 3. The primary aim in this article is to introduce a new estimation method for ridge parameter and hence provide an alternative estimation method for unknown regression parameters to overcome multicollinearity in the linear regression. Then well known ridge estimators reviewed in this article are compared. The organization of the paper is as follows. We propose the new ridge parameter in Section 2. A numerical example and ISBN-1391-4987 IASSL simulation study has been conducted to study the performance of proposed ridge parameter by comparing it with the other ridge parameters. Performance of various ridge parameters through different ridge estimators in terms of mean square error (MSE) criterion are given in Section 3. Finally some concluding remarks are given in Section 4.

Proposed estimator for ridge parameter
It is well known that 2 max 2  is the upper bound of ridge parameter used in ridge regression stated by Hoerl and Kennard [15]. Based on the same upper bound Alkhamisi and Shukur [2] present a new method to estimate the ridge parameter k , which is presented below: Furthermore, by taking the maximum of both terms as an alternative to the above method they derived the following estimator of the ridge parameter In this article, we present a new method to estimate the ridge parameter k . Initially, we suggest the modification by taking the minimum instead of taking the maximum of the term used in above (22), that is . Also, it is well known that 2  plays an important role in most of the ridge parameters suggested by various researchers. We suggest our ridge parameter by multiplying 2  to the reciprocal of , which is presented below:

Numerical Example
To illustrate the performance of our proposed estimator of k , we give a numerical example to investigate the estimators discussed in the dataset which was discussed in Gruber [10]. Condition number of the dataset is 146.4233, which clearly indicates that data exists with multicollinearity. Data shows Total National Research and Development Expenditures as a Percent of Gross National Product by Country: 1972Country: -1986. It represents the relationship between the dependent variable Y the percentage spent by the United States and the four other independent variables X 1 , X 2 , X 3 and X 4 .
It is convenient to make comparison among proposed ridge parameter and other ridge parameters given in (9) to (22). We compute estimated MSE values of well known ridge regression estimator RR  using (5) at different ridge parameters, and the values are reported in Table 1.    [16]. But, at the same time, we have to note that the estimated MSE value of RR  at the  k is always smaller than at the remaining ridge parameters.

Simulation Study Part A:
We are now ready to illustrate the behavior of the proposed ridge parameter via a simulation study. The simulation is carried out under different degrees of multicollinearity. Proposed estimator for estimating ridge parameter k is then compared in the sense of MSE criterion to the other ridge parameters reviewed in this article. We consider the true model as where, u ij are independent standard normal pseudo-random numbers and  is specified so that the theoretical correlation between any two explanatory variables is given by  2 .
To examine the robustness of all parameters under consideration, random numbers are generated for the error terms () from each of the normal, t, F, Chi-square and exponential distributions respectively. In this study, to investigate the effects of different degrees of multicollinearity on the estimators, we consider two different  Tables 2 and 3. We consider the method that leads to the minimum AMSE to the best from the MSE point of view.
From Tables 2 and 3 (See Appendix), we observe that performance of our proposed ridge parameter  k is equivalent but slightly better than ridge parameter 1 k proposed by Hoerl et al. [16]. However, table values clearly shown that  k is more efficient in terms of MSE than rest of the ridge parameters used in the simulation study for various values of triplet (  , n, distribution of  ).

Part B:
In part A, of the simulation study we compared proposed and other ridge parameters among themselves in the sense of MSE criterion. Here, we demonstrate the performance different estimation methods alternate to OLS estimator which are used to estimate unknown regression parameters in the presence of multicollinearity. Obviously, in the present comparative study we used different well known ridge estimators computed using different ridge parameters including proposed ridge parameter  k . We consider a simulated data on two different models I and II. For model I and II we have generated random samples respectively from N 4 (0, Σ 1 ) and N 3 (0, Σ 2 ). For both models, we have generated observations on a response variable Y with the following specifications on number of predictors (p), sample size (n) and distribution of the error term (  ).
Obviously, multicollinearity is present in the simulated data generated from both the models. Using this data, we compute different estimators given in (4) and (6) to (8) by using each ridge parameters given in (9) to (22) Table 4 for both the models I and II. From

Conclusion
In this article we suggest a new approach to obtain an estimator of the ridge parameter k . New estimator is evaluated and compared with well-known existing estimators in terms of MSE criterion. Also, well-known existing estimators of the unknown regression parameters are evaluated using different ridge parameters including proposed ridge parameter. The investigation has been carried out using Monte Carlo simulations. Here we try to suggest the best estimation method for estimating unknown regression parameters based on proper choice of ridge parameter. Finally, from a numerical example and the simulation study, we found that the performance of the proposed estimator of the ridge parameter k is