Confidence Band for Additive Regression Model

Additive model is widely recognized as an effective tool for di- mension reduction. Existing methods for estimation of additive regression function, including backfitting, marginal integration, projection and spline methods, do not provide any level of uniform confidence. In this paper a sim- ple construction of confidence band is proposed for the additive regression function based on polynomial spline estimation and wild bootstrap. Monte Carlo results show three desirable properties of the proposed band: excellent coverage of the true function, width rapidly shrinking to zero with increasing sample size, and minimal computing time. These properties make he pro- cedure is highly recommended for nonparametric regression with confidence when additive modelling is appropriate.


Introduction
Non-and semi-parametric smoothing estimation of unknown regression functions has found many applications in modelling nonlinearity which exists widely in various disciplines such as econometrics (Feng 2004, Racine andLi 2004), and biological sciences (Liang 2004, Liang, Wang, Robins andCarroll 2004).One common reservation about using nonparametric smoothing is the "curse of dimensionality", which refers to the lack of accuracy in estimating functions of general nonparametric multivariate form.Additive modelling is a very effective technique to alleviate the curse of dimensionality without constraining the regression function to simple linear form, see for example, Hastie and Tibshirani (1990) and the more recent works of Chen and Tsay (1993), Tjøstheim and Auestad (1994), Linton and Nielsen (1995), Yang, Härdle and Nielsen (1999), Sperlich, Tjøstheim and Yang (2002), and Yang, Sperlich and Härdle (2003), which applied various kernel smoothing techniques.Huang and Yang (2004), on the other hand, employed spline smoothing methods.
As with any estimation procedure, confidence regions provide more information than a single estimate can.With nonparametric smoothing, however, confidence bands for the regression functions have proven to be one of the more challenging problems, and existing literature is rather limited and narrow in scope.The early works of Hall and Titterington (1988), Härdle (1989) established asymptotic confidence band of nonparametric regression function which are univariate, based on Nadaraya-Watson kernel smoothing.This line of research has been extended to the more efficient local polynomial kernel smoothing in Xia (1998), which has further developed into bootstrap procedure in Claeskens and Van Keilegom (2003).All of the aforementioned methods involve local averaging by kernel weights, and requires solving a linear least squares problem for every design point.In contrast, Wang and Yang (2006) propose new confidence bands based on the polynomial spline method, which obliges the user to solving merely one linear least squares problem, and is practically as easy to implement and fast as a simple linear regression with slowly increasing number of parameters.Such nonparametric confidence bands, however, does not exist for estimation of additive regression function.Techniques based on kernel smoothing reply on complicated backfitting or averaging algorithms [also known as projection in Tjøstheim and Auestad (1994) and marginal integration in Linton and Nielsen (1995)], from which it is extremely difficult to construct confidence bands.Spline technique is therefore chosen for tackling this difficult problem.
In this paper the spline confidence band for univariate regression in Wang and Yang (2006) is extended to confidence band of multivariate additive regression.In section 2, I propose an asymptotically conservative confidence band for the nonparametric autoregression function.Intuitive justifications are given for the procedure, which is based on wild bootstrap.In section 3, I examine the properties of the procedure through simulation study and discuss the issues in implementation.Section 4 concludes.

Additive Confidence Band
The additive regression model is of the form .., X id } n i=1 generated from (2.1), estimators of the functions {m α (x α )} d α=1 and m (x) can be constructed by employing either kernel or spline smoothing.In this section, I describe the spline estimator m (x) of m (x) and the construction of a conservative confidence band based on m (x).
One needs to first introduce the spline basis of order p (p ≥ 1) as the following set of piecewise polynomials are equally spaced points on the range of variable X β , called knots.To be precise, denote the compact support of the density where the number of knots 2p+3) , in which k is a tuning constant.One then define for any where the coefficients λ0,1 , λ1,1 , ..., λp+N,d are solutions of the following least squares problem Although splines of any order p ≥ 1 can be employed, the simplicity of linear splines (with p = 1), which are piecewise linear functions, makes them especially appealing.Thus in this paper I have used only linear splines.The tuning constant is set to k = 1, which is not crucial based on observation from extensive simulation.In what follows, a confidence level 1 − α ∈ (0, 1) is fixed, whose default is 0.95.The main goal is to construct simultaneous confidence intervals for the value of function m (x) at each point in other words, a confidence band for the function m (x) over X .
For univariate spline regression (d = 1), Huang (2003) had obtained asymptotic pointwise confidence intervals of level 2) while σ m (x) is its standard error, whose explicit formula is contained in Huang (2003) in the conditional form, and Wang and Yang (2006) in the unconditional form.The latter had employed the Brownian bridge strong approximation theorem of Tusnády (1977) to obtain the following simultaneous confidence intervals based on linear spline estimator where I denote by χ 2 t,1−α the (1 − α)-quantile of the chi-square distribution with t degrees of freedom.Comparing the pointwise confidence interval of Huang (2003) and the simultaneous confidence intervals (i.e., a confidence band) of Wang and Yang (2006), one clearly see that both are centered at m (x), while simultaneity is achieved by inflating the pointwise interval at the ratio of Huang (2003) and Wang and Yang (2006) that σ m (x) is of order n −2/5 , hence the confidence band has a bound of order n −2/5 log (n) for its width throughout the interval [a 1 , b 1 ].
In the case of additive model (2.1), the deviation of m (x) from m (x) is to be bounded on a total of (N + 1) d d-dimensional intervals.On each interval, the deviation comes from each of the d variables.Hence the degree of freedom for the chi-square quantile in the inflation ratio is 2d, while the tail probability is 1 − α/ (N + 1) d .Hence the inflation ratio is naturally z −1 1−α/2 χ 2 2d,1−α/(N +1) d .In the case of univariate regression (i.e.d = 1), the ratio is the exact same one proved in Wang and Yang (2006).In addition, the confidence band should have a bound of order n −2/5 log (n) for its width throughout the compact set A rigorous proof of this asymptotic result would be feasible if strong approximation result as sharp as Tusnády (1977) would be available for d + 1 dimensions.Wang and Yang (2006) has used a plug-in procedure to estimate σ m (x) in the construction of the confidence band.The plug-in method, however, requires additional smoothing steps which may render the results less accurate, and hence has been replaced in this paper with wild bootstrap for improved performance.The steps are described here.In the following, I define the residuals εi = Y i − m (X i ) , 1 ≤ i ≤ n, and denote a predetermined large integer by n B , whose default value is 400.
Step 4 Denote the wild bootstrap (1 − α) confidence band for the function m (x) over X as m L,α/2 (x) , m U,α/2 (x) , x ∈ X .As mentioned earlier, the uniform confidence band needs to be wider than the pointwise confidence interval by a factor of z −1 1−α/2 χ 2 2d,1−α/(N +1) d when localized at any point, hence I define where The justification for making use of the wild bootstrap sample (2.4) is the same as in Sperlich, Tjøstheim and Yang (2002), i.e., the sample {Y i,b , X i } n i=1 mimmicks the dependence structure of sample {Y i , X i } n i=1 up to second order.Intuitively this is seen from the aforementioned fact that E (δ i,b ) ≡ 0, var (δ i,b ) ≡ 1 and therefore each δ i,b εi has the same mean and variance as εi , approximately 0 and σ 2 .In the next section, I will demonstrate the excellent practical performance of the proposed band by applying it to simulated examples.

Monte Carlo Study
To evaluate the performance of the confidence band defined in (2.6) and (2.7), the following Monte Carlo study has been carried out.The data generating model is an additive regression model of the form are generated according to model (3.1) for sample sizes n = 50, 100, 200, 400 and dimensions d = 2, 4, and the 95% conservative confidence band computed for each replication.All computing has been carried out in the environment of XploRe [see, for example, Härdle, Hlavka and Klinke (2000)] or visit the link http://www.quantlet.com/mdstat/scripts/xag/html/xaghtml.html The simulation attempts to address three questions: First, does the conservative confidence band given in (2.6) and (2.7) contains the additive function with empirical frequencies higher than the predetermined confidence level?Second, is the conservative confidence band too wide?Third, how much computing is needed to produce the confidence band?
The answer to the first question is extremely positive: for all the experiments, it is observed that the empirical frequencies of coverage are 100% in all cases.Hence the constructed band is indeed conservative in terms of coverage.
To address the second and third questions, I have plotted the bands as well as the true additive function against the first variable X (20) i,1 , in Figures 1 and 2, for the 20-th replication of each dimension d, sample size n combination.To be precise, the plots are made by connecting points (thin lines) and (thick line) respectively.For each dimension d, sample size n combination, I have also recorded in Table 1 the average runtime of one replication, as well as .
mean width of confidence band is defined as From Figures 1 and 2, one can see that the band narrows for the 20-th sample as sample size n increases.This is corroborated by the confidence bands' mean widths in Table 1, which decreases as sample size increases, thus the confidence bands are not "too conservative" so to speak.Table 1 also shows that the runtime for constructing one confidence band is less than one second when n = 400, hence the computing burden is extremely light.

Conclusions
In this paper, a method is proposed for constructing confidence band of additive nonparametric regression function, filling one significant current gap in statistical literature.The band covers the true regression function with frequency much higher than the nominal confidence level, can be computed within fractions of a second, and becomes narrower rather rapidly with increasing sample size.These three features make the proposed confidence band very appealing for application.
The idea of this paper could be easily extended to related regression models.For instance, the recent works of Xue and Yang (2006a,b) have made both kernel and spline estimation available for the so-called additive coefficient model, a class of model that includes as special cases additive model, functional coefficient model, and varying coefficient model.It is conceivable that similar confidence band could be constructed for this broad model class.
the response Y i in (2.3) with Y i,b and then solve the following least ,b B L,α (x α ) .

Table 1 :
Asymptotic performance and computing cost of the additive confidence band