There was a post on TalkStats quite a while ago that I ended up recalling for another thread. I thought I’d post it here for a better reference and because the idea is important. A poster asked when it is appropriate to run a regression without an intercept. There was some discussion but mainly we decided that it’s not usually a good idea. Bryangoodrich posted this:

“If you purposely exclude it, then you’re basically forcing the regression to be (0, 0) when all the X terms are 0″

And I replied with:

And note that even though in a lot of situations this sounds desirable – it isn’t a reason that should force you to not include an intercept. The reason being that you are forcing an added restriction on your model. You’re already making the assumption that the response is linear with respect to the predictors. This might be a reasonable approximation locally but if all of your predictors are very far away from 0 then this greatly influences the fit of the model.

Consider the following fake dataset:

Let’s look at the predictions for the models with an intercept and the predictions for a model without the intercept.

Now it might sound silly to fit a model with no intercept to this data. But what if I told you that the data was generated from a model where y would be 0 when x is 0. The model is . Now this is a quadratic model but the linear fit with an intercept gives a good local approximation for the data we had. If you knew that the response needed to be 0 when x was 0 and forced that into your model you’re only hurting yourself in this situation.

Normality is, perhaps, the most common assumption in statistics. Many tests and models are somehow related to a normal distribution. Testing for normality is a common procedure and we all know (or should know) how to test for univariate normality: we have the Shapiro-Wilk test, the Shapiro-Francia test, Skewness and Kurtosis coefficients and my personal favorites, graphical displays. So, no problem there. The issues start when we talk about multivariate normality.

Many researchers don’t test for multivariate distributions, maybe because some multivariate techniques are very robust against deviations from multivariate normality, like discriminant analysis, for example. Still, it is important to verify the assumption in order to understand the severity of  violations. Also, we should remember that some multivariate procedures are really, really, sensitive to non-normal distributions, such as Box’s M statistic, which is almost useless even with some minor deviations from multivariate normality. Ok, testing normality is good, but how can I test the multivariate hypothesis?

The most common way of testing multivariate normality, one that I really do not like but that many authors recommend, is testing for univariate and bivariate normality. Normality on each of the variables separately is a necessary, but not sufficient, condition for multivariate normality to hold. That is, each of the individual variables must be normally distributed for the variables to follow a multivariate normal distribution. Another property of a multivariate normal distribution imply that all pairs of variables must be bivariate normal. Bivariate normality, for correlated variables, implies that the scatterplots for each pair of variables will be elliptical; the higher the correlation, the thinner the ellipse. So, as a partial check on multivariate normality, one could verify univariate normality in every single variable and then obtain the scatterplots for pairs of variables and see if they are approximately elliptical.

As I wrote, many authors support this procedure. Gnanadesikan (1977) stated: “In practice, except for rare or pathological examples, the presence of joint (multivariate) normality is likely to be detected quite often by methods directed at studying the marginal (univariate) normality of the observations on each variable”. Johnson and Wichern (1992) agreed: “Moreover, for most practical work, one-dimensional and two-dimensional investigations are ordinarily sufficient. Fortunately, pathological data sets that are normal in lower dimensional representations but non normal in higher dimensions are not frequently encountered in practice”.

Personally, I don’t think a univariate analysis should be the standard procedure for analyzing multivariate normality, since there are many other good methods available for studying this distribution directly, not only through hypothesis testing but also with some graphical tools.

Indeed, there are tests developed for testing multivariate normality. Mardia (1970) produced a generalization of both skewness and kurtosis measures for multivariate normal distributions. From these, two statistics were developed: Mardia’s statistic of multivariate kurtosis and Mardia’s statistic of multivariate skewness. An omnibus test was later proposed. There’s also a multivariate Shapiro-Wilk test, proposed by Royston (1983). This one combines the univariate tests creating a statistic H which has an approximate  distribution with non-integer degrees of freedom e. Using HATCO’s data, a common data set taken from an industrial enterprise and used in the multivariate statistics book from Hair, we can calculate some of these statistics:

 Mardia’s Statistic of Multivariate Skewness= 25.3078

Mardia’s Statistic of Multivariate Kurtosis=74.6837

So far, the most popular statistic for multivariate normality (at least to my humble knowledge) is the Doornik-Hansen test (2008). To obtain this statistic the multivariate observations are transformed and then univariate skewness and kurtosis for each of the transformed variables are computed. These measures are finally combined in an approximate χ2 statistic:

Doornik-Hansen Multivariate Normality Test

As we can see, results tend to agree: our dataset does not seem to adjust a multivariate normal distribution. Sadly, most of this values are not included in common statistics software. If you happen to fall in this case, don’t worry, there are more solutions. Stevens (1986) included in the first two editions of his book “Applied Multivariate Statistics” a graphical procedure to assess for multivariate normality. The result is a graph very similar to those obtained with normal probability plots.

Remember that the Mahalanobis distance of a multivariate vector with a mean vector and covariance matrix S is formally defined as:

Well, the trick is simple: first, calculate Mahalanobis Distance of your data and plot it against ordered quantiles of a Chi-square distribution. I don’t recall any function in R that can immediately obtain this graph, though we have the mahalanobis and qchisq functions. The resulting graph for the HATCO data is something like this:

This graph is a powerful way to assess for multivariate normality. As in any normality plot, we wish to see a straight 45° line that ensures multivariate normality. I like it because it not only tells you whether your data is multivariate normal or not, but it may also show what’s wrong. In our dataset, there are two observations that deviate heavily from multivariate normality, possibly outliers, certainly cases that require further attention.

As you can see, there are several ways to test for multivariate normality and some are easy to implement. Please don’t forget that distributional assumptions are important and if these are not met, your analysis may not be as powerful as you wished.

]]> http://www.statisticspedia.com/articles/basic-statistics/testing-multivariate-normality/feed/ 1 http://www.statisticspedia.com/articles/statistical-research/pca-step-by-step3/ http://www.statisticspedia.com/articles/statistical-research/pca-step-by-step3/#comments Wed, 25 May 2011 20:57:40 +0000 terzi http://www.statisticspedia.com/?p=283

You can find the first two sections of this tutorial here:

Principal Components Analysis – Step by Step (Part I)

Principal Components Analysis – Step by Step (Part II)

Hi! This is the last part of this tutorial, a tiny introduction to Principal Components Analysis. So far we have seen the basics of the analysis and some interpretations. Now, we’ll see some post estimation statistics and values that are usually used to qualify the overall results of our PCA. In fact these methods look more like pre estimation commands rather than post estimation ones, yet these are commonly used at the end to confirm the results.

I think the most common post estimation measure for PCA is the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy, so let’s start with that. KMO takes values between 0 and 1, with small values indicating that overall the variables have too little in common to warrant a PCA analysis. This is done by comparing the partial correlations and the correlations of our variables. Historically (or at least according STATA’s help file), the following labels are often given to values of KMO:

0.00 to 0.49 >>> Unacceptable

0.50 to 0.59 >>> Miserable

0.60 to 0.69 >>> Mediocre

0.70 to 0.79 >>> Middling

0.80 to 0.89 >>> Meritorious

0.90 to 1.00 >>> Marvelous

I wonder why almost all labels start with an “M”. Anyway, for our example, the KMO resulted in 0.8153 which is “Meritorious”. Wow, it sound like we would get a medal for that, so let’s just call it OK. Usually, when KMO turns out to be low that is also seen in the unexplained variance. Analysis with low KMO values will usually need a higher number of components to obtain a good representation. Remember that KMO is calculated based on correlations so it is not influenced by the number of components retained in a PCA.

There are other ways to check whether the analysis properly described our data, such as using a post estimation procedure to understand the residuals. Wait, residuals? In some forms, PCA can be seen as a model, actually Pearson first developed it as some form of regression model. It is possible to invert the equation that produces PC from data, in order to produce a formula that can compute the data from the Principal Components. However, the original data matrix will only be reproduced when all the components are retained. Certainly, that almost never happens. Because of that, it is useful to estimate our data using only the retained components, then analyzing the distance between these predicted outcomes and the real values. These residuals can be can be tested by means of the sum of squares of the residuals. This is called the Q-statistic or Rao statistic.

The values for the Q-statistic in our problem are shown above for every observation:

There’s a way to obtain a critical value, in order to test for problems or outliers in Q-statistics. In this case, it will be enough to notice that two states have remarkably higher values, that may indicate that these states are not properly represented by the first two components. If any of this states were absolutely important for our results, it may be necessary to retain more PC’s. Since that’s not our case, it should be enough to comment in our conclusions that the states of Tabasco and Yucatán were not properly represented by our analysis.

In a very similar way, t is also possible to analyze the fitted correlation matrix to understand which variables and relationships were not fully incorporated in these results. We can now calculate the residual correlation matrix, whose elements are the difference between the actual and the fitted correlations. For our example, we get the following matrix or residuals:

Most relationships have been properly represented, with most residuals below 0.06. The misrepresentation is mostly due to the variation in the variables, which are not fully accounted. Some variables like sewer coverage, water coverage and life expectancy have some large residuals (in bold). Does this sound familiar? Remember that when we analyzed the unexplained variance accounted for each variables we got these exact same numbers. Now you know where they came from.

This additional post estimation analysis must be incorporated in any Principal Components Analysis or ate least some of this results. This information is important in order to understand the solution that was obtained. As you can imagine, most analysis will drop several components but using these measures the analyst can easily understand the impact of that unexplained variance.

Well, that is about everything I can contribute regarding how to perform a good Principal Components Analysis. Even when there are many things we didn’t discuss, such as analysis of characteristic roots, inferential procedures, rotations, etc. I still hope this humble practical guide will help you in understanding the concepts that lies beneath this beautiful statistical tool. Don’t forget to check the references out in order to gain some deeper knowledge about PCA and remember that we have a whole forum and a this great website in order to help those in statistical need.

Have a nice day!

REFERENCES

• Cahill, Miles B. et al Using principal components to produce an economic and social development index: An application to Latin America and the U.S. Atlantic Economic Journal Volume 29, Number 3.
• Jackson, J. Edward A User’s Guide to Principal Components. Wiley – Interscience, 1991.
• Lattin, James Analyzing Multivariate Data Duxbury Press, 2002.
• Rabe-Heskett, Sophia A Handbook of Statistical Analyses using STATA. CRC Press, 2004.
• Venables, W.N. & Ripley, B.D. Modern Applied Statistics with S. Springer, 2002.

Hi again! This time I’ll continue with some results and interpretations from a Principal Component Analysis, hoping this tiny tutorial can help you understand some of the basics. Although we won’t cover every aspect, some ideas will be useful when you happen to need PCA. The application that we’ve been working is a study aimed to compare development among the states in Mexico. Last time, we started our analysis and decided to retain two principal components. You can check that last post here. It is time to continue with interpretations.

Last time we decided to keep two components, so it is time check the first two eigenvectors that resulted from our Singular Value Decomposition. First of all, it is important that we retain only two principal components. I mean, it is important to let the software know it. By doing that, most programs will show us some important information regarding the unexplained variance. Recall that our first two components will allow us to represent 87% of the variation, so there’s still a 13% hanging around. After adjusting the solution for two components, we can see them in order to understand the variables involved:

Let’s begin analyzing the unexplained variance. As you see, sewer access is the variable that was less explained by our two components. The rest have only a mere 15% of variance unexplained at most. Since there is no single variable with a high unexplained rate, we can assume that most variables will be correctly represented.

One of the most important interpretations in PCA is that of the components, the eigenvectors. Each component requires an interpretation: based on the variables that are important we can even “name” our components. As you can see, in PC1 almost all variables have the same value. Remember that one of our main reasons for analyzing the components is to understand the numbers in the index related to it, the scores that we will discuss below. The higher numbers, in absolute value, mean that those particular variables are more important to that component. In PC1 all variables seem to contribute and most have positive sign, which means that a state with larger values in GDP, access to services and life expectancy will get larger numbers in that index. Notice that the variable illiteracy has a negative sign: a state with high illiteracy will have low values in this index. It is important to realize how cool PCA is, since no one ever told the software that illiteracy was a bad sign for development. Anyway, our first PC is an index of “overall development” or wellbeing, where states with higher scores can be seen as those with a higher quality of life.

One of the reasons why we kept two components is because the interpretation of the first one was quite easy, don’t you think? I’d bet most books have examples like it. The second PC is something that resembles some real situations and I find its interpretation more worthy for the purpose of this tutorial. Two variables have almost zero values: water and illiteracy, which means that their effect in the component is almost nonexistent, we can ignore them. Sewer and Electricity have positive values. GDP and life expectancy have negative values. This component is a contrast. The analysis detected some states with low economic performance but great infrastructure. Politics, certainly. The key is to understand what a score would mean for this index. Low numbers in this score would mean great GDP and life expectancy in a state with poor infrastructure for its citizens. High values would mean great infrastructure but an economic performance not so good. The ideal for any state would be near zero values, which would mean some balance… or maybe slightly negative, everyone wants to be rich, right? This weird situation arose since GDP is the least correlated variable in our group. Check the correlation matrix and you’ll see yet another proof that richer people aren’t always better.

From the principal components we can obtain one of the most important results in PCA: principal component scores. Those are our “indexes”. Scores represent the values that each unit, in this case, each state would get in each component. This is crucial, specially since we now understand what the values in the components would mean. With the first scores, we can see the valuation each state got in our “overall development” index, our first PC:

It is easy to tell which are the five most developed states and the five least developed. I live in Veracruz, by the way so I’m starting to regret analyzing this data. Anyway, remember that this first scores only give us a 74% of the real variation. In order to get that 87% we wish, we should check both indexes at once. That is usually done by graphing indexes, i.e. both scores, in a scatterplot, which is commonly named Principal Components Score Plot or Principal Components Biplot. We can see it here:

There is a lot to learn from this graph. First of all, who’s enjoying the best life? Well, our first score is represented by the x-axis, so states at the left have lowest values in our overall well being index, our first PC. So, the best states for living should be in the right part of the graph: too bad for Veracruz. Do you remember what we concluded regarding PC2? Low numbers would mean great GDP and life expectancy but poor infrastructure and high values would mean great infrastructure but low economic performance. Since it was stated that the ideal for any state would be near zero or slightly negative, those states above Jalisco may be considered less developed. One could say that Jalisco, Nayarit, Distrito Federal and the states in the low right corner are the winners: the most developed areas. Another really interesting result are the clusters that appear in the graph. It is easy to notice three or maybe four different groups: PCA is a good way to start a cluster analysis. From the graph you can also find the situation of any individual state. For instance, Distrito Federal, which is the official name of the capital of Mexico, Mexico City, is certainly one of the most developed areas. Points that are near in the graph represent similar situations in those estates.  It is easy to compare states with each others or even geographic areas.

Notice that two states are colored in gray in the graph. That is because a residual analysis showed that these two points were not properly represented: these are part of that 13% PCA could not get. But wait, Residual analysis in PCA? Yes, we’ll see that when we get to the final part of this tutorial: Assessing the fit in the solution.

Principal Components Analysis – Step by Step (Part III)

Hi everyone. For my first post I’ll start a basic step by step tutorial regarding the use of principal component analysis, an old technique that can be dated back to 1901 with the first approaches by Pearson. It was Hotelling who derived a definitive solution about three decades later. Although the mathematical background in the analysis isn’t too complex, I’ll try to avoid formulas to focus on a single application that may be useful to understand the basics of the analysis. I hope this turns out helpful at all.

Mexico is divided in 32 states (Actually 31 and a federal district, but let us consider 32 states). Some places have a better quality of life and a higher level of development than others, as it happens in every country in the world. Therefore, one interesting subject of study would be a comparison of that development among the states. But, what exactly is development? Well, that isn’t just a number, right? That would include many things: life expectancy in the state, Gross Domestic Product, percentage of people with access to fresh water, portion of population that is illiterate. It seems we should compare all these things at once in order to perform a good comparison. And that’s what PCA is good for. The data we will use has the following form:

As you can see, there are 6 variables, all in continuous scale. I just used those for convenience, but some analysis will include twenty, forty or even hundreds. How would you compare all of them at once? One way would be to produce an index and then compare that index, right? But another question arises: how do we get that index? We could get the sum of all the variables, like some psychological tests do. We could get an average, or sum all the variables and then subtract the year I was born. Those are good indexes, but which one would be the best?

In PCA we want to obtain the best indexes, that is, the linear combinations that can reproduce the highest possible amount of variance present in our data. Hotelling suggested using a Singular Value Decomposition of the covariance matrix in order to obtain that result. When used as a descriptive tool, PCA has no distributional assumptions. So, the first step will require an analysis of the covariance matrix. Here it is:

Great! We should perform an SVD on this matrix and our results will be a PCA! Well, it’s not that simple. You can see the variances in bold. It is easy to notice that certain variable that involves money has a huge variance, simply because it is measured with a different scale. Is that a problem? Yeah, because PCA wants to represent the total variance, and here the total variance is almost entirely due to the GDP. The rest of the variables will be somehow ignored by the analysis, more exactly they will receive a lower weight in the results. We can perform a PCA of this matrix but our indexes will be almost totally about GDP and nothing more. In order to obtain results that give the same importance to every variable in the study, we should standardize the variables. That way no variable will have a larger influence in our results. Here we have the standardized covariance matrix:

No, you’re not wrong. The standardized covariances are the correlations. That is why your software will ask for a covariance or correlation matrix, depending on whether you standardize your data. As you can see, it will largely depend on the scales you are using. I’ll take the correlation matrix, since I don’t want the first variable to overtake the rest. So, SVD on the R matrix and, voila! PCA! Your stats software will get you something like this:

Six variables generated six eigenvalues. These eigenvalues originated from the SVD and are related to six principal components that can express the total amount of variation in our data. Every time you run a Principal Component Analysis, you will get as many components as variables. Certainly, with PCA we want to reduce dimension, so why having six indexes? The key is that these contain information regarding several variables. As you can see, the first eigenvalue, attached to the first principal component, can positively express 74.06% of our total variance. One single value will express ¾ of our six measures of well being! Since the principal components are orthogonal, the amount of total variance expressed by the first two PC is 87.37%, the sum of the proportions explained by them individually. Two indexes will give us almost 90% of the information! So, analyzing fewer pieces of information will give us almost the same results as analyzing the whole set of variables.

In practice, you can work with as many components as you wish. A good rule of thumb suggests that you should keep only the components that have an eigenvalue larger than 1 (only if you based your PCA on the correlation matrix). Others suggest to look for a breaking point in a screeplot of eigenvalues, or simply retaining the components that will give you the amount of explained variance that you find suitable. Our screeplot for this case is shown below:

We look for a breaking point, a moment where the line starts flattening. That is achieved with 2 eigenvalues. Some may say that it would be enough to work just with the first principal component but for the purpose of this tutorial, I’ll keep the first two in order to show some different interpretations. But this is getting way longer than a first post should be, so I’ll continue next time with the interpretation and understanding of principal components analysis.

Principal Components Analysis – Step by Step (Part II)

Principal Components Analysis – Step by Step (Part III)

Sometimes it can be advantageous to write some code in C instead of R.  R is great but sometimes its speed is lacking when it comes to certain tasks.  If you find yourself in a situation like this it can be nice to drop down to a lower level language like C to do some of the programming because you’ll find a substantial speed boost in terms of execution time.  However, coding in C is somewhat more of a headache than coding in R.  You have to get your hands dirty with memory allocation and all the fun stuff that goes along with that.  That being said if you have a task that takes forever to run you might want to spend a little time optimizing your algorithms using C.  We’ll start by doing a simple example that you wouldn’t actually need C to do: computing the fourth power of a vector of numbers.  I won’t go into the details of writing C code here – just how to write C code and compile it in such a way that you can use the code from R.  I’ll be assuming some sort of *nix environment.  The first thing we need to do is write our function in C.

/* n: The length of the array x
 * x: The array of numbers we want to take the forth power of */
void forth(int *n, double *x){
    int i;
    int tmp;
    for(i = 0; i < n[0]; i++){
        tmp = x[i] * x[i];
        x[i] = tmp * tmp;
    }
}

All inputs to our functions are pointers. Also note that we don’t actually return any values from our function. After we call the C function in R we get a list returned with all the inputs in their modified form. In this case n will be unchanged but x will have all of its elements to the forth power.

Now assume we saved that in a file called forth.c. To compile this code to be used in R we execute the following command from the command line:

 R CMD SHLIB forth.c

which will create a .so file in the same directory. That .so file is what we’ll call when we use this in R.

It is most convenient to write a wrapper function to call the C function to do some error checking for us.

dyn.load("forth.so")
forth <- function(x){
  if(!is.numeric(x)){
    stop("Argument x must be numeric")
  }
  ans <- .C("forth",
            n = as.integer(length(x)),
            x = as.double(x))
  return(ans$x)
}

Typically I would save this code in a file called forth.R and source that in if I wanted to use this function. There are a few things to note here: 1) You need to use dyn.load before you have access to the .so file you created and 2) You use .C to call the function itself. It is possible to use .Call and there are a few subtle differences between .C and .Call that I’m not going to discuss. It is good practice to make sure that when you use .C that you explicitly cast your inputs to the correct type required by the C function.

Note that one could do some other helpful things for the user. For instance if you input a numeric matrix into our function the output will be a vector. There are a variety of ways to fix this depending on what you would expect this function should do.

It is also possible to call R functions from within your C code. I’ll get to that in the next post.

Hi All,
This is Vinux ( Richie). Hope all TS contributers joined blog. Please reply if you have joined the blog.

Regards
Richie

Note:

I will delete this post, once the quality articles started filling the blog

What is StatisticsPedia?

StatisticsPedia is a free online service which has been written collaboratively by members of Talk Stats Forum.

While the talkstats forum is a great place to interact with other users, focusing on statistics related questions, StatisticsPedia is viewed as a repository for statistics-related information provided by users whom are willing and able to share this information based on their experience.

The ‘pedia allows users to post and edit articles related to specific statistics-related topics and will provide a useful source of statistical definitions, tips, tricks, tutorials, and other free stuff for anyone interested in statistics.

So welcome to StatisticsPedia, and have fun!

StatisticsPedia is a website devoted to statistics and sharing statistics related information.
We are currently accepting articles in all areas of statistics. If you have experience in a particular field of statistics, have knowledge of a particular piece of software or have just discovered some new tips or tricks that you would like to share, we would love to hear from you!

StatisticsPedia will also accept contributions to our on-line dictionary/glossary, and written tutorials.

Is there a word limit?

Strictly speaking, no; however we would encourage articles to be kept to a few hundred words.

First you need to register and log in to the control dashboard. The “Add Post” link is located on the top left corner of the dashboard. When adding a post, you can enter article title, content, and make sure to slelect the appropriate category.  We suggest that you write and edit the article using a program on your computer, then copy and paste the article into the add post page.