Why can't we just perform multiple ANOVAs?


Post Hoc Tests For ANOVA


In the case where the explanatory variable represents more than two groups, a significant ANOVA does not tell us which groups are different from the others.
To determine which groups are different from the others, we would need to perform a post hoc test.
A post hoc test conducts post hoc paired comparisons.
Post hoc means after the fact. And these post hoc paired comparisons must be conducted in a particular way in order to prevent excessive type 1 error.
Type 1 error, as you'll recall, occurs when you make an incorrect decision about the null hypothesis. That is, you reject the null hypothesis when the null hypothesis is true.

Why can't we just perform multiple ANOVAs?

As you know, we accept significance and reject the null hypothesis at P less than or equal to 0.05. A 5% chance that we're wrong and have committed a type 1 error.
There's actually a 5% chance of making a type 1 error for each analysis of variance that we conduct on this question.
Therefore, performing multiple tests means that our overall chance of committing type 1 error, could be far greater than 5%. Here's how it works out.







Where C the number of comparisons, Alpha type 1 error.

Using the formula displayed under this table, you can see that while one test has a Type 1 Error Rate of 0.05, by the time we've conducted ten tests on this question, our chance of rejecting the null hypothesis when the null hypothesis is true is up to 40%.

This increase in the Type 1 error rate is called the family-wise error rate and is the error rate for the group of pair comparison.

Post hoc tests are designed to evaluate the difference between pairs of means while protecting against inflation of Type 1 errors. And there are a lot of post hoc tests to choose from, when it comes to analysis of variance. There's :
  1. Sidak test
  2. Holm T test. 
  3. Fisher's Least Significant Difference test
  4. Tukey's Honestly Significant Difference test
  5. The Scheffe test. 
  6. The Newman-Keuls test
  7. Dunnett's Multiple Comparison test
  8. The Duncan Multiple Range test
  9. The Bonferroni Procedure.
While there are certainly differences in how conservative each test is in terms of protecting against type one error, in many cases it's far less important which post hoc test you conduct and far more important that you do conduct one.

Comments

Post a Comment

Popular posts from this blog

Variables Types

Variables Types Quantitative variables Numerical, measurable quantities in which arithmetic operations often make sense. Continuous, could take on any value within an interval, many possible values. Discrete, countable value, finite number of values. Categorical or qualitative variables Classifies items into different groups Ordinal, groups have an order or ranking Nominal, groups are merely names, no ranking.  Examples Employees were asked to report their typical daily commute time, in minutes. What type of variable would their response be considered? Employees were asked to report their typical daily mode of transportation to and from work (i.e. Car, Bike, Bus, etc.). What type of variable would their response be considered? The company wanted to know how employees perceived the work of upper management. Employees were asked to report the satisfaction of upper management using a 1 to 5 scale (with the following representations: 1 - Extremely Unsat...
Read more

Confounder Variable

Confounder Variable  A confounder  is a variable that is related to both the independent and dependent variable, and partially,  or even entirely, accounts for the relationship between these two. An important thing to note about confounders, is that they  are not included in the hypothesis and they are generally not measured.  This makes it impossible to determine what the actual effect of a confounder was. The only thing to do is to repeat the study, and control the confounder  by making sure it takes on the same value for all participants. Example If we want to estimate house price and in our data set we have those variables : Size  Number of bedrooms  Address ZipCode Age of house  We might find that size is strongly correlated to the price of the house, but in our dataset, the size also correlated to the zip code. and if we loop deeper we might find that the size of the hou...
Read more

Scientific Method

Scientific Method The scientific method is based on systematic observation and   consistent logic.   Applying the scientific method increases our chances of coming up   with valid explanations.   It also provides a way to evaluate the plausibility of our scientific claims,   or hypotheses.   And the strength of the empirical evidence that we provide for   these hypotheses in our empirical study or research. Empirically testable Replicable Objective Transparent Falsifiable logically consistent Empirically Testable This means that it should be possible to collect empirical or physical evidence or observations that will either support or contradict the hypothesis. Replicable A study and its findings should be replicable. Meaning we should be able to consistently repeat the original study. If the expected result occurs only once, or in very few cases, then the result could just have been coincidental...
Read more