The Fisher Exact Test looks at a contingency table which displays how different treatments have produced different outcomes. Its null hypothesis is that treatments do not affect outcomes-- that the two are independent. Reject the null hypothesis (i.e., conclude treatment affects outcome) if p is "small".
The usual approach to contingency tables is to apply the X2 statistic to each cell of the table. You should probably use the X2 approach, unless you have a special reason. The most common reason to avoid X2 is because you have small expectation values. This problem and some solutions are discussed here.
In the exact method, we view the particular contingency table found by assigning treatments and observing outcomes as embedded in a universe of similar tables that have the same outcome probabilities as our table (i.e., have the same row totals) and the same distribution of treatments (i.e., have the same column totals). In the re-binned example from a previous page we have treatments: A=control (no treatment) and B=intervention including careful removal of clearly affected branches and outcomes: 1=tree death within four years, 2=tree alive after four years. We found this particular 2×2 table:
A B
1: 7 12 19
2: 0 5 5
7 17 24
or more simply:
7 12 0 5The universe of similar tables includes just six tables:
7 12 | 6 13 | 5 14 | 4 15 | 3 16 | 2 17 0 5 | 1 4 | 2 3 | 3 2 | 4 1 | 5 0In an exact method you calculate the probability of each table and then sum the probability of the given table and every other "unusual" table. (The "unusual" tables are those that have probabilities less than or equal to the given table.) If the total probability of such unusual tables is "small" --that is if it is rare to have observed such unlikely tables-- we can reject the null hypothesis that the outcome is independent of the treatment.
Note that the universe of a 2×2 table can be arranged in the linear form shown above so the term: "more unusual than" is well defined without reference to any particular statistical measure of "unusualness" like X2. For our particular table, there is nothing to the left of it, so the universe contains nothing more unusual than our table. If our found table was
3 16 4 1we would need to sum the probability of that table and all tables to the right of it. (Note that we only sum the probabilities for tables to one side or the other of our found table. In this sense our Exact Test is "one-sided".)
The universe of tables grows rapidly with the size of the contingency table. For example, the 3×3 discussed on a previous page:
5 3 2 2 3 4 0 2 3inhabits a universe with 756 tables. The most likely (p=.025) table in this universe is
3 3 4 3 3 3 1 2 2followed with six tables each with p=.019
2 4 4 | 3 3 4 | 3 3 4 | 3 4 3 | 3 4 3 | 4 3 3 3 3 3 | 2 3 4 | 2 4 3 | 2 3 4 | 3 2 4 | 2 3 4 2 1 2 | 2 2 1 | 2 1 2 | 2 1 2 | 1 2 2 | 1 2 2Note that likely tables have, as much as possible, equal cell counts. Unlikely tables have evacuated some cells and maximized others. The six least likely tables have probabilities of (respectively) 5.3 × 10-9, 1.1 × 10-8, 1.4 × 10-8, 4.3 × 10-8, 4.3 × 10-8, and 4.8 × 10-8.
2 8 0 | 1 0 9 | 7 3 0 | 0 1 9 | 0 1 9 | 2 0 8 0 0 9 | 1 8 0 | 0 0 9 | 2 7 0 | 7 2 0 | 0 8 1 5 0 0 | 5 0 0 | 0 5 0 | 5 0 0 | 0 5 0 | 5 0 0The given table:
5 3 2 2 3 4 0 2 3turns out to be not particularly unusual. Its p is .0038; 689 of the 756 tables have probabilities equal to or smaller that. The total probability of these "unusual" tables is .385-- we do not come close to rejecting the null hypothesis. On the other hand, if the given table were:
6 3 1 1 4 4 0 1 4which has p=.0003 we would find 443 of the 756 tables are considered unusual. The total probability of these tables is .034-- which is commonly taken as small enough to reject the null hypothesis. Notice that disturbingly few switches were needed to convert "insignificant" results into "significant" results and that "re-binning" this 3×3 to a 2×2 in the way described above also converts a "significant" result into a "insignificant" result.
In the unlikely event you want to look at all 756 tables and their probabilities, you can click here.
The number of tables in the universe grows very rapidly with the size of the contingency table. The computational problems of enumerating many billions of tables can overwhelm existing computers. This problem was long thought to be a show-stopper, until Mehta and Patel [J. Am. Stat. Assoc. 78 (1983) 427-434] found a clever recursive method of summing the probability in the required tables. Mehta and Patel's methods have been released as F77 code: Algorithm 643 FEXACT in ACM Transactions on Mathematical Software. The version used on this server is due to Clarkson, Fan, and Joe [19 (1993) 484-488].
In the case of 2×2 tables there is an intrinsically defined meaning to the set of tables more unusual than the given table, which allows "one-sided" probabilities to be calculated. There are no well-defined "sides" in more general contingency tables so the below on-line calculator is "two-sided" even if applied to a 2×2 table.
For additional mathematical details of the exact test, click here