Optimal Error Rate Calculator

This tool calculates the optimal Type I error rate (alpha) that minimizes overall error (Type I and II) for different kinds of t-Tests. Optimal alpha is calculated from sample size, effect size, ratio of the costs of making Type I vs. Type II errors, and prior probabilities of the null hypothesis (i.e., no change) being true. Click the Effect Size tab for help calculating effect size for your data. Click the More Info tab for background and references.

Input parameters

Optimal Error Rate Output


            
Optimal Error Rate Graph

            

          

          

            
Effect Size Explained

            Effect size is the magnitude of difference between two groups. For a useful statistical test, you should first determine what a meaningful effect size is. 
                    There are many different ways to calculate effect size. The optimal error rate tool uses 
                    the Cohen's D standardized effect size which is a
                    unitless number that is a difference in means divided by a pooled standard deviation. Depending on the test and options, there are different forumulas for
                    calculating Cohen's D. You can use the calculator below and input the result in the Effect Size box at the left.
            

            
Effect Size Inputs

            
            
            
            

              

                
                
                
                
                
                
              

              

                

                  
                  
                  
                  
                  
                  
                

                

                  

                    
                    
                  

                  

                    
                    
                  

                  
                

              

            

            

              Calculated effect size is: 
              
            

            

            
Helpful Links

            
          

          

            
Cost Ratio Calculator

            
This is a simple calculator for determining cost ratios to use in the optimal alpha tool. The costs considered here are those related to making a Type I or Type II error.While some error costs are easy to quantiy in monetary terms (direct costs), others are difficult to put a dolar amount to (indirect costs). Examples of indirect costs may include time lost due to appeals and litigation, loss of credibility, damage to partnerships and working relationships. Despite being challenging to quantify, you should make an effort to put a number to them. Alternatively, costs could be specified in relative terms.

            

            
Cost Ratio Inputs

            

              

                
Type I Error Costs

                
                
                
                
                
                
                
                
              

              

                
Type II Error Costs

                
                
                
                
                
                
              

            

            

              Calculated cost ratio is: 
              
            

          

          

            
In the context of natural resource monitoring, failing to detect a change (Type II error) can be more harmful and costly than falsely claiming that a change occurred (Type I error).
                                Rather than using arbitrary error rates (e.g., alpha = 0.05) for statitical analyses and tests which may lead to increased likelihood of making Type II errors, both error rates should be set at levels that minimize the chance of making any type of error.
                                Because Type I and II error rates are related, the optimal error rate is the alpha level (Type I error rate) that minimizes the probability of Type I and Type II errors for given sample and effect sizes
                                (see Mudge et al. 2012a, 2012b). The optimal error rate can be adjusted to account for differential costs of the different types of errors and for expectations 
                                (i.e., prior probabilities) of the likelihood of change occurring (or not occurring).

            
This tool uses R code provided by Mudge et al. (2012a) to calculate optimal Type I error rates. Specifically, this Shiny tool executes the following function:

            
optab(n1,n2,d,type,tails,T1T2cratio,HaHopratio

            
where n1 and n2 are sample sizes, d is Cohen's effect size, type is the type of t-Test (one sample, two-sample, paired), tails refers to whether the test is for a one-tailed or two-tailed alternative, T1T2cratio = the cost ration of Type I to Type II errors, and HaHopratio is the ratio of prior probabilities. A companion function was written to produce the plot showing how error rates change as Type I error rate goes from zero to one:

            
optab.plot(n1,n2,d,type,tails,T1T2cratio,HaHopratio)

            
References

            
  • Mudge, Joseph F., Leanne F. Baker, Christopher B. Edge, and Jeff E. Houlahan. “Setting an Optimal Α That Minimizes Errors in Null Hypothesis Significance Tests.” Edited by Zheng Su. PLoS ONE 7, no. 2 (February 28, 2012): e32734. doi:10.1371/journal.pone.0032734.

  • Mudge, Joseph F., Timothy J. Barrett, Kelly R. Munkittrick, and Jeff E. Houlahan. “Negative Consequences of Using Α = 0.05 for Environmental Monitoring Decisions: A Case Study from a Decade of Canada’s Environmental Effects Monitoring Program.” Environmental Science & Technology 46, no. 17 (September 4, 2012): 9249–55. doi:10.1021/es301320n.