Decision rule – a brief overview of the p-value approach

Just to be clear, the expected result from a hypothesis test is whether the researcher (you) is rejecting or failing to reject the null hypothesis. The heuristic adopted to go for one or the other is called the decision rule; basically, the rule used to either reject the null hypothesis or not.

The most common decision rule is to fix a limit to the significance level in advance. This limit can be seen as the greater probability of committing a type I error you're willing to risk; that is, the greater chance to reject the null hypothesis while it was actually right. Common numbers are 5% and 1% (these are usually arbitrary).

With your number at hand, you can check whether the p-value output by the test is greater or lower in comparison to this limit (which is the significance level, α). Let's say that you picked 5% and you get a p-value of 0.1 (10%); hence, you may say that you failed to reject the null hypothesis within 5% level of significance. But what if you get a p-value of 0.001 (0.1%)? You could even say that you were able to reject the null hypothesis within 1% level of significance. It's a good practice to show the actual p-value to the audience.

In the immediately previous example, the p-value was 0.06313; hence, we should fail to reject the null hypothesis within 5% level of significance. This means that we don't deny that the true mean could be equal to ten as our null hypothesis assumed. There may be more reasonable (and laborious) ways to set your significance level. 5% seems to be a widely adopted number among researchers in various fields and it's definitely a magical number.

Clearly stating your decision and showing the test statistic, p-value, and the confidence interval is probably the better way to go. You can also set your alpha based on the expected outcomes from not rejecting the H₀ while it's true versus rejecting it (while it's true). This may be challenging but pretty cool and goes very well with a thing called test's power (β).