How To Calculate Statistical Significance (And Its Importance)
If you’re trying to determine the effectiveness of something, consider calculating statistical significance. Though it’s known for being taught in statistics coursework, it can be used for a variety of different industries including business.
In this article, we define statistical significance, its importance and how to calculate it by hand.
What is statistical significance?
Statistical significance refers to the likelihood that a relationship between two or more variables is not caused by random chance. In essence, it’s a way of proving the reliability of a certain statistic. Its two main components are sample size and effect size. In the use of statistical hypothesis testing, a data set’s result can be deemed statistically significant if you have reached a certain level of confidence in the result. In statistical hypothesis testing, this means the hypothesis is unlikely to have occurred given the null hypothesis. According to a null hypothesis, there is no relationship between the variables in question.
Why is statistical significance important?
In regards to business, statistical significance is important because it helps you know that the changes you’ve implemented can be positively attributed to various metrics. For example, if you’ve recently implemented a new application to help your office work more efficiently, statistical significance provides you with the confidence in knowing that it made a positive impact on your company’s overall workflow. That is, the app’s impact was statistically significant and provided value. If it turns out the app wasn’t statistically significant, this means your business dollars and the app are at risk. Make sure to measure the statistical significance for every result to get a more comprehensive calculation and result.
To help you make business decisions in the future, consider using business relevance along with statistical significance. This will ensure your decisions are not based on statistical significance alone.
How to calculate statistical significance
Calculating the statistical significance is rather extensive if you calculate it by hand and this is why it’s typically calculated using a calculator. When you calculate it by hand, however, it will help you more fully understand the concept. Here are the steps for calculating statistical significance:
- Create a null hypothesis.
- Create an alternative hypothesis.
- Determine the significance level.
- Decide on the type of test you’ll use.
- Perform a power analysis to find out your sample size.
- Calculate the standard deviation.
- Use the standard error formula.
- Determine the t-score.
- Find the degrees of freedom.
- Use a t-table.
1. Create a null hypothesis
The first step in calculating statistical significance is to determine your null hypothesis. Your null hypothesis should state that there is no significant difference between the sets of data you’re using. Keep in mind that you don’t need to believe the null hypothesis.
2. Create an alternative hypothesis
Next, create an alternative hypothesis. Typically, your alternative hypothesis is the opposite of your null hypothesis since it’ll state that there is, in fact, a statistically significant relationship between your data sets.
3. Determine the significance level
Your next step involves determining the significance level or rather, the alpha. This refers to the likelihood of rejecting the null hypothesis even when it’s true. A common alpha is 0.05 or five percent.
4. Decide on the type of test you’ll use
Next, you’ll need to determine if you’ll use a one-tailed test or a two-tailed test. Whereas the critical area of distribution is one-sided in a one-tailed test, it’s two-sided in a two-tailed test. In other words, one-tailed tests analyze the relationship between two variables in one direction and two-tailed tests analyze the relationship between two variables in two directions. If the sample you’re using lands within the one-sided critical area, the alternative hypothesis is considered true.
5. Perform a power analysis to find out your sample size
You’ll then need to do a power analysis to determine your sample size. A power analysis involves the effect size, sample size, significance level and statistical power. For this step, consider using a calculator. This type of analysis allows you to see the sample size you’ll need to determine the effect of a given test within a degree of confidence. In other words, it’ll let you know what sample size is suitable to determine statistical significance. For example, if your sample size ends up being too small, it won’t give you an accurate result.
6. Calculate the standard deviation
Next, you’ll need to calculate the standard deviation. To this, you’ll use the following formula:
standard deviation = √((∑|x−μ|^ 2) / (N-1))
where:
∑ = the sum of the data
x = individual data
μ = the data’s mean for each group
N = the total sample
Performing this calculation will let you know how to spread out your measurements are about the mean or expected value. If you have more than one sample group, you’ll also need to determine the variance between the sample groups.
7. Use the standard error formula
Next, you’ll need to use the standard error formula. For our purposes, let’s say you have two standard deviations for your two groups. The standard error formula is as follows:
standard error = √((s1/N1) + (s2/N2))
where:
s1 = the standard deviation of your first group
N1 = group one’s sample size
s2 = the standard deviation of your second group
N2 = group two’s sample size
8. Determine t-score
For the next step, you’ll need to find the t-score. The equation for this is as follows:
t = ((µ1–µ2) / (sd))
where:
t = the t-score
µ1 = group one’s average
µ2 = group two’s average
sd = standard error
9. Find the degrees of freedom
Next, you’ll need to determine the degrees of freedom. The formula for this is as follows:
degrees of freedom = (s1 + s2) – 2
where:
s1 = samples of group 1
s2 = samples of group 2
10. Use a t-table
Finally, you’ll calculate the statistical significance using a t-table. Start by looking at the left side of your degrees of freedom and find your variance. Then, go upward to see the p-values. Compare the p-value to the significance level or rather, the alpha. Remember that a p-value less than 0.05 is considered statistically significant.
Examples of statistical significance
Consider the following examples of statistical significance:
Example 1
Let’s say you want to attract more customers to your business, so you decide to run an ad campaign. In doing so, you consider how many advertisements should be made in print and how many should be made digitally. You rely on past ad campaigns to forecast how many you’ll need of each. If you determine that your p-value is above 0.05 or 5%, you’d end up with a result that is not statistically significant. This means that there’s a greater than 5% chance that the relationship between the two types of ads was left up to chance. Therefore, this result would indicate that it’s not reasonable to use the previous ad campaign as a guide.
Example 2
Next, let’s say you’ve created a new company website design with the hopes of attracting more customers. You’ve determined that there was a statistically significant increase in the number of customers since the new website’s implementation. Your calculation of the statistical significance resulted in a p-value of 3% or 0.03. Given that it’s below 0.05, this is a statistically significant result meaning that the increase in customers was not left to random chance.
How to Understand & Calculate Statistical Significance [Example]
Have you ever presented results from a marketing campaign and been asked, “But are these results statistically significant?” As data-driven marketers, we’re not only asked to measure the results of our marketing campaigns but also to demonstrate the validity of the data — exactly what statistical significance is.
While there are several free tools out there to calculate statistical significance for you, it’s helpful to understand what they’re calculating and what it all means. Below, we’ll geek out on the numbers using a specific example of statistical significance to help you understand why it’s crucial for marketing success.
Statistical Significance Example
Say you’re going to be running an ad campaign on Facebook, but you want to ensure you use an ad that’s most likely to bring desired results. So, you run an A/B test for 48 hours with ad A as the control variable, and B as the variation. These are the results I get:
Even though we can see based on the numbers that ad B received more conversions, you want to be confident that the difference in conversions is significant, and not due to random chance. If I plug these numbers into a chi-squared test calculator (more on that later), my p-value is 0.0, meaning that my results are significant, and there is a difference in performance between ad A and ad B that is not due to chance.
When I run my actual campaign, I would want to use ad B.
If you’re anything like me, you need more explanation as to what p-value and 0.0 mean, so we’ll go through an in-depth example below.
How to Calculate Statistical Significance
- Determine what you’d like to test.
- Determine your hypothesis.
- Start collecting data.
- Calculate Chi-Squared results.
- Calculate your expected results.
- See how your results differ from what you expected.
- Find your sum.
- Report on statistical significance to your teams.
1. Determine what you’d like to test.
First, decide what you’d like to test. This could be comparing conversion rates on two landing pages with different images, click-through rates on emails with different subject lines, or conversion rates on different call-to-action buttons at the end of a blog post. The choices are endless.
My advice would be to keep it simple; pick a piece of content that you want to create two different variations of and decide your goal — a better conversion rate or more views are good places to start.
You can certainly test additional variations or even create a multivariate test, but, for this example, we’ll stick to two variations of a landing page with the goal being increasing conversion rates. If you’d like to learn more about A/B testing and multivariate tests
2. Determine your hypothesis.
Before I start collecting data, I find it helpful to state my hypothesis at the beginning of the test and determine the degree of confidence I want to test. Since I’m testing out a landing page and want to see if one performs better, I hypothesize that there is a relationship between the landing page the visitors receive and their conversion rate.
3. Start collecting your data.
Now that you’ve determined what you’d like to test, it’s time to start collecting your data. Since you’re likely running this test to determine what piece of content is best to use in the future, you’ll want to pull a sample size. For a landing page, that might mean picking a set amount of time to run your test (e.g., make your page live for three days).
For something like an email, you might pick a random sample of your list to randomly send variations of your emails to. Determining the right sample size can be tricky, and the right sample size will vary between each test. As a general rule of thumb, you want the expected value for each variation to be greater than 5. (We’ll cover expected values further down.)
4. Calculate Chi-Squared results.
There are several different statistical tests that you can run to measure the significance of your data, and picking one depends on what you’re trying to test and the type of data you’ll collect. In most cases, you’ll use a Chi-Squared test since the data is discrete.
Discrete is a fancy way of saying that your experiment can produce a finite number of results. For example, a visitor will either convert or not convert; there aren’t varying degrees of conversion for a single visitor.
You can test based on varying degrees of confidence (sometimes referred to as the alpha of the test). If you want the requirement for reaching statistical significance to be high, your alpha will be lower. You may have seen statistical significance reported in terms of confidence.
For example, “The results are statistically significant with 95% confidence.” In this scenario, the alpha was .05 (confidence is calculated as one minus the alpha), meaning there’s a one in 20 chance of making an error in the stated relationship.
After I’ve collected the data, I put it in a chart to make it easy to organize. Since I’m testing out two different variations (A and B) and there are two possible outcomes (converted, did not convert), I’ll have a 2×2 chart. I’ll total each column and row so I can easily see the results in aggregate.
Once I’ve created my chart, the next step is to run the equation using the chi-squared formula.
Statistical Significance Formula
The image below is the chi-squared formula for statistical significance:
In the equation,
- Σ means sum,
- O = observed, actual values,
- E = expected values.
When running the equation, you calculate everything after the Σ for each pair of values and then sum (add) them all up.
Statistical hypothesis testing is the a result that is attained when a p – value is lesser than the significance level, denoted by , alpha. p – value is the probability of getting at least as extreme results that is provided that the null hypothesis is true. Statistical significance is the mean to get sure that the statistic is reliable.
If there is a large sample size, then small difference in the research findings can be negligible if you are very sure that the differences did not arise out of fluke. This formula helps us determine that there is a relationship in the differences or variations. Depending upon the sample size, to know how moderate, weak or strong is the relationship, statistical significance is used.
Statistical significance is also referred to as type 1 error. The formula and terminologies related to this formula is given as:
Solved Examples
Question: Find out the statistical significance using the z test if the sample mean is 15, is μ = 12, σ is 4 and the sample size is 30?
Solution
Given parameters are
5. Calculate your expected values.
Now, I’ll calculate what the expected values are. If there were no relationship between what landing page visitors saw and their conversion rate in the example above, we would expect to see the same conversion rates with versions A and B. From the totals, we can see that 1,945 people converted out of the 4,935 total visitors, or roughly 39% of visitors.
To calculate the expected frequencies (E in the chi-squared formula) for each version of the landing page, we can multiply the row total for that cell by the column total and divide it by the total number of visitors. In this example, to find the expected value of conversion on version A, I would use the following equation:
(1945*2401)/4935 = 946
6. See how your results differ from what you expected.
To calculate Chi-Square, I compare the observed frequencies (O in the chi-squared equation) to the expected frequencies (E in the chi-squared equation). This comparison is done by subtracting the observed from the expected value, squaring the result, and dividing it by the expected frequency value.
Essentially, I’m trying to see how different my actual results are from what we might expect. Squaring the difference amplifies the effects of the difference, and dividing by what’s expected normalizes the results. As a refresher, The equation looks like this: (observed – expected)*2)/expected
7. Find your sum.
I then sum the four results to get my Chi-Square number. In this case, it’s .95. To see whether or not the conversion rates for my landing pages are different with statistical significance, I compare this with the value from a Chi-Squared distribution table based on my alpha (in this case, .05) and the degrees of freedom.
Degrees of freedom are based on how many variables you have. With a 2×2 table like in this example, the degree of freedom is 1.
In this case, the Chi-Square value would need to be equal to or exceed 3.84 for the results to be statistically significant. Since .95 is less than 3.84, my results are not statistically different. This means that there is no relationship between what version of landing page a visitor receives and the conversion rate with statistical significance.
8. Report on statistical significance to your teams.
After running your experiment, the next step is to report your results to your teams to ensure everyone is on the same page about next steps. So, continuing with the previous example, I would need to let my teams know that the type of landing page we use in our upcoming campaign will not impact our conversion rate because our test results were not significant.
If results were significant, I would inform my teams that landing page version A performed better than the others, and we should opt to use that one in our upcoming campaign.
Why Statistical Significance Is Significant
You may be asking yourself why this is important if you can just use a free tool to run the calculation. Understanding how statistical significance is calculated can help you determine how to best test results from your own experiments.
Many tools use a 95% confidence rate, but for your experiments, it might make sense to use a lower confidence rate if you don’t need the test to be as stringent.
Understanding the underlying calculations also helps you explain why your results might be significant to people who aren’t already familiar with statistics.
Significance Level
In statistical tests, statistical significance is determined by citing an alpha level, or the probability of rejecting the null hypothesis when the null hypothesis is true. For this example, alpha, or significance level, is set to 0.05 (5%).
The formula for the t-test is as follows.
In this equation, x̄ is the sample mean, μ is the population mean, s is the sample standard deviation, and n is the number of observations in the sample. The sample mean (75), the sample standard deviation (9.3), and the number of observations in the sample (9) are all known. Assume the average height of students in the school is 69 inches:
The calculated t-value can be used to test the original hypotheses and determine statistical significance. The first step is to look at a t-table and find the value associated with 8 degrees of freedom (sample size – 1) and our alpha level of 0.05. Because the test determines statistical difference between sample mean (class) and population mean (class), this is considered a two-tailed test. For this reason, the alpha level is divided in half (0.05/2 = 0.025) and then located on the t-table to find our critical value, which comes out to be 2.306. Because the t-value is lower than the critical value on the t-table, we fail to reject the null hypothesis that the sample mean and population mean are statistically different at the 0.05 significance level.
The other statistical test that could be used is a z-test, but this test is only appropriate when the sample size is above 30 and the standard deviation of the population is known. The formula for the z-test is the same as the t-test formula.
Mọi chi tiết liên hệ với chúng tôi :
TRUNG TÂM GIA SƯ TÂM TÀI ĐỨC
Các số điện thoại tư vấn cho Phụ Huynh :
Điện Thoại : 091 62 65 673 hoặc 01634 136 810
Các số điện thoại tư vấn cho Gia sư :
Điện thoại : 0902 968 024 hoặc 0908 290 601
Để lại một phản hồi