What is a statistically significant change?
A statistically significant change means we're confident something real caused an increase or decrease in your metrics.
A bigger sample size means even a small change can be statistically significant
When we have lots of data, we can be more confident about the results.
For example, if we have 1,000 responses, we might only need a 3% change to be confident the change is significant.
A smaller sample size needs a bigger change to be statistically significant
With less data, we need a larger change to be confident it's significant. This is because we have less information to work with.
For instance, with 200 responses, a 3% difference might not be significant due to higher chances of random variation. We might need a 7% change to be confident.