š§¢ Coach T's Recap
Not every shift in your data is meaningful. Statistical significance helps you understand which movements are real signals versus normal variation, so you can focus on changes that actually matter.
What is statistical significance?
Statistical significance helps you determine whether a change in your brand data is likely to be real or whether it could have happened by chance.
If a result is statistically significant, it means the change is large enough, and supported by enough data, that itās unlikely to be random. In other words, itās a shift worth paying attention to.
How statistical significance works in Tracksuit
Whenever you view movement in your metrics across any Timeline or Comparison page, Tracksuit runs a statistical test behind the scenes.
It takes into account:
The sample size
The starting value of the metric
The size of the change
With more responses, even small changes can be meaningful. With fewer responses (for example, when filtering by demographics), changes need to be larger to confidently rule out normal variation.
If a shift is mathematically meaningful, we flag it in the dashboard as statistically significant so you know it's a signal worth paying attention to.
š” Tip
Understand meaningful shifts in the data at a glance with our color-coded arrows. Significant uplifts are displayed as green arrows and significant declines are displayed as red arrows.
How to interpret a statistically significant change
When somethingās flagged as significant in the dashboard, thatās your cue to get curious.
Here are some smart questions to guide your thinking š
Whatās changed recently?
Connect the data to real-world context.
Have there been changes in media spend, messaging, or targeting?
Did a competitor launch, reposition, or enter the market?
Could seasonality or broader market conditions be influencing the result?
Which segment is driving the shift?
Zoom into the data to understand who is behind the movement:
Is it driven by a specific age group, region, or demographic segment?
Are new audiences discovering your brand, or are existing customers leaning in?
Does this align with our strategy?
Now bring it back to your goals.
Is the movement aligned with what youāve been trying to achieve?
Does it validate recent activity, or suggest the need to adjust course?
Is growth happening at the right stage of the funnel?
Why isn't my change statistically significant?
It can feel surprising when you see a 5 percentage point movement, but the dashboard tells you it isn't statistically significant.
That's because there isn't one fixed threshold for statistical significance. Every result is calculated using the sample size, the starting value of the metric, and the size of the change. As those factors change, so does the threshold for what's considered statistically significant.
One of the most common times you'll notice this is when you're looking at a subset of respondents, for example after applying demographic filters or viewing metrics that are only asked of part of your audience. Because statistical significance is recalculated for each result, the same movement may be statistically significant in one chart but not another.
Here are a few examples š
Audience segments
Audience segments
When you filter the dashboard, the result is calculated using only respondents in that audience segment.
ā
In this example, a 5 percentage point movement isn't statistically significant because, when the sample size, starting value, and size of the change are all considered together, there isn't enough evidence to confidently say this is a meaningful shift.
Statements
Statements
In this example, the 'Is easy to find' brand statement is measured only among respondents who consider the brand.
ā
Although the statement performance decreases by 4 percentage points, the result is based on this subset of respondents. When the sample size, starting value, and size of the change are all considered together, the movement isn't classified as statistically significant.
ā
What if something isnāt significant?
A non-significant result doesn't mean nothing happened. It simply means there isn't yet enough evidence to confidently say the movement reflects a real shift rather than normal survey variation.
ā
Treat these movements as directional signals, not firm conclusions. If you're trying to understand whether a change is meaningful, zoom out and look at the longer-term trend. A movement that isn't statistically significant today may become a meaningful trend as more data is collected over time.
At Tracksuit, we take care of the statistical testing behind the scenes so you can focus on what matters most: understanding how your brand is changing over time.