Measuring Security Awareness Effectiveness
Reviewed by the Fully Compliance editorial team
Measuring security awareness requires tracking four metric levels: activity (what you did), awareness (what people learned), behavior (what people actually do), and business outcomes (whether incidents decreased). Most organizations only measure completion rates, which prove the program exists but say nothing about whether it works. Move toward behavior and outcome metrics to measure real impact.
Completion Rates Prove Existence, Not Effectiveness
You have been running a security awareness program for a year. You conducted mandatory training for all 500 employees. You ran quarterly phishing simulations. You sent monthly security tips. The question that matters is: did any of this actually work? Is your organization more secure? Are employees making better security decisions? Are incidents decreasing?
Most organizations measure what is easy to measure. Training completion rates: 94% of employees took the course. Phishing simulation metrics: click rates declined from 25% to 18%. Compliance checkboxes: yes, we have an awareness program. But these metrics do not prove that the program is working or that behavior has actually improved. They prove that the program exists and people participated. The Ponemon Institute's 2024 Cost of a Data Breach Report found that employee training was one of the top factors associated with reducing breach costs, with organizations investing in training saving an average of $232,867 per breach -- but only when the training produced actual behavior change, not just completion metrics. The difference between measuring effort and measuring impact is fundamental, and most organizations miss it because true impact measurement is genuinely difficult.
There are fundamentally different levels of metrics, and they measure different things. Activity metrics measure effort: how many training sessions did you run, how many newsletters did you send? These metrics tell you whether the program is active. They do not tell you anything about impact. You could send 12 security newsletters per year and have no one read them. Awareness metrics measure knowledge: did people learn the material? Quiz scores immediately after training, simulation click rates, survey responses about security practices. These tell you whether people absorbed information, but absorbing information during training is different from applying it in real situations. A person scores 85% on a security training quiz and falls for phishing the next week when they are distracted and under time pressure. Behavior metrics measure actual actions: are people doing the security practices you trained them on? Phishing reporting rates, password strength audits, access request patterns, incident reports submitted. Behavior metrics tell you whether change has happened. Business outcome metrics measure real impact: phishing-based breach incidents, cost of incidents, dwell time, mean time to detection and response. These tell you whether the program is actually reducing risk, but they depend on factors beyond awareness -- technical controls, threat actor behavior, and whether you get targeted in a given period.
Most organizations focus on activity and awareness metrics because they are easy to measure. The goal should be to move toward behavior and outcome metrics.
Phishing Simulations: What Click Rates and Reporting Rates Actually Tell You
Phishing simulation click rate is one of the few metrics that measures behavioral response to a specific scenario. A click rate showing decline over time suggests people are getting better at recognizing phishing. The Verizon 2024 DBIR found that the median time for a user to fall for a phishing email was less than 60 seconds, which means recognition has to be nearly instinctive -- and measuring whether that instinct is developing requires consistent simulation.
But click rate has significant limitations. It is specific to simulated phishing, and real phishing looks different. A declining click rate might reflect that people now know simulations happen and are more cautious generally, not that they are better at real phishing. If you had a 30% click rate in simulation one and a 15% rate in simulation two, did awareness improve? Or is simulation two just harder to fall for?
More valuable than click rate alone is phishing reporting rate -- what percentage of people who receive suspicious emails actually report them to security. This shows active behavioral change. If phishing reporting increases while click rates decrease, you have evidence that people are not only avoiding the bait but taking action. If click rates decrease but reporting stays flat, people are deleting suspicious emails rather than recognizing the threat.
The metrics work better together than separately. Track both click rates and reporting rates. Look at trends over time. Track incident trends -- are phishing-based breaches actually decreasing? A dashboard that shows "click rate down 20%, reporting rate up 30%, no phishing-based breaches in past 12 months" tells a much clearer story than any single metric.
Training Completion, Comprehension, and the Retention Problem
Training completion is what most organizations measure: what percentage of people clicked through the course and finished it? It is clean and easy. It is also mostly meaningless. A person who clicks through an online training module while checking email every 30 seconds has completed the training but learned nothing.
Comprehension -- did people understand the material -- is harder to measure but more meaningful. Quiz scores immediately after training indicate comprehension. But immediate recall is not the same as understanding, and understanding is not the same as remembering. Retention -- do people remember the training weeks or months later -- is what actually matters for behavior change. A quiz immediately after training tells you if someone paid attention. A quiz two weeks later tells you if they retained anything. Most organizations do not measure retention because it requires follow-up.
The practical approach is: measure completion because it is easy, measure comprehension because it is informative, and assume retention decays without reinforcement. If someone took training in January and received monthly reminders, they are more likely to retain the information in June than if they took training in January and heard nothing else until July. Research on training effectiveness consistently shows that single-exposure training decays rapidly -- within weeks, most knowledge is lost without reinforcement.
Behavior Change and the Attribution Problem
Behavior change is the goal but hard to measure directly. You can look at indirect indicators that suggest behavior has changed. For phishing awareness: simulation click rates decline, phishing reporting rates increase, phishing-based incidents decrease, employees report suspicious emails proactively. For password security: audits show increasing percentages of accounts with strong passwords, shared account incidents decrease. For incident reporting: reported incidents increase as more people report, time to detection decreases as internal detection beats external discovery, incident severity decreases for reported incidents because they are caught faster.
These indicators suggest behavior change but do not prove it. Click rates decline for reasons unrelated to awareness. Password strength improves because of technical controls like password managers and complexity requirements rather than behavior change. But multiple indicators pointing the same direction build a body of evidence.
Even if you see improvements, can you attribute them to your awareness program? Or did they result from better email filtering, new technical controls, endpoint detection deployment, or threat actor changes? True attribution requires controlling for other variables, which is nearly impossible in real organizations. The best practical approach is to build a body of evidence: do incident reports from caught phishing suggest people are more aware, when you interview employees do they demonstrate better security knowledge, are captured phishing emails showing higher sophistication suggesting attackers are working harder because easy targets are gone?
The FBI IC3's 2023 report showed that phishing and its variants remained the most reported cybercrime category with over 298,000 complaints, which means the threat is not declining even if your organization's susceptibility is. Research on awareness program effectiveness shows programs reduce fall-for rates by 20-30%, but effects decay without reinforcement. This is valuable improvement but not transformational.
Building a Practical Measurement Framework
A practical measurement framework has multiple metric types, each answering different questions. Activity metrics track what you do -- training offered, simulations sent, communications distributed. Use these to assess program scope and frequency. Awareness metrics track whether people are paying attention -- quiz results, survey responses, self-reported awareness of specific threats. Use these to assess whether information is reaching people. A spike in quiz scores immediately after training and a decline months later tells you retention is fading and you need more reinforcement. Behavior metrics track actions people take -- phishing reporting rates, access review participation, password change patterns, incident reports filed. Use these to assess whether behavior is actually changing. Outcome metrics track business impact -- phishing-based breaches, incident cost, dwell time, mean time to detection. These are hard to measure and attribution is complicated, but they are what ultimately matters.
Track metrics over time to show trends. A single data point is meaningless. Showing that click rates have gone from 30% to 15% over a year shows improvement. Showing that click rates dropped sharply in the month after training and gradually increased over the following months shows the decay pattern that research predicts. Set improvement targets: "reduce phishing click rate by 5% per year," "increase phishing reporting by 10% annually," "decrease phishing-based incidents by 20% annually." Targets give direction and create accountability. Without targets, metrics become historical documentation rather than management tools.
Regular review -- monthly or quarterly assessment of metrics -- keeps the program active and allows adjustment. If you notice click rates increasing in June, that is a signal to increase reinforcement. If phishing reporting is flat despite declining click rates, that is a signal you need to encourage reporting more actively. A framework with multiple metric types gives better visibility than any single metric. You cannot evaluate program effectiveness from completion rates alone. But when you track activity, awareness, behavior, and outcomes over time, and see trends pointing in a consistent direction, you have evidence that the program is working.
Frequently Asked Questions
What is the single most important metric for measuring awareness program effectiveness?
Phishing reporting rate -- the percentage of people who report suspicious emails to security rather than just ignoring or deleting them. This measures active behavioral change, not just avoidance. Combined with declining click rates, increasing reporting rates provide the strongest evidence that awareness training is translating into real behavior.
How do we separate awareness program impact from improvements in technical controls?
You cannot fully separate them. The best approach is tracking multiple metrics that technical controls do not explain. Technical controls explain why fewer phishing emails reach inboxes, but they do not explain why employees who receive suspicious emails report them more often. Behavioral metrics like reporting rates and interview responses provide evidence of awareness impact beyond technical improvements.
How often should we measure awareness metrics?
Activity and awareness metrics should be reviewed monthly or quarterly. Behavior metrics should be assessed quarterly. Outcome metrics should be evaluated annually because incidents are rare enough that shorter timeframes produce unreliable data. The key is consistent measurement over time to identify trends.
What is a good phishing simulation click rate to target?
Industry benchmarks for moderately sophisticated simulations range from 5% to 15%. More important than hitting a specific number is showing consistent improvement over time with comparable simulation difficulty. A declining trend from 25% to 12% over 18 months is more meaningful than a single 8% result.
How do we measure retention when employees forget training quickly?
Run comprehension assessments at multiple intervals -- immediately after training, then at 30 days, 60 days, and 90 days. The decay curve shows how fast knowledge is lost and where reinforcement is needed. Organizations that combine formal training with monthly micro-reinforcements see significantly less decay than those relying on annual training alone.