How should a business compare security awareness training providers in terms of content quality, frequency, and measurable impact?

Comparing security awareness training providers requires moving past basic compliance checklists to evaluate how effectively a program changes employee behavior. Evaluating them across content quality, frequency , and measurable impact ensures your investment reduces organizational risk.

1. Content Quality

High-quality content must be engaging, relevant, and adapted to modern threat landscapes rather than treating training as an annual, check-the-box chore.

• Relevance & Localization: Look for providers offering micro-learning modules (3–5 minutes) that reflect real-world scenarios (e.g., modern phishing, QR-code phishing/quishing, deepfakes, and business email compromise). Content should match your industry's specific threat profile.
• Engagement & Interactivity: Dry, legacy 45-minute videos fail to retain attention. Prioritize vendors utilizing gamification, interactive storytelling, and localized or multi-lingual support if you operate globally.
• Freshness of Library: Cyber threats evolve weekly. Check how often the provider updates their catalog with fresh, timely modules rather than recycling years-old content.

2. Frequency & Reinforcement

A single annual or quarterly session does not alter human behavior. True awareness is built through continuous, low-friction reinforcement.

• Continuous Micro-learning: The cadence should be drip-fed—short, digestible pieces of content delivered monthly or bi-weekly rather than heavy blocks of training.
• Automated Phishing Simulations: Consistent, randomized phishing tests (ranging from easy to highly sophisticated) keep employees vigilant. Top providers adapt the simulation frequency and difficulty based on individual user risk scores (remediation training triggered instantly upon a failed test).
• Just-in-Time Training: When an employee clicks a simulated phishing link or flags a suspicious email, the system should immediately deliver micro-guidance right at that teaching moment.

3. Measurable Impact & Metrics

A good provider doesn't just track who completed the training; they quantify risk reduction and behavioral change over time.

• Behavioral Phish-Prone Percentage (PPP): Look for robust analytics that track baseline vs. ongoing click rates on phishing simulations. A declining PPP over 6 to 12 months indicates a real shift in organizational posture.
• Culture and Reporting Metrics: Measure the "reporting rate"—how often employees actively report suspicious emails to the security operations center (SOC). An increase in reporting accompanied by a decrease in clicking is the gold standard of cultural shift.
• Integration and Risk Scoring: The platform should offer risk-scoring algorithms that identify your "hottest" human targets (frequent clickers or high-access departments) and integrate data with your existing security stack (like SIEM or email security tools) via APIs or xAPI/SCORM standards.

If you'd like, let me know:

• Your company size or industry
• Whether you are currently using an existing platform (like KnowBe4, Proofpoint, or Cofense)
• Your primary pain point (e.g., low completion rates, high repeat clickers, or lack of reporting)

I can help you build a custom evaluation scorecard or vendor comparison matrix.

A business should compare security awareness training providers as a risk-reduction investment, not just as a library of courses. The strongest evaluation looks at three dimensions: content quality, training frequency, and measurable behavioral impact. Completion rates alone are usually insufficient because they show delivery, not whether employees behave more securely.

1. Compare content quality

Evaluate whether the provider’s material is relevant, engaging, and aligned with your organization’s risks.

A large content catalog is less valuable than content that changes employee decisions in realistic situations. Look for providers that connect training to phishing simulations, reporting workflows, and real-world behavior.

2. Compare training frequency and delivery model

Annual compliance training is generally a weak benchmark because secure behavior requires reinforcement over time. NIST Computer Security Resource Center Compare providers on:

• Initial onboarding: How quickly new employees receive training.
• Continuous reinforcement: Whether employees receive regular microlearning, reminders, and simulations.
• Phishing simulations: Frequency, realism, and whether scenarios adapt based on results.
• Risk-based targeting: Whether high-risk groups receive additional coaching.
• Campaign flexibility: Ability to increase training around current threats or incidents.

A mature program might combine:

• Annual foundational training
• Monthly or quarterly awareness activities
• Periodic phishing simulations
• Targeted remediation for employees who need additional coaching

3. Compare measurable impact

The provider should demonstrate whether behavior improves, not just whether training was completed. Useful metrics include:

Behavioral metrics

• Phishing click rate: Does risky behavior decline over time?
• Reporting rate: Are employees reporting suspicious messages more often?
• Time to report: Are employees identifying threats faster?
• Repeat-risk rate: Do employees who fail simulations improve after coaching?
• Quality of reports: Are employees correctly identifying real threats?

Learning metrics

• Knowledge assessment improvement
• Retention over time
• Completion rates by department and role
• Training engagement rates

Business-risk metrics

• Reduction in user-driven security incidents
• Faster detection of phishing attempts
• Fewer repeat mistakes
• Evidence that training supports audit, insurance, and governance requirements

A good provider should help establish a baseline before deployment and show trends afterward rather than only providing a final completion report.

4. Evaluate reporting and integrations

Ask how the provider connects with your security ecosystem:

• Can it integrate with email security tools?
• Can employee phishing reports flow into your security operations process?
• Does it integrate with your identity provider or HR systems?
• Can managers see team-level risk trends?
• Are executive dashboards available?

The goal is to connect awareness training with detection and response, not operate it as a standalone learning system.

5. Use a weighted scorecard

A sample evaluation model:

Then require finalists to demonstrate:

• A sample employee training experience
• Example dashboards
• A phishing simulation workflow
• How they handle repeat risky behavior
• How they prove improvement over 6–12 months

The best provider is usually not the one with the most courses—it is the one that can show measurable reduction in human-related security risk.

Comparing security awareness training providers can feel like wading through a sea of checkbox compliance vendors. To find a program that actually changes employee behavior rather than just ticking a box for an auditor, you need a structured evaluation framework across your three core pillars: content quality, frequency , and measurable impact.

1. Content Quality

Great content doesn't just look professional—it resonates with human psychology, drives engagement, and adapts to real-world threat landscapes. Look for:

• Relevance and Realism: Avoid dry, 45-minute compliance videos from 2018. Content should mirror modern threat vectors like sophisticated spear-phishing, QR code phishing (quishing), deepfakes, and business email compromise (BEC).
• Bite-Sized Delivery (Microlearning): Look for 3-to-5-minute modules. Modern attention spans and busy workdays mean high-retention, short-form training wins over long, once-a-year sessions.
• Localization and Diversity: Content should support multiple languages, regional nuances, and varied roles (e.g., finance vs. engineering vs. customer service face different risks).
• Interactivity and Tone: The best programs use relatable scenarios, humor, or gamification instead of fear-mongering and punitive language.

2. Frequency

An annual or quarterly module isn't enough to combat daily threat evolution. A modern cadence blends consistency with automation:

• Continuous Micro-Training: Delivering fresh, short content monthly or bi-weekly keeps security top-of-mind without causing "training fatigue."
• Just-In-Time (JIT) Training: Automated interventions are crucial. When an employee actually clicks a simulated phishing link, the system should instantly launch a 30-second "teachable moment" right then and there.
• Adaptive Cadence: High-risk employees—those who repeatedly fail simulations—should automatically cycle into more frequent, targeted reinforcement tracks compared to security-conscious peers.

3. Measurable Impact

If you can’t prove the training reduces risk, it’s hard to justify the budget to leadership. Evaluate providers on their analytics depth:

• Phishing-Prone Percentage (PPP): Track how baseline click rates drop over time across simulated campaigns.
• Behavioral Reporting Metrics: Look past click rates to see reporting rates . A healthy culture is measured by how fast employees actively report suspicious emails to the IT/security desk.
• Risk Scoring and Trending: The platform should offer individual and departmental risk scores, helping you identify organizational blind spots or departments needing extra support.
• Integration and Actionability: Can the platform integrate with your email gateway (like Microsoft Defender or Google Workspace) to feed real data back into your metrics?

If you'd like, let me know:

• Your current company size or industry
• Whether you have a dedicated security team or if IT handles it
• Your biggest pain point right now (e.g., low engagement, repeated clickers, or no budget approval)

I can help you build a custom RFP scorecard or list top-tier vendors that fit your specific environment.

A business should compare security awareness training providers as a behavior-change and risk-reduction program, not simply as a library of compliance videos. NIST specifically cautions that completion rates are easy to measure but don't necessarily demonstrate real-world effectiveness.

1. Compare content quality

Look for content that is:

• Relevant to your actual threats: phishing, business email compromise, credential theft, ransomware, social engineering, data handling, physical security, insider risk, and whatever risks are prominent in your industry.
• Role-based: finance, executives, developers, HR, administrators, and general employees shouldn't necessarily receive identical training.
• Short and engaging: favor concise modules, scenarios, decision-making exercises, and interactive material over long lectures.
• Current: the provider should update content when attack techniques, regulations, or major incidents change. NIST recommends updating training regularly and after events such as security incidents, system changes, or regulatory changes.
• Action-oriented: employees should leave knowing what to do, not merely what a threat is.
• Accessible and customizable: assess language support, accessibility, branding, policy customization, and integration with your environment.

Ask each vendor to provide 2–3 sample modules rather than relying on screenshots or a catalog description. Have actual employees from different roles score them for relevance, clarity, engagement, and usefulness.

2. Compare frequency and reinforcement

Don't evaluate vendors solely on whether they offer "annual training."

A stronger program typically combines:

• Initial onboarding training
• Periodic short awareness modules
• Regular but varied phishing/social-engineering simulations
• Just-in-time training after risky behavior
• Targeted training for high-risk roles
• Updates when threats or organizational policies change

The important question is whether frequency produces sustained behavior change, rather than simply how many modules the platform can deliver. NIST's current learning-program guidance emphasizes an ongoing lifecycle and behavior change rather than a one-time compliance event.

Be cautious of vendors selling "more training" as automatically better. A 2025 IEEE study found that commonly deployed phishing-training approaches can have limited practical effect, illustrating why frequency needs to be evaluated alongside actual behavioral outcomes.

3. Compare measurable impact

This is probably the most important differentiator.

Separate metrics into three levels:

NIST's research found completion rates were the most commonly used measure, but also found organizations using phishing reporting, incident trends, and other behavioral measures. NIST argues that effectiveness should go beyond compliance metrics.

For phishing in particular, don't look only at click rate. Measure:

• Click/failure rate
• Report rate
• Time to report
• Repeat-failure rate
• Performance by department/role
• Performance over time
• Difficulty of the simulated phish

NIST's Phish Scale is useful here because a very difficult phishing simulation can naturally produce more failures than an obvious one; comparing raw click rates without accounting for difficulty can therefore be misleading.

4. Make vendors prove their claims

During an RFP or trial, ask:

1. What behavior changes have your customers demonstrated?
2. What metrics do you recommend beyond completion rates?
3. Can we export raw, longitudinal data?
4. Can we segment results by role, department, location, and risk level?
5. How do you normalize phishing results for simulation difficulty?
6. Can we run a baseline measurement before training and compare it afterward?
7. How quickly do you update content after significant threats emerge?
8. Can training automatically adapt after an employee fails a simulation?
9. What independent research supports the effectiveness of your approach?
10. Can you provide customer results showing improvement over 6–12 months?

I'd give extra weight to vendors willing to run a pilot with a measurable baseline, rather than vendors whose primary evidence is high completion rates.

5. Use a weighted scorecard

A practical evaluation might look like:

• Content quality/relevance — 30%
• Behavior-change methodology — 20%
• Measurement and analytics — 25%
• Frequency/reinforcement capabilities — 10%
• Customization/integration — 10%
• Cost and administration — 5%

Then establish success criteria before selecting the provider. For example: "Within six months, increase simulated-phishing reporting by 30%, reduce repeat failures by 25%, and maintain ≥95% completion."

That prevents the vendor from defining success after the fact.

Bottom line

The strongest provider isn't necessarily the one with the largest content library or most frequent training. Prefer the provider that can demonstrate a credible chain from relevant content → repeated practice → changed behavior → reduced security risk, with transparent data proving each step. This aligns with NIST's emphasis on behavior change and risk management rather than "check-the-box" awareness.

A business should compare security awareness training (SAT) providers by looking beyond “does it have training videos?” The strongest evaluations focus on whether the program improves employee behavior, reduces human risk, and provides evidence of that improvement. www.nist.govwww.ciopages.com A practical comparison framework is:

How to score content quality

A good content review should examine:

• Accuracy: Does the material reflect current threats such as business email compromise, social engineering, deepfake-enabled scams, ransomware delivery methods, and cloud-service risks?
• Relevance: Can training be targeted to employees’ actual responsibilities?
• Engagement: Are lessons interactive, concise, and memorable?
• Freshness: How frequently are modules, phishing templates, and scenarios updated?
• Accessibility: Are subtitles, translations, and accessibility features available?

Providers differ in approach. Some emphasize large content libraries and administrator control, while others emphasize adaptive learning and behavior change. For example, platforms such as KnowBe4, Proofpoint, and Hoxhunt position their solutions around different mixes of content breadth, ecosystem integration, and continuous behavioral training.

How to compare frequency and reinforcement

Ask vendors to explain their recommended operating model:

Weak model

• Annual compliance video
• Quiz completion
• Certificate issued

Stronger model

• Monthly or continuous learning moments
• Regular phishing simulations
• Immediate feedback after risky actions
• Targeted follow-up training based on behavior

The goal is to create habits, not simply document attendance.

How to measure measurable impact

Avoid relying only on:

• Training completion rate
• Number of employees trained
• Quiz scores

Those show activity, not necessarily reduced risk.

More meaningful metrics include:

Employee behavior metrics

• Phishing click rate trend
• Credential submission rate
• Phishing reporting rate
• Time from receiving a suspicious message to reporting it
• Repeat risky behavior reduction

Program effectiveness metrics

• Improvement from baseline testing
• Risk reduction by department or role
• Engagement rates
• Training difficulty progression
• Correlation between training and real incident reporting

The National Institute of Standards and Technology has highlighted that phishing metrics should account for human factors and difficulty differences rather than treating click rates alone as a complete measure of effectiveness.

Suggested vendor evaluation scorecard

Weight providers on a 100-point scale:

• Content quality and relevance — 25 points
• Behavior-change approach — 20 points
• Measurement and analytics — 20 points
• Training frequency and reinforcement — 15 points
• Ease of administration — 10 points
• Integrations and compliance support — 10 points

Questions to ask during a proof-of-concept

1. “Show us a 90-day sample training program.”
2. “How do you measure improvement beyond completion rates?”
3. “What happens after an employee fails a phishing simulation?”
4. “How do you prevent employees from becoming familiar with simulation patterns?”
5. “Can we see behavior trends before and after deployment?”
6. “What metrics would you recommend reporting to executives or the board?”

The best provider is usually the one that can demonstrate a sustained reduction in risky behavior—not simply the one with the largest course library or the most compliance reports.

Evaluating security awareness training providers requires looking beyond the checkbox of "compliance" and focusing on real behavioral change. Here is a structured approach to comparing providers across your three core criteria:

• Content quality: Assess how engaging, relevant, and modern the training material is. Look for micro-learning modules (3–5 minutes) rather than grueling annual 45-minute videos. The content should feature realistic scenarios (phishing, vishing, social engineering) that reflect modern threat landscapes, and it should be localized or customizable to your company’s specific industry and internal policies.
• Frequency: Look at how the provider operationalizes continuous learning rather than "point-in-time" cramming. Effective programs blend baseline annual training with automated, bite-sized monthly modules and continuous, dynamic phishing simulations calibrated to employee risk levels.
• Measurable impact: Evaluate the depth of the vendor's analytics and reporting dashboard. Look for metrics that go past raw completion rates—such as individual risk scoring, repeat-offender tracking, reduction in click rates over time on simulations, and how quickly employees report suspicious emails using a phishing reporting tool.

To help narrow down the best vendors for your organization, let me know:

• What is your company size and industry?
• Are you prioritizing regulatory compliance or deep cultural behavioral change?
• Do you have an existing LMS (Learning Management System) it needs to integrate with?

A good comparison should treat security awareness training as a behavior-change program, not simply a library of compliance videos. NIST explicitly recommends evaluating whether programs influence security behavior, rather than relying primarily on completion rates.

1. Compare content quality

Look for content that is:

• Behavior-focused: Teaches employees what to do—verify requests, report suspicious messages, protect credentials—not just cybersecurity terminology.
• Relevant to your risks: Phishing/BEC, credential theft, ransomware, data handling, physical security, social engineering, AI-enabled scams, etc., prioritized according to your threat profile.
• Role-specific: Finance, executives, developers, IT administrators, HR, and general staff shouldn't necessarily receive identical training. NIST specifically recommends tailoring training to users' duties and responsibilities.
• Engaging and usable: Short modules, realistic scenarios, good accessibility, mobile support, and minimal "check-the-box" feel. NIST research found that keeping training engaging and tailoring it to diverse workforces are significant challenges.
• Current: Ask how frequently the provider refreshes content and how quickly it incorporates new attack techniques, incidents, regulations, and emerging threats. NIST recommends updating training when security incidents, system changes, or regulatory changes warrant it.

Vendor test: Don't judge from a sales demo. Have each provider give you 2–3 representative modules and score them against the same rubric.

2. Compare frequency and delivery model

Don't equate "more training" with "better training."

A strong program usually combines:

The important question is whether frequency produces sustained behavior change. A 2025 large-scale study found that simply completing annual awareness training was not significantly associated with better performance in simulated phishing, illustrating why frequency alone shouldn't be treated as evidence of effectiveness.

3. Compare measurable impact

This is where providers should differentiate themselves most.

Weak metrics:

• % completing training
• Number of modules assigned
• Training hours
• Number of phishing simulations conducted

These demonstrate activity or compliance, not necessarily security improvement. NIST research found completion rates were the most common effectiveness measure, despite organizations' difficulty determining actual impact.

Better metrics:

• Phishing report rate
• Phishing failure/click rate
• Change in behavior over successive simulations
• Time to report suspicious messages
• Repeat-risk rate—how many people repeatedly make the same mistake
• Knowledge retention
• Reporting of real-world suspicious activity
• Security incident trends associated with targeted behaviors
• Differences between high-risk and low-risk groups

For phishing specifically, don't blindly compare raw click rates between vendors. NIST's Phish Scale exists because the difficulty of the simulated phishing message can materially affect click rates.

4. Require vendors to prove attribution

Ask:

"How will you demonstrate that your program changed employee behavior, rather than merely increasing training completion?"

A strong provider should be able to establish a baseline, measure behavior over time, segment results by risk group, and show trends after interventions.

Ideally, your evaluation looks something like:

Baseline → training → simulation/behavior measurement → targeted reinforcement → repeat measurement → business-risk correlation

Also ask whether you can export the underlying data and integrate it with your SIEM, identity platform, HR system, or security operations workflows.

5. Use a weighted scorecard

For a typical enterprise, I'd weight the evaluation approximately:

• Content quality & relevance — 30%
• Behavior-change methodology — 20%
• Measurement & analytics — 25%
• Frequency/adaptability — 10%
• Administration/integration — 10%
• Price — 5%

I'd deliberately keep price relatively low in the initial scoring. A cheaper provider that produces little measurable behavior change can be substantially more expensive from a risk perspective.

The key procurement question

Instead of asking vendors:

"How many courses and phishing simulations do you provide?"

ask:

"Show us, using customer data, how your program has changed risky employee behavior, how you measure that change, and how you distinguish genuine improvement from differences in simulation difficulty or employee participation."

That question pushes vendors away from feature-counting and toward measurable security outcomes—the direction supported by NIST's current learning-program guidance.

Comparing security awareness training providers requires a structured evaluation across the dimensions that actually drive behavioral change: content quality, frequency , and measurable impact.

Here is how a business can systematically compare and vet different platforms:

1. Content Quality

• Relevance and Realism should feature modern scenarios reflecting current threat landscapes (e.g., deepfakes, sophisticated spear-phishing, SMS/quishing attacks) rather than outdated, generic email scams.
• Engagement and Delivery must go beyond static 30-minute annual modules. Look for micro-learning formats (3 to 5 minutes), gamification, interactive quizzes, and high-production-video storytelling.
• Localization and Customization are critical; the content should support multiple languages, regional cultural contexts, and the ability to inject company-specific policies or brand elements.

2. Frequency (Cadence & Automation)

• Continuous vs. Periodic training is essential. Annual or quarterly compliance check-the-box training fails. Top platforms automate continuous micro-learning drips, periodic pulse surveys, and "just-in-time" training triggered when a user fails a simulated test.
• Simulated Phishing Agility should allow customizable, automated phishing campaigns that scale in difficulty based on individual or department risk profiles rather than blasting the same template company-wide.
• Administrative Burden matters. Evaluate how much manual scheduling your security team has to do versus what the platform automates based on risk triggers.

3. Measurable Impact

• Phish-Prone Percentage (PPP) tracking must be standard, measuring baseline vs. ongoing click rates on simulated phishing tests over time.
• Behavioral Metrics should look deeper than test scores—tracking metrics like organic (real-world) phishing incident reporting rates, time-to-report, and repeat offender remediation.
• Integration and Analytics should include robust dashboards, API integrations with your SIEM, email gateway (e.g., Microsoft Defender, Google Workspace), or HRIS systems to correlate training data with actual security incidents and risk reduction ROI.

If you'd like, let me know:

• Your company size or industry
• Any specific compliance frameworks you need to meet (e.g., SOC 2, HIPAA, ISO 27001)

I can recommend specific vendor categories or a tailored evaluation scorecard to help you choose.

Comparing security awareness training providers requires looking beyond check-the-box compliance to evaluate how effectively they change employee behavior. Here is a framework to evaluate vendors across your three key criteria:

• Content quality : Look for engaging, realistic, and up-to-date material. Avoid dry, 45-minute annual modules that employees click through mindlessly. Quality content means micro-learning modules (3 to 7 minutes long), high production value, interactive scenarios, and localized or role-based content tailored to different departments (e.g., finance vs. engineering).
• Frequency : Move away from once-a-year or quarterly compliance dumps. Continuous, spaced repetition yields the best retention. Effective providers support automated, rolling training schedules, bite-sized monthly modules, and "just-in-time" training triggered immediately when an employee fails a simulated phishing test.
• Measurable impact : Look past completion rates (which only measure compliance) to true behavioral metrics. Key indicators include your organization's Phish-Prone Percentage (PPP) over time, how quickly employees report suspicious emails using an integrated phishing-reporting button, and whether repeat offenders show improvement after targeted remediation.

Would you like me to help you build a vendor evaluation scorecard or list specific top-tier providers known for these criteria?

A business should evaluate security awareness training providers as a behavior-change program, not simply as a library of compliance courses. NIST specifically warns that completion rates alone don't demonstrate effectiveness and recommends measuring whether training changes workforce attitudes and behaviors.

1. Content quality

Score providers on whether the training is:

• Relevant to your actual risks: phishing/BEC, password and MFA behavior, data handling, social engineering, insider risk, physical security, remote work, etc.
• Role-based: executives, finance, developers, privileged users, and general staff shouldn't necessarily receive identical material. NIST recommends tailoring training to users' roles and responsibilities.
• Realistic and actionable: employees should practice recognizing and responding to situations they'll actually encounter, rather than memorizing policy language.
• Current: ask how quickly content is updated after major threats, incidents, regulatory changes, or changes to your environment. NIST explicitly calls for regular content updates and updates triggered by relevant events.
• Engaging and accessible: short modules, good examples, mobile/accessibility support, appropriate localization, and minimal "death by PowerPoint."
• Customizable: ideally you can incorporate your policies, terminology, applications, and threat scenarios.

Provider questions:
"What percentage of your library is updated annually?" "Can we review representative modules before purchase?" "How do you map content to our risk profile?" "Can we create role-specific learning paths?"

2. Frequency and delivery model

Don't automatically choose the provider offering the most training.

A stronger model is continuous reinforcement: initial onboarding + periodic core training + short, targeted interventions + realistic simulations + event-driven training.

NIST's current guidance leaves the exact frequency to the organization, but calls for training after onboarding and when relevant system/security changes or events occur.

Compare vendors on:

The key question is whether frequency produces retention and behavior change, not whether a vendor can send 12 modules a year.

3. Measurable impact

This should receive the greatest weight in your evaluation.

Use a hierarchy of metrics:

Level 1 — Participation

• Completion rate
• Time to completion
• Assessment scores

Useful for compliance, but weak evidence of risk reduction. NIST research found completion rates were among the most commonly used measures, while noting the broader challenge of measuring actual effectiveness.

Level 2 — Knowledge

• Pre/post assessment improvement
• Knowledge retention over time
• Ability to apply concepts to scenarios

Level 3 — Behavior

• Phishing click rate
• Phishing reporting rate
• Repeat-click rate
• Time to first report
• Appropriate reporting of real suspicious messages
• Reduction in risky behaviors

NIST's Phish Scale is particularly useful here because it helps account for the difficulty of the simulated phishing message; a raw click rate isn't very meaningful if one provider uses easy simulations and another uses highly sophisticated ones.

Level 4 — Business/security outcomes

• Security incidents attributable to human error
• Account compromise trends
• BEC incidents
• Data-handling violations
• Help-desk/security-team indicators of employee understanding
• Incident reporting and response speed

This is where a mature program demonstrates value. NIST research describes the transition from compliance metrics toward actual workforce behavior and organizational impact.

4. Make vendors prove their claims

Before selecting a provider, give each finalist the same evaluation scenario:

1. Establish your baseline metrics.
2. Run a pilot with a representative employee group.
3. Give vendors comparable populations and roughly equivalent phishing difficulty.
4. Measure knowledge, behavior, and reporting—not just completion.
5. Re-test several weeks/months later to assess retention.
6. Require a dashboard showing trends over time and by business unit/risk group.
7. Ask the vendor to demonstrate how its data can be exported into your GRC/SIEM/analytics environment.

For phishing in particular, don't accept vendor-to-vendor benchmark comparisons at face value. SANS notes that click/report rates aren't reliably comparable unless simulation difficulty and program maturity are normalized.

A practical weighting model

I'd use something like:

• Content quality & relevance — 35%
• Measurement/analytics & demonstrated impact — 35%
• Frequency, personalization & delivery — 20%
• Administration, integrations & cost — 10%

And make "can demonstrate behavior change" a minimum requirement, rather than allowing an inexpensive provider with excellent completion reporting to win despite weak evidence of effectiveness.

The simplest executive-level test is:

Does the provider make employees more capable of recognizing and reporting threats, and can it prove that improvement with credible before-and-after data?

If the answer is mostly "we get 98% completion," you're measuring training administration—not security impact.