We Have A Security Awareness Problem

The Hidden Challenges of Measuring Security Awareness Training Effectiveness

Posted by:

Tim Marley

Introduction

In the ever-evolving landscape of cybersecurity, organizations have universally recognized the importance of security awareness training. However, despite the widespread adoption of these programs, a significant problem remains: many organizations are not measuring the true effectiveness of their efforts. Instead, they often mistake compliance metrics, such as training completion rates, for success, without ensuring that these programs lead to meaningful behavioral change.

The disconnect between compliance and actual security improvement is more than just a theoretical concern—it’s a real-world problem that I’ve encountered firsthand. Even when employees receive direct, hands-on training, the results can be unexpectedly poor, with many still falling victim to phishing and social engineering attacks. This underscores a critical issue: security awareness training, as it stands today, is frequently inadequate, leaving organizations vulnerable despite their best efforts.

This blog will explore the hidden challenges of security awareness training, drawing on both industry insights and personal experience to highlight the urgent need for a more effective, behavior-focused approach. As we dive into the data and real-world scenarios, it becomes clear that we have a problem—one that demands immediate and strategic action.

Background

Security awareness training is not just a best practice—it’s a legal requirement for many organizations across various industries. Numerous regulations mandate that companies implement formal security awareness programs to educate their workforce on protecting sensitive information and adhering to security protocols. These requirements, while essential for ensuring a baseline level of knowledge, often focus on the necessity of providing training rather than evaluating its effectiveness in changing behaviors.

  • Health Insurance Portability and Accountability Act (HIPAA): Under HIPAA, covered entities and their business associates are required to implement a security awareness and training program for all workforce members. This ensures that employees are informed about the importance of protecting patient health information and adhering to privacy and security rules.
  • Payment Card Industry Data Security Standard (PCI DSS): PCI DSS mandates that all employees are made aware of the importance of cardholder information security. This requirement is crucial for organizations that handle payment card transactions, aiming to prevent data breaches and fraud.
  • Sarbanes-Oxley Act (SOX): SOX requires public companies to have internal controls and procedures for financial reporting, which includes security awareness training. This training helps ensure that employees understand the importance of maintaining accurate financial records and protecting the integrity of financial information.
  • Gramm-Leach-Bliley Act (GLBA): GLBA mandates that financial institutions protect customer information through various security measures, including training employees. The focus is on ensuring that staff members are aware of and adhere to protocols that safeguard customer data.
  • Federal Information Security Management Act (FISMA): FISMA requires federal agencies to develop, document, and implement an agency-wide program to provide information security for the information systems that support the operations and assets of the agency. Security awareness training is a critical component of these programs, ensuring that federal employees are equipped to protect government information systems.
  • ISO/IEC 27001 & 27002: These international standards require organizations to provide appropriate awareness training and regular updates in organizational policies and procedures. This ensures that employees remain informed about the latest security practices and organizational expectations.
  • Federal Trade Commission (FTC) Red Flags Rule: As part of an Identity Theft Prevention Program, the FTC Red Flags Rule requires organizations to train their employees on how to detect and respond to warning signs of identity theft, commonly referred to as “red flags.”
  • General Data Protection Regulation (GDPR): GDPR requires organizations that process the personal data of EU citizens to implement security measures, including training employees on data protection practices. This training is essential for ensuring that employees understand their responsibilities under GDPR and can effectively protect personal data.
  • California Consumer Privacy Act (CCPA): CCPA mandates that businesses which collect personal data from California residents provide training to employees who handle such data, ensuring they understand and comply with the law’s requirements for data privacy and protection.


These regulations, among others, underscore the critical role that security awareness training plays in protecting sensitive information across industries. While the specific requirements may vary, the common thread is clear: organizations must educate their workforce on security practices. However, it’s important to note that these examples are not exhaustive, and organizations may be subject to additional requirements depending on their industry and geographic location..

Key Challenges and Strategic Responses

Threats: 

  • Compliance Over Impact:
  • Many organizations fall into the trap of equating training completion with success. This focus on compliance metrics, such as the percentage of employees who have completed their training, creates a false sense of security. While it’s important to ensure that all employees participate in required training, this metric alone fails to capture whether the training has led to meaningful behavioral change. The real goal of security awareness training should be to foster a culture of security mindfulness, but compliance-focused metrics often miss this critical objective.
  • Challenges in Measurement:
  • Accurately measuring the effectiveness of security awareness training is a significant challenge. Many organizations struggle to determine what metrics truly reflect the success of their training programs. The common reliance on metrics like training completion rates or quiz scores doesn’t necessarily correlate with improved security behaviors. A particularly problematic metric is the “click rate” from phishing simulations. While it can provide some insight into susceptibility, click rates alone often fail to capture the full picture of an employee’s understanding and behavior. For example, a low click rate might suggest effectiveness, but it could also result from employees becoming overly cautious and not engaging with legitimate emails. Conversely, a high click rate might not fully reflect the complexity of the phishing attempts used in simulations. These nuances highlight the limitations of relying on click rates as a primary measure of success, leading to inconsistent or incomplete assessments of training effectiveness.

Prevention/Mitigation Strategies:

  • Holistic Measurement Approaches:
  • To move beyond the limitations of compliance-focused metrics, organizations should adopt a more holistic approach to measuring the effectiveness of security awareness training. This involves using a combination of metrics that go beyond simple participation rates. For example, organizations can monitor the outcomes of phishing simulations, track the frequency and nature of security incident reports, and gather employee feedback on their understanding and application of security practices. By diversifying the metrics used, organizations can gain a more accurate picture of how well their training is translating into real-world behavior.
  • Behavioral Change Focus:
  • Shifting the focus from compliance to actual behavioral change is crucial for the success of any security awareness program. This can be achieved by incorporating ongoing assessments that continuously evaluate and reinforce security behaviors. Tailored training programs that address specific gaps in knowledge or behavior are also essential. Rather than a one-size-fits-all approach, training should be personalized to meet the needs of different roles within the organization, ensuring that the content is relevant and impactful for every employee.

Industry Insights

Measuring the Effectiveness of U.S. Government Security Awareness Programs: A Mixed-Methods Study, National Institute of Standards and Technology, July 14, 20221

Challenges in Measuring Effectiveness:

“Forty-four percent of survey participants rated determining what to measure and how to measure program effectiveness as very or moderately challenging. Only 14% rated it not at all challenging”.

Compliance vs. Behavioral Change:

“Despite almost half of survey participants believing compliance is the most important indicator of success, many participants in both the focus groups and survey voiced a concern that compliance metrics in the form of training completion rates, although required, do not demonstrate long-term attitude or behavior change, which should be the real goals of security awareness training”.

Manager Preferences for Demonstrating Effectiveness:

“Managers most frequently listed behavioral data, such as phishing click rates and user security incidents, as the most helpful in demonstrating the effectiveness of the security awareness program”.

Use of Multiple Measures:

“Sixty-four percent of organizations use at least five different measures to assess the effectiveness of their security awareness programs, with training completion rates and phishing simulation click rates being the most popular”.

Leadership Focus on Compliance:

“Over half of responding participants (56%) agreed or strongly agreed that their organization’s leadership thought compliance was the most important indicator of security awareness program success”.

First Hand Experience

In a real-world scenario from my experience as a Chief Information Security Officer (CISO) at a multi-office organization, I encountered firsthand the complexities of implementing an effective security awareness training program. The organization had three offices located in New York City, Dallas, and Norman, Oklahoma. As part of my responsibilities, I provided hands-on security training to employees, supplemented by professionally produced videos, but focused this training exclusively on the Norman office, which consisted of around 100 employees.

To assess the effectiveness of the training, I enlisted an external consultant to conduct a phishing exercise. The consultant crafted an email that spoofed my own email address and asked recipients to sign into a fake website by providing their credentials. The exercise targeted a sample of employees from both the Norman and Dallas offices, despite the Dallas office (also ~100 employees) not receiving the specialized training provided in Norman.

The results were both surprising and revealing. Despite the Norman employees receiving direct, hands-on training—including specific instructions that neither I nor anyone from IT would ever request their passwords—50% of those tested in Norman still fell for the phishing attack. Even more concerning, 60% of the employees in the Dallas office, who had not received the targeted training, also failed the exercise.

This case underscores the inherent challenges in security awareness training. Despite clear and direct training, a significant portion of the workforce still failed to recognize a phishing attempt. The results highlight that even well-intentioned and thoughtfully designed training programs can fall short, particularly when employees are faced with sophisticated social engineering attacks that exploit their trust and familiarity with internal processes.

The takeaway from this experience is clear: Security awareness training is not a one-size-fits-all solution, and achieving the desired behavioral change is often more difficult than anticipated. Continuous reinforcement, realistic phishing simulations, and a multi-faceted approach to training are essential to improving the overall security posture of an organization. Even with the best training, human error remains a significant risk, which is why it’s critical to combine awareness programs with strong technical defenses and ongoing assessments to mitigate potential vulnerabilities.

Future outlook

Evolving Cyber Threat Landscape:

  • Increased Complexity of Threats: As cyber threats continue to evolve, particularly with the rise of sophisticated phishing attacks, social engineering, and ransomware, the need for effective security awareness training will intensify. Organizations will need to ensure their training programs are agile and can adapt quickly to emerging threats.
  • AI-Driven Threats: One of the most alarming developments in the cyber threat landscape is the use of advanced AI to simulate video and audio calls that are almost indistinguishable from those of a real person. This technology, known as deepfake AI, can create highly convincing replicas of voices and faces, making it possible for attackers to impersonate trusted individuals with unprecedented accuracy. For example, an AI-generated video call could mimic the voice and appearance of a company executive, instructing an employee to transfer funds or share sensitive information. The increasing realism of these AI-generated simulations poses a significant challenge to traditional security awareness training, as employees may find it difficult to detect these sophisticated deceptions. As these threats grow, organizations must enhance their training programs to address the risks associated with AI-driven impersonation, teaching employees how to verify the authenticity of communications and encouraging a culture of skepticism towards unexpected requests.
  • Regulatory Pressures: With new regulations on the horizon, and existing ones becoming more stringent (e.g., updates to GDPR or the introduction of CMMC 2.0), organizations will be required to demonstrate not just compliance but also the effectiveness of their security measures, including awareness training.

Innovative Solutions and Technologies:

  • AI-Driven Training Platforms: The future of security awareness training will likely involve AI-driven platforms that personalize training content based on individual employee behavior and risk profile. These platforms can provide real-time feedback, adjust difficulty levels, and even predict which employees might need more intensive training.
  • Advanced Analytics: Organizations will increasingly rely on advanced analytics to measure the impact of their training programs. This will involve not just tracking completion rates but also analyzing behavioral data, such as response times to simulated attacks, the frequency of security incident reports, and changes in security posture over time.
  • Continuous Learning Models: The traditional once-a-year training model may give way to continuous learning environments where employees receive regular, bite-sized training sessions that keep security top-of-mind. These sessions could be integrated into daily workflows, making training a more seamless part of an employee’s routine.

Holistic Integration with Other Security Measures:

  • Integration with Incident Response: Future training programs may be more closely integrated with incident response protocols, ensuring that employees not only know how to prevent incidents but also how to respond effectively when they occur.
  • Cross-Departmental Collaboration: Security awareness training will increasingly involve collaboration between different departments, such as HR, IT, and legal, to ensure a comprehensive approach to security that addresses all aspects of the business.

Conclusion

Security training is a critical component of any organization’s cybersecurity strategy, encompassing more than just raising awareness. Effective training programs should not only inform employees about potential threats but also equip them with the skills and knowledge needed to recognize, respond to, and prevent security incidents. As we’ve explored, there are significant challenges in ensuring the effectiveness of these programs. The overemphasis on compliance metrics, such as training completion rates, often leads to a false sense of security, while the real goal—achieving meaningful behavioral change—remains elusive.

To truly protect against threats, it’s essential to shift the focus from compliance-driven training to approaches that prioritize real-world impact. This means adopting holistic measurement strategies that look beyond participation rates and focusing on changing the security behaviors of employees through continuous reinforcement and tailored training. By recognizing and addressing these issues, organizations can create a security awareness program that not only meets regulatory requirements but also fosters a culture of security mindfulness that is crucial for defending against today’s sophisticated cyber threats.

Next Steps

If your organization is struggling with the effectiveness of its security awareness training, it’s time to rethink your approach. Don’t let compliance metrics lull you into a false sense of security. Instead, take proactive steps to ensure that your training programs are driving the behavioral change necessary to protect your organization from the inside out.

Contact us today to discuss how we can help you develop a more effective, behavior-focused security awareness program tailored to your organization’s unique needs. Let’s work together to build a stronger, more resilient security culture that goes beyond compliance to deliver real-world protection.

References
  1. https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=934952 ↩︎

Timothy J. Marley, CPA, CISSP, CISA, CISM, CIA, CDPSE

Founder and Principal Consultant, Prism One

Timothy J. Marley, founder and principal consultant of Prism One, is a seasoned cybersecurity executive with over two decades of experience in information technology, risk management, and compliance. Tim’s extensive expertise helps organizations navigate the complex landscape of cybersecurity and risk management.

Tim’s mission is to empower organizations to build resilient security postures in an ever-evolving threat environment. At Prism One, he continues to provide tailored, high-quality cybersecurity solutions to meet the unique needs of each client.

Similar Posts