Lessons Learned from ROE Development
The foundation of any successful red team assessment lies in well-defined Rules of Engagement (ROE). Drawing from my experience developing ROEs, I'll share valuable lessons learned, real-world scenarios, and common pitfalls.
3/12/20259 min read
An ROE document serves as a formal guide that defines the scope, boundaries, and acceptable methods for conducting red team assessments—ensuring system safety and clear communication between red teams and organizations. While this discussion will help you overcome typical ROE development challenges, it isn't a comprehensive guide. For a detailed breakdown of ROE development, check out the Cyber Red Teaming book.
Use the PDF template in the Resources > Rules of Engagement section. This template was created for the book but can be easily adapted for your specific requirements.
Lessons Learned
Explore key insights and practical knowledge gained from developing and implementing Rules of Engagement across numerous red team assessments. These lessons, derived from real-world experiences, highlight both successful strategies and common pitfalls to avoid when creating ROEs for security assessments.
Rules of Engagement (ROE): Purpose and Importance
The Rules of Engagement (ROE) are the foundation of a red team assessment. They define the boundaries, responsibilities, and guidelines for both the red team and the organization. Think of the ROE as a “contract” that ensures everyone is on the same page about what is allowed and what is off-limits.
Why It’s Important:
Protects the Organization: By setting clear limits, the ROE prevents disruptions to critical operations or breaches of sensitive areas.
Protects the Red Team: The ROE ensures the red team operates legally and ethically by defining permissible activities.
Builds Trust: It fosters a collaborative relationship between the red team and the organization, ensuring both work together toward improving security.
Example: Suppose a company wants its production systems tested. The ROE would define specific times (e.g., during non-business hours) and methods (e.g., no denial-of-service attacks) to ensure the tests don’t interfere with operations.
Scope Definition: What’s In and Out
The scope is arguably the most critical part of the ROE because it tells the red team what they can and cannot test. A poorly defined scope can lead to wasted efforts or unintended consequences.
What to Include:
In-Scope: Systems, networks, and assets that the red team is allowed to test.
Out-of-Scope: Specific assets or systems that are off-limits, such as third-party vendor systems or sensitive personal data.
Why It Matters:
Without a clear scope, the red team might inadvertently disrupt critical systems or expose customer data, which could result in legal or financial consequences.
Example: If a company’s website is in-scope, the red team might test for SQL injection vulnerabilities. However, if the HR system is out-of-scope, the red team must avoid accessing employee data.
Authorized Actions vs. Restricted Actions
The ROE specifies what the red team can and cannot do during the assessment. This ensures the organization’s assets are protected while still allowing for comprehensive testing.
Authorized Actions:
Activities like reconnaissance, vulnerability exploitation, and phishing campaigns.
Example: Sending phishing emails to employees to test their susceptibility to social engineering.
Restricted Actions:
Prohibited activities that could cause harm, such as denial-of-service (DoS) attacks or accessing personal employee data.
Example: The red team is not allowed to corrupt databases or disrupt critical systems during business hours.
Incident Response Procedures
The ROE must define what happens if the red team accidentally triggers a real security incident. This ensures the situation is handled quickly and transparently.
Key Steps:
Cease Activities: The red team should immediately stop testing.
Preserve Evidence: Logs and other data must be preserved for analysis.
Notify Stakeholders: The incident response team and key contacts (e.g., the Trusted Agent) must be informed right away.
Assist with Resolution: The red team should help investigate and resolve the incident.
Example: If the red team discovers a critical vulnerability that allows unauthorized access to customer data, they should report it immediately and assist the organization in patching the issue.
Communication Protocols
Clear communication is essential for a smooth red team engagement. The ROE must define how and when updates are shared.
What to Include:
Regular Updates: Weekly or daily closeout status reports.
Critical Findings: Immediate reporting of high-risk vulnerabilities, such as exposed admin credentials.
Emergency Contacts: A hotline or secure messaging channel for urgent issues.
Example: If the red team identifies a vulnerability that allows attackers to bypass login authentication, they should notify the organization within one hour, as defined in the ROE.
Adversary Emulation
Adversary emulation involves simulating the tactics, techniques, and procedures (TTPs) of real-world attackers. This provides a realistic test of the organization’s defenses.
Why It’s Important:
By mimicking real attackers, the red team can identify weaknesses that would be exploited in a real-world scenario.
Example: The red team might emulate a known Advanced Persistent Threat (APT) group by using phishing emails to gain initial access, followed by lateral movement to exfiltrate sensitive data.
Data Handling and Destruction
Sensitive data collected during the assessment must be handled and destroyed properly to protect the organization’s privacy and comply with regulations.
Key Points:
Encryption: All data must be encrypted during storage and transit.
Secure Deletion: Data must be securely deleted at the end of the engagement, with a certificate of destruction provided to the organization.
Example: If the red team collects employee credentials during a phishing test, they must encrypt the data and securely delete it after the assessment.
Physical Security Testing
This involves testing the organization’s physical security measures, such as access controls and surveillance systems.
Key Considerations:
Ensure all activities are pre-approved in the ROE.
Avoid causing alarm or disrupting daily operations.
Example: The red team might attempt to gain unauthorized access to a data center by tailgating an employee or bypassing a card reader.
Post-Engagement Debriefing
Once the assessment is complete, the red team should provide a detailed report and conduct a debriefing session to share findings and recommendations.
What to Include:
Findings: Vulnerabilities discovered during the assessment.
Recommendations: Steps to remediate vulnerabilities and improve defenses.
Lessons Learned: Insights for both the red team and the organization.
Example: The red team might report that employees were highly susceptible to phishing emails and recommend additional security awareness training.
Best Practices for Developing ROE
Creating an effective ROE requires collaboration and careful planning. Here are some tips:
Be Specific: Use clear, unambiguous language to avoid misunderstandings.
Involve Stakeholders: Include input from security teams, IT, legal, and client leadership.
Plan for Flexibility: Allow for adjustments if the scope or objectives change.
Document Everything: Keep detailed records of all agreements, approvals, and activities.
Scenario
These scenarios represent common situations you'll need to resolve during assessments. I've experienced all of them, some frequently, so it's important to have a plan ready. With practice, you'll become skilled at managing these situations and handling any conflicts that arise.
Scenario 1: Defining Scope and Boundaries
Scenario:
You are conducting a red team engagement for a financial institution. During the pre-scoping meeting, the client insists that their production systems be included in the assessment, but they want to avoid any disruptions to business operations.
Questions:
How would you define the scope to balance the need for testing production systems with the client’s concern about disruptions?
What safeguards would you include in the ROE to minimize the risk of unintended downtime?
Scope Definition:
Include production systems in the scope but limit testing to non-disruptive methods (e.g., passive reconnaissance, vulnerability scanning without exploitation).
Specify blackout periods during business hours (e.g., 9:30 AM - 4:00 PM EST) to avoid impacting critical operations.
Safeguards:
Establish stop conditions to immediately cease testing if critical systems show signs of instability.
Require real-time monitoring and communication with the client to detect potential issues early.
Scenario 2: Handling Incident Response
Scenario:
During a red team engagement, your team inadvertently triggers a real security incident, leading to unauthorized access to sensitive customer data. The client’s Security Operations Center (SOC) detects the issue and escalates it as a critical incident.
Questions:
What steps should your team take immediately upon discovering the incident?
How should this situation be communicated to the client, and what role does the ROE play in this process?
Immediate Steps:
Cease all red team activities immediately.
Preserve logs, screenshots, and other evidence related to the incident.
Notify the Incident Response (IR) team and the Trusted Agent as outlined in the ROE.
Communication:
Provide a detailed briefing to the client, including:
The nature of the incident.
The systems and data potentially affected.
Actions taken by the red team.
The ROE serves as a guide, ensuring the situation is handled transparently and within agreed-upon protocols.
Scenario 3: Testing Physical Security
Scenario:
As part of a red team engagement, your team is tasked with testing the physical security of the client’s headquarters. The ROE specifies that no damage should occur, and testing must not disrupt daily operations. Upon attempting to bypass a card reader, you notice employees becoming suspicious.
Questions:
How should you proceed to complete the test without causing undue alarm or disruption?
What steps can you take to ensure the physical security test aligns with the ROE?
Proceeding with the Test:
Cease the current activity to avoid escalating suspicion.
Use alternative, less conspicuous methods (e.g., tailgating during employee entry) to continue testing.
Alignment with ROE:
Ensure all physical security testing methods are pre-approved and documented.
Carry identification and authorization letters to present if confronted by employees or security personnel.
Scenario 4: Communication and Reporting
Scenario:
Midway through the engagement, your team discovers a critical vulnerability in a public-facing web application that could allow attackers to exfiltrate sensitive data. The ROE requires critical findings to be reported immediately.
Questions:
What information should be included in your initial report to the client?
How should you follow up after the initial report to ensure the issue is properly addressed?
Initial Report:
Include the following details:
A brief description of the vulnerability.
The potential impact (e.g., data exfiltration).
Recommended immediate actions to mitigate the risk.
Deliver the report through the agreed-upon secure communication channel (e.g., encrypted email).
Follow-Up:
Provide a detailed technical write-up within 24 hours, including steps to reproduce the issue and remediation guidance.
Offer to assist the client’s technical team in implementing and validating the fix.
Scenario 5: Managing Scope Changes
Scenario:
During the engagement, the client requests additional testing of a newly deployed cloud environment that was not included in the original scope. The client’s legal team has not yet reviewed the change.
Questions:
What steps should you take to handle this request?
How can the ROE help guide the process for modifying the scope?
Handling the Request:
Inform the client that testing of the new environment cannot proceed until the scope change is formally approved.
Document the request and submit it for review by all relevant stakeholders, including the client’s legal team.
Role of ROE:
Refer to the ROE’s “Approval Process” section, which outlines the workflow for requesting and approving scope changes.
Ensure all approvals are documented before proceeding with the additional testing.
Scenario 6: Data Handling and Destruction
Scenario:
At the conclusion of the engagement, your team has collected sensitive data, including credentials and internal documents, as part of the assessment. The ROE specifies that all data must be securely destroyed within five business days.
Questions:
What steps should you take to securely delete the data?
How can you provide assurance to the client that the data has been properly handled and destroyed?
Data Deletion:
Use secure deletion tools (e.g., shred or sdelete) to overwrite data on all storage devices.
Ensure backups and temporary files are also securely deleted.
Providing Assurance:
Provide a certificate of destruction, documenting the methods used and confirming compliance with the ROE.
Offer to allow the client to audit the deletion process if required.
Pitfalls
Drawing from the Pareto Principle—which states that 80% of outcomes stem from 20% of causes—I've identified the most common pitfalls you need to understand to prevent friction points throughout your red team assessments.
Overlooking the importance of scope definition.
Tip: Clearly define in-scope and out-of-scope systems, networks, and assets during the planning phase. This ensures testing focuses on critical areas without unintentionally affecting excluded systems.
A well-defined scope prevents confusion and ensures both the organization and the red team operate within agreed boundaries, reducing the risk of unintended disruptions.
Failing to establish stop conditions or safe words.
Tip: Include explicit stop conditions and safe words in the ROE to halt testing immediately if critical systems or operations are at risk.
Stop conditions ensure that both the organization and the red team can quickly address potential damage or operational disruptions during the engagement.
Inadequate communication protocols.
Tip: Define clear communication protocols, including regular updates, escalation procedures, and emergency contacts, to keep all parties informed throughout the engagement.
Proper communication ensures transparency, builds trust, and allows for quick resolution of any issues that arise during the assessment.
Neglecting to address regulatory compliance.
Tip: Consult with legal teams to ensure ROEs comply with all relevant regulations (e.g., GDPR, PCI-DSS) and explicitly restrict actions that could lead to violations.
Adhering to legal and regulatory requirements protects the organization from liability and ensures the engagement is conducted ethically.
Ignoring the impact of testing on business operations.
Tip: Schedule testing activities during non-critical hours and establish blackout periods to avoid disrupting key business functions.
Proper scheduling minimizes the risk of operational downtime while still allowing thorough testing.
Skipping stakeholder involvement in ROE development.
Tip: Engage all relevant stakeholders, including security teams, IT, legal, and business leaders, to ensure the ROE aligns with organizational objectives and constraints.
Involving stakeholders ensures the ROE addresses the organization's unique priorities and operational needs.
Overlooking data handling and destruction protocols.
Tip: Specify how sensitive data will be encrypted, stored, and securely destroyed after the engagement, and provide a certificate of destruction to the organization.
Proper data handling builds trust and ensures compliance with privacy and security standards.
Conducting overly aggressive testing without safeguards.
Tip: Avoid destructive testing methods (e.g., DoS attacks) unless explicitly approved, and document all actions thoroughly to ensure accountability.
Safeguards protect critical systems and data while preventing unintended harm during the assessment.
Failing to provide actionable reports.
Tip: Ensure final reports include clear findings, risk levels, and practical recommendations for remediation. Provide a debriefing session to explain the results.
Actionable reports help the organization address vulnerabilities effectively and improve its security posture.
Not simulating realistic threat scenarios.
Tip: Base threat simulations on current threat intelligence and emulate adversaries’ tactics, techniques, and procedures (TTPs) to provide a realistic assessment.
Realistic simulations ensure the engagement reflects real-world risks and provides valuable insights into the organization’s defenses.
Summary
The success of a red team engagement heavily depends on well-crafted Rules of Engagement that establish clear boundaries, communication protocols, and expectations. Through these scenarios and common pitfalls, we've seen how ROEs serve as the foundation for effective assessments. When developed ROEs transform red team engagements from potentially risky exercises into valuable partnerships that strengthen an organization's security posture.
Remember that ROEs are living documents that should evolve based on lessons learned, changing threat landscapes, and organizational needs. The key to successful red team engagements lies not just in technical expertise, but in the ability to operate within established boundaries while delivering meaningful results that drive security improvements.
