Update CCADB Incident Reporting Guidelines (Fall 2024)

[ryancdickson]

In October 2023, the CCADB Steering Committee, with valuable feedback from the CCADB Public community, updated the CCADB Incident Reporting Guidelines (IRGs). While the resulting updates have led to some reports becoming more useful and effective, Root Store Operators have continued to stress the importance of high-quality incident reports during CA/Browser Forum Face-to-Face updates and elsewhere.

In the spirit of continuous improvement, the CCADB Steering Committee has worked over the past few months to further enhance the effectiveness of the IRGs.

Objectives for this update to the IRGs include:

• Clarifying Root Store Operator expectations
• Aligning report format and content with those expectations
• Improving clarity regarding the difference between “preliminary" and “full" reports, and the timelines for disclosing these reports
• Improving Root Cause Analysis
• Tracking commitments made by CA Owners in response to incidents
• Increasing accountability and generating more actionable insights
• Improving consistency in report quality
• Emphasizing continuous improvement and lessons learned
• Encouraging familiarity with historical incident reports
• Defining a standard process for closing reports
• Aligning the incident reporting format with Steering Committee objectives planned for 2025+

The set of proposed updates are described in this Pull Request.

These changes should not be considered “final”, but instead a “work in-progress” that we hope to enhance through community contributions. Feedback is welcome either on this Pull Request (preferred), or via the discussion thread on public [at] ccadb [dot] org by January 15, 2025.

[aarongable]

Hi Ryan! I've gone ahead and left a bevy of comments here. The vast majority are just line-edits around things like "i.e.", "e.g.", and markdown whitespace; just a couple are more substantive comments.

I think my overall feedback is:

1. The updated instructions and textual requirements are really good improvements!
2. The distinction between "action items" and "ongoing commitments" feels arbitrary and awkward -- if I make a code change two weeks after the incident ended that prevents this from happening ever again, is that an action item (specific task with a deadline) or an ongoing commitment (designed to prevent a future incident)? Also this distinction is only used once: in the incident closure template. It's never referenced again, not even in the incident report template. Also, the incident closure template is embedded in the midst of the requirements, instead of being with the other templates at the bottom of the doc. Altogether this leads to the "ongoing commitments" section being jarring and out-of-place when reading the doc.
3. The new incident report template is difficult to read. There is so much going on there, and the use of square brackets to indicate text that should be replaced is confusing in a markdown context where square brackets generally denote hyperlinks. It's also unclear whether we're supposed to simply address each bullet point, or copy each bullet point into our report.

[aarongable]

This section is somewhat confusing to me. It's very good material! But it doesn't contain any MUST/SHOULD/etc requirements, and its content seems to overlap with advice in the new "incident-reporting.md" doc below. Can we consolidate advice into one doc, and requirements into the other?

[aarongable]

This feels like a strong re-definition of the term "Action Item" away from how we've been using it in this community. For the most part, by the time an incident report has been uploaded, all action items aimed at resolving the immediate incident have long since been completed. It's only the forward-looking things (putting new processes in place, adding safeguards, deleting old systems, etc) that generally get listed as Action Items at the bottom of a report.

Are we saying that all of those are now Ongoing Commitments instead?

[wthayer]

I agree - how can these two concepts be clearly distinguished?

[romanf]

Shouldn't it be "Audit Finding Reporting"?
A finding does not have to be an incident... (applies also for the rest of the template)

[ChristopherRC]

@romanf I believe the intent of using the naming "Audit Incident Report" is to differentiate that type of report and its contents from a "Incident Report". Depending on Root Program Policies the "Audit Incident Report" may be required for any non-conformities, qualifications, or modified opinions included in an Audit. The "Audit Incident Report" may include one or more findings, but it is focused on the root cause of each and action items to address the root cause. The other type of report (i.e., "Incident Report") intends to include even more content.

[ryancdickson]

In response to @aarongable's comment:

Thank you for the speedy and detailed review! We also very much appreciate the set of suggested edits made to the draft, many of which have been accepted.

The distinction between "action items" and "ongoing commitments" feels arbitrary and awkward -- if I make a code change two weeks after the incident ended that prevents this from happening ever again, is that an action item (specific task with a deadline) or an ongoing commitment (designed to prevent a future incident)? Also this distinction is only used once: in the incident closure template. It's never referenced again, not even in the incident report template. Also, the incident closure template is embedded in the midst of the requirements, instead of being with the other templates at the bottom of the doc. Altogether this leads to the "ongoing commitments" section being jarring and out-of-place when reading the doc.

Thanks for sharing this perspective. The Steering Committee can discuss this further and someone will report back on this comment.

The new incident report template is difficult to read. There is so much going on there, and the use of square brackets to indicate text that should be replaced is confusing in a markdown context where square brackets generally denote hyperlinks. It's also unclear whether we're supposed to simply address each bullet point, or copy each bullet point into our report.

One of the goals of consolidating the reporting expectations within the template was to promote more reliable collection of the required information.

As an example, many Incident Reports filed over the last year (and beyond) have failed to include a narrative on how the root cause(s) of a subject incident avoided detection, yet this expectation is described on CCADB.org (below).

The Root Cause Analysis section must contain a detailed analysis of the conditions which combined to give rise to the issue. It is unusual for an incident to have a single root cause; often there must be a confluence of several issues such as a software bug, insufficient checks, and a malformed request. Make sure that all contributing causes are identified and described, including noting when they first arose and how they avoided detection until they were discovered or identified.

Our thought was that perhaps because the reporting expectations were separate from the reporting template, it contributed to the inadvertent omission of required information.

Questions:

1. Would you consider the added specificity in the reporting template an effective measure at addressing the described problem?
2. Is it your preference that the CCADB continues to keep the reporting expectations and IR Markdown templates separate?
3. Given the problem description, do you have any other thoughts as to how we can ensure report content more reliably satisfies the reporting expectations, without introducing needless complexity to the template itself?

Thanks again!

[wthayer]

Given the more prescriptive requirements being added elsewhere, I wonder if "weekly" is sufficiently descriptive of this expectation. For example, can someone post an update on a Monday and then on Friday of the following week? I'm not a fan of making this strictly 'within 7 days' either. Perhaps SHOULD weekly and MUST within 14 days would be a good balance here.

[wthayer]

Frankly, this feels punitive. Is there much value in getting this info every 3 days rather than every 7?

[wthayer]

I find this requirement quite confusing. How is "public reporting" (presumably by the CA) a form of measurability that is verifiable? Requiring external verifiability seems to greatly restrict actions the can be ongoing commitments. In the example below, I fail to see how "The presence of linting failures where certificates were issued more than 7 days after a new lint was released will represent a failure of this ongoing commitment." meets this requirement, unless we're saying that "verifiable" only means that the community will notice if/when something goes wrong that implies that the commitment was not met, and that seems pointless.

[aarongable]

1. Would you consider the added specificity in the reporting template an effective measure at addressing the described problem?

I agree that having the requirements and template in separate places probably did lead to inadvertent omission of required information. However I don't think that moving the requirements into the template will be an effective solution. Details below.

2. Is it your preference that the CCADB continues to keep the reporting expectations and IR Markdown templates separate?

Yes, simply because I think that formatting requirements-within-comments-within-markdown-within-a-code-block-within-markdown makes those requirements difficult to read and understand. I think that this arrangement is a good idea in theory, but I think in practice it will result in difficult-to-read reports that poorly copy-paste bullets from the template and accidentally leave formatting or prompt text in place.

Also, I cannot imagine a world in which all of the requirements (e.g. how often reports need to be updated) end up within the template. As such, people writing incident reports will need to be referencing requirements in multiple places anyway. Having to cross-reference requirements from multiple sources increases the cognitive load required to write the report, and will result in slower and less well-constructed reports. The more onerous the report-writing process becomes, the more rules-as-written (as opposed to spirit-of-the-rules) the reports will be.

3. Given the problem description, do you have any other thoughts as to how we can ensure report content more reliably satisfies the reporting expectations, without introducing needless complexity to the template itself?

I would follow RFC 3647's lead on tackling this issue. The requirements document should have the exact same set of headings as the template, and the contents of each section should be the requirements concerning exactly that section. Then the templates can just be (markdown codeblock formatted) lists of empty headings, contained in an appendix.

[wthayer]

It might be helpful to add that fields that are not applicable (e.g. Appendix) shall be included in the report and marked as such. I'm not a big fan of being so prescriptive, but I think the "MUST be completed." language takes us down this road.

[wthayer]

Where do ongoing commitments belong?