# Cyber AI Learning — Research Planning Worksheet

Use this worksheet to turn a practical observation into a testable, reproducible research question. Suggested planning time: 45–60 minutes; a full study requires additional work.

## 1. Define the question

Example: “Under a fixed review-time budget, does a model-based ranking improve the number of correctly resolved synthetic incidents compared with a rule-based ranking?”

- Your question:
- Population and operating context:
- Decision or task:
- Proposed contribution:
- Falsifiable hypothesis:

## 2. Choose the comparison

- Baseline and why it is credible:
- Proposed method:
- Conditions held constant:
- Unit of analysis:
- Primary outcome:
- Secondary outcomes, including effort and cost:
- Failure criteria:

Pre-declare the principal comparison and scoring rules. Separate exploratory findings from confirmatory tests.

## 3. Describe the data

- Source, permission, and intended use:
- Synthetic or observed:
- Labeling procedure and uncertainty:
- Class balance and sampling strategy:
- Repeated users, devices, or related events:
- Train/validation/test separation:
- Temporal or domain shift to evaluate:
- Information available at prediction time:
- Sensitive information and data-minimization plan:

The site's 20-record synthetic dataset supports teaching and software checks. It is too small and artificial to substantiate claims about real-world detection effectiveness.

## 4. Specify evaluation

- Metrics and their connection to the decision:
- Threshold-selection procedure using validation data:
- Independent or blinded review plan:
- Repeated runs and random seeds, if applicable:
- Sample-size rationale:
- Uncertainty estimates appropriate to the design:
- Error analysis by meaningful case categories:
- Ablations or alternative explanations:

Avoid choosing the final threshold or model by repeatedly inspecting the test set. Explain what a negative result would mean.

## 5. Review the system and people involved

- Assets and trust boundaries:
- Tools and permissions:
- Human review and escalation:
- Potential impact on participants or real systems:
- Required institutional review or organizational authorization, if applicable:
- Failure containment and stopping conditions:

## 6. Plan reproducibility

Record code and dependency versions, data provenance, transformations, prompts, model identifiers, settings, evaluation scripts, raw outputs, and exclusions. Preserve unsuccessful runs and document why any result was excluded. State which materials can be shared and provide a reproduction procedure.

## 7. Bound the claim

Complete this sentence:

“Under __________ conditions, using __________ data and __________ evaluation procedure, we observed __________ compared with __________. This does not establish __________.”

List the three most important threats to validity:

1.
2.
3.

## 8. Set the next concrete milestone

- Smallest useful pilot:
- Required evidence:
- Owner and review date:
- Decision after the pilot:

## Suggested outputs

A one-page protocol, a data statement, an evaluation script or scoring rubric, a reproducibility checklist, and an honest limitations section.

## References

- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)
- [MITRE ATLAS](https://atlas.mitre.org/)
- [scikit-learn User Guide](https://scikit-learn.org/stable/user_guide.html)