Free Claude CCAR-P Exam Questions (2026 Update)

Explanation

C. Define the hypothesis and primary success metric. Before random assignment, the experiment must pre-specify a clear hypothesis and a single primary metric. Anthropic's official prompt engineering documentation explicitly states that "defining the task and success criteria" is the first and most critical step, with success criteria needing to be specific and measurable to guide subsequent evaluation and optimization. If the metric is chosen after seeing the data, it constitutes post-hoc selection and renders the experiment invalid.

A. Determine the minimum detectable effect size and sample size. Sample size must be calculated before the experiment to ensure sufficient statistical power to distinguish genuine effects from noise. LLM outputs are stochastic, and small prompt differences are often subtle, making insufficient sample sizes likely to produce false conclusions. Experimentation methodology literature points out that running underpowered tests is one of the most common experimental design errors. The sample size formula requires pre-input of the minimum detectable effect size (MDE) and significance level.

Why the Other Options Are Incorrect

B. Decide to promote, reject, or iterate — This is a post-experiment decision that can only be made after analyzing the results, not a pre-experiment step.

D. Analyze results against predefined metrics — This is a post-data collection analysis activity that occurs after the experiment is completed.

E. Document recommendations, trade-offs, and alternatives — This is post-decision documentation work, belonging to the final stage of the experiment lifecycle.

Explanation:

The dominant cost drivers identified in the question are the long, repeated system prompt and the large retrieved context per request. Option C directly addresses both: moving the stable system prompt into a cacheable prefix allows subsequent requests to reuse the cached computation, with cache-read tokens billed at a substantially lower rate—Anthropic's own pricing documentation confirms cached reads cost a fraction of standard input pricing. Meanwhile, trimming retrieved context to only query-relevant spans reduces the volume of input tokens per request, which is precisely the variable driving the other half of the cost structure. Research on retrieval depth tradeoffs shows that always using heavy retrieval wastes budget on unnecessary context, while per-query routing can reduce billed tokens by 26% without quality loss.

Why the other options are incorrect:

A increases retrieval depth, which worsens the dominant cost driver by adding more retrieved tokens per request—the opposite of optimization.

B adds more repeated content to the system prompt, increasing the very token volume that needs reduction.

D switches to the heaviest model, raising per-token costs without addressing the structural waste from redundant prompts and excessive retrieval.

References:

- Anthropic prompt caching documentation: cache reads are billed at reduced rates and the cache follows a prefix-matching hierarchy.

Explanation:

The core of this question is "silent quality drift in safety-related outputs"—it is invisible, has serious consequences, and is specifically triggered by model version upgrades. Option D maintains an adversarial evaluation set and performs version-attributed scoring, which directly addresses two key requirements: first, using the same fixed adversarial test set ensures that results are longitudinally comparable across different model versions, rather than the evaluation criteria drifting with each upgrade; second, version attribution ensures that each score can be traced back to a specific model version, making it impossible for degradations to be obscured by "overall metrics look fine." Research shows that silent updates to LLM services can introduce behavioral drift, leading to regressions in safety constraints and other aspects, and deployer-side compatibility gates (blocking updates unless safety/performance standards are met) are an effective governance tool. The Anthropic platform documentation also explicitly states that even if model weights are fixed, serving infrastructure updates (including safety classifiers) may produce observable behavioral differences—this further demonstrates the necessity of fixing the evaluation set and scoring by version.

Why the other options are incorrect:

- A disabling adversarial evaluation entirely eliminates the only means of detecting safety drift; quality regressions will only be discovered after entering production.

- B random rotation of adversarial inputs makes cross-version comparisons impossible (each upgrade is tested with different questions), so degradation cannot be attributed to version changes.

- C skipping evaluation and relying on user complaints essentially makes production the testing ground; by the time problems are discovered, safety-related harm has already occurred.

References:

- arXiv (2026): deployer-side governance framework proposes "risk-category-based testing suite" and "compatibility gates" to address silent updates and safety regressions.

Explanation:

This question requires mapping the four observations to the CCAR-P business value pillars (Efficiency, Productivity, Solution Cost, Performance SLA).

- Observation 1 (summarization time decreased from 47 to 6 minutes): This is a direct efficiency gain. Efficiency measures the speed or resource cost to complete a specific task.

- Observation 2 (associates spend less time, but billable output has not changed): This shows that productivity has not yet been realized. Productivity means increasing total output (revenue, billables) with the same resources. Saving time is only an efficiency gain unless that saved time is redeployed to generate more value.

- Observation 3 (infrastructure costs increased by 22% due to retry logic): This is a solution-cost issue. The technological solution is consuming more budget than planned due to inefficiencies (redundant retries).

- Observation 4 (attorney correction adds 8 minutes per document): This is an efficiency loss (rework) that partially offsets the gains from Observation 1. The net time saved is actually 33 minutes (47 - 6 - 8), not 41 minutes.

Why the other options are incorrect:

- A misclassifies Observation 4 as a performance-SLA issue. SLA refers to system latency or uptime contracts, not human rework time.

- B misclassifies Observation 3 as SLA degradation. Increased API costs do not inherently violate a latency/uptime SLA. It also misclassifies Observation 2, which is a productivity gap, not an efficiency gain.

- C incorrectly states Observation 4 is an SLA improvement (it is a loss) and misclassifies the other pillars.

References:

- CCAR-P Domain 1: Business value pillars. Efficiency (doing the same task faster), Productivity (increasing business output), Solution Cost (financial cost of the tech), and Performance SLA (system reliability/latency).

Explanation:

The core of the Discernment competency is critically evaluating the quality of AI outputs, rather than merely counting how often outputs are accepted. In this scenario, the "89% first-response acceptance rate" only shows that specialists initially accepted the recommendations; it does not prove that the recommendations were correct.

Option C (scope gap in the input space) directly addresses the root cause revealed by the follow-up surveys: specialists frequently override the recommendations "after considering criteria the assistant did not evaluate." This means that the assistant's input space lacks the key evaluation dimensions that specialists actually use. This is a Discernment judgment about the completeness of the problem definition—the output may "look reasonable," but it does not cover the decision factors that truly matter.

Option D (acceptance does not establish correctness after review) directly targets the misleading nature of the acceptance rate. Discernment requires judging the "Product" quality of the output, not the initial reaction. If the recommendations are overturned at a high rate during specialist review, then the high acceptance rate is merely a superficial metric and cannot support launch readiness.

Why the other options are incorrect:

- A contradicts the core of Discernment: acceptance rate is a process metric and cannot replace judgment about output quality.

- B concerns statistical representativeness, which pertains to survey methodology, not to the Discernment competency's critical evaluation of AI output quality.

- E concerns the adequacy of the pilot duration, which is an empirical sufficiency issue for reliability and does not directly address the substantive problem that recommendations are frequently overturned due to omitted evaluation criteria.

References:

- Discernment competency definition: evaluating AI outputs critically, including Product (output quality) and Process (reasoning) dimensions.

Explanation:

The core clue is "malformed JSON began to appear after the scheduled maintenance window," which suggests a configuration change introduced during maintenance (rather than a model-side issue or a long-standing bug). The two investigative steps should directly verify "what changed" and "whether it is reproducible."

Option B (compare pre- and post-maintenance configurations) directly targets the most likely root cause. Malformed JSON typically originates from a change in the output schema, system prompt, or tool definitions. Claude's structured output relies on tool schemas or output_config constraints; if the schema was modified, reordered, or serialized non-deterministically during maintenance, the model's JSON output may no longer conform to expectations. Comparing against the last known good version is the fastest way to localize the discrepancy.

Option E (replay pre-maintenance requests and inspect raw output) is another key step. It distinguishes "model behavior change" from "downstream parsing issue": by replaying known-good requests, if the raw output is already malformed, the problem lies in the model's input configuration; if the raw output is valid JSON but downstream parsing fails, the problem lies in the parsing logic. This directly isolates the failure point.

Why the other options are incorrect:

- A (clearing the prompt cache) targets cache hits and cost, not JSON format correctness; cache invalidation does not cause JSON to become malformed.

- C (switching model tiers) is a costly workaround that neither confirms the root cause nor excludes provider-side factors; there is no evidence in the question pointing to model changes.

- D (increasing max_tokens) addresses output truncation, but truncation typically manifests as incomplete JSON (missing closing braces) rather than "malformed"; moreover, the timing of onset after maintenance does not support truncation as the primary cause.

References:

Structured outputs and schema adherence: Changes to schema or prompt configurations can directly affect JSON validity.

Explanation:

The deployment explicitly states its priority as "accuracy first, latency second." This is the decisive constraint for the judgment. Configuration X achieves a 38% p95 latency reduction but comes with a measured 2% accuracy benchmark decline. Since accuracy is the first priority, any measurable accuracy regression cannot be accepted unless explicitly authorized to trade off accuracy. Configuration Y maintains existing retrieval depth and the heavier model, with unchanged latency and no measurable accuracy change, fully aligning with the stated priorities.

Anthropic's official documentation supports this judgment logic: it is recommended to first establish a performance ceiling with the most capable model and unconstrained prompt, and only after achieving desired output quality should you then experiment with smaller models or shorter prompts to reduce latency and cost as needed. This means that model downgrading or retrieval reduction should be second-order optimization after confirming accuracy is unaffected, rather than proactively making trade-offs when accuracy is the first priority. In compliance Q&A scenarios, the potential cost of a 2% accuracy regression (providing incorrect compliance guidance) far exceeds the benefit of a 38% latency improvement.

Why the other options are incorrect:

- A assumes "2% falls within normal benchmark variance," but the question clearly states this is a "measured" decline, and the deployment prioritizes accuracy first; this assumption lacks a basis.

- C directly contradicts the deployment's stated priorities, using a general preference to override explicit requirements.

- D similarly imposes a "general rule" that conflicts with the deployment's stated priorities; in compliance scenarios, accuracy typically cannot be traded for latency.

References:

- Anthropic prompt engineering documentation: it is recommended to first establish a performance ceiling with the most capable model, then consider smaller models or shorter prompts to reduce latency and cost as needed.

Explanation:

The question requires reducing cost without affecting accuracy, and the optimization should target the two largest cost contributors. Profiling shows that the repeated 6,000-token policy preamble accounts for 45% of cost, and the average 3,000-token retrieval of historical sales chunks accounts for 30%.

Option C (enabling prompt caching on the static policy preamble) directly targets the largest cost item. Prompt caching reuses static prefix content across calls, with cached reads billed at a fraction of standard input pricing—typically 10% of base input token price, with some models as low as 2.5% or 5%. The 6,000-token preamble is identical on every call and is an ideal cache candidate. Crucially, caching does not alter the model's input content; it only avoids redundant computation, so accuracy remains unchanged.

Option E (caching common retrieved sales chunks) targets the second-largest cost item. If many daily queries retrieve overlapping historical sales chunks, caching these shared chunks avoids re-billing the same content on every call. This likewise preserves the exact retrieved content sent to the model, so accuracy is unaffected.

Why the other options are incorrect:

- A (reducing retrieved chunks) would cut retrieval depth, which likely changes the information available to the model and risks accuracy regression—explicitly prohibited by the stakeholder requirement.

- B (truncating the policy preamble) removes clauses, altering the instructions the model receives and potentially changing output quality.

- D (switching to a smaller model) changes the model itself, which is the most direct way to affect accuracy and violates the requirement.

References:

- Anthropic prompt caching documentation: cache reads billed at 0.1× base input price (model-dependent exceptions apply).

Explanation:

The core constraint of this question is: per-turn latency is the dominant performance bottleneck, and quality on routine turns is already acceptable. Therefore, the optimization direction should be to directly reduce latency without disproportionately damaging quality.

Option B addresses latency from two dimensions simultaneously: reducing retrieval depth to the top-k passages that historically cover the answer can cut the time consumed by retrieval and the input tokens for model processing—research shows that per-query adaptive retrieval depth reduces billed tokens by 26% and latency by 34% compared to always using heavy retrieval, while maintaining answer quality parity; caching stable system prompt content directly reduces time-to-first-token through prompt caching, with official Anthropic data showing that multi-turn conversations with long system prompts experience latency dropping from approximately 10 seconds to approximately 2.5 seconds (75% reduction) after caching. Both measures act on the latency bottleneck without touching the model itself or quality baseline, and therefore will not cause disproportionate damage to quality.

Why the other options are incorrect:

- A disabling prompt caching causes the system prompt to be reprocessed on every turn, which instead increases latency—official data shows that caching significantly reduces time-to-first-token.

- C increasing retrieval depth to the corpus maximum directly increases tokens and processing time per turn, worsening the latency constraint.

- D switching to the heaviest model increases inference latency, moving in the opposite direction of the latency optimization goal.

References:

- Anthropic official blog: prompt caching reduces latency by up to 85% for long prompts, with a measured 75% time-to-first-token reduction in multi-turn conversation scenarios.

Explanation:

The question asks which steps must be completed before running the deployment against the evaluation dataset. This maps directly to the "specification" phase of the evaluation pipeline—defining what will be measured and preparing the measurement instrument itself, before any model outputs are generated.

Option D (define metrics and slices) is the foundational specification step. Anthropic's evaluation guidance explicitly states that prompt engineering should begin with building a rigorous eval set, not writing prompts. You cannot interpret a score without first defining what dimension it measures (accuracy, tone, safety) and which data slices it covers (representative, edge, adversarial). Without this definition, running the deployment produces numbers with no agreed-upon meaning.

Option B (curate and label the evaluation dataset to match the defined slices) is the direct preparation of the evaluation instrument. Once metrics and slices are defined, the dataset must be labeled with golden answers or reference outputs so that scoring can occur. The dataset must align with the slices defined in D—otherwise the metrics will be computed over the wrong population.

Why the other options are incorrect:

- A (publish metrics and gate releases) occurs after scoring and aggregation—it is a post-evaluation governance action, not a pre-run specification step.

- C (review failure cases with SMEs to refine rubric) happens after outputs exist to review. It is an iterative refinement activity, not a prerequisite specification step.

- E (score outputs and aggregate metrics) is the execution of evaluation itself. It cannot happen before the dataset and metrics exist, and it is definitionally after the deployment has run.

References:

- Anthropic empirical performance evaluations: eval lifecycle begins with building the eval set before prompt iteration; eval components include input prompts, outputs, golden answers, and scores.