Managing Risks of Generative AI in Software Testing practice questions

Reasoning errors occur when LLMs misinterpret logical structures; the root cause is that LLMs rely on pattern matching rather than true logical reasoning.

Reasoning errors are misinterpretations of cause-and-effect, conditional logic or step-by-step processes. The syllabus explains that LLMs, unlike humans, lack true logical reasoning and rely on pattern matching, so tasks that require logic, such as test planning and test case prioritization, are exposed. Training-data skew explains bias, and temperature and context window are separate properties that do not create or remove the lack of reasoning.

1. Identify the defect type named in the question: reasoning error.
2. Recall the syllabus cause: pattern matching instead of true logical reasoning.
3. Reject causes that belong to bias (training data) or to output variability (temperature).

Exam tip: LLMs pattern-match; they do not truly reason, so logic-heavy test tasks need checking.

LLM bias comes from the training data and makes output favour certain information, approaches or assumptions.

The syllabus traces bias to the training data: a model trained mostly on English-language data, for example, may underrepresent non-English perspectives. In testing this shows up when generating test data or refining acceptance criteria. Temperature, seed and tokenization are inference mechanics; they influence how output is produced but are not where systematic preferences come from.

1. Recall the definition of bias in 3.1.1.
2. The origin named is the training data.
3. Inference settings such as temperature and seed affect variability, not bias.

Exam tip: Bias in = bias out: it starts in the training data.

Non-deterministic behaviour makes hallucinations, reasoning errors and biases hard to fix: they can seem fixed in one output and reappear in another conversation.

LLM output comes from probabilistic sampling, so the same prompt can give different responses. The syllabus warns that this is why these defects are difficult to fix: a prompt change that removes a hallucination in one conversation does not guarantee it is gone in the next. Retraining is not indicated, prompt changes do matter, and inventing a rule that is not in the test basis is a hallucination rather than a reasoning error.

1. Note that the prompt and the LLM are unchanged, but the output differs.
2. Same input, different output = non-deterministic behaviour.
3. Conclude that a single clean run does not prove the defect is fixed; keep verifying output.

Exam tip: One clean response does not prove a hallucination is fixed — LLM output varies run to run.

Test cases that verify non-existent acceptance criteria are a typical hallucination in software testing.

The syllabus lists "suggesting test cases that verify non-existent acceptance criteria" as a way hallucinations show up in testing. Fingerprint login is not in the user story, so the test case is fictitious with respect to the test basis. A reasoning error would misread one of the three given criteria; bias would show as a systematic skew; prompt injection is a security attack covered elsewhere.

1. List what the test basis actually contains: three acceptance criteria.
2. Compare the output: one test case targets a feature not in the basis.
3. Invented, irrelevant content = hallucination.

Exam tip: A test for a requirement nobody wrote is a hallucination.

Reasoning errors are faulty logic in tasks such as prioritization; the content is real but the conclusion drawn from it is wrong.

The syllabus names test case prioritization as a test task that needs logical reasoning and where LLMs make reasoning errors, because they pattern-match rather than reason. Here all inputs are reproduced correctly, so it is not a hallucination, and there is no systematic skew pointing to bias. The model simply failed to apply the ordering rule it was given.

1. Check whether any content is invented: no, all IDs and risk levels come from the input.
2. Check the logic: medium-risk TC3 is placed before high-risk TC1 and TC4.
3. Correct inputs with a wrong conclusion = reasoning error.

Exam tip: Real inputs, wrong ordering or calculation = reasoning error.

Bias makes LLM output favour certain information or assumptions, for example underrepresenting non-English or non-US data in generated test data.

The syllabus explains that models trained primarily on certain data may underrepresent other perspectives, and that bias can influence generated test data. Fifty records that all follow US formats for a worldwide shop are a systematic skew of exactly this kind. Invented records are expected for synthetic data, so this is not a hallucination, and no logic was misapplied.

1. Note the requirement: a worldwide customer base.
2. Note the output: one region's formats throughout.
3. A systematic, one-sided skew = bias.

Exam tip: Skewed, one-sided test data = bias.