Test Question Design to Disrupt Generative AI Misuse
Sidebar
Generative AI tools (such as ChatGPT, Claude and Gemini) don’t think or understand course concepts. They operate on probabilistic next-token prediction. Basically, they calculate the most likely sequence of words mathematically based on the patterns present in their training data. Standard textbook definitions, classic case studies and generic essay prompts are available in millions of instances online, enabling AI to generate correct answers with extremely high confidence. After all, in most cases these are the exact resources used to train the model.
To make questions “AI-resistant”, you can write prompts and questions that force these tools into low-probability prediction territory.
5 Strategies to Break (or Bend) AI Probability Mechanics
Large language models rely on familiar word pairings and common patterns. Introducing custom scenarios, unique fictional entities, or local/institutional variables forces the AI into unfamiliar territory where generic pattern-matching is more likely to fail.
Original Example Question: What are the key ethical considerations when a company decides to move its manufacturing from one city to another?
Revision: A fictional Louisville brewery, “River City Ales”, is considering moving bottling to a facility in Southern Indiana to lower transport costs but must balance local union agreements discussed in Lecture 4. Which factor creates the primary conflict?
Large language models excel at generating plausible, fluent text but struggle to identify subtle logical leaps or errors in judgment.
Original Example Question: Which of the following is the correct definition of confirmation bias?
Revision: Below is an excerpt from a team meeting. An analyst concludes that “Confirmation Bias” caused the project delay. Evaluate this claim and determine how the analyst misapplied the concept from the options below.
When asked a question with a single clear answer, AI answers easily. When forced to weigh multiple, competing, mutually exclusive constraints defined specifically in class, AI tends to default to generic hedging rather than apply a formal framework.
Original Example Question: What is the primary advantage of adopting a just-in-time inventory model?
Revision: A firm faces a 20% spike in raw material volatility, strict ESG reporting rules, and a 48-hour delivery SLA. Based on the framework from Chapter 5, which inventory adjustment presents the necessary tradeoff?
Copying and pasting text into AI is a common form of quick cheating. Embedding diagrams, annotated tables, or flowcharts creates physical friction.
Original Example Question: Interpret the outcome when Supply Curve A shifts left while Demand Curve B remains stationary.
Revision: Refer to the custom decision tree image attached. If variable Y drops by 10%, which path must the operator follow?
Standard questions ask “Which of these is correct?” Inverted questions ask students to identify the exception, construct a counter-example, or select which condition would cause a standard rule to break down.
Original Example Question: Which statistical test should be used to compare the means of two independent groups?
Revision: Under which specific data distribution anomaly would applying an Independent Samples t-test produce a false positive result, based on our class lab dataset?
Quick Reference for Common Question Types
| Question Format | Vulnerable to AI When… | AI-Resistant Adaptation |
|---|---|---|
| Multiple Choice | Distractors are obvious or stem uses standard textbook phrasing. | Use scenario-based stems with plausible-sounding distractors. |
| Multiple Select | Evaluating isolated factual statements. | Require students to select all conditions that satisfy a complex, multi-constraint rule simultaneously. |
| Matching | Matching standard terms directly to definitions. | Match the novel mini-scenarios to the applicable concept. |