English academy teachers have a lot to do besides teaching students.
One of those tasks is writing English questions for their students’ exams.
This was a pain point for the teachers. Writing exam questions takes creativity and a considerable amount of time. The question types follow certain patterns, but the English passage changes every time. At some point, I thought, “Isn’t this the kind of thing LLMs are good at?” I decided to see if I could free the teachers from this tedious, repetitive work.
The feature was simple. Given an English passage, it would generate several types of questions. I built the first MVP with Codex in three hours, and the output looked fairly convincing.
But when we actually used it, it was not good enough for the classroom.
Convincing Results
When the teachers and I reviewed the questions generated by the LLM, we found several problems.
- The question types did not match the forms teachers normally expected.
- The questions were far too easy for the requested grade level.
- The answers were correct, but the expressions being tested had little educational value.
- More than one answer could be correct depending on the context.
We decided to address them one at a time.
1. The question types did not match the forms teachers normally expected.
The LLM knew what a question type such as “identifying the main idea” was. But the teachers reviewing the questions spent more time designing the wrong choices than the correct one.
Teachers have patterns for designing wrong choices. For example, each wrong choice in a main-idea question should be wrong for a different reason.
1. The correct answer must represent the passage as a whole.
2. Each wrong choice must be wrong for a different reason.
- A detail from the passage
- Information not found in the passage
- An overgeneralization
- A sentence with only similar keywords
This was the kind of information that AI could not consistently apply to the desired standard unless it was stated explicitly. I needed to structure the domain knowledge that the teachers had built over many years, turn it into a template, and present it in a form the AI could understand.
2. The questions were too easy for the requested grade level.
I did not know how to tell an LLM about an abstract concept like “difficulty.”
I could not even explain the difference between questions for first- and third-year high school students, so the AI could not have known exactly what I meant by “first-year level.”
I asked one of the teachers I had been working with for help. They gave me past exam questions organized by grade and question type, and I analyzed them with the AI. Some patterns began to emerge.
| Category | First year of high school | Third year of high school |
|---|---|---|
| Subject-verb agreement | The subject and verb are close together. | The subject and verb are far apart. |
| Tense | The tense is easy to identify from a time expression. | The time must be inferred from context. |
| Voice | The agent and object are clear. | Omitted information must be reconstructed. |
| Non-finite verbs | The question tests basic usage. | Both time and voice must be considered. |
| Relative clauses | The question distinguishes between who and which. | Both the preposition and relative pronoun must be considered. |
| Sentence structure | A simple clause structure | A complex structure with clauses inside clauses |
| Parallelism | The elements are close together. | The elements are far apart. |
At least for grammar questions, difficulty depended more on the complexity of analyzing the sentence structure and reconstructing the context than on which grammar point was being tested.
For example, a first-year question may place the subject (The students) and verb (is) right next to each other, making the error easy to identify. Since the subject is plural, are would be correct.
The students is waiting outside.
In the third-year example, the student must set aside several modifiers and a relative clause to find that The increase is the actual subject before deciding whether have is correct.
The increase in online transactions, together with several unexpected policy changes that affected consumer behavior, have caused new security concerns.
3. The expressions being tested had little educational value.
This issue was related to difficulty. Creating a grammatically incorrect expression was easy. Changing one word in the original passage or removing an article was enough to make a grammar question.
For example, the sentence below needs the article a before book.
She bought book at the store.
The sentence is grammatically wrong, but a high school student can find the answer by checking a single article without analyzing the sentence structure or context. A sentence being grammatically incorrect and a question being worth learning from were two different things.
At first, I only told the AI to modify the source text and create an incorrect expression. It tended to choose the easiest expression to change from the many available candidates, such as removing an article or switching between singular and plural.
I therefore limited the grammar points that it could use to the following list.
- Subject-verb agreement
- Choice and structure of relative pronouns and relative adverbs
- Active and passive voice in present and past participles
- Inversion caused by negative and limiting expressions
Once I combined these limits with the grade-level difficulty criteria from earlier, the quality of the generated questions improved considerably.
4. More than one answer could be correct depending on the context.
No matter how carefully I wrote the prompt, questions generated by an LLM did not guarantee a unique answer or logical consistency. We needed a process that solved each generated question from another perspective and checked whether it reached the same conclusion.
I set up three validation agents to solve each question in separate calls without seeing one another’s answers or the answer provided by the generator. Of course, three agents choosing the same answer does not guarantee that it is correct. But checking whether they independently reached the same conclusion allowed us to filter out clear logical conflicts in advance.
For example, the validation agents shared the following goals, with separate checks assigned to each role.
Shared goals
- Solve the question independently without seeing the generator’s answer.
- Provide the answer they reached and the reasoning behind it.
- Check whether any other choice could also be correct.
Checks by role
| Role | What to check |
|---|---|
| Answer and choice validation | Confirm that there is exactly one correct answer and that every other choice is clearly wrong. |
| Evidence and source-text validation | Confirm that the passage contains evidence for the answer and that the meaning of the source text has not been changed. |
| Difficulty validation | Confirm that the question is appropriate for the requested grade and requires the reasoning expected at that level. |
The Teacher Makes the Final Decision
AI clearly helped generate and validate questions quickly. But the standard for a good question can vary even within the same grade depending on the students’ level or how far the class has progressed. Some areas are still difficult to structure and require the teachers’ judgment. I therefore decided that the process needed final approval from a person.
Teachers could also review questions that failed automated validation along with the reason for the failure. To keep the validation result from replacing human judgment, regeneration occurred only when a teacher requested it.
What We Need to Keep Generating Good Questions
Once the feature was working, the next step was to record operational metrics for the service.
Both introducing a system and using an LLM cost money. Someone running an academy cannot ignore that cost. The system is still in testing, so I recorded the data below to at least compare the cost of tokens with how much teacher time the system saved.
(We still need to determine how much the English question generator actually reduces teachers’ working time. We are currently discussing what approval rate would make it cost-effective.)
- Tokens used to generate one question
- Time taken to generate one question
- Validation success rate for generated questions
- Number of regeneration attempts
- First-pass approval rate
To be honest, the results so far are not yet satisfactory. After introducing templates for each question type and splitting the validation process into more detailed steps in June, both tokens per question and generation time increased. The validation success rate and first-pass approval rate also rose. I cannot say what caused the change, but the number of regeneration attempts did decrease.
| Tokens per question | Average time | Validation success rate | Regeneration attempts | First-pass approval rate | |
|---|---|---|---|---|---|
| 2026.05 | 2428 | 4.2s | 21% | 3.9 | 12% |
| 2026.06 | 8247 | 7.4s | 58% | 2.1 | 38% |
What I Had to Understand Before the AI Could
At first, I thought expensive models and the latest techniques would produce good questions. But when the instructions were unclear, using a better model did not improve the results.
What mattered most was understanding what practitioners actually do in the classroom.
Only after I understood how teachers write questions, choose good ones, and make those judgments could I give the AI precise instructions. If my request is vague, the AI gives a plausible but evasive response. The clearer my request is, the better its answer. Anthropic also emphasizes this principle in its official documentation.
Source: Claude prompting best practices
AI has given us a powerful tool, but I do not think the essential role of an engineer has changed very much.
It is still about listening to people in the field, separating what we know from what we do not know, finding the real problem, and solving it.
That may be the kind of engineer I should work toward becoming.