Hydra tutorial series — Part 6: Troubleshooting & escalation
What to do when the pipeline gets stuck: updating a stale PR, re-queueing a review stage or a build, and unblocking a needs-input stage — ordered from cheap to expensive. Sixth of seven short modules.
In part 5 the pipeline was running. This part is about the other case: what do you do when it doesn't go green? It gives you the decision tree — ordered from cheapest intervention to most expensive — and the patterns that get a stuck issue moving again.
What do the outcomes mean?
The two lanes of hydra-sequencer end differently, and it helps to keep them apart.
The build lane ends in one of four labels. None of them is needs-input:
| Label | What happened | Is there a PR? |
|---|---|---|
build:pass | The gates were green, straight away or after the one fix pass. | Yes |
build:fail | The gates were still red after the one fix pass. | Yes, with the gate output in its body |
build:no-change | The agent changed nothing, so there was nothing to push. | No |
build:blocked | The issue body names no openspec/changes/<name>, that change does not exist on the base branch, or the build could not run. | No |
The review lane (code review → security review → applier) ends each stage in :pass, :fail or needs-input. needs-input is not a verdict on the code. It means the stage produced no usable verdict:
- the verdict file
hydra-verdict.jsonis missing or unreadable; - the verdict does not list
hydra-gatesinchecks_run, so there is no proof the gates ran; - the stage could not run at all.
A :fail is a real verdict: the reviewer looked and said no. Nothing further is queued after a :fail, and after build:pass nothing is queued at all: you add code-review:queued yourself.
Your role as a human: find out which of these you are looking at, and pick the cheapest lever below that fits.
The intervention ladder
Every lever is a label. Set it with scripts/hydra-label.sh, so the project board follows.
| # | Situation | Intervention | Cost | What you keep |
|---|---|---|---|---|
| 1 | Development has moved on and the PR is stale (new lint rules, ADRs, fixtures on development) | gh pr update-branch <N>, or wait for cron-update-prs.sh (hourly) | Free | Everything. Only the base is refreshed. |
| 2 | build:pass and nobody reviewed yet | Add code-review:queued | Up to three review stages | The build. |
| 3 | Review contracts changed, or a review should run again | Remove that stage's :pass/:fail (and needs-input), add <stage>:queued | One to three review stages | The build. On a pass the sequencer queues the next stage again. |
| 4 | A review stage ended in needs-input | First remove that stage's :queued label, fix the cause, then remove needs-input and add :queued again (see below) | One stage | The build and the earlier verdicts. |
| 5 | The build went wrong: build:fail, build:no-change, build:blocked, or a build:pass that is wrong anyway | Remove the build:* verdict, add build:queued | A full build | Nothing from the old build. The sequencer builds again from the base branch and pushes feature/<issue>/hydra-build. |
| 6 | You closed the PR yourself and stale labels are still on the issue | Wait for reconcile.sh (every 10 min) | Next reconcile run | check_stage_without_open_pr sees the inconsistency and sets build:queued. Self-healing. |
| 7 | Pipeline infra is broken (flow switched off, a workload step that cannot start, a credential the broker cannot resolve) | Fix the infra first, and only THEN pick one of 1–5 | Time | — |
Order: always the cheapest that fits. Reach for a new build only if the build itself was wrong; otherwise you throw away good work.
Re-queueing a review stage
Say the code review failed and you think it should run again, because an ADR changed or you pushed a fix to the branch by hand:
./scripts/hydra-label.sh ConductionNL/<app> <N> remove code-review:fail
./scripts/hydra-label.sh ConductionNL/<app> <N> add code-review:queued
The sequencer picks it up on one of its next ticks. If the review passes, it adds security-review:queued itself, and the chain continues from there.
A review stage that ended in needs-input
:queued label firstA stage that ends in needs-input keeps its :queued label. The sequencer only looks for that label, so it picks the stage up again on the next tick — every five minutes — until someone removes it. Take the label off before you start investigating.
# 1. Stop the re-runs
./scripts/hydra-label.sh ConductionNL/<app> <N> remove security-review:queued
# 2. Fix the cause: gates not run, no verdict file, or a stage that could not start
# 3. Queue it again
./scripts/hydra-label.sh ConductionNL/<app> <N> remove needs-input
./scripts/hydra-label.sh ConductionNL/<app> <N> add security-review:queued
Always pass the bare label name. With HYDRA_LABEL_PREFIX set, hydra-label.sh adds the prefix itself, so wilco-needs-input would become wilco-wilco-needs-input. And since the flows only read bare labels, leave the prefix unset anyway (part 5).
Re-queueing the build
./scripts/hydra-label.sh ConductionNL/<app> <N> remove build:fail
./scripts/hydra-label.sh ConductionNL/<app> <N> add build:queued
Replace build:fail with whichever build verdict is on the issue. If there are also old review labels on it, ./scripts/hydra-label.sh ConductionNL/<app> <N> transition build:queued does it in one call: it strips every other queued label, needs-input and the old build verdict, and leaves build:queued.
What the push does with an existing feature/<issue>/hydra-build branch, and whether a new PR is opened next to an old one, is decided outside the flow definitions. If you want a clean PR, close the old one first.
What about retry:queued and rebuild:queued?
They do nothing today. No flow reads either label. Earlier versions of this part described what they did under the retired shell orchestrator: retry:queued compiled a feedback.md from hydra.json and ran the builder once more in HYDRA_MODE=fix, and rebuild:queued closed the PR and hard-reset the branch. That orchestrator is gone, and so is that behaviour; scripts/lib/build-feedback-brief.py has no caller.
create-labels.sh still creates both labels, and cron-update-prs.sh still puts rebuild:queued on the issue of a PR with merge conflicts, together with a comment. Nothing consumes that label, so treat the comment as the signal: resolve the conflict, or re-queue the build (rung 5).
When is it the gate, and when is it the builder?
A recurring pitfall from our own retrospectives: the same issue fails three times on the same mechanical gate — typically a missing SPDX header or a similar hygiene check. Each re-queued build reproduces the same pattern.
In that case a fourth build is not the right move. Be disciplined:
- Sanity-check the gate itself. Is this a false positive? (See part 3.) A classic: a grep without a word boundary that matches more than intended. The gates live in the
conduction/hydra-gatespackage (inConductionNL/.github); fix the gate there, or its skill inhydra/. - Hand-fix the mechanical violation. Often it's faster to add the SPDX headers yourself, commit on
feature/<issue>/hydra-build, and queue the code review (rung 2 or 3). Time: minutes. - Consider a stronger agent. If the same class of mistake shows the same blind spot across multiple repos, that's a signal that the build prompt or the mechanical gate isn't enough. Open an issue in
hydra/to tighten the gate or the brief.
Watch-list patterns
Patterns we've already seen and that are worth comparing against your situation first. The first three come from the shell pipeline that the flows replaced; the lessons still hold.
Diagnostics: where do you look?
There is no log file on your machine to tail. Quick checklist when an issue is stuck:
# 1. Which labels are currently set?
gh issue view <N> --repo ConductionNL/<app> --json labels --jq '[.labels[].name]'
# 2. Is there a PR, and what does its body say (gate output)?
gh pr list --repo ConductionNL/<app> --head feature/<N>/hydra-build --state all
# 3. What does hydra.json on the feature branch say about the last recorded stage?
gh api "repos/ConductionNL/<app>/contents/openspec/changes/<slug>/hydra.json?ref=feature/<N>/hydra-build" \
--jq '.content' | base64 -d | jq '.cycles[-1]'
The fourth place is the flow runs of hydra-sequencer in the OpenRegister instance: every tick is one run there. Look there when the labels and hydra.json don't explain what happened — for instance when a stage could not start, or its verdict could not be read.
Reading hydra.json (schema v2)
After a stage's labels have moved, the sequencer calls the hydra-record-stage flow, which appends an entry to openspec/changes/<slug>/hydra.json on feature/<issue>/hydra-build and commits it through the API. It refuses a stage without an owner: the record has to say who ran it, on whose credential. Each recorded stage becomes a new entry in cycles[]:
{
"schema_version": 2,
"app": "decidiq",
"repo": "ConductionNL/decidiq",
"issue": 15,
"spec_slug": "p2-meeting-management",
"cycles": [
{
"cycle": 3,
"outcome": "failed", // ← the headline: passed | failed
"outcome_reason": "build: 2 gate(s) failed",
"started_at": "<timestamp>",
"stages": [
{
"stage": "build",
"verdict": "fail",
"failures": 2, // number of failed gates
"exit_code": 2,
"output_tail": "…the gate output…", // what the gates actually said
"owner": "<nextcloud-uid>",
"credential_owner": "<nextcloud-uid>",
"credential_name": "<credential name>"
}
]
}
]
}
Read it top-down: the last cycles[] entry's outcome and outcome_reason tell you what happened, and output_tail tells you which gates said so. Two caveats. The record step currently names every stage build, review stages included, so don't read the stage field as proof of which stage ran. And a record write that fails does not stop the run, so a missing entry does not mean the stage didn't run; check the labels and the flow runs.
When is "human takes over" simply the right call?
Not every stuck issue deserves an automated path. Sometimes the most pragmatic action is: a human takes over, fixes it in a commit on the feature branch, pushes, and queues the review.
Examples:
- A small semantic decision the builder can't make without context outside the change (e.g. "should this field be optional or required?").
- An interaction with an external party (open ticket with a vendor).
- A gate that is a false positive and requires a fix in the gate package itself (which you won't resolve within one build).
Hydra is a factory, not an ideology. Take the human shortcut when it's the fairer call.
Test yourself
Four short questions to check whether you've grasped this part. Stuck? Click Hint. Curious about the answer? Click Answer.
1. In what order do you try the interventions in the ladder, and why?
Hint
One dimension: cost. The other: how much already-done work you lose.
Answer
From cheap to expensive, with the goal of preserving as much already-done work as possible:
gh pr update-branch— free. Everything stays; only the base is refreshed.- Add
code-review:queuedafter abuild:pass— nothing queues it for you. - Re-queue a review stage — remove its verdict, add
<stage>:queued. The build stays. - Unblock a
needs-inputstage — remove its:queuedfirst, fix the cause, then queue it again. - Re-queue the build — remove the
build:*verdict, addbuild:queued. A full new build from the base branch. - Manual fix — when the same gate keeps failing, or a human needs to make a semantic decision.
Reach for a new build only if the original build was wrong; otherwise you waste good work. retry:queued and rebuild:queued are not on the list: no flow reads them.
2. What's the difference between needs-input and a :fail on a review stage?
Hint
The question is: did the reviewer actually deliver a verdict?
Answer
:fail: the reviewer ran, produced a readable verdict withhydra-gatesinchecks_run, and said no. That is a judgement on the code. Nothing further is queued; you fix the code (or disagree with the verdict) and re-queue the stage.needs-input: the stage produced no usable verdict — the verdict file was missing or unreadable,checks_rundid not namehydra-gates, or the stage could not run. That says nothing about the code. The stage also keeps its:queuedlabel, so it runs again every tick until you remove it.
The build lane never sets needs-input; its outcomes are build:pass, build:fail, build:no-change and build:blocked.
3. When does it make more sense to do the fix by hand instead of another build?
Hint
If the same pattern keeps repeating on the same mechanical gate, throwing more model time at it is not the answer.
Answer
When the same issue has failed 3 times or more for the same reason — typically a mechanical gate like the SPDX header check. Each new build reproduces the same pattern.
In that case:
- First check whether it's a false-positive gate (see part 3). If so: fix the gate.
- If not: fix it by hand (e.g. commit the SPDX headers yourself on
feature/<issue>/hydra-build) and queue the code review.
Time: minutes. Cheaper than another model run on something that keeps going wrong.
4. The same review stage runs again every five minutes on one issue. What is going on, and what do you do?
Hint
What does the sequencer search for, and what stays on the issue when a stage ends blocked?
Answer
The stage ended in needs-input and kept its :queued label. The sequencer only searches for that label, and it has no terminal-state guard that skips needs-input issues, so it picks the stage up again on every tick.
What you do: remove the stage's :queued label first, so the re-runs stop. Then find the cause (no verdict file, gates not run, stage could not start), fix it, remove needs-input and queue the stage again. Re-queueing without removing the label first changes nothing; a missing guard is an infra problem, worth an issue in hydra/.
Done — what now?
You've worked through troubleshooting. One module left, plus the wider context: