Regression testing by risk: what to run when there is not enough time
Regression testing is not “run everything again”. In real releases there is rarely enough time, so the valuable skill is choosing what must be checked first and what risk remains.
- Read what changed
- Map affected flows
- Pick high-impact checks
- Name what was not covered
Start from change, not from habit
A regression suite often grows into a ritual. Risk-based regression starts with the release notes, commits, tickets or a conversation with the developer: what changed and what can it touch?
The same checkout bug is more important after a payment refactor than after a text change in the footer.
What is high risk
High risk means high impact or high probability. Money, login, permissions, personal data, orders, payments and integrations usually go first.
- Flows that block users from doing the main job.
- Flows that move money or irreversible data.
- Areas touched by the current release.
- Areas with recent or repeated defects.
- Integrations where failure is hard to see in UI.
How to say no safely
A middle QA does not silently skip checks. They say what was not covered and why. That turns an invisible gap into a shared release decision.
The phrase “not tested” is not shameful when it is explicit. It is dangerous only when everyone assumes someone else checked it.
A compact regression checklist
For a small release, cover smoke flows, changed functionality, nearby functionality, permissions, key negative cases and one end-to-end business path.
For an API release, add schema compatibility, error handling, idempotency and old clients if they still exist.
What to improve after release
Every escaped bug teaches the regression suite. Ask which signal was missing: no test case, weak test data, unclear owner, missing log or no automated guard.
Regression gets better when it is edited, not when it only grows.
Likelihood × impact matrix
Use a simple 1–3 scale. Likelihood rises for changed, complex, historically faulty or unstable areas. Impact rises for money or data loss, security, blocked core journeys and broad user reach. The score aids comparison but does not replace discussion: 3×3 comes before 1×3.
Add confidence. An unfamiliar integration may move upward even without defect history. Record the reason for every score or the matrix becomes decoration.
Scenario P I Score Decision\nRepeated payment callback 3 3 9 test first\nOld mobile client 2 3 6 contract + smoke\nOrder-history filter 2 1 2 sample\nFooter copy 1 1 1 out of scope
Example: 90 minutes for payment regression
Spend 10 minutes on the diff, ticket and rollout plan; 15 on login, catalogue and checkout smoke; 40 on success, decline, timeout and retry, duplicate callback, idempotency key and reconciliation; 15 on an old client and permissions; 10 on logs, metrics and summary.
Rare currencies and the full promotion set remain uncovered. State why they rank lower, who accepts the risk and what monitoring can detect trouble during rollout.
A release-ready test summary
Include build and environment, scope, critical results, open defects, untested areas, metric observations and a QA recommendation. QA does not magically authorise release; the team or risk owner decides.
Useful wording: “Critical payment flows passed on build 1842. Blocker PAY-91 remains: a repeated callback charges twice. Stop rollout. JPY and Android 11 are untested; the release owner accepts that risk if rollout continues.”
Worked example: a release changes payment rounding
The fee calculation service changed. There are 90 minutes before release while the full suite takes two days. QA must build a focused set from the impact and state the remaining risk, not pretend everything was checked.
- Identify changed rules and downstream consumers.
- Rank by likelihood and impact.
- Exercise critical paths and adjacent behavior.
- Publish results and residual risk.
Map impact before running tests
Read the diff, ticket and incident history. A fee result may be consumed by transfers, refunds, cancellations, statements and notifications. Mark currency, storage unit, rounding order and old-client behavior. Even if the diff only changes a formula, ask which consumers cache a result or retry after timeout.
Reduce scope deliberately
Start with sign-in, account access and a basic transfer. Then test rounding boundaries, smallest currency unit, exact remaining balance, insufficient funds, retry with the same key, refund and statement. Divide cases into release-critical and safe to run during rollout. Prioritize loss of funds or balance errors, not simply short tests.
Use an independent oracle
Calculate the expected fee independently from the API response using an approved formula and preselected values. The treatment of 1.005 depends on the agreed rounding rule; a test without it is meaningless. Compare response, both balances, transaction record and statement. Two green UI screens do not prove the ledger agrees internally.
Write a decision-ready summary
Give build, environment, critical results, open defects and untested areas. Do not conceal an unchecked old Android client or rare currency under “regression passed”. Name the release decision owner, post-release signals and rollback threshold. If a double charge appears, recommend halting rollout regardless of the pass percentage.
Build 1842 / QA / payment-fee rollout\nPassed: transfer, insufficient funds, boundary, retry, refund\nBlocked: none\nNot checked: Android 11, JPY rounding, delayed export\nResidual risk: old client may parse fee field differently\nMonitor: duplicate charges, ledger mismatch, 5xx, fee delta\nRecommendation: staged rollout with agreed rollback threshold
Why is this risk-based regression?
The set follows the change, consumers and potential harm rather than the historical checklist order. It cannot promise complete safety, but makes checked behavior, gaps and production signals visible to the team.
Second case: CRM lead conversion changes
A seemingly small change creates a manager task when a lead converts. The action also touches customer, deal, owner assignment, notifications and retries. Regression should follow connected state; a green Convert button can hide damaged data.
Draw the dependency graph
List lead, customer, deal, activity, task and notification. Mark create/update actions and linking keys. Determine which effects are synchronous and which use a queue. If a task arrives later, wait for observable state up to a bounded timeout rather than sleeping five fixed seconds. Check that the task owner can see the newly created customer.
Choose control scenarios
At least test a first conversion, blocked repeat, existing customer email and invalid required field. Add another manager’s role and failure while creating a task. If the customer was created but the task failed, the expected result depends on the contract: full rollback or retry. Clarify it before reporting a partial state as a defect.
Check identity and object counts
After conversion expect exactly one customer, deal and task in the correct states. Replay the request and compare ids and counts: “already converted” alone does not prove no duplicate. Check task.customer_id, deal.customer_id and owner_id. UI email uniqueness does not protect two concurrent API requests, so test the race separately.
Check due dates and reminders
An overdue task should not become ordinary because of a time-zone conversion. Use a fixed date in the test environment; compare due_at in API, manager view and overdue sorting. Clarify DST and local-day rules. Verify that message redelivery does not create duplicate notifications.
Report release gaps
If there is no time for multi-tenant and old-lead migration, name those gaps. Make repeat conversion and access to the new customer release-critical because errors alter data and permissions. After rollout monitor failed conversions, duplicate customers and task-queue depth. This is more useful than “CRM regression passed”.
Case Client Deal Task Lead state\nfirst convert +1 +1 +1 converted\nrepeat same lead +0 +0 +0 converted\ninvalid lead +0 +0 +0 unchanged\ntask worker fails contract-defined retry or rollback\nsecond manager no access unless assigned
Why is this regression?
It checks connected-object invariants, roles, asynchronous events and repeated calls rather than only one button. The matrix makes risk understandable to engineering and the release owner and can evolve with the sales workflow.
Common regression questions
How is smoke different from regression?
Smoke quickly decides whether the build is testable and core functions are available. Regression searches for change side effects. Smoke is usually part of regression, not a replacement.
Should the entire automated suite always run?
Often yes when it is fast, stable and cheap, but green automation does not replace impact analysis, new-risk exploration or repair of slow and flaky checks.
Who accepts residual risk?
Not QA alone. QA makes evidence and gaps visible; product, engineering or a release owner makes the decision under the team process.
Good regression testing is visible prioritisation. It protects the product and tells the team exactly what risk is still being shipped.