Silent partial failure
The CRM step can succeed while the alert step fails. The workflow reports completion even though the sales team never sees the lead.
This is a synthetic example, not a client result. It is designed to show the structure and level of evidence an HDP Workflow Reliability Audit can produce without exposing customer systems or private data.
Example system
Lead intake → CRM → team alert → follow-up
Scenario: a service business relies on a web lead form that creates a CRM record, writes a tracking row, alerts the sales team, and starts an automated email follow-up. The workflow succeeds most of the time, but staff still check it manually because failures are difficult to detect.
1. Web form
Lead submits
Lead submits
2. Webhook
Payload received
Payload received
3. CRM
Contact + deal
Contact + deal
4. Alert
Team notified
Team notified
5. Follow-up
Email sequence
Email sequence
Duplicate creation
Webhook retries have no durable idempotency key. A timeout followed by retry can create two deals and trigger duplicate follow-up.
Weak recovery path
Transient API failures retry only once. Failed payloads are not retained in a durable recovery queue.
Manual babysitting
Staff compare form notifications against CRM records because there is no reliable exception dashboard or reconciliation signal.
Evidence map
Observed failure modes and likely business impact
| Finding | Evidence to reproduce | Business risk | Priority |
|---|---|---|---|
| Alert can fail after CRM success | Force alert provider 500 response after CRM record creation | Lead exists but receives no timely human response | High |
| Duplicate deal on retry | Delay CRM response until webhook sender retries same payload | Duplicate outreach, reporting distortion, customer confusion | High |
| No durable failed-event queue | Disconnect CRM API beyond retry window | Lead data can require manual reconstruction | Medium |
| No end-to-end reconciliation | Compare daily form count with CRM + follow-up count | Humans remain the monitoring system | Medium |
Prioritized repair plan
What we would harden first
- Add a durable event ID and idempotency check before any side-effecting CRM or messaging action.
- Separate workflow completion from business completion so each required downstream step has an explicit success state.
- Add bounded retries with a recovery queue for transient API failures instead of dropping the event after one retry.
- Add an exception alert and reconciliation check comparing submitted leads against CRM creation and follow-up initiation.
- Document manual fallback and ownership so a failed automation does not become an undefined operational emergency.
Audit output
The goal is evidence, not a generic automation review.
A real HDP Workflow Reliability Audit begins with the actual workflow, reproduces realistic failures where practical, maps business impact, and produces a prioritized repair/hardening plan. Repair work is scoped separately so the diagnostic remains bounded.