bledaone

Zoho 6 Problem

Nov 17th, 2026 (edited)
53
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 23.61 KB | None | 0 0
  1. Zoho 6 - Course Fee Payment Reminder (coursemo, n8n flow.h5.sg): analysis by Antigravity, model Gemini 4 Hydrogen (XHigh), 2026-11-17.
  2.  
  3. How it was produced:
  4. - CEO Bot exported read-only evidence from live n8n (login + GET only): the live workflow, all 14 retained executions (status, errors, per-node counts; no student data), every workflow's schedule trigger, and instance settings.
  5. - Antigravity analysed only those files. It had no n8n access. Nothing on the live workflow was changed, run or retried.
  6.  
  7. Key points:
  8. - The 06:00 move worked for the crash part: today's 06:00 run (732844) ran 127 s, past the ~73 s point where the 05:00 runs were killed.
  9. - It still fails because of a separate bug: when the Zoho Deal search finds no Deal, n8n gets an empty success, 'Update Deals' calls a URL with no Deal ID, gets HTTP 404 and stops the whole run. Today: 94 selected, stopped at item 18, 77 not queued.
  10. - Fix first: an IF node that checks the Deal ID before 'Update Deals', and On Error = continue on 'Update Deals'. Details below.
  11.  
  12. Full report (Markdown) follows and is attached.
  13.  
  14. ======================================================================
  15.  
  16. # Diagnostic Report: "Zoho 6 - Course Fee Payment Reminder" (`xseQrIUNKZde9lvh`)
  17.  
  18. ## 1. Executive Summary
  19.  
  20. - **Two independent failures broke the workflow:** (1) `crashed` runs at `05:00 SGT` (7 of 14 retained runs) were killed ~72–74 s after trigger—consistent with the daily Docker/n8n update restart interrupting a multi-minute loop; and (2) `error` runs (6 of 14) abort immediately on `Update Deals` HTTP `404` whenever `GET CRM Deal ID` finds no Deal for an `sfc_id`, because Zoho CRM's empty search returns HTTP `204` (`{}`) on the **success** output and bypasses `onError: continueErrorOutput`.
  21. - **Did moving `05:00 -> 06:00` fix the collision?** On the first `06:00 SGT` run (`732844`, `2026-11-17`), it ran past the ~73 s crash mark (`127.348 s`) without crashing and does not collide with any active n8n schedule in [all_schedule_triggers.json](all_schedule_triggers.json), though 1 run is preliminary without host update logs (6 earlier `05:00` runs also did not crash).
  22. - **Why it still fails:** Moving the schedule does not touch the `Update Deals` `404` bug, so `732844` still errored at item 18 of 94 (`77` students unqueued); secondary gaps (no `errorWorkflow`/alerts, 3-day unconditional re-queuing, no payment check, and a broken disconnected email branch) remain unresolved.
  23.  
  24. ---
  25.  
  26. ## 2. Evidence
  27.  
  28. ### Workflow & Instance Configuration
  29. - **Workflow ([zoho6_workflow.json](zoho6_workflow.json)):**
  30. - ID `xseQrIUNKZde9lvh`, `active: true`, `updatedAt: "2026-11-16T14:24:50.561Z"` (`2026-11-16 22:24:50.561 SGT`), active version `b49ec7d6-7d6d-4aa3-b101-53c0c8711882` (activated `2026-11-16T14:24:59.024Z`).
  31. - `settings`: `{"executionOrder": "v1", "binaryMode": "separate"}` — **no `errorWorkflow`**, no `saveExecutionProgress` override.
  32. - **Active path:** `Schedule Trigger` (`cronExpression: "0 6 * * *"`) -> `MySQL: Fetch Unpaid Students` (`MySQL OLA (Read Only)`) -> `Loop Over Items` (`splitInBatches` v3, default batch size `1`, `done` output `0` disconnected) -> `GET CRM Deal ID` (`httpRequest` v4.3, `onError: "continueErrorOutput"`, output `1` -> `Loop Over Items`) -> `Update Deals` (`httpRequest` v4.3, **no `onError`**, **no `retryOnFail`**) -> `Wait 5 Sec` (`wait` v1.1, default 5 s) -> `Loop Over Items`.
  33. - **Disconnected branch:** `Get a deal` -> `Format Links & Date` -> `Send email` has no incoming connections (0 runs across all executions).
  34. - **Instance ([settings.json](settings.json), [notes_n8n_installation.md](notes_n8n_installation.md)):**
  35. - Upgraded from `2.44.6` (`2026-10-25`) to `2.47.7` (`2026-11-17`); `timezone: "Asia/Singapore"`, `executionMode: "regular"`, `isDocker: true`, `maxExecutionTimeout: 3600`, `saveDataErrorExecution: "all"`, `saveDataSuccessExecution: "all"`, `pruning: {isEnabled: true, maxAge: 336, maxCount: 10000}`.
  36. - Execution progress is **not** saved ([notes_n8n_installation.md](notes_n8n_installation.md#L35)).
  37. - **Other Scheduled Workflows ([all_schedule_triggers.json](all_schedule_triggers.json)):**
  38. - Active explicit schedules: `XI0ba65cUsD4jRgH` (`MySQL 1 - Update Deal on Class End`, `02:00 SGT`), `JjbOZKuOWG4kRvM4` (`Send Class Cancellation`, `11:45 SGT`), `o2hkHcribohOTCR1` (`Send Class Reminders`, `12:00 SGT`), and `xseQrIUNKZde9lvh` (`Zoho 6`, `06:00 SGT`).
  39. - Active with empty schedule object: `K3YW6fqzeR6t66KL` (`Zoho 7 - Update Phone +65`, `interval: [{}]` — no hour saved).
  40. - Inactive `I49mDOR73V85mdGW` (`Zoho 6b - Course Fee Payment Reminder`) sits at `triggerAtHour: 5`; inactive `wY4QG4IMhHd0OdiU` references `errorWorkflow: "VKJUEQNFd1yCKu9a"`.
  41.  
  42. ### Retained Executions (14 total: 1 `success`, 6 `error`, 7 `crashed`)
  43. >From [zoho6_executions_list.json](zoho6_executions_list.json) and [zoho6_execution_summaries.json](zoho6_execution_summaries.json) (all `mode: "trigger"`, `retryOf: null`):
  44.  
  45. | Exec ID | Status | Version | Started (UTC -> SGT) | Stopped (SGT) | Duration | JSON Bytes | MySQL Out | Runs (`Loop` / `GET` / `Upd` / `Wait`) | Outcome / Error |
  46. | :--- | :--- | :--- | :--- | :--- | ---: | ---: | ---: | :--- | :--- |
  47. | `732844` | `error` | `b49ec7d6` | `2026-11-16T22:00:00.029Z` (`2026-11-17 06:00:00.029`) | `06:02:07.377` | `127.348 s` | 130,922 | 94 | `18 / 18 / 18 / 17` | `Update Deals` HTTP `404` on item 18 (`77` unqueued) |
  48. | `732104` | `error` | `bc89a048` | `2026-11-15T21:00:00.028Z` (`2026-11-16 05:00:00.028`) | `05:00:30.701` | `30.673 s` | 71,312 | 112 | `4 / 4 / 4 / 3` | `Update Deals` HTTP `404` on item 4 (`109` unqueued) |
  49. | `731671` | `error` | `bc89a048` | `2026-11-14T21:00:00.032Z` (`2026-11-15 05:00:00.032`) | `05:04:01.823` | `241.791 s` | 202,506 | 97 | `33 / 33 / 33 / 32` | `Update Deals` HTTP `404` on item 33 (`65` unqueued) |
  50. | `731235` | `crashed` | `bc89a048` | `2026-11-13T21:00:00.036Z` (`2026-11-14 05:00:00.036`) | `05:01:13.306` | `73.270 s` | 9,292 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
  51. | `730801` | `crashed` | `bc89a048` | `2026-11-12T21:00:00.024Z` (`2026-11-13 05:00:00.024`) | `05:01:12.578` | `72.554 s` | 9,290 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
  52. | `729977` | `crashed` | `bc89a048` | `2026-11-11T21:00:00.031Z` (`2026-11-12 05:00:00.031`) | `05:01:11.581` | `71.550 s` | 9,292 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
  53. | `728921` | `crashed` | `bc89a048` | `2026-11-10T21:00:00.037Z` (`2026-11-11 05:00:00.037`) | `05:01:13.645` | `73.608 s` | 9,293 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
  54. | `728231` | `crashed` | `bc89a048` | `2026-11-09T21:00:00.027Z` (`2026-11-10 05:00:00.027`) | `05:01:13.501` | `73.474 s` | 9,291 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
  55. | `727646` | `success` | `bc89a048` | `2026-11-08T21:00:00.030Z` (`2026-11-09 05:00:00.030`) | `05:07:26.138` | `446.108 s` | 325,378 | 62 | `63 / 62 / 62 / 62` | All 62 updated (`Loop Over Items` done pass = run 63) |
  56. | `727551` | `error` | `bc89a048` | `2026-11-07T21:00:00.038Z` (`2026-11-08 05:00:00.038`) | `05:01:37.741` | `97.703 s` | 102,636 | 69 | `14 / 14 / 14 / 13` | `Update Deals` HTTP `404` on item 14 (`56` unqueued) |
  57. | `727085` | `error` | `bc89a048` | `2026-11-06T21:00:00.032Z` (`2026-11-07 05:00:00.032`) | `05:02:09.705` | `129.673 s` | 111,596 | 43 | `18 / 18 / 18 / 17` | `Update Deals` HTTP `404` on item 18 (`26` unqueued) |
  58. | `726263` | `error` | `bc89a048` | `2026-11-05T21:00:00.028Z` (`2026-11-06 05:00:00.028`) | `05:02:23.950` | `143.922 s` | 120,249 | 43 | `20 / 20 / 20 / 19` | `Update Deals` HTTP `404` on item 20 (`24` unqueued) |
  59. | `725593` | `crashed` | `bc89a048` | `2026-11-04T21:00:00.025Z` (`2026-11-05 05:00:00.025`) | `05:01:13.087` | `73.062 s` | 9,286 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
  60. | `724709` | `crashed` | `bc89a048` | `2026-11-03T21:00:00.032Z` (`2026-11-04 05:00:00.032`) | `05:01:12.287` | `72.255 s` | 9,292 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
  61.  
  62. *\*In all 7 `crashed` executions, `1` is a synthetic `isArtificialRecoveredEventItem` placeholder (`jsonSizeBytes` 9,286–9,293 B; `first_start` values within 0–5 s of trigger) rather than real row/node counts ([notes_zoho6_live_check_2026-11-16.md](notes_zoho6_live_check_2026-11-16.md#L34)).*
  63.  
  64. ---
  65.  
  66. ## 3. Root-Cause Analysis
  67.  
  68. ### (a) The `"crashed"` runs (7 of 14: `724709`, `725593`, `728231`, `728921`, `729977`, `730801`, `731235`)
  69. 1. **Long serial runtime collides with ~05:00–05:01 SGT restart window:** Each student takes ~`7.16–8.14 s` serially (`GET CRM Deal ID` + `Update Deals` + `Wait 5 Sec`), so typical batches of 43–112 students require **~5 to 15 minutes** (e.g., `727646` took `446.108 s` for 62 students).
  70. 2. **Mid-flight process termination and startup recovery:** In `regular` execution mode on Docker with execution progress saving disabled, when the n8n container stops/restarts around `05:00–05:01 SGT` (matching the owner's daily update/restart context and the observed upgrade from `2.44.6` to `2.47.7`), the active run is aborted before finishing. When the restarted container boots (~`71.55–73.61 s` after `05:00:00`, i.e., `05:01:11.581–05:01:13.645 SGT` across all 7 crashes), n8n's startup crash-recovery marks any unfinished execution row in Postgres as `status: "crashed"`, sets `stoppedAt` to the recovery timestamp, and writes dummy `isArtificialRecoveredEventItem` placeholders (`9,286–9,293 B`), wiping visibility into how many students were already queued before termination.
  71. 3. **Why some `05:00` runs did not crash (and status of the `06:00` move):** `732104` errored in `30.673 s` (before `05:01:12`), whereas `726263`, `727085`, `727551`, `727646`, and `731671` ran `97.7–446.1 s` past `05:01:13` without crashing—showing the container did not restart every single day at `05:01` (inferred: conditional/image-pull update or intermittent job). Run `732844` at `06:00:00 SGT` ran `127.348 s` without crashing, indicating `06:00` clears the ~`05:01` window, though 1 run alone does not prove `06:00–06:15` is free of host jobs without host cron/container logs.
  72.  
  73. ### (b) The `"error"` runs (6 of 14: `726263`, `727085`, `727551`, `731671`, `732104`, `732844`)
  74. 1. **MySQL selects rows without verifying CRM Deal presence:** `MySQL: Fetch Unpaid Students` selects all `SFC` rows with `Pass_Fail IN (2, 12)` (`2 = R`, `12 = R-Unpaid`), non-cancelled class, and `Start_Date` between `CURDATE() + 1` and `CURDATE() + 3` (`43–112` rows, no `ORDER BY`).
  75. 2. **Empty Deal search emits `{}` on the SUCCESS branch:** `GET CRM Deal ID` searches `GET https://www.zohoapis.com/crm/v2/Deals/search?criteria=(SFC_ID:equals:{{ $json.sfc_id }})` and sets `"onError": "continueErrorOutput"`, routing output `1` (error) back to `Loop Over Items`. However, when Zoho CRM v2 finds **zero matching Deals** for an `sfc_id`, it responds with HTTP `204 No Content` (empty body), which is a 2xx HTTP success. n8n's HTTP Request node emits `{}` on **output `0` (success)** and `0` items on output `1` (`"items_per_output_last": [1, 0]` in all 6 error runs; confirmed directly for `731671` search 33 in [notes_zoho6_live_check_2026-11-16.md](notes_zoho6_live_check_2026-11-16.md#L33-L35)).
  76. 3. **`Update Deals` interpolates a missing ID and gets HTTP `404`:** Output `0` feeds `Update Deals`, whose URL is `=https://www.zohoapis.com/crm/v2/Deals/{{ $json.data[0].id }}`. With `{}` input, `$json.data[0].id` is missing, so the request hits Zoho without a Deal ID and receives HTTP `404` (`INVALID_URL_PATTERN`, reported by n8n as `NodeApiError: The resource you are requesting could not be found` / `Please check if the URL trying to access is a correct one`).
  77. 4. **Whole-workflow abort:** Because `Update Deals` has **no `onError`** setting (default `stopWorkflow`) and **no `retryOnFail`**, a single unmatched student stops the entire execution and drops all remaining students (`24` dropped in `726263`, `26` in `727085`, `56` in `727551`, `65` in `731671`, `109` in `732104`, `77` in `732844`). Only `727646` (Nov 9, 62 students) succeeded because every student in that day's window happened to match a Deal.
  78.  
  79. ### (c) Secondary Design & Logic Defects
  80. - **Silent failures & undetected crashes:** No `errorWorkflow` in workflow `settings`; `Loop Over Items` output `0` (`done`) is disconnected.
  81. - **Unconditional 3-day re-queuing & no payment check:** `CURDATE()+1..+3` matches each student on 3 consecutive mornings and overwrites `wa_inv_reminder = "queued"` without reading `wa_inv_reminder` or payment status from the Deal already fetched by `GET CRM Deal ID`. Nor does MySQL check `SFC.PaymentID` (noted in [notes_zoho6_live_check_2026-11-16.md](notes_zoho6_live_check_2026-11-16.md#L31) as unchecked, while `MySQL 1` only syncs TMS->Deal 2 days *after* class end).
  82. - **Broken disconnected email branch:** `Get a deal` -> `Format Links & Date` -> `Send email` is disconnected and defective (`toEmail` hardcodes `<[email protected]>`, `Format Links & Date` reads nonexistent `$json.Invoice`, `Send email` has stray `{{ $('Loop Over Items').item.json.VenueName }}` and reads `$json.AgentName` from the wrong node).
  83.  
  84. ---
  85.  
  86. ## 4. Advice (Prioritised Fixes)
  87.  
  88. ### P0 — Fix Immediately (Unblocks Execution & Stops Batch Aborts)
  89.  
  90. 1. **[Safe Quick Win] Guard Deal ID with an `IF` node between `GET CRM Deal ID` (output `0`) and `Update Deals`**
  91. - Insert an `IF` node (`n8n-nodes-base.if` v2+, e.g., `Has Valid Deal ID?`):
  92. - Condition (Boolean `is true`):
  93. ```text
  94. {{ Boolean($json.data && Array.isArray($json.data) && $json.data.length > 0 && $json.data[0]?.id) }}
  95. ```
  96. *(Or String `is not empty` on `{{ $json.data?.[0]?.id ?? '' }}` — optional chaining `?.` prevents a `TypeError` when `$json` is `{}`.)*
  97. - **`true` output** -> `Update Deals`
  98. - **`false` output** -> `Set` / `Code` node recording the unmatched student (`sfc_id: {{ $('Loop Over Items').item.json.sfc_id }}`, `student: {{ $('Loop Over Items').item.json.Student_Name }}`) -> `Loop Over Items` (skips `Update Deals` and `Wait 5 Sec` cleanly).
  99. - Also wire `GET CRM Deal ID` error output (output `1`) through a failure-recording node (capturing `sfc_id` and `$json.error`) before looping back to `Loop Over Items`, so Zoho search errors (e.g. 401/429/5xx) are recorded rather than silently dropped.
  100.  
  101. 2. **[Safe Quick Win] Add `onError` and Safe Interpolation on `Update Deals`**
  102. - Set `Update Deals` **On Error** (`onError`) to `"continueErrorOutput"` (instead of `stopWorkflow`), so a single Deal update failure (404/400/5xx) never aborts the remaining 50–100 students.
  103. - Route `Update Deals` error output (output `1`) -> record failed `sfc_id` & `$json.error.message` -> `Loop Over Items` (or via `Wait 5 Sec` if rate-limiting failed calls too).
  104. - Harden `Update Deals` URL expression to `=https://www.zohoapis.com/crm/v2/Deals/{{ $json.data?.[0]?.id }}`.
  105.  
  106. 3. **[Safe Quick Win] Enable `retryOnFail` on Network / DB Nodes**
  107. - On `MySQL: Fetch Unpaid Students`, `GET CRM Deal ID`, and `Update Deals`, enable **Retry On Fail** (`retryOnFail: true`):
  108. - `maxTries`: `3`
  109. - `waitBetweenTries`: `5000` (5,000 ms)
  110. - *Interaction note:* n8n exhausts `maxTries` before routing to `continueErrorOutput`. For `Update Deals`, this recovers transient Zoho `429`/`5xx`/OAuth glitches while the P0 `IF` guard prevents wasting retries on empty-search `404`s.
  111.  
  112. ### P1 — Observability, Alerting, and Schedule Hardening
  113.  
  114. 4. **[Safe Quick Win] Attach an Active Error Workflow (`settings.errorWorkflow`)**
  115. - In **Workflow Settings**, assign an active `Error Trigger` workflow (`settings.errorWorkflow`).
  116. - Check whether existing `VKJUEQNFd1yCKu9a` (seen on `wY4QG4IMhHd0OdiU` in [all_schedule_triggers.json](all_schedule_triggers.json#L69)) is an active shared alert workflow; if not, create a minimal `Error Trigger` -> `Send Email` (using existing SMTP credential `iOnCWj0QGgXdd68L`, `AWS SES @ SGP1 ([email protected])`) or Telegram/Slack alert.
  117.  
  118. 5. **Add a Run-Summary & Dead-Man Alert on `Loop Over Items` (`done` output `0`)**
  119. - Note that n8n's `errorWorkflow` triggers on `error` executions, **not** on `crashed` runs killed mid-flight (nor on runs that finish `success` with skipped items via `continueErrorOutput`).
  120. - Connect `Loop Over Items` output `0` (`done`) to an aggregation / `Send Email` (or messaging) node that reports:
  121. - `Total fetched`: `{{ $('MySQL: Fetch Unpaid Students').all().length }}`
  122. - `Updated / Queued`, `Skipped (no CRM Deal ID)`, `Skipped (already queued/paid)`, and `Errors` (list the exact `sfc_id`s so missing CRM `SFC_ID` links can be fixed in Zoho/TMS).
  123. - **Dead-man monitoring:** Send this summary daily (or at least alert whenever skipped/error > 0, and send a lightweight heartbeat/summary email or trigger a Healthchecks.io/Uptime monitor ping on `done`). If a container restart ever crashes the run before `done`, the missing heartbeat raises an immediate alert.
  124.  
  125. 6. **Verify Update Window & Clean Up Triggers**
  126. - **Host schedule check:** Confirm the exact cron/Watchtower/Plesk restart window on the host before trusting `06:00 SGT`. Give Zoho 6 at least a **20–30 minute buffer** away from any container update/backup restart (112 students take ~15 min at ~7.5 s/item; `06:00` or `06:30` or `07:00 SGT` works if the update window is strictly `05:00–05:15 SGT`).
  127. - **[Safe Quick Win] Archive/clean up `Zoho 6b`** (`I49mDOR73V85mdGW`, still configured for hour `5`, currently inactive) so it isn't accidentally re-enabled into the `05:00` collision window.
  128. - Check `Zoho 7 - Update Phone +65` (`K3YW6fqzeR6t66KL`, active with empty `interval: [{}]`) to ensure its default schedule is intentional and not burning Zoho API quota concurrently.
  129. - *Optional crash-forensics aid:* Enabling **Save execution progress** (`saveExecutionProgress: true` in workflow settings) lets you see which loop iteration was last reached if a crash ever recurs (trade-off: extra Postgres writes per loop iteration, so keep off or disable once host schedule collision is confirmed resolved).
  130.  
  131. ### P2 — Idempotency, Re-Queue Behavior, and Data Quality
  132.  
  133. 7. **Prevent Unnecessary Re-Queues & Skip Wait on Skipped Items**
  134. - `GET CRM Deal ID` (`/crm/v2/Deals/search`) already fetches the Deal's fields in `$json.data[0]`. Before `Update Deals`, check `$json.data[0].wa_inv_reminder` (extend the P0 `IF` node or add a second `IF` node):
  135. - If business logic intends **only 1 queue trigger** per unpaid booking (or if a crashed/retried run restarts from row 1 on the same day): skip `Update Deals` **and** `Wait 5 Sec` when:
  136. ```text
  137. {{ $json.data[0]?.wa_inv_reminder === 'queued' }}
  138. ```
  139. *(Adjust condition if Zoho CRM resets `wa_inv_reminder` to `'sent'` / `'done'` after sending and you only want 1 reminder instead of 3 consecutive daily reminders on D-3, D-2, D-1.)*
  140. - Also check whether the Deal already shows paid in CRM (e.g., `Stage` or payment fields on `$json.data[0]`) and skip if paid.
  141. - **[Safe Quick Win] Deterministic SQL ordering:** Add `ORDER BY c.Start_Date ASC, s.id ASC` to `MySQL: Fetch Unpaid Students` so failures/skips are reproducible and nearest-starting classes (`Start_Date = CURDATE() + 1`) are reminded first if a run is ever interrupted.
  142. - **Verify `PaymentID` in MySQL (requires schema/ops confirmation first):** Since Zoho 6 selects only `Pass_Fail IN (2, 12)` and `MySQL 1` runs only 2 days post-class, verify with TMS owners whether `AND (s.PaymentID IS NULL OR s.PaymentID = '')` should be added to `MySQL: Fetch Unpaid Students` to stop reminding students who already paid in TMS before class.
  143. - **Do NOT connect the dead email branch (`Get a deal` -> `Format Links & Date` -> `Send email`) as saved:** Move it out or delete it until `toEmail` (`[email protected]`), `paynow_link` (`$json.Invoice`), `AgentName` scope, `formatted_date`, and HTML template bugs are explicitly fixed and approved.
  144.  
  145. ---
  146.  
  147. ## 5. Verification Plan (For the Owner)
  148.  
  149. 1. **Test the missing-Deal and already-queued branches safely (pre-activation):**
  150. - Duplicate the workflow (or test manually in Editor with `Update Deals` temporarily disabled / mocked, or test with 2 known `sfc_id` values: 1 valid Deal and 1 missing `sfc_id`).
  151. - Run `GET CRM Deal ID` -> `Has Valid Deal ID?` and verify the missing `sfc_id` outputs `{}` on output `0`, routes to the `false` branch (skip recorded), bypasses `Update Deals` and `Wait 5 Sec`, and loops cleanly to the next item.
  152. 2. **Test `Update Deals` `onError` + `retryOnFail`:**
  153. - Temporarily test a forced invalid Deal ID past the `IF` guard (or mock a 404/500 in a test copy) to confirm `Update Deals` retries `3` times, routes to output `1` (`continueErrorOutput`), logs the error, and continues the loop instead of stopping the workflow.
  154. 3. **Verify `errorWorkflow` & `done` summary alert:**
  155. - Trigger a deliberate error in a test workflow linked to the configured `errorWorkflow` and confirm the error notification arrives.
  156. - Run a controlled small batch (`LIMIT 3` in a test copy or manual run once `Update Deals` logic is verified) and confirm `Loop Over Items` output `0` (`done`) fires once at the end with accurate counts (`fetched`, `updated`, `skipped_no_deal`, `skipped_already_queued`, `errors`).
  157. 4. **Verify host schedule & monitor 7 consecutive live runs:**
  158. - Confirm from host cron / Docker config / container logs that no update/restart/backup runs within `06:00–06:30 SGT` (or adjust `Schedule Trigger` cron to a clean window).
  159. - Activate the updated workflow and verify the next **7 consecutive daily runs** in n8n Executions show:
  160. - `status: "success"`, `finished: true`
  161. - `MySQL: Fetch Unpaid Students` count matches `updated + skipped + errored`
  162. - `0` `crashed` runs and `0` unhandled `404` aborts
  163. - Expected CRM `wa_inv_reminder` transitions and summary email/alert received each morning.
  164.  
  165. ---
  166.  
  167. ## 6. Open Questions / Missing Evidence
  168.  
  169. 1. **Host update/restart schedule & container logs:** We do not have host crontab, `docker-compose.yml`, Watchtower config, Plesk scheduled tasks, or container/n8n logs for `05:00–05:02 SGT` (why crashes happened on Nov 4–5 and Nov 10–14, but not Nov 6–9 or Nov 15–16) or `06:00–06:30 SGT` (to confirm `06:00` has zero host collision over a full week).
  170. 2. **CRM `wa_inv_reminder` automation rules:** How does Zoho CRM consume `"wa_inv_reminder": "queued"`? Does it reset the field (e.g., to `"sent"`), ignore repeated `"queued"` writes, or send duplicate WhatsApp messages across all 3 days (`Start_Date` +1..+3)?
  171. 3. **Unmatched `sfc_id` root cause in CRM/TMS:** Which specific `sfc_id`s failed in `732844` (item 18) and `732104` (item 4), and why did they lack a matching `Deals` record in Zoho CRM (`Zoho 4` writeback failure, non-Zoho registration, or deleted/merged Deal)?
  172. 4. **TMS `PaymentID` vs `Pass_Fail` & CRM payment fields:** Can `MySQL: Fetch Unpaid Students` safely filter `s.PaymentID IS NULL` (or check a CRM payment/Stage field on `GET CRM Deal ID`) without missing partial/SFC-pending payments?
  173. 5. **Side effects of crashed runs:** Because `crashed` runs (`724709`, `725593`, `728231`, `728921`, `729977`, `730801`, `731235`) retain only `isArtificialRecoveredEventItem` placeholders, Zoho CRM audit logs or container logs are needed to see how many reminders were actually queued during those 7 mornings.
  174. 6. **Existing shared workflows:** Is `VKJUEQNFd1yCKu9a` (referenced in [all_schedule_triggers.json](all_schedule_triggers.json#L69)) an active, maintained Error Workflow, and what schedule is intended for active `K3YW6fqzeR6t66KL` (`Zoho 7 - Update Phone +65`) which currently has an empty `interval: [{}]`?
Advertisement