Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- Zoho 6 - Course Fee Payment Reminder (coursemo, n8n flow.h5.sg): analysis by Antigravity, model Gemini 4 Hydrogen (XHigh), 2026-11-17.
- How it was produced:
- - CEO Bot exported read-only evidence from live n8n (login + GET only): the live workflow, all 14 retained executions (status, errors, per-node counts; no student data), every workflow's schedule trigger, and instance settings.
- - Antigravity analysed only those files. It had no n8n access. Nothing on the live workflow was changed, run or retried.
- Key points:
- - The 06:00 move worked for the crash part: today's 06:00 run (732844) ran 127 s, past the ~73 s point where the 05:00 runs were killed.
- - It still fails because of a separate bug: when the Zoho Deal search finds no Deal, n8n gets an empty success, 'Update Deals' calls a URL with no Deal ID, gets HTTP 404 and stops the whole run. Today: 94 selected, stopped at item 18, 77 not queued.
- - Fix first: an IF node that checks the Deal ID before 'Update Deals', and On Error = continue on 'Update Deals'. Details below.
- Full report (Markdown) follows and is attached.
- ======================================================================
- # Diagnostic Report: "Zoho 6 - Course Fee Payment Reminder" (`xseQrIUNKZde9lvh`)
- ## 1. Executive Summary
- - **Two independent failures broke the workflow:** (1) `crashed` runs at `05:00 SGT` (7 of 14 retained runs) were killed ~72–74 s after trigger—consistent with the daily Docker/n8n update restart interrupting a multi-minute loop; and (2) `error` runs (6 of 14) abort immediately on `Update Deals` HTTP `404` whenever `GET CRM Deal ID` finds no Deal for an `sfc_id`, because Zoho CRM's empty search returns HTTP `204` (`{}`) on the **success** output and bypasses `onError: continueErrorOutput`.
- - **Did moving `05:00 -> 06:00` fix the collision?** On the first `06:00 SGT` run (`732844`, `2026-11-17`), it ran past the ~73 s crash mark (`127.348 s`) without crashing and does not collide with any active n8n schedule in [all_schedule_triggers.json](all_schedule_triggers.json), though 1 run is preliminary without host update logs (6 earlier `05:00` runs also did not crash).
- - **Why it still fails:** Moving the schedule does not touch the `Update Deals` `404` bug, so `732844` still errored at item 18 of 94 (`77` students unqueued); secondary gaps (no `errorWorkflow`/alerts, 3-day unconditional re-queuing, no payment check, and a broken disconnected email branch) remain unresolved.
- ---
- ## 2. Evidence
- ### Workflow & Instance Configuration
- - **Workflow ([zoho6_workflow.json](zoho6_workflow.json)):**
- - ID `xseQrIUNKZde9lvh`, `active: true`, `updatedAt: "2026-11-16T14:24:50.561Z"` (`2026-11-16 22:24:50.561 SGT`), active version `b49ec7d6-7d6d-4aa3-b101-53c0c8711882` (activated `2026-11-16T14:24:59.024Z`).
- - `settings`: `{"executionOrder": "v1", "binaryMode": "separate"}` — **no `errorWorkflow`**, no `saveExecutionProgress` override.
- - **Active path:** `Schedule Trigger` (`cronExpression: "0 6 * * *"`) -> `MySQL: Fetch Unpaid Students` (`MySQL OLA (Read Only)`) -> `Loop Over Items` (`splitInBatches` v3, default batch size `1`, `done` output `0` disconnected) -> `GET CRM Deal ID` (`httpRequest` v4.3, `onError: "continueErrorOutput"`, output `1` -> `Loop Over Items`) -> `Update Deals` (`httpRequest` v4.3, **no `onError`**, **no `retryOnFail`**) -> `Wait 5 Sec` (`wait` v1.1, default 5 s) -> `Loop Over Items`.
- - **Disconnected branch:** `Get a deal` -> `Format Links & Date` -> `Send email` has no incoming connections (0 runs across all executions).
- - **Instance ([settings.json](settings.json), [notes_n8n_installation.md](notes_n8n_installation.md)):**
- - Upgraded from `2.44.6` (`2026-10-25`) to `2.47.7` (`2026-11-17`); `timezone: "Asia/Singapore"`, `executionMode: "regular"`, `isDocker: true`, `maxExecutionTimeout: 3600`, `saveDataErrorExecution: "all"`, `saveDataSuccessExecution: "all"`, `pruning: {isEnabled: true, maxAge: 336, maxCount: 10000}`.
- - Execution progress is **not** saved ([notes_n8n_installation.md](notes_n8n_installation.md#L35)).
- - **Other Scheduled Workflows ([all_schedule_triggers.json](all_schedule_triggers.json)):**
- - Active explicit schedules: `XI0ba65cUsD4jRgH` (`MySQL 1 - Update Deal on Class End`, `02:00 SGT`), `JjbOZKuOWG4kRvM4` (`Send Class Cancellation`, `11:45 SGT`), `o2hkHcribohOTCR1` (`Send Class Reminders`, `12:00 SGT`), and `xseQrIUNKZde9lvh` (`Zoho 6`, `06:00 SGT`).
- - Active with empty schedule object: `K3YW6fqzeR6t66KL` (`Zoho 7 - Update Phone +65`, `interval: [{}]` — no hour saved).
- - Inactive `I49mDOR73V85mdGW` (`Zoho 6b - Course Fee Payment Reminder`) sits at `triggerAtHour: 5`; inactive `wY4QG4IMhHd0OdiU` references `errorWorkflow: "VKJUEQNFd1yCKu9a"`.
- ### Retained Executions (14 total: 1 `success`, 6 `error`, 7 `crashed`)
- >From [zoho6_executions_list.json](zoho6_executions_list.json) and [zoho6_execution_summaries.json](zoho6_execution_summaries.json) (all `mode: "trigger"`, `retryOf: null`):
- | Exec ID | Status | Version | Started (UTC -> SGT) | Stopped (SGT) | Duration | JSON Bytes | MySQL Out | Runs (`Loop` / `GET` / `Upd` / `Wait`) | Outcome / Error |
- | :--- | :--- | :--- | :--- | :--- | ---: | ---: | ---: | :--- | :--- |
- | `732844` | `error` | `b49ec7d6` | `2026-11-16T22:00:00.029Z` (`2026-11-17 06:00:00.029`) | `06:02:07.377` | `127.348 s` | 130,922 | 94 | `18 / 18 / 18 / 17` | `Update Deals` HTTP `404` on item 18 (`77` unqueued) |
- | `732104` | `error` | `bc89a048` | `2026-11-15T21:00:00.028Z` (`2026-11-16 05:00:00.028`) | `05:00:30.701` | `30.673 s` | 71,312 | 112 | `4 / 4 / 4 / 3` | `Update Deals` HTTP `404` on item 4 (`109` unqueued) |
- | `731671` | `error` | `bc89a048` | `2026-11-14T21:00:00.032Z` (`2026-11-15 05:00:00.032`) | `05:04:01.823` | `241.791 s` | 202,506 | 97 | `33 / 33 / 33 / 32` | `Update Deals` HTTP `404` on item 33 (`65` unqueued) |
- | `731235` | `crashed` | `bc89a048` | `2026-11-13T21:00:00.036Z` (`2026-11-14 05:00:00.036`) | `05:01:13.306` | `73.270 s` | 9,292 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
- | `730801` | `crashed` | `bc89a048` | `2026-11-12T21:00:00.024Z` (`2026-11-13 05:00:00.024`) | `05:01:12.578` | `72.554 s` | 9,290 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
- | `729977` | `crashed` | `bc89a048` | `2026-11-11T21:00:00.031Z` (`2026-11-12 05:00:00.031`) | `05:01:11.581` | `71.550 s` | 9,292 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
- | `728921` | `crashed` | `bc89a048` | `2026-11-10T21:00:00.037Z` (`2026-11-11 05:00:00.037`) | `05:01:13.645` | `73.608 s` | 9,293 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
- | `728231` | `crashed` | `bc89a048` | `2026-11-09T21:00:00.027Z` (`2026-11-10 05:00:00.027`) | `05:01:13.501` | `73.474 s` | 9,291 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
- | `727646` | `success` | `bc89a048` | `2026-11-08T21:00:00.030Z` (`2026-11-09 05:00:00.030`) | `05:07:26.138` | `446.108 s` | 325,378 | 62 | `63 / 62 / 62 / 62` | All 62 updated (`Loop Over Items` done pass = run 63) |
- | `727551` | `error` | `bc89a048` | `2026-11-07T21:00:00.038Z` (`2026-11-08 05:00:00.038`) | `05:01:37.741` | `97.703 s` | 102,636 | 69 | `14 / 14 / 14 / 13` | `Update Deals` HTTP `404` on item 14 (`56` unqueued) |
- | `727085` | `error` | `bc89a048` | `2026-11-06T21:00:00.032Z` (`2026-11-07 05:00:00.032`) | `05:02:09.705` | `129.673 s` | 111,596 | 43 | `18 / 18 / 18 / 17` | `Update Deals` HTTP `404` on item 18 (`26` unqueued) |
- | `726263` | `error` | `bc89a048` | `2026-11-05T21:00:00.028Z` (`2026-11-06 05:00:00.028`) | `05:02:23.950` | `143.922 s` | 120,249 | 43 | `20 / 20 / 20 / 19` | `Update Deals` HTTP `404` on item 20 (`24` unqueued) |
- | `725593` | `crashed` | `bc89a048` | `2026-11-04T21:00:00.025Z` (`2026-11-05 05:00:00.025`) | `05:01:13.087` | `73.062 s` | 9,286 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
- | `724709` | `crashed` | `bc89a048` | `2026-11-03T21:00:00.032Z` (`2026-11-04 05:00:00.032`) | `05:01:12.287` | `72.255 s` | 9,292 | 1* | `1 / 1 / 1 / 1`* | Placeholder recovery (`lastNodeExecuted: Update Deals`) |
- *\*In all 7 `crashed` executions, `1` is a synthetic `isArtificialRecoveredEventItem` placeholder (`jsonSizeBytes` 9,286–9,293 B; `first_start` values within 0–5 s of trigger) rather than real row/node counts ([notes_zoho6_live_check_2026-11-16.md](notes_zoho6_live_check_2026-11-16.md#L34)).*
- ---
- ## 3. Root-Cause Analysis
- ### (a) The `"crashed"` runs (7 of 14: `724709`, `725593`, `728231`, `728921`, `729977`, `730801`, `731235`)
- 1. **Long serial runtime collides with ~05:00–05:01 SGT restart window:** Each student takes ~`7.16–8.14 s` serially (`GET CRM Deal ID` + `Update Deals` + `Wait 5 Sec`), so typical batches of 43–112 students require **~5 to 15 minutes** (e.g., `727646` took `446.108 s` for 62 students).
- 2. **Mid-flight process termination and startup recovery:** In `regular` execution mode on Docker with execution progress saving disabled, when the n8n container stops/restarts around `05:00–05:01 SGT` (matching the owner's daily update/restart context and the observed upgrade from `2.44.6` to `2.47.7`), the active run is aborted before finishing. When the restarted container boots (~`71.55–73.61 s` after `05:00:00`, i.e., `05:01:11.581–05:01:13.645 SGT` across all 7 crashes), n8n's startup crash-recovery marks any unfinished execution row in Postgres as `status: "crashed"`, sets `stoppedAt` to the recovery timestamp, and writes dummy `isArtificialRecoveredEventItem` placeholders (`9,286–9,293 B`), wiping visibility into how many students were already queued before termination.
- 3. **Why some `05:00` runs did not crash (and status of the `06:00` move):** `732104` errored in `30.673 s` (before `05:01:12`), whereas `726263`, `727085`, `727551`, `727646`, and `731671` ran `97.7–446.1 s` past `05:01:13` without crashing—showing the container did not restart every single day at `05:01` (inferred: conditional/image-pull update or intermittent job). Run `732844` at `06:00:00 SGT` ran `127.348 s` without crashing, indicating `06:00` clears the ~`05:01` window, though 1 run alone does not prove `06:00–06:15` is free of host jobs without host cron/container logs.
- ### (b) The `"error"` runs (6 of 14: `726263`, `727085`, `727551`, `731671`, `732104`, `732844`)
- 1. **MySQL selects rows without verifying CRM Deal presence:** `MySQL: Fetch Unpaid Students` selects all `SFC` rows with `Pass_Fail IN (2, 12)` (`2 = R`, `12 = R-Unpaid`), non-cancelled class, and `Start_Date` between `CURDATE() + 1` and `CURDATE() + 3` (`43–112` rows, no `ORDER BY`).
- 2. **Empty Deal search emits `{}` on the SUCCESS branch:** `GET CRM Deal ID` searches `GET https://www.zohoapis.com/crm/v2/Deals/search?criteria=(SFC_ID:equals:{{ $json.sfc_id }})` and sets `"onError": "continueErrorOutput"`, routing output `1` (error) back to `Loop Over Items`. However, when Zoho CRM v2 finds **zero matching Deals** for an `sfc_id`, it responds with HTTP `204 No Content` (empty body), which is a 2xx HTTP success. n8n's HTTP Request node emits `{}` on **output `0` (success)** and `0` items on output `1` (`"items_per_output_last": [1, 0]` in all 6 error runs; confirmed directly for `731671` search 33 in [notes_zoho6_live_check_2026-11-16.md](notes_zoho6_live_check_2026-11-16.md#L33-L35)).
- 3. **`Update Deals` interpolates a missing ID and gets HTTP `404`:** Output `0` feeds `Update Deals`, whose URL is `=https://www.zohoapis.com/crm/v2/Deals/{{ $json.data[0].id }}`. With `{}` input, `$json.data[0].id` is missing, so the request hits Zoho without a Deal ID and receives HTTP `404` (`INVALID_URL_PATTERN`, reported by n8n as `NodeApiError: The resource you are requesting could not be found` / `Please check if the URL trying to access is a correct one`).
- 4. **Whole-workflow abort:** Because `Update Deals` has **no `onError`** setting (default `stopWorkflow`) and **no `retryOnFail`**, a single unmatched student stops the entire execution and drops all remaining students (`24` dropped in `726263`, `26` in `727085`, `56` in `727551`, `65` in `731671`, `109` in `732104`, `77` in `732844`). Only `727646` (Nov 9, 62 students) succeeded because every student in that day's window happened to match a Deal.
- ### (c) Secondary Design & Logic Defects
- - **Silent failures & undetected crashes:** No `errorWorkflow` in workflow `settings`; `Loop Over Items` output `0` (`done`) is disconnected.
- - **Unconditional 3-day re-queuing & no payment check:** `CURDATE()+1..+3` matches each student on 3 consecutive mornings and overwrites `wa_inv_reminder = "queued"` without reading `wa_inv_reminder` or payment status from the Deal already fetched by `GET CRM Deal ID`. Nor does MySQL check `SFC.PaymentID` (noted in [notes_zoho6_live_check_2026-11-16.md](notes_zoho6_live_check_2026-11-16.md#L31) as unchecked, while `MySQL 1` only syncs TMS->Deal 2 days *after* class end).
- - **Broken disconnected email branch:** `Get a deal` -> `Format Links & Date` -> `Send email` is disconnected and defective (`toEmail` hardcodes `<[email protected]>`, `Format Links & Date` reads nonexistent `$json.Invoice`, `Send email` has stray `{{ $('Loop Over Items').item.json.VenueName }}` and reads `$json.AgentName` from the wrong node).
- ---
- ## 4. Advice (Prioritised Fixes)
- ### P0 — Fix Immediately (Unblocks Execution & Stops Batch Aborts)
- 1. **[Safe Quick Win] Guard Deal ID with an `IF` node between `GET CRM Deal ID` (output `0`) and `Update Deals`**
- - Insert an `IF` node (`n8n-nodes-base.if` v2+, e.g., `Has Valid Deal ID?`):
- - Condition (Boolean `is true`):
- ```text
- {{ Boolean($json.data && Array.isArray($json.data) && $json.data.length > 0 && $json.data[0]?.id) }}
- ```
- *(Or String `is not empty` on `{{ $json.data?.[0]?.id ?? '' }}` — optional chaining `?.` prevents a `TypeError` when `$json` is `{}`.)*
- - **`true` output** -> `Update Deals`
- - **`false` output** -> `Set` / `Code` node recording the unmatched student (`sfc_id: {{ $('Loop Over Items').item.json.sfc_id }}`, `student: {{ $('Loop Over Items').item.json.Student_Name }}`) -> `Loop Over Items` (skips `Update Deals` and `Wait 5 Sec` cleanly).
- - Also wire `GET CRM Deal ID` error output (output `1`) through a failure-recording node (capturing `sfc_id` and `$json.error`) before looping back to `Loop Over Items`, so Zoho search errors (e.g. 401/429/5xx) are recorded rather than silently dropped.
- 2. **[Safe Quick Win] Add `onError` and Safe Interpolation on `Update Deals`**
- - Set `Update Deals` **On Error** (`onError`) to `"continueErrorOutput"` (instead of `stopWorkflow`), so a single Deal update failure (404/400/5xx) never aborts the remaining 50–100 students.
- - Route `Update Deals` error output (output `1`) -> record failed `sfc_id` & `$json.error.message` -> `Loop Over Items` (or via `Wait 5 Sec` if rate-limiting failed calls too).
- - Harden `Update Deals` URL expression to `=https://www.zohoapis.com/crm/v2/Deals/{{ $json.data?.[0]?.id }}`.
- 3. **[Safe Quick Win] Enable `retryOnFail` on Network / DB Nodes**
- - On `MySQL: Fetch Unpaid Students`, `GET CRM Deal ID`, and `Update Deals`, enable **Retry On Fail** (`retryOnFail: true`):
- - `maxTries`: `3`
- - `waitBetweenTries`: `5000` (5,000 ms)
- - *Interaction note:* n8n exhausts `maxTries` before routing to `continueErrorOutput`. For `Update Deals`, this recovers transient Zoho `429`/`5xx`/OAuth glitches while the P0 `IF` guard prevents wasting retries on empty-search `404`s.
- ### P1 — Observability, Alerting, and Schedule Hardening
- 4. **[Safe Quick Win] Attach an Active Error Workflow (`settings.errorWorkflow`)**
- - In **Workflow Settings**, assign an active `Error Trigger` workflow (`settings.errorWorkflow`).
- - Check whether existing `VKJUEQNFd1yCKu9a` (seen on `wY4QG4IMhHd0OdiU` in [all_schedule_triggers.json](all_schedule_triggers.json#L69)) is an active shared alert workflow; if not, create a minimal `Error Trigger` -> `Send Email` (using existing SMTP credential `iOnCWj0QGgXdd68L`, `AWS SES @ SGP1 ([email protected])`) or Telegram/Slack alert.
- 5. **Add a Run-Summary & Dead-Man Alert on `Loop Over Items` (`done` output `0`)**
- - Note that n8n's `errorWorkflow` triggers on `error` executions, **not** on `crashed` runs killed mid-flight (nor on runs that finish `success` with skipped items via `continueErrorOutput`).
- - Connect `Loop Over Items` output `0` (`done`) to an aggregation / `Send Email` (or messaging) node that reports:
- - `Total fetched`: `{{ $('MySQL: Fetch Unpaid Students').all().length }}`
- - `Updated / Queued`, `Skipped (no CRM Deal ID)`, `Skipped (already queued/paid)`, and `Errors` (list the exact `sfc_id`s so missing CRM `SFC_ID` links can be fixed in Zoho/TMS).
- - **Dead-man monitoring:** Send this summary daily (or at least alert whenever skipped/error > 0, and send a lightweight heartbeat/summary email or trigger a Healthchecks.io/Uptime monitor ping on `done`). If a container restart ever crashes the run before `done`, the missing heartbeat raises an immediate alert.
- 6. **Verify Update Window & Clean Up Triggers**
- - **Host schedule check:** Confirm the exact cron/Watchtower/Plesk restart window on the host before trusting `06:00 SGT`. Give Zoho 6 at least a **20–30 minute buffer** away from any container update/backup restart (112 students take ~15 min at ~7.5 s/item; `06:00` or `06:30` or `07:00 SGT` works if the update window is strictly `05:00–05:15 SGT`).
- - **[Safe Quick Win] Archive/clean up `Zoho 6b`** (`I49mDOR73V85mdGW`, still configured for hour `5`, currently inactive) so it isn't accidentally re-enabled into the `05:00` collision window.
- - Check `Zoho 7 - Update Phone +65` (`K3YW6fqzeR6t66KL`, active with empty `interval: [{}]`) to ensure its default schedule is intentional and not burning Zoho API quota concurrently.
- - *Optional crash-forensics aid:* Enabling **Save execution progress** (`saveExecutionProgress: true` in workflow settings) lets you see which loop iteration was last reached if a crash ever recurs (trade-off: extra Postgres writes per loop iteration, so keep off or disable once host schedule collision is confirmed resolved).
- ### P2 — Idempotency, Re-Queue Behavior, and Data Quality
- 7. **Prevent Unnecessary Re-Queues & Skip Wait on Skipped Items**
- - `GET CRM Deal ID` (`/crm/v2/Deals/search`) already fetches the Deal's fields in `$json.data[0]`. Before `Update Deals`, check `$json.data[0].wa_inv_reminder` (extend the P0 `IF` node or add a second `IF` node):
- - If business logic intends **only 1 queue trigger** per unpaid booking (or if a crashed/retried run restarts from row 1 on the same day): skip `Update Deals` **and** `Wait 5 Sec` when:
- ```text
- {{ $json.data[0]?.wa_inv_reminder === 'queued' }}
- ```
- *(Adjust condition if Zoho CRM resets `wa_inv_reminder` to `'sent'` / `'done'` after sending and you only want 1 reminder instead of 3 consecutive daily reminders on D-3, D-2, D-1.)*
- - Also check whether the Deal already shows paid in CRM (e.g., `Stage` or payment fields on `$json.data[0]`) and skip if paid.
- - **[Safe Quick Win] Deterministic SQL ordering:** Add `ORDER BY c.Start_Date ASC, s.id ASC` to `MySQL: Fetch Unpaid Students` so failures/skips are reproducible and nearest-starting classes (`Start_Date = CURDATE() + 1`) are reminded first if a run is ever interrupted.
- - **Verify `PaymentID` in MySQL (requires schema/ops confirmation first):** Since Zoho 6 selects only `Pass_Fail IN (2, 12)` and `MySQL 1` runs only 2 days post-class, verify with TMS owners whether `AND (s.PaymentID IS NULL OR s.PaymentID = '')` should be added to `MySQL: Fetch Unpaid Students` to stop reminding students who already paid in TMS before class.
- - **Do NOT connect the dead email branch (`Get a deal` -> `Format Links & Date` -> `Send email`) as saved:** Move it out or delete it until `toEmail` (`[email protected]`), `paynow_link` (`$json.Invoice`), `AgentName` scope, `formatted_date`, and HTML template bugs are explicitly fixed and approved.
- ---
- ## 5. Verification Plan (For the Owner)
- 1. **Test the missing-Deal and already-queued branches safely (pre-activation):**
- - Duplicate the workflow (or test manually in Editor with `Update Deals` temporarily disabled / mocked, or test with 2 known `sfc_id` values: 1 valid Deal and 1 missing `sfc_id`).
- - Run `GET CRM Deal ID` -> `Has Valid Deal ID?` and verify the missing `sfc_id` outputs `{}` on output `0`, routes to the `false` branch (skip recorded), bypasses `Update Deals` and `Wait 5 Sec`, and loops cleanly to the next item.
- 2. **Test `Update Deals` `onError` + `retryOnFail`:**
- - Temporarily test a forced invalid Deal ID past the `IF` guard (or mock a 404/500 in a test copy) to confirm `Update Deals` retries `3` times, routes to output `1` (`continueErrorOutput`), logs the error, and continues the loop instead of stopping the workflow.
- 3. **Verify `errorWorkflow` & `done` summary alert:**
- - Trigger a deliberate error in a test workflow linked to the configured `errorWorkflow` and confirm the error notification arrives.
- - Run a controlled small batch (`LIMIT 3` in a test copy or manual run once `Update Deals` logic is verified) and confirm `Loop Over Items` output `0` (`done`) fires once at the end with accurate counts (`fetched`, `updated`, `skipped_no_deal`, `skipped_already_queued`, `errors`).
- 4. **Verify host schedule & monitor 7 consecutive live runs:**
- - Confirm from host cron / Docker config / container logs that no update/restart/backup runs within `06:00–06:30 SGT` (or adjust `Schedule Trigger` cron to a clean window).
- - Activate the updated workflow and verify the next **7 consecutive daily runs** in n8n Executions show:
- - `status: "success"`, `finished: true`
- - `MySQL: Fetch Unpaid Students` count matches `updated + skipped + errored`
- - `0` `crashed` runs and `0` unhandled `404` aborts
- - Expected CRM `wa_inv_reminder` transitions and summary email/alert received each morning.
- ---
- ## 6. Open Questions / Missing Evidence
- 1. **Host update/restart schedule & container logs:** We do not have host crontab, `docker-compose.yml`, Watchtower config, Plesk scheduled tasks, or container/n8n logs for `05:00–05:02 SGT` (why crashes happened on Nov 4–5 and Nov 10–14, but not Nov 6–9 or Nov 15–16) or `06:00–06:30 SGT` (to confirm `06:00` has zero host collision over a full week).
- 2. **CRM `wa_inv_reminder` automation rules:** How does Zoho CRM consume `"wa_inv_reminder": "queued"`? Does it reset the field (e.g., to `"sent"`), ignore repeated `"queued"` writes, or send duplicate WhatsApp messages across all 3 days (`Start_Date` +1..+3)?
- 3. **Unmatched `sfc_id` root cause in CRM/TMS:** Which specific `sfc_id`s failed in `732844` (item 18) and `732104` (item 4), and why did they lack a matching `Deals` record in Zoho CRM (`Zoho 4` writeback failure, non-Zoho registration, or deleted/merged Deal)?
- 4. **TMS `PaymentID` vs `Pass_Fail` & CRM payment fields:** Can `MySQL: Fetch Unpaid Students` safely filter `s.PaymentID IS NULL` (or check a CRM payment/Stage field on `GET CRM Deal ID`) without missing partial/SFC-pending payments?
- 5. **Side effects of crashed runs:** Because `crashed` runs (`724709`, `725593`, `728231`, `728921`, `729977`, `730801`, `731235`) retain only `isArtificialRecoveredEventItem` placeholders, Zoho CRM audit logs or container logs are needed to see how many reminders were actually queued during those 7 mornings.
- 6. **Existing shared workflows:** Is `VKJUEQNFd1yCKu9a` (referenced in [all_schedule_triggers.json](all_schedule_triggers.json#L69)) an active, maintained Error Workflow, and what schedule is intended for active `K3YW6fqzeR6t66KL` (`Zoho 7 - Update Phone +65`) which currently has an empty `interval: [{}]`?
Advertisement