gpt2

A detailed technical flow diagram of a web scraping pipeline
for Thai real estate data. Clean minimal style, white background,
light gray connectors, soft shadows on cards.

Show two parallel scraping tracks side by side:

LEFT TRACK labeled "Facebook Groups":
Step 1: "APScheduler triggers job" (clock icon)
Step 2: "Playwright launches stealth browser" (browser icon)
Step 3: "Load Facebook group page"
Step 4: "Dismiss login wall overlay" (shield icon)
Step 5: "Scroll feed × 6, expand See More buttons"
Step 6: "Extract post text + image URLs per article"
Step 7: "Strip header / footer boilerplate"
Step 8: "Save to ks_raw_posts" (database icon, teal)

RIGHT TRACK labeled "DDproperty Portal":
Step 1: "APScheduler triggers job" (clock icon)
Step 2: "Playwright launches stealth browser"
Step 3: "Navigate zone URL (Sukhumvit, Silom, Rama9...)"
Step 4: "Wait for JS render — networkidle"
Step 5: "Extract listing cards via data-testid selectors"
Step 6: "Paginate through results (page=1..10)"
Step 7: "Parse raw_title, raw_price, raw_area, raw_html"
Step 8: "Save to ks_raw_posts" (database icon, teal)

Both tracks merge at the bottom into one box:
"ks_raw_posts — raw_text stored, parse_status = pending"

On the right side show a vertical "Proxy Layer" panel with:
- Block detector (HTTP 403 / CAPTCHA / selector missing)
- IP rotation arrow
- Circuit breaker states: CLOSED → OPEN → HALF-OPEN

Color: teal for Facebook track, blue for DDproperty track,
coral for proxy panel. Rounded rectangle cards.
Arrow connectors with labels. Sans-serif font (Inter or similar).
16:9 aspect ratio. High resolution infographic.

A clean minimal flowchart diagram on white background showing
a web scraper block detection and recovery system.

Main flow (left to right):
"Make request" → diamond "Response OK?"
→ YES path: "Record success in Redis" → "Continue scraping"
→ NO path: leads to detection layer

Detection layer (vertical checklist boxes):
Box 1: "Check HTTP status — 403 / 429 / 503?"
Box 2: "Check page title — contains captcha / blocked?"
Box 3: "Check selector — expected card found?"
Box 4: "Check result count — less than 50% of 7-day baseline?"

Each box has YES/NO branches.
Any YES leads to: "Classify block type"
(4 labels: HTTP_ERROR / CAPTCHA / SILENT_REDIRECT / SOFT_BLOCK)

Then: "Get next proxy from Redis pool" →
"Exponential backoff (30s → 5min → 30min)" →
"Increment retry counter" →
diamond "Retry count > 3?" →
YES: "Open circuit breaker → Dead letter queue" →
NO: loop back to "Make request"

Color: light blue for normal flow, amber for detection,
red for blocked states, green for success.
Minimal rounded cards, thin gray connectors,
clean sans-serif labels. White background.
High resolution, 16:9.