API tools faq
paste
Guest User

Untitled

a guest
Jul 2nd, 2026
3,196
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 92.88 KB | None | 0 0
raw download clone embed print report
  1. PERSONALOS RECREATION SPECIFICATION
  2. Sanitized, implementation-oriented version
  3.  
  4. Purpose of this document
  5. ========================
  6.  
  7. This file specifies the PersonalOS system as a reusable private research and
  8. tooling environment. It is written so another person can recreate the same kind
  9. of system for their own life archive without receiving any private facts from
  10. the original workspace.
  11.  
  12. This document intentionally does not include:
  13.  
  14. - The original subject's name.
  15. - Names of family members, friends, employers, schools, cities of residence, or
  16. other identifying entities.
  17. - Raw excerpts from private messages, notes, emails, voice transcripts,
  18. documents, photos, health records, finance records, or generated analyses.
  19. - Absolute local paths that reveal a machine username.
  20. - Account IDs, project IDs, API keys, tokens, hashes of private archives, or
  21. credential scan findings.
  22. - Concrete life events, diagnoses, financial amounts, travel routes, addresses,
  23. or exact personal chronology facts from the source workspace.
  24.  
  25. Where a concrete identifier would have appeared, this spec uses placeholders:
  26.  
  27. - <workspace>: the root directory for the new PersonalOS workspace.
  28. - <subject>: the person whose archive is being modeled.
  29. - <subject_slug>: a filesystem-safe slug for <subject>.
  30. - <counterpart>: another person in a message or relationship record.
  31. - <source_family>: a source family such as messages, writing, voice, finance,
  32. health, location, media, or generated analysis.
  33. - <model>: the selected language, embedding, vision, or local ML model.
  34.  
  35. What the original project completed, stated generically
  36. =======================================================
  37.  
  38. The original PersonalOS work built a private end-to-end system with these
  39. completed or partially completed layers:
  40.  
  41. 1. Reorganized the workspace into a clean personal/project split.
  42. 2. Documented raw-source preservation rules, privacy posture, canonical paths,
  43. generated-output locations, and legacy path translation.
  44. 3. Collected and canonicalized multiple source families: writing, messages,
  45. voice, calendar, mail, photos metadata, finance, rideshare/location, health
  46. inventories, AI conversations, and selected generated analyses.
  47. 4. Built platform-specific normalization for major message platforms, including
  48. repaired social-media exports and stable source locators.
  49. 5. Built a cross-platform merged message-history pipeline with normalized event
  50. rows, raw identity rows, person identity edges, conversation rows, manual
  51. identity override queues, and top-person history files.
  52. 6. Built message activity analytics, group/cross-platform salience windows, a
  53. metadata-only relationship graph, closeness rankings, friend-mention indexes,
  54. and model-ranked relationship chunks/subchunks.
  55. 7. Built a chronology system: source acquisition, broad seed events, filtered
  56. candidate packs, chronology drafts/addenda, mail and finance backfills,
  57. travel/movement ledgers, flight-route guess ledgers, daily location modeling,
  58. and visual trip/movement surfaces.
  59. 8. Built day-level activity indexing from generated evidence packets and model
  60. labels, including deterministic no-evidence and location-only rows.
  61. 9. Built media layers: image caption manifests, model caption batches,
  62. caption/FTS SQLite indexes, text/OCR pass, important-media-text promotion,
  63. emotional/life-moment tags, anonymous face detection/clustering, and local
  64. media review surfaces.
  65. 10. Built location layers: common place indexes, merchant/location records,
  66. message-location mention aggregates, RAG-backed naming, grouped location
  67. resolution, and per-location activity timelines.
  68. 11. Built context packs from primary sources, life moments, relationship
  69. subchunks, writing, voice transcripts, and synthesis backfill, including a
  70. medium default pack for seeded chats.
  71. 12. Built a staged contrastive synthesis harness with source cards, primary
  72. moments, a claim ledger, contrasts, theme dossiers, skeptical audit, and a
  73. final report.
  74. 13. Built higher-interpretation layers: psychological/formulation dossiers,
  75. skeptical audits, structured ledgers, introspective-question extraction,
  76. open-ended question answering, and final conclusion ledgers.
  77. 14. Built a local RAG index with source-aware formatting, FTS, exact search,
  78. embedding search, hybrid/evidence retrieval, salience roles, vector matrix
  79. generation, cost guards, and regression checks.
  80. 15. Built local chat over the RAG index with default medium-pack seeding and
  81. retrieval-as-tool behavior.
  82. 16. Built a no-build static React dashboard with a unified local Python server
  83. that mounts dashboard, raw stream review, raw plain view, chat, search,
  84. media caption review, relationship graph, chronology globe, rideshare map,
  85. presence map, finance summaries, daily timeline, and final conclusions.
  86. 17. Added privacy/security tooling: generated-artifact secret scanning,
  87. local-only raw APIs, optional authentication, cleanup/status reports, and
  88. explicit pre-export rules.
  89. 18. Added miscellaneous local utilities such as a browser path blocker and
  90. vendored export tools.
  91.  
  92. Core idea
  93. =========
  94.  
  95. PersonalOS is a private local system for turning a person's scattered archive
  96. into source-grounded context. It is not a generic notes app, CRM, diary, or
  97. public website. It is a research environment that:
  98.  
  99. 1. Preserves raw source exports as evidence.
  100. 2. Normalizes heterogeneous sources into typed records with stable source
  101. locators.
  102. 3. Builds derived indexes over chronology, relationships, locations, media,
  103. daily activity, and life moments.
  104. 4. Uses model-generated analysis, but keeps it downstream of evidence.
  105. 5. Separates primary source material, derived indexes, model interpretation,
  106. and orientation/profile material.
  107. 6. Supports local search, retrieval-augmented chat, dashboards, and review
  108. tools without publishing private data.
  109.  
  110. The strongest outputs should function as indexes back into primary sources, not
  111. as replacements for those sources. Every important claim should point back to a
  112. source path, local locator, confidence label, and uncertainty/counterevidence
  113. when available.
  114.  
  115. Design principles
  116. =================
  117.  
  118. Evidence rules
  119. --------------
  120.  
  121. - Never rewrite or delete raw exports unless the user explicitly asks.
  122. - Treat every generated artifact as sensitive by default.
  123. - Preserve distinction between:
  124. - direct self-expression;
  125. - observed behavior;
  126. - third-party reports;
  127. - platform metadata;
  128. - generated interpretation.
  129. - Use model analysis for synthesis and hypotheses, not as final authority.
  130. - Prefer falsifiable claims over smooth narrative.
  131. - Store source paths, row numbers, byte offsets, message IDs, local line ranges,
  132. JSON pointers, or other stable locators wherever possible.
  133. - Make uncertainty visible. A weak signal should remain weak until corroborated.
  134. - Do not let one high-volume platform dominate the entire self-model.
  135. - Keep relationship-specific behavior separate from broad personality claims.
  136.  
  137. Privacy rules
  138. -------------
  139.  
  140. - Default posture: local-only.
  141. - Do not commit raw archives or large private generated outputs to a normal git
  142. repository.
  143. - Do not publish dashboards, context packs, synthesis reports, RAG indexes, or
  144. model prompts without a separate redaction/export workflow.
  145. - Raw-content APIs must bind to loopback by default.
  146. - Remote/tunnel access must require authentication.
  147. - Pre-export secret scanning is mandatory for generated artifacts.
  148. - Secret scanners must report only path, rule, severity, and line number; they
  149. must not print or persist matched secret values.
  150. - Avoid storing API keys, OAuth tokens, session cookies, or private keys inside
  151. source files, generated prompts, model responses, or shell history.
  152.  
  153. Visual/interface style
  154. ----------------------
  155.  
  156. For local websites, timelines, dashboards, and review tools, use a dense
  157. monochrome information-design style:
  158.  
  159. - linear ledger-like layouts;
  160. - square controls;
  161. - thin rules;
  162. - compact typography;
  163. - low ornament;
  164. - color used selectively for state, severity, grouping, or focus;
  165. - avoid big rounded cards, pill shapes, decorative gradients, bubble timelines,
  166. oversized whitespace, and marketing-style hero sections;
  167. - expose state and internal workings instead of hiding complexity.
  168.  
  169. Canonical workspace layout
  170. ==========================
  171.  
  172. Create the root directory:
  173.  
  174. <workspace>/
  175.  
  176. The root should stay clean. The canonical top-level layout is:
  177.  
  178. <workspace>/
  179. README.md
  180. AGENTS.md
  181. personal/
  182. project/
  183.  
  184. Do not recreate legacy root-level folders such as raw message exports, outputs,
  185. scripts, or source-family folders at the workspace root. Keep the split:
  186.  
  187. - personal/: raw personal store and human-facing reading surface.
  188. - project/: code, pipelines, generated artifacts, docs, sites, and tools.
  189.  
  190. Personal store
  191. --------------
  192.  
  193. personal/
  194. README.md
  195. raw/
  196. conversations/
  197. library/
  198.  
  199. Raw source folders:
  200.  
  201. personal/raw/
  202. writing/
  203. apple_notes/
  204. google_docs/
  205. other_writing/
  206. messages/
  207. imessage/
  208. signal/
  209. snapchat/
  210. discord/
  211. discord_dce/
  212. instagram/
  213. twitter_raw/
  214. twitter_data/
  215. other_platforms/
  216. voice/
  217. voice_memos/
  218. finance/
  219. bank_statements/
  220. paypal/
  221. other_accounts/
  222. health/
  223. lab_results/
  224. clinical_records/
  225. location/
  226. rideshare/
  227. maps_takeout/
  228. other_location_exports/
  229. media/
  230. photos_exports/
  231. social_media/
  232.  
  233. Rules for personal/raw:
  234.  
  235. - It is evidence storage.
  236. - Add new exports here.
  237. - Do not normalize in place.
  238. - Do not move source files during pipeline work.
  239. - If a source-family folder already contains old charts or analysis created
  240. during earlier extraction, leave it unless there is an explicit migration.
  241.  
  242. Conversation folders:
  243.  
  244. personal/conversations/
  245. ai_chats/
  246. terminal_context_chats/
  247. web_context_chats/
  248. other_model_conversations/
  249.  
  250. Human library folders:
  251.  
  252. personal/library/
  253. chronology/
  254. synthesis/
  255. context_packs/
  256. life_moments/
  257. message_histories/
  258. profile_seeds/
  259.  
  260. The library is a reading surface. It may contain symlinks into project/outputs.
  261. The canonical generated artifacts still live with project code.
  262.  
  263. Project folders
  264. ---------------
  265.  
  266. project/
  267. README.md
  268. docs/
  269. scripts/
  270. pipelines/
  271. outputs/
  272. sites/
  273. tools/
  274. deploy/
  275. tests/
  276. tmp/
  277.  
  278. Use these rules:
  279.  
  280. - Single-file utilities and drivers go in project/scripts.
  281. - Multi-file pipelines go in project/pipelines.
  282. - Cross-project generated artifacts go in project/outputs.
  283. - Pipeline-owned outputs may live under project/pipelines/<pipeline>/outputs.
  284. - Local web tools go in project/sites.
  285. - Vendored or local binary tools go in project/tools.
  286. - Deployment shims go in project/deploy.
  287. - Scratch files go in project/tmp and must not become durable state.
  288.  
  289. Path helper module
  290. ------------------
  291.  
  292. Add a shared path helper such as:
  293.  
  294. project/scripts/personalos_paths.py
  295.  
  296. It should expose:
  297.  
  298. - ROOT: <workspace>.
  299. - PERSONAL_DIR: <workspace>/personal.
  300. - RAW_DIR: <workspace>/personal/raw.
  301. - CONVERSATIONS_DIR: <workspace>/personal/conversations.
  302. - LIBRARY_DIR: <workspace>/personal/library.
  303. - PROJECT_DIR: <workspace>/project.
  304. - SCRIPTS_DIR: <workspace>/project/scripts.
  305. - PIPELINES_DIR: <workspace>/project/pipelines.
  306. - OUTPUTS_DIR: <workspace>/project/outputs.
  307. - TMP_DIR: <workspace>/project/tmp.
  308. - source-family convenience paths.
  309. - workspace_root(file): find <workspace> from any script path.
  310. - canonicalize_workspace_path(value): translate old paths to canonical paths.
  311.  
  312. Every new script should use this helper rather than assuming a hard-coded root.
  313.  
  314. Legacy path map
  315. ---------------
  316.  
  317. If the project previously had root-level folders, keep a documentation-only
  318. legacy path map:
  319.  
  320. project/docs/LEGACY_PATHS.md
  321.  
  322. Do not recreate old paths. The map should translate old references to current
  323. canonical paths for older generated artifacts.
  324.  
  325. Environment and dependencies
  326. ============================
  327.  
  328. Minimum practical environment:
  329.  
  330. - macOS or Linux-like local environment.
  331. - Python 3.11+.
  332. - Node.js 20+ or current local equivalent.
  333. - SQLite with FTS5 support.
  334. - ripgrep for fast local search.
  335. - Optional local nginx or hosts entry for a friendly local domain.
  336. - Optional Poppler, ffmpeg, OCR tools, image libraries, and local ML packages
  337. for media/voice work.
  338. - API access for selected model providers.
  339. - For browser/dashboard QA, Playwright or another browser automation tool.
  340.  
  341. Python packages commonly needed:
  342.  
  343. - standard library: argparse, csv, json, sqlite3, pathlib, dataclasses,
  344. datetime, email, mailbox, re, hashlib, zipfile, plistlib, http.server.
  345. - data: pandas, numpy, openpyxl.
  346. - text/token utilities: tiktoken or equivalent tokenizer.
  347. - model clients: provider SDKs as needed.
  348. - PDF/finance extraction: pypdf, pdfplumber, or platform tools.
  349. - media/vision: pillow, imageio, OpenCV as needed.
  350. - face detection/clustering: insightface, onnxruntime/CoreML provider,
  351. scikit-learn for DBSCAN.
  352. - web/server: no framework required; the current style uses Python
  353. ThreadingHTTPServer plus static vendored frontend assets.
  354.  
  355. Node packages/tools commonly needed:
  356.  
  357. - vanilla Node scripts for chart/workbook/timeline generation.
  358. - local vendored React builds for no-build static apps.
  359. - optional force-graph or Three.js vendored locally for graph/globe views.
  360. - optional workbook/artifact generation libraries.
  361.  
  362. There is no single root package manager file in the current style. Prefer small
  363. scripts with explicit dependencies and status docs. If you introduce a package
  364. manager later, keep raw archives outside dependency tooling and avoid scanning
  365. node_modules into RAG or secret reports.
  366.  
  367. Configuration
  368. =============
  369.  
  370. Create a subject config file before building:
  371.  
  372. project/config/subject.json
  373.  
  374. Recommended fields:
  375.  
  376. {
  377. "subject_slug": "<subject_slug>",
  378. "display_name": "<subject>",
  379. "timezone": "<IANA timezone>",
  380. "birth_date": "<YYYY-MM-DD or null>",
  381. "known_life_phase_boundaries": [],
  382. "identity_seed_path": "personal/library/profile_seeds/<subject_slug>.md",
  383. "canonical_email_addresses": [],
  384. "canonical_phone_numbers": [],
  385. "self_platform_ids": {},
  386. "redaction_policy": {
  387. "external_exports_default": "blocked",
  388. "raw_quotes_default": "minimal",
  389. "remote_sensitive_api_default": "disabled"
  390. }
  391. }
  392.  
  393. The current project has some older scripts where subject-specific filenames or
  394. seed facts are embedded directly. For a clean recreation, parameterize those
  395. scripts to read subject config and data files. Hard-coded identity facts should
  396. be replaced with:
  397.  
  398. project/config/identity_seed_events.jsonl
  399.  
  400. Identity seed event schema:
  401.  
  402. event_id
  403. date_start
  404. date_end
  405. precision
  406. title
  407. description
  408. domains
  409. people
  410. locations
  411. source_type
  412. source_path
  413. locator
  414. evidence_basis
  415. confidence
  416. importance_hint
  417. citation
  418. interpretive_risk
  419.  
  420. Do not put private seed facts in this public spec. Each user should create their
  421. own local seed file.
  422.  
  423. Data model overview
  424. ===================
  425.  
  426. Use a small number of durable record types across the system.
  427.  
  428. SourceRef
  429. ---------
  430.  
  431. Every derived row that points to evidence should support:
  432.  
  433. source_path: workspace-relative path to raw or derived source.
  434. source_family: high-level family.
  435. source_id: script-defined source identifier.
  436. locator: stable pointer such as row number, JSON pointer, byte offset,
  437. message ID, line range, file member, or timestamp.
  438. source_line: optional local line number.
  439. source_member: optional member path inside an archive.
  440. evidence_basis: direct_self_expression, observed_behavior,
  441. other_person_report, platform_metadata, generated_index,
  442. model_interpretation, or mixed.
  443. confidence: low, medium, high, or needs_review.
  444. interpretive_risk: why this row may be misleading.
  445.  
  446. Chronology event
  447. ----------------
  448.  
  449. Use a broad event row for chronology seeds:
  450.  
  451. event_id
  452. date_start
  453. date_end
  454. precision
  455. title
  456. description
  457. domains
  458. people
  459. locations
  460. source_type
  461. source_id
  462. source_path
  463. locator
  464. evidence_basis
  465. confidence
  466. importance_hint
  467. citation
  468. interpretive_risk
  469. raw
  470.  
  471. Keep seed events inclusive. Candidate filtering and narrative chronology are
  472. separate downstream steps.
  473.  
  474. Normalized message event
  475. ------------------------
  476.  
  477. Message normalization should converge on:
  478.  
  479. event_id
  480. platform
  481. source_family
  482. source_path
  483. source_line
  484. raw_message_id
  485. timestamp_utc
  486. timestamp_local
  487. conversation_id
  488. conversation_kind
  489. conversation_title
  490. sender_raw_identity_id
  491. sender_person_id
  492. sender_label
  493. is_from_me
  494. text
  495. participant_person_ids
  496. related_person_ids
  497. relation_scope_by_person
  498. source_quality
  499. confidence_flags
  500.  
  501. The key field is related_person_ids. It answers: whose per-person history should
  502. include this event in the deterministic first pass?
  503.  
  504. Relation scopes:
  505.  
  506. direct_conversation
  507. small_group_context
  508. medium_group_context
  509. group_sender
  510. server_sender
  511. additive_sender_only
  512.  
  513. Raw identity row
  514. ----------------
  515.  
  516. raw_identity_id
  517. platform
  518. raw_label
  519. raw_platform_id
  520. normalized_label
  521. person_id
  522. person_label
  523. merge_confidence
  524. evidence
  525.  
  526. Person identity edge
  527. --------------------
  528.  
  529. person_id
  530. person_label
  531. raw_identity_id
  532. platform
  533. raw_label
  534. raw_platform_id
  535. confidence
  536. method
  537. note
  538.  
  539. Conversation row
  540. ----------------
  541.  
  542. conversation_id
  543. platform
  544. source_family
  545. conversation_kind
  546. conversation_title
  547. raw_conversation_id
  548. source_path
  549. participant_person_ids
  550. participant_raw_identity_ids
  551. message_count
  552. my_message_count
  553. first_timestamp_utc
  554. last_timestamp_utc
  555. source_quality
  556. notes
  557.  
  558. Summary node
  559. ------------
  560.  
  561. Hierarchical summaries should use:
  562.  
  563. summary_id
  564. source_modality
  565. node_kind
  566. level
  567. summary_status
  568. summary_text
  569. time_start_utc
  570. time_end_utc
  571. conversation_ids
  572. conversation_titles
  573. conversation_kinds
  574. person_ids
  575. person_labels
  576. platforms
  577. event_count
  578. my_event_count
  579. other_event_count
  580. episode_ids
  581. child_summary_ids
  582. source_refs_sample
  583. source_ref_count
  584. event_ids_sample
  585. event_id_count
  586. salience
  587. life_moment_candidate
  588. search_tags
  589. input_hash
  590.  
  591. RAG source/chunk
  592. ----------------
  593.  
  594. RAG indexing should keep separate source and chunk tables.
  595.  
  596. Source table fields:
  597.  
  598. source_id
  599. path
  600. family
  601. artifact_type
  602. content_provenance
  603. is_model_generated
  604. embedding_salience
  605. evidence_role
  606. mtime
  607. size
  608. sha256
  609. formatter_version
  610.  
  611. Chunk table fields:
  612.  
  613. chunk_id
  614. source_id
  615. chunk_index
  616. text
  617. formatted_text
  618. start_line
  619. end_line
  620. start_byte
  621. end_byte
  622. timestamp_start
  623. timestamp_end
  624. people
  625. locations
  626. tags
  627. embedding_salience
  628. evidence_role
  629. content_provenance
  630. input_hash
  631.  
  632. Content provenance:
  633.  
  634. primary_source
  635. derived_artifact
  636. model_generated
  637. project_doc
  638. orientation
  639. prior_chat
  640. unknown
  641.  
  642. Evidence role:
  643.  
  644. primary
  645. derived_index
  646. analysis
  647. orientation
  648. prior_chat
  649.  
  650. Source acquisition layer
  651. ========================
  652.  
  653. The source acquisition layer imports raw archives into canonical storage without
  654. changing the originals.
  655.  
  656. Writing
  657. -------
  658.  
  659. Sources:
  660.  
  661. - Apple Notes export.
  662. - Google Docs or Drive export.
  663. - Markdown/text files.
  664. - PDFs or documents containing personal writing.
  665.  
  666. Desired derived artifacts:
  667.  
  668. - merged text files for large source families;
  669. - per-document manifests with title, path, created/modified times, token counts,
  670. and source refs;
  671. - optional ranking/tagging outputs for emotional salience, self-reflection,
  672. chronology relevance, and inclusion/exclusion in context packs.
  673.  
  674. Implementation notes:
  675.  
  676. - Preserve original export directories.
  677. - Convert documents into readable plain text where possible.
  678. - Keep document-level metadata separate from extracted body text.
  679. - Do not overwrite the raw export with normalized text.
  680.  
  681. Messages
  682. --------
  683.  
  684. Sources:
  685.  
  686. - local chat database exports;
  687. - phone/cloud archive exports;
  688. - Signal exports;
  689. - Discord account export and DiscordChatExporter JSON;
  690. - Snapchat data export;
  691. - Instagram data export;
  692. - Twitter/X archive exports;
  693. - other platforms as needed.
  694.  
  695. For each platform:
  696.  
  697. 1. Put raw export under personal/raw/messages/<platform>/.
  698. 2. Write a platform normalizer that emits normalized records with:
  699. timestamp, sender, participants, conversation, text, attachments/placeholder
  700. events, reactions, calls/system rows where available, and source locators.
  701. 3. Preserve message text in normalized event ledgers when needed for local
  702. analysis, but avoid writing message bodies into metadata-only reports.
  703. 4. Route every platform through the merged message histories pipeline.
  704.  
  705. Special source-family notes:
  706.  
  707. - iMessage/live databases may need a main export plus additive cloud/archive
  708. rows not present in the live database. Deduplicate by stable message IDs where
  709. available.
  710. - Discord DMs/group DMs should use recipient metadata from the account package
  711. when available instead of inferring participants only from authored rows.
  712. - Snapchat may contain both chat-history rows and content-free activity
  713. metadata. Keep media decode separate from message normalization.
  714. - Instagram exports often need mojibake repair. Repair strings in memory and
  715. write normalized derived rows; never edit raw Meta files.
  716. - Twitter/X participants may be numeric IDs until manually resolved.
  717.  
  718. Voice
  719. -----
  720.  
  721. Sources:
  722.  
  723. - Voice memo manifest.
  724. - Audio cache.
  725. - Transcripts.
  726. - Suspect transcript folder.
  727. - Prior analysis outputs.
  728.  
  729. Derived artifacts:
  730.  
  731. project/outputs/voice/
  732. canonical_voice_manifest.csv
  733. canonical_voice_manifest.jsonl
  734. voice_transcript_triage.md
  735. voice_cleanup_candidates.csv
  736. voice_repeat_artifacts.csv
  737. voice_repeat_artifacts.md
  738. STATUS.md
  739.  
  740. Voice pipeline behavior:
  741.  
  742. - Join manifest rows to audio files and transcript files.
  743. - Flag missing audio, missing transcript, zero-token transcript, short
  744. transcript, likely truncation, repeated-word artifacts, and tiny clips.
  745. - Do not delete raw audio.
  746. - Treat cleanup as analysis triage: ignore/exclude generated empty clips unless
  747. the user manually reviews them.
  748. - Feed high-value transcripts into context packs, synthesis, RAG, chronology,
  749. and question extraction.
  750.  
  751. Calendar, mail, photos, and location metadata
  752. ---------------------------------------------
  753.  
  754. Sources:
  755.  
  756. - Local calendar database.
  757. - Google Calendar Takeout.
  758. - Photos library metadata or photo export metadata.
  759. - Email takeout mbox.
  760. - Rideshare/location export archives.
  761. - Other maps/location histories.
  762.  
  763. Derived artifacts:
  764.  
  765. project/outputs/chronology/data_acquisition/
  766. apple_calendar_events.csv/jsonl/ics
  767. apple_calendar_hidden_events.csv/jsonl
  768. google_calendar_takeout_events.csv/jsonl
  769. photos_metadata.csv/jsonl
  770. photos_daily_counts.csv
  771. google_mail_mbox_index.csv/jsonl
  772. google_mail_chronology_hits_yYYYY.md
  773. google_mail_targeted_backfill_candidates.jsonl/tsv
  774. google_mail_curated_evidence.md
  775. rideshare/
  776. rides_redacted.csv/jsonl
  777. orders_redacted.csv/jsonl
  778. city_presence_days.csv/jsonl
  779. monthly_summary.csv/jsonl
  780. chronology_signals.jsonl
  781.  
  782. Calendar rules:
  783.  
  784. - Read local calendar DB read-only.
  785. - Exclude non-personal calendars such as holidays/birthdays unless explicitly
  786. useful.
  787. - Hide legacy family/shared calendars that do not represent <subject>'s own
  788. chronology.
  789. - Preserve hidden rows in audit sidecars.
  790.  
  791. Email rules:
  792.  
  793. - Index mbox headers and byte offsets first.
  794. - Store message IDs, dates, sender/recipient metadata, subject, labels, and
  795. byte offsets.
  796. - Do not load entire mbox bodies into dashboards by default.
  797. - Extract broad year-sliced chronology hits.
  798. - Create targeted backfill candidates for school, admissions, work, travel,
  799. legal, finance, health, and other chosen domains.
  800. - Manually curate a small evidence pack from noisy hits.
  801.  
  802. Photos rules:
  803.  
  804. - Start with metadata: asset ID, created date, media type, dimensions, GPS if
  805. available, favorite/hidden flags, source path, derivative path.
  806. - Use metadata for activity density and location corroboration.
  807. - Direct visual review/captioning is a separate media pipeline.
  808.  
  809. Rideshare/location rules:
  810.  
  811. - Store raw export archive under personal/raw/location/<source>/.
  812. - Normalize into redacted outputs.
  813. - Omit full addresses, phone/email, card numbers, device IDs, IP addresses,
  814. precise analytics coordinates, and item-level order customizations unless
  815. explicitly needed.
  816. - Round coordinates or city-bucket them in derived artifacts.
  817. - Treat rideshare metadata as strong location/logistics evidence and weak
  818. purpose/meaning evidence.
  819.  
  820. Finance
  821. -------
  822.  
  823. Sources:
  824.  
  825. - Bank statement PDFs.
  826. - PayPal CSV exports.
  827. - Other card/bank/brokerage/payroll exports.
  828.  
  829. Derived artifacts:
  830.  
  831. project/outputs/chronology/data_acquisition/bank_statements/
  832. plain_text/
  833. bank_statement_manifest.csv
  834. bank_statement_transactions.csv
  835. bank_statement_monthly_summary.csv
  836. bank_statement_analysis.md
  837. bank_statement_chronology_addendum.md
  838. charts/
  839.  
  840. project/outputs/chronology/data_acquisition/paypal/
  841. paypal_source_manifest.csv
  842. paypal_transactions.csv
  843. paypal_monthly_summary.csv
  844. paypal_history.xlsx
  845. paypal_history_analysis.md
  846.  
  847. Finance rules:
  848.  
  849. - Parse statements into transaction-like rows.
  850. - Keep account numbers masked or reduced to non-sensitive account buckets.
  851. - Preserve source statement path and page/line/source locator.
  852. - Treat merchant/location parsing as heuristic.
  853. - Separate physical point-of-sale from remote/subscription/web payments.
  854. - Use finance data for chronology and location corroboration, not for exposing
  855. raw spending details in default dashboards.
  856.  
  857. Health
  858. ------
  859.  
  860. Sources:
  861.  
  862. - Lab results.
  863. - appointment exports.
  864. - clinical records if the user wants them included.
  865.  
  866. Rules:
  867.  
  868. - Store under personal/raw/health.
  869. - Do not index into public or shared artifacts.
  870. - In RAG, classify as primary evidence but high sensitivity.
  871. - In dashboards, default to aggregate inventory/status only.
  872. - Clinical-style interpretation may be done locally only with source labels and
  873. confidence/counterevidence.
  874.  
  875. Pipeline: merged message histories
  876. ==================================
  877.  
  878. Purpose
  879. -------
  880.  
  881. Build a graph-first, cross-platform message event ledger and per-person history
  882. files.
  883.  
  884. Inputs:
  885.  
  886. personal/raw/messages/...
  887. project/outputs/instagram/instagram_dm_events.jsonl
  888. project/outputs/instagram/instagram_conversations.jsonl
  889. platform-specific normalized exports
  890. identity_overrides.csv
  891.  
  892. Main command pattern:
  893.  
  894. python3 project/pipelines/merged_message_histories/build_merged_message_histories.py \
  895. --top-n 100 \
  896. --output project/pipelines/merged_message_histories/outputs
  897.  
  898. Useful options:
  899.  
  900. --limit-sources imessage signal
  901. --limit-sources instagram
  902. --top-n <N>
  903. --output <dir>
  904.  
  905. Outputs:
  906.  
  907. project/pipelines/merged_message_histories/outputs/
  908. message_events.jsonl
  909. conversations.jsonl
  910. raw_identities.jsonl
  911. person_identity_edges.jsonl
  912. person_index.csv
  913. top_people_seed.csv
  914. top100_index.csv
  915. run_summary.json
  916. review/
  917. identity_collapse_report.csv
  918. identity_merge_candidates.csv
  919. high_confidence_identity_merge_candidates.csv
  920. medium_confidence_identity_merge_candidates.csv
  921. platform_metadata_aliases.csv
  922. identity_resolution_worklist.csv
  923. identity_resolution_worklist.md
  924. identity_review_queue.csv
  925. service_or_spam_candidates.csv
  926. unresolved_platform_ids.csv
  927. top100/
  928. 001_person_slug.md
  929. 001_person_slug.events.jsonl
  930. 001_person_slug.history.tsv
  931.  
  932. Identity strategy
  933. -----------------
  934.  
  935. The pipeline should be conservative:
  936.  
  937. - raw identities are platform-specific labels, handles, phone numbers, emails,
  938. user IDs, or display names;
  939. - people are canonical candidates;
  940. - raw identities map to people through confidence-labeled edges;
  941. - deterministic merges are allowed for exact contact-resolved labels and stable
  942. raw platform IDs;
  943. - uncertain matches go to review queues;
  944. - manual merges live in identity_overrides.csv.
  945.  
  946. identity_overrides.csv schema:
  947.  
  948. platform,raw_label,person_label,confidence,note
  949.  
  950. Example placeholder:
  951.  
  952. signal,Alex Example,Alex Example,manual_high,same person as another platform contact
  953.  
  954. Group chat attribution
  955. ----------------------
  956.  
  957. Use tiered inclusion:
  958.  
  959. - direct/private conversation: include all rows in the counterpart's history;
  960. - very small group: include all rows for all known participants;
  961. - medium group: include sender rows for sender, and <subject>-authored rows for
  962. all participants only below a configured participant cap;
  963. - large group/server: include sender rows only by default;
  964. - additive cloud/archive rows without reliable membership: sender-only.
  965.  
  966. This avoids treating every large-group message as relationship evidence for
  967. every participant.
  968.  
  969. Per-person files
  970. ----------------
  971.  
  972. For the top people by related event count:
  973.  
  974. - write a Markdown coverage summary without message bodies;
  975. - write an events JSONL with message text for local analysis;
  976. - write a history TSV for model chunking and local review.
  977.  
  978. Validation:
  979.  
  980. - every JSONL row parses;
  981. - required schema fields present;
  982. - no duplicate event IDs;
  983. - no missing UTC timestamps where source timestamp exists;
  984. - raw identity IDs do not link to multiple final people;
  985. - top-N output files exist and are non-empty;
  986. - run_summary counts match event files.
  987.  
  988. Pipeline: message activity and salience analytics
  989. =================================================
  990.  
  991. Activity reports
  992. ----------------
  993.  
  994. Command pattern:
  995.  
  996. python3 project/pipelines/merged_message_histories/build_activity_reports.py \
  997. --input project/pipelines/merged_message_histories/outputs \
  998. --output project/pipelines/merged_message_histories/outputs/analytics \
  999. --top-n 100
  1000.  
  1001. Outputs:
  1002.  
  1003. analytics/monthly_activity.csv
  1004. analytics/monthly_sent_received.svg
  1005. analytics/monthly_top5_people.csv
  1006. analytics/monthly_top5_people.svg
  1007. analytics/three_month_top5_people.csv
  1008. analytics/three_month_top5_people.svg
  1009. analytics/top100_by_messages.csv
  1010. analytics/top100_by_direct_messages.csv
  1011. analytics/analytics_summary.md
  1012. analytics/run_summary.json
  1013.  
  1014. Definitions:
  1015.  
  1016. - Monthly sent/received counts are unique normalized events.
  1017. - Word/character counts come from normalized text.
  1018. - Direct thread episode endings use inactivity windows.
  1019. - Monthly top people are ranked by related-history events, so small-group
  1020. context may count one source message toward several people.
  1021.  
  1022. Salience windows
  1023. ----------------
  1024.  
  1025. Command pattern:
  1026.  
  1027. python3 project/pipelines/merged_message_histories/build_salience_windows.py
  1028.  
  1029. Outputs:
  1030.  
  1031. analytics/identity_review_summary.json
  1032. analytics/group_salience_windows.csv/jsonl
  1033. analytics/cross_platform_episode_windows.csv/jsonl
  1034. analytics/salience_run_summary.json
  1035. analytics/salience_analytics.md
  1036.  
  1037. Group-chat salience:
  1038.  
  1039. - split non-direct conversations by inactivity gap;
  1040. - rank by volume, participant breadth, <subject> participation, and text flags;
  1041. - text flags include mentions, planning, conflict, emotion, decisions,
  1042. transitions, links, and platform shifts;
  1043. - do not write message bodies.
  1044.  
  1045. Cross-platform episodes:
  1046.  
  1047. - per-person bursts among top related people;
  1048. - require multiple platforms;
  1049. - split by inactivity gap;
  1050. - cap long windows so constant relationships do not become multi-month blobs.
  1051.  
  1052. Pipeline: relationship graph
  1053. ============================
  1054.  
  1055. Purpose
  1056. -------
  1057.  
  1058. Build a metadata-only graph of relationship candidates and shared group context.
  1059.  
  1060. Command:
  1061.  
  1062. python3 project/pipelines/merged_message_histories/build_relationship_graph.py
  1063.  
  1064. Inputs:
  1065.  
  1066. person_index.csv
  1067. conversations.jsonl
  1068. run_summary.json
  1069. service/spam review CSV
  1070. optionally message_events.jsonl for temporal metadata only
  1071.  
  1072. Outputs:
  1073.  
  1074. project/outputs/relationship_graph/
  1075. relationship_graph.json
  1076. relationship_graph_summary.json
  1077. closeness_rankings.csv
  1078. friend_mention_index.csv
  1079. friend_mention_index.json
  1080. friend_mention_summary.md
  1081. STATUS.md
  1082.  
  1083. Rules:
  1084.  
  1085. - Message text is not embedded in graph JSON.
  1086. - Every person with at least one related event gets an ego edge.
  1087. - Ego edge weight uses direct volume, context volume, total volume,
  1088. reciprocity, recency, duration, platform breadth, conversation breadth, and
  1089. confidence.
  1090. - A separate closeness score emphasizes sustained direct contact, active direct
  1091. months, direct episodes, direct recency, reciprocity, relationship duration,
  1092. platform breadth, and capped group-context support.
  1093. - Person-person links are inferred only from shared non-direct conversations in
  1094. a bounded participant range.
  1095. - Cap the number of shared group links to keep the visualization usable.
  1096. - Low-quality labels and service/spam candidates stay labeled as low confidence.
  1097.  
  1098. Pipeline: friend mentions and birthdays
  1099. =======================================
  1100.  
  1101. Friend mention index:
  1102.  
  1103. - Build aliases from the top relationship/contact set.
  1104. - Scan message text locally for mentions of those aliases.
  1105. - Aggregate mention counts, platforms, source refs, and time buckets.
  1106. - Do not write raw message text into mention outputs.
  1107. - Use for relationship graph enrichment and evidence discovery.
  1108.  
  1109. Birthday index:
  1110.  
  1111. - Extract birthday candidates from message/event evidence where useful.
  1112. - Store known vs unresolved rows separately.
  1113. - Include source refs and confidence.
  1114. - Keep evidence review local and do not expose third-party private facts by
  1115. default.
  1116.  
  1117. Pipeline: relationship chunk salience
  1118. =====================================
  1119.  
  1120. Purpose
  1121. -------
  1122.  
  1123. Use model scoring to identify the most salient relationship-history chunks,
  1124. then compress those chunks into tighter subchunks for context packs and review.
  1125.  
  1126. Inputs:
  1127.  
  1128. project/pipelines/merged_message_histories/outputs/top100/*.history.tsv
  1129. relationship graph metadata
  1130. prior relationship takeaways
  1131. optional psychological/relationship dossiers
  1132. compact global profile/chronology orientation
  1133.  
  1134. Stage 1 command pattern:
  1135.  
  1136. python3 project/scripts/gpt5mini_relationship_chunk_salience.py
  1137.  
  1138. Stage 1 outputs:
  1139.  
  1140. project/outputs/relationship_chunking/
  1141. relationship_chunk_manifest.jsonl
  1142. relationship_chunk_scores_gpt5mini.jsonl
  1143. relationship_chunk_scores_gpt5mini_latest.jsonl
  1144. relationship_chunk_scores_gpt5mini_latest_summary.json
  1145. relationship_chunk_rankings_gpt5mini_latest.csv
  1146. relationship_chunk_rankings_gpt5mini_latest.md
  1147. relationship_chunking_gpt5mini_latest_audit.md
  1148. STATUS.md
  1149.  
  1150. Stage 2 command:
  1151.  
  1152. python3 project/scripts/build_relationship_salience_subchunks.py
  1153.  
  1154. Stage 2 outputs:
  1155.  
  1156. relationship_top150_diverse_ultra_tight_subchunks_pack.md
  1157. relationship_top150_diverse_ultra_tight_subchunks_manifest.jsonl
  1158. relationship_top150_diverse_ultra_tight_subchunks.csv
  1159. relationship_top150_diverse_ultra_tight_subchunks_stats.json
  1160. relationship_top150_diverse_ultra_tight_subchunks_audit.md
  1161. relationship_top150_diverse_ultra_tight_breakdown_by_relationship.md/csv
  1162.  
  1163. Also keep broader fallback packs for review:
  1164.  
  1165. relationship_top300_ultra_tight_subchunks_pack.md
  1166. relationship_top300_high_notable_subchunks_pack.md
  1167. relationship_top300_salient_subchunks_pack.md
  1168.  
  1169. Subchunk rules:
  1170.  
  1171. - Start from ranked parent chunks.
  1172. - Prefer high notable source ranges.
  1173. - Merge nearby overlaps.
  1174. - Use compact speaker labels.
  1175. - Preserve dates, platform, direct/group context, and source row ranges.
  1176. - Keep the pack small enough for downstream model context.
  1177.  
  1178. Pipeline: hierarchical summaries
  1179. ================================
  1180.  
  1181. Purpose
  1182. -------
  1183.  
  1184. Build a summary-node tree over normalized sources. The first implemented slice
  1185. is messages, using merged-message history outputs as source of truth.
  1186.  
  1187. Command examples:
  1188.  
  1189. python3 project/pipelines/hierarchical_summaries/build_message_summary_tree.py --dry-run
  1190. python3 project/pipelines/hierarchical_summaries/build_message_summary_tree.py --person-limit 10
  1191. python3 project/pipelines/hierarchical_summaries/build_message_summary_tree.py --include-all-people
  1192.  
  1193. Inputs:
  1194.  
  1195. message_events.jsonl
  1196. conversations.jsonl
  1197. person_index.csv
  1198.  
  1199. Outputs:
  1200.  
  1201. project/outputs/hierarchical_summaries/messages/
  1202. message_summary_nodes.jsonl
  1203. message_summary_edges.jsonl
  1204. message_summary_aliases.jsonl
  1205. life_moment_candidates.jsonl
  1206. message_summary_manifest.json
  1207. message_summary_status.md
  1208.  
  1209. Behavior:
  1210.  
  1211. - Split platform threads into episode nodes using inactivity gaps.
  1212. - Build calendar rollups at multiple window sizes, such as 1w, 2w, 1m, 2m, 4m,
  1213. 6m, and 1y.
  1214. - Write aliases instead of duplicating summary prose when wider windows contain
  1215. only one child.
  1216. - In the metadata-first pass, read message bodies locally for scoring but do
  1217. not write message bodies into summary outputs.
  1218. - Emit life-moment candidates as a filtered subset of summary nodes.
  1219.  
  1220. Context packages for future model summarization:
  1221.  
  1222. python3 project/pipelines/hierarchical_summaries/build_message_summary_contexts.py sample --count 20 --run
  1223.  
  1224. Context package fields:
  1225.  
  1226. global_context
  1227. node_metadata
  1228. person_thread_context
  1229. life_context_near_window
  1230. local_continuity_context
  1231. child_summary_context
  1232. episode_raw_material
  1233. output_budget_hint
  1234. prompt
  1235.  
  1236. These prompt packages may include raw local message transcripts and must be
  1237. treated as private generated artifacts.
  1238.  
  1239. Pipeline: life-moment examples
  1240. ==============================
  1241.  
  1242. Purpose
  1243. -------
  1244.  
  1245. Extract reviewable life-moment examples from source-family packs. A life moment
  1246. is not a final biographical truth; it is a source-grounded candidate with
  1247. evidence pointers.
  1248.  
  1249. Command pattern:
  1250.  
  1251. python3 project/scripts/extract_life_moment_examples.py
  1252.  
  1253. Inputs:
  1254.  
  1255. - synthesis source manifest;
  1256. - selected source-family packs;
  1257. - message/history packs;
  1258. - Instagram derived packs;
  1259. - voice/writing/context sources.
  1260.  
  1261. Outputs:
  1262.  
  1263. project/outputs/life_moment_examples/
  1264. life_moment_examples_report.md
  1265. life_moment_sources.json
  1266. run_summary.json
  1267. sources/<source_id>.json
  1268. full_source_packs/<source_id>.txt
  1269. prompts/
  1270. raw_responses/
  1271. STATUS.md
  1272.  
  1273. Rules:
  1274.  
  1275. - Keep source IDs stable.
  1276. - Keep source refs and why-review notes.
  1277. - Treat weak attention/social signals as weak unless corroborated.
  1278. - Use life-moment candidates from hierarchical summaries as a future preferred
  1279. input layer.
  1280.  
  1281. Pipeline: Instagram integration
  1282. ===============================
  1283.  
  1284. Purpose
  1285. -------
  1286.  
  1287. Normalize a raw Instagram/Meta export into useful derived artifacts without
  1288. editing the raw export.
  1289.  
  1290. Command:
  1291.  
  1292. python3 project/scripts/normalize_instagram_export.py \
  1293. --raw-root personal/raw/messages/instagram \
  1294. --output project/outputs/instagram \
  1295. --timezone <timezone>
  1296.  
  1297. Outputs:
  1298.  
  1299. project/outputs/instagram/
  1300. instagram_dm_events.jsonl/csv
  1301. instagram_conversations.jsonl/csv
  1302. instagram_dm_pack.txt
  1303. instagram_public_records.jsonl
  1304. instagram_public_pack.txt
  1305. instagram_attention_events.jsonl
  1306. instagram_attention_pack.txt
  1307. instagram_chronology_signals.jsonl
  1308. instagram_identity_review_candidates.csv/jsonl
  1309. instagram_inventory_summary.json
  1310. validation_summary.json
  1311. integration_validation_summary.json/md
  1312. STATUS.md
  1313.  
  1314. Rules:
  1315.  
  1316. - Repair common export encoding issues in memory.
  1317. - Convert timestamp_ms to UTC ISO and configured local time.
  1318. - Create deterministic event IDs from thread path, index, timestamp, sender,
  1319. and text hash.
  1320. - Use JSON pointers such as relative/path.json#/messages/N for review.
  1321. - Preserve message text in local event JSONL/CSV, but reports should avoid raw
  1322. message bodies by construction.
  1323. - Create placeholders for shares, reactions, photos, videos, audio, calls, and
  1324. unavailable/deleted content.
  1325. - Distinguish inbox threads from message requests.
  1326. - Mark deleted/anonymous/brand/service/spam-looking labels as low confidence.
  1327.  
  1328. Downstream use:
  1329.  
  1330. - Merged message histories import normalized DM events and conversations.
  1331. - Synthesis uses three derived sources: DMs, public/profile surface, and weak
  1332. attention graph.
  1333. - Chronology starts with chronology signals; bulk DMs should flow through
  1334. selected life-moment/synthesis outputs, not direct chronology insertion.
  1335. - RAG indexes the derived packs and JSONL records.
  1336.  
  1337. Validation:
  1338.  
  1339. python3 project/scripts/validate_instagram_integration.py
  1340.  
  1341. Pipeline: chronology
  1342. ====================
  1343.  
  1344. Purpose
  1345. -------
  1346.  
  1347. Build an inclusive event database, then narrow it into a readable, cited life
  1348. chronology.
  1349.  
  1350. Core outputs:
  1351.  
  1352. project/outputs/chronology/
  1353. chronology_seed_events.jsonl/csv
  1354. chronology_seed_summary.json
  1355. chronology_candidate_events.jsonl
  1356. chronology_candidate_pack.md
  1357. chronology_candidate_summary.json
  1358. chronology_draft_v0.1.md
  1359. chronology_draft_v0.2.md
  1360. chronology_draft_v0.3.md
  1361. chronology_addenda/
  1362. STATUS.md
  1363.  
  1364. Stage 1: export local chronology sources
  1365. ----------------------------------------
  1366.  
  1367. Command:
  1368.  
  1369. python3 project/scripts/export_chronology_sources.py
  1370.  
  1371. What it does:
  1372.  
  1373. - Reads local calendar DB read-only.
  1374. - Reads local Photos metadata DB read-only.
  1375. - Writes visible calendar rows, hidden-calendar audit rows, ICS export, photos
  1376. metadata, and count summaries.
  1377. - Applies conservative filters for non-personal calendars and unsupported legacy
  1378. shared-calendar rows.
  1379.  
  1380. Stage 2: parse Google Calendar Takeout
  1381. --------------------------------------
  1382.  
  1383. Command:
  1384.  
  1385. python3 project/scripts/parse_google_calendar_takeout.py
  1386.  
  1387. What it does:
  1388.  
  1389. - Parses .ics files from Takeout.
  1390. - Emits CSV/JSONL rows and a summary.
  1391. - Preserves calendar/source locators.
  1392.  
  1393. Stage 3: index Google Mail Takeout
  1394. ----------------------------------
  1395.  
  1396. Commands:
  1397.  
  1398. python3 project/scripts/index_google_mail_mbox.py
  1399. python3 project/scripts/extract_google_mail_chronology_hits.py
  1400. python3 project/scripts/extract_google_mail_targeted_backfill.py
  1401.  
  1402. What they do:
  1403.  
  1404. - Build an mbox index with byte offsets.
  1405. - Extract broad year-sliced chronology hits.
  1406. - Extract targeted backfill candidates.
  1407. - Support manual curation into google_mail_curated_evidence.md.
  1408.  
  1409. Stage 4: finance acquisition
  1410. ----------------------------
  1411.  
  1412. Commands:
  1413.  
  1414. python3 project/scripts/extract_chase_bank_statements.py
  1415. python3 project/scripts/export_paypal_history.py
  1416. node project/scripts/build_paypal_history_workbook.mjs
  1417. python3 project/scripts/prepare_chase_chart_data.py
  1418. node project/scripts/build_chase_financial_charts_workbook.mjs
  1419.  
  1420. What they do:
  1421.  
  1422. - Extract statement text.
  1423. - Parse statement metadata and transaction rows.
  1424. - Build monthly summaries.
  1425. - Write chronology addenda.
  1426. - Build review workbooks/charts.
  1427.  
  1428. Stage 5: rideshare/location acquisition
  1429. ---------------------------------------
  1430.  
  1431. Command pattern:
  1432.  
  1433. python3 project/scripts/normalize_uber_export.py
  1434.  
  1435. Important implementation note:
  1436.  
  1437. - In the current style, some older normalizers may run when invoked without
  1438. subcommands. Add normal argparse support before sharing with another user so
  1439. --help never mutates outputs.
  1440.  
  1441. What it does:
  1442.  
  1443. - Reads a raw export zip from personal/raw/location/<source>/.
  1444. - Writes redacted ride/order/city/monthly summaries.
  1445. - Writes chronology signals.
  1446. - Excludes full addresses, exact sensitive coordinates, device/network IDs,
  1447. payment details, and other sensitive fields.
  1448.  
  1449. Stage 6: build chronology seed
  1450. ------------------------------
  1451.  
  1452. Command:
  1453.  
  1454. python3 project/scripts/build_chronology_seed.py
  1455.  
  1456. Inputs:
  1457.  
  1458. - identity seed events;
  1459. - primary-source moments;
  1460. - life-moment examples;
  1461. - voice manifest;
  1462. - calendar rows;
  1463. - photos metadata/counts;
  1464. - email curated evidence;
  1465. - bank/PayPal addenda;
  1466. - Instagram chronology signals;
  1467. - rideshare chronology signals;
  1468. - other source-family extraction outputs.
  1469.  
  1470. Output schema is the chronology event row described above.
  1471.  
  1472. Stage 7: build candidate pack
  1473. -----------------------------
  1474.  
  1475. Command:
  1476.  
  1477. python3 project/scripts/build_chronology_candidate_pack.py
  1478.  
  1479. Behavior:
  1480.  
  1481. - Reads chronology_seed_events.jsonl.
  1482. - Keeps identity, primary moments, curated evidence, high-signal calendar rows,
  1483. dated life moments, longer voice records, high-volume photo days, finance and
  1484. location corroboration, and selected platform signals.
  1485. - Drops routine calendar noise, generic reminders, routine meetings, and weak
  1486. metadata unless useful as corroboration.
  1487. - Writes JSONL plus a GPT-ready Markdown pack.
  1488.  
  1489. Stage 8: write chronology narrative
  1490. -----------------------------------
  1491.  
  1492. Use the candidate pack as source material for a model or human pass. The prompt
  1493. should require:
  1494.  
  1495. - chronological ordering;
  1496. - short titles;
  1497. - source citations;
  1498. - confidence labels;
  1499. - contradictions/needs-review ledger;
  1500. - gaps section;
  1501. - no future events treated as having occurred;
  1502. - no unsupported interpretations.
  1503.  
  1504. Keep drafts versioned. Add source-family addenda rather than overwriting the
  1505. main draft when a new source is first integrated.
  1506.  
  1507. Travel, movement, and location sublayers
  1508. ----------------------------------------
  1509.  
  1510. Major trip index:
  1511.  
  1512. project/outputs/chronology/major_trips_index_v0.1.md
  1513. project/outputs/chronology/major_trips_index_v0.2.md
  1514.  
  1515. Flight route guess ledger:
  1516.  
  1517. python3 project/scripts/build_flight_route_guess_ledger.py
  1518.  
  1519. Travel movement ledger:
  1520.  
  1521. node project/scripts/build_travel_movement_events.mjs
  1522.  
  1523. Daily location model:
  1524.  
  1525. python3 project/scripts/build_daily_location_model.py
  1526.  
  1527. Major trips visual timeline:
  1528.  
  1529. node project/scripts/build_major_trips_visual_timeline.mjs
  1530.  
  1531. Trip globe data:
  1532.  
  1533. node project/scripts/build_trip_globe_timelapse_data.mjs
  1534.  
  1535. Rules:
  1536.  
  1537. - Split trips into explicit origin-to-destination movement legs where possible.
  1538. - Insert continuity_gap rows where location continuity is unresolved.
  1539. - Use confidence labels.
  1540. - Treat flight numbers/airport pairs as guesses unless source evidence is
  1541. explicit.
  1542. - Do not reconstruct exact routes from redacted endpoints as truth; label them
  1543. approximate.
  1544. - For visual timelines, use dense ledger-style rows instead of decorative
  1545. timeline cards.
  1546.  
  1547. Pipeline: daily activity index
  1548. ==============================
  1549.  
  1550. Purpose
  1551. -------
  1552.  
  1553. Build a source-grounded day-by-day activity ledger from generated indexes,
  1554. including deterministic no-evidence rows.
  1555.  
  1556. Command sequence:
  1557.  
  1558. python3 project/scripts/build_daily_activity_index.py --today <YYYY-MM-DD> prepare
  1559. python3 project/scripts/build_daily_activity_index.py --today <YYYY-MM-DD> run --model <model> --reasoning none --assemble
  1560. python3 project/scripts/build_daily_activity_index.py --today <YYYY-MM-DD> fetch-pending --assemble
  1561. python3 project/scripts/build_daily_activity_index.py --today <YYYY-MM-DD> assemble
  1562. python3 project/scripts/build_daily_activity_index.py --today <YYYY-MM-DD> status
  1563.  
  1564. Outputs:
  1565.  
  1566. project/outputs/daily_activity_index/
  1567. daily_evidence_packets.jsonl
  1568. model_labels.jsonl
  1569. daily_activity_index.jsonl
  1570. daily_activity_index.csv
  1571. daily_activity_index.md
  1572. manifest.json
  1573. daily_activity_index_summary.json
  1574. prompts/
  1575. raw_model_responses/
  1576. pending_model_responses/
  1577. errors/
  1578. README.md
  1579.  
  1580. Label categories:
  1581.  
  1582. administrative_financial_day
  1583. future_scheduled
  1584. health_or_appointment_day
  1585. inconclusive
  1586. media_or_capture_only
  1587. no_evidence
  1588. routine_communications
  1589. social_or_relationship_day
  1590. substantive_day
  1591. travel_or_location_day
  1592. work_or_school_day
  1593.  
  1594. Rules:
  1595.  
  1596. - Read generated indexes, not raw exports directly.
  1597. - Cap snippets in evidence packets.
  1598. - Final rows point back to local refs.
  1599. - Future rows are schedule/context, not confirmation that future events
  1600. happened.
  1601. - Use deterministic labels for no-evidence and location-only days.
  1602. - Use model labels only for evidence-bearing days.
  1603.  
  1604. Pipeline: location index
  1605. ========================
  1606.  
  1607. Purpose
  1608. -------
  1609.  
  1610. Index places that repeatedly appear as stays, rideshare endpoints, purchase
  1611. venues, merchants, photos, calendar rows, chronology rows, and message-location
  1612. mentions.
  1613.  
  1614. Commands:
  1615.  
  1616. python3 project/scripts/build_common_location_index.py
  1617. python3 project/scripts/name_location_index_with_rag.py --rag --output-dir project/outputs/location_index/place_naming
  1618. python3 project/scripts/resolve_location_groups_with_rag_web.py --web-context --reuse-llm-adjudications --output-dir project/outputs/location_index/location_resolution
  1619. python3 project/scripts/build_location_activity_index.py --output-dir project/outputs/location_index/location_activity
  1620.  
  1621. Outputs:
  1622.  
  1623. project/outputs/location_index/
  1624. common_location_index.jsonl/csv
  1625. all_location_records.jsonl/csv
  1626. merchant_location_index.jsonl/csv
  1627. message_location_mentions.jsonl/csv
  1628. location_evidence_events.jsonl/csv
  1629. location_index_summary.json
  1630. place_naming/
  1631. location_resolution/
  1632. location_activity/
  1633. README.md
  1634.  
  1635. Commonness rules can be tuned:
  1636.  
  1637. - stay places: minimum modeled days;
  1638. - purchase places: minimum purchase count;
  1639. - rideshare endpoints: minimum endpoint touches or days;
  1640. - message-only places: minimum mention count.
  1641.  
  1642. Privacy:
  1643.  
  1644. - Do not copy raw message text.
  1645. - Use redacted rideshare inputs.
  1646. - Round coordinates.
  1647. - Redact street-address-looking private anchors in generated titles.
  1648. - Public reverse geocoding is context, not proof of personal meaning.
  1649.  
  1650. Pipeline: media captioning, OCR/text pass, and face clustering
  1651. ==============================================================
  1652.  
  1653. Purpose
  1654. -------
  1655.  
  1656. Create source-aware search context for images/media while preserving source
  1657. pointers, uncertainty, and local-only review.
  1658.  
  1659. Media captioning commands:
  1660.  
  1661. python3 project/pipelines/media_captioning/media_captioning.py build-manifest
  1662. python3 project/pipelines/media_captioning/media_captioning.py sample --count 24 --run --model <vision_model> --detail high
  1663. python3 project/pipelines/media_captioning/media_captioning.py prepare-inline-batches --model <vision_model> --detail high
  1664. python3 project/pipelines/media_captioning/media_captioning.py submit-batches project/outputs/media_captioning/batches/<batch_dir>
  1665. python3 project/pipelines/media_captioning/media_captioning.py batch-statuses project/outputs/media_captioning/batches/<batch_dir>
  1666. python3 project/pipelines/media_captioning/media_captioning.py fetch-batches project/outputs/media_captioning/batches/<batch_dir>
  1667. python3 project/pipelines/media_captioning/media_captioning.py collect-batch-outputs project/outputs/media_captioning/batches/<batch_dir>
  1668. python3 project/pipelines/media_captioning/media_captioning.py prepare-retry-batches ...
  1669.  
  1670. Outputs:
  1671.  
  1672. project/outputs/media_captioning/
  1673. manifests/
  1674. samples/
  1675. batches/
  1676. captions/
  1677. media_captions.sqlite
  1678. RUN_STATUS.md
  1679.  
  1680. Caption records should include:
  1681.  
  1682. - media_id;
  1683. - source path;
  1684. - media source family;
  1685. - date/context hint;
  1686. - generated caption;
  1687. - visible people/activity/mood/location hints;
  1688. - uncertainty;
  1689. - tags;
  1690. - parse/review flags;
  1691. - model usage metadata in raw output files, not in RAG-facing formatted text.
  1692.  
  1693. Text/OCR pass commands:
  1694.  
  1695. python3 project/pipelines/media_captioning/media_text_pass.py build-manifest
  1696. python3 project/pipelines/media_captioning/media_text_pass.py sample --count <N> --run --model <vision_model>
  1697. python3 project/pipelines/media_captioning/media_text_pass.py prepare-inline-batches --model <vision_model>
  1698. python3 project/pipelines/media_captioning/media_text_pass.py submit-batches <batch_dir>
  1699. python3 project/pipelines/media_captioning/media_text_pass.py fetch-batches <batch_dir>
  1700. python3 project/pipelines/media_captioning/media_text_pass.py collect-outputs
  1701. python3 project/pipelines/media_captioning/media_text_pass.py prepare-retry-batches
  1702. python3 project/pipelines/media_captioning/media_text_pass.py index-sql
  1703.  
  1704. Important media text:
  1705.  
  1706. python3 project/scripts/build_important_media_text.py
  1707.  
  1708. Rules:
  1709.  
  1710. - General generated captions are selective/lower-salience RAG material.
  1711. - Strict OCR/text records with important personal content can be promoted into
  1712. default-salience RAG.
  1713. - Promotion requires actual transcribed text, non-low confidence, and high
  1714. personal salience.
  1715. - Utility text, receipts, generic school exercises, product listings, maps,
  1716. tickets, weather, games, memes, comics, and generic news/articles should be
  1717. excluded from default high-salience RAG unless explicitly needed.
  1718.  
  1719. Emotional/life-moment tag layer:
  1720.  
  1721. python3 project/scripts/tag_media_emotional_moments.py --sync-sqlite
  1722.  
  1723. Rules:
  1724.  
  1725. - Derived tag JSONL should store media IDs, source paths, priority, and tags,
  1726. but not raw caption text or OCR text.
  1727.  
  1728. Face clustering commands:
  1729.  
  1730. python3 project/pipelines/media_captioning/face_clustering.py build-candidates
  1731. python3 project/pipelines/media_captioning/face_clustering.py detect
  1732. python3 project/pipelines/media_captioning/face_clustering.py cluster
  1733. python3 project/pipelines/media_captioning/face_clustering.py index-sql
  1734.  
  1735. Outputs:
  1736.  
  1737. project/outputs/media_face_clustering/
  1738. face_candidate_manifest_*.jsonl
  1739. face_detections_*.jsonl
  1740. clusters_*/
  1741. media_face_clusters.sqlite
  1742. media_face_clusters.summary.json
  1743. STATUS.md
  1744.  
  1745. Rules:
  1746.  
  1747. - Cluster IDs are anonymous by default.
  1748. - Do not assign person names without explicit visual/manual review.
  1749. - Keep all detections so stricter/looser clustering can be rerun without
  1750. redetecting.
  1751. - Treat large clusters as candidates, not identity truth.
  1752.  
  1753. Pipeline: context packs
  1754. =======================
  1755.  
  1756. Purpose
  1757. -------
  1758.  
  1759. Build compressed, source-aware context packs for future model sessions.
  1760.  
  1761. Output family:
  1762.  
  1763. project/outputs/context_pack/
  1764. <subject_slug>_context_prompt_primary_source_pack.md
  1765. <subject_slug>_context_prompt_primary_source_pack_v2.md
  1766. <subject_slug>_context_prompt_primary_source_pack_v3.md
  1767. <subject_slug>_context_prompt_primary_source_pack_v4.md
  1768. <subject_slug>_context_prompt_primary_source_pack_v4_medium_50k.md
  1769. *_stats.json
  1770. *_coverage_audit.md
  1771.  
  1772. Builder family:
  1773.  
  1774. python3 project/scripts/build_personal_context_doc.py
  1775. python3 project/scripts/build_personal_context_doc_v2.py
  1776. python3 project/scripts/build_personal_context_doc_v3.py
  1777. python3 project/scripts/build_personal_context_doc_v4.py
  1778. python3 project/scripts/build_personal_context_doc_medium.py
  1779.  
  1780. Evolution:
  1781.  
  1782. - v1: primary source moment selections and supplemental sections.
  1783. - v2: better dedupe, source displays, special high-priority source sections,
  1784. life-moment sections, synthesis backfill, coverage audit.
  1785. - v3: expanded life sections and source-stat coverage improvements.
  1786. - v4: source-card taxonomy refresh after moving from legacy relationship slices
  1787. to canonical source-family cards.
  1788. - medium: deterministic broad sample from the full v4 pack, sized for default
  1789. seeded chats; retrieval is still required for specific claims.
  1790.  
  1791. Rules:
  1792.  
  1793. - Context packs are orientation, not primary truth.
  1794. - Include source-use cautions and evidence hierarchy.
  1795. - Preserve source map and section boundaries.
  1796. - Deduplicate repeated source blocks.
  1797. - Record stats: token counts, byte size, SHA-256, selected groups, omitted
  1798. groups, version, source pack hash.
  1799. - Write coverage audits listing included/skipped source groups.
  1800. - Do not include old per-person source slices in the active v4 taxonomy unless
  1801. deliberately re-enabled.
  1802.  
  1803. Docs/notes/voice specialized packs:
  1804.  
  1805. python3 project/scripts/build_docs_notes_voice_context_pack.py
  1806. python3 project/scripts/tag_notes_and_voice_sources.py build-manifest
  1807. python3 project/scripts/tag_notes_and_voice_sources.py ...
  1808. python3 project/scripts/build_tagged_docs_notes_voice_context_pack.py
  1809.  
  1810. Use these to:
  1811.  
  1812. - rank notes/voice sources by emotional, self-reflective, chronology, and
  1813. inclusion salience;
  1814. - include transcripts with quality flags;
  1815. - exclude tiny/empty/repeated-artifact voice cases;
  1816. - write audit files explaining included/excluded items.
  1817.  
  1818. Pipeline: contrastive synthesis
  1819. ===============================
  1820.  
  1821. Purpose
  1822. -------
  1823.  
  1824. Build a staged, skeptical, contrastive synthesis over source families.
  1825.  
  1826. Command sequence:
  1827.  
  1828. python3 project/scripts/gpt55_synthesis.py plan
  1829. python3 project/scripts/gpt55_synthesis.py status --verbose
  1830. python3 project/scripts/gpt55_synthesis.py run --stage source-cards
  1831. python3 project/scripts/gpt55_synthesis.py run --stage primary-moments
  1832. python3 project/scripts/gpt55_synthesis.py run --stage claim-ledger
  1833. python3 project/scripts/gpt55_synthesis.py run --stage contrasts
  1834. python3 project/scripts/gpt55_synthesis.py run --stage theme-dossiers
  1835. python3 project/scripts/gpt55_synthesis.py run --stage audit
  1836. python3 project/scripts/gpt55_synthesis.py run --stage final-report
  1837.  
  1838. Or:
  1839.  
  1840. python3 project/scripts/gpt55_synthesis.py run --stage all
  1841.  
  1842. Dry run:
  1843.  
  1844. python3 project/scripts/gpt55_synthesis.py run --stage source-cards --only <source_id> --dry-run
  1845.  
  1846. Fetch interrupted background response:
  1847.  
  1848. python3 project/scripts/gpt55_synthesis.py fetch --response-id <response_id> --stage <stage> --job-id <job_id>
  1849.  
  1850. Outputs:
  1851.  
  1852. project/outputs/synthesis/
  1853. synthesis_manifest.json
  1854. synthesis_token_report.md
  1855. synthesis_run_summary.json
  1856. inputs/
  1857. prompts/
  1858. raw/
  1859. response_ids/
  1860. errors/
  1861. source_cards/
  1862. primary_source_moments/
  1863. claim_ledger/claim_ledger.jsonl
  1864. relationship_contrasts/
  1865. theme_dossiers/
  1866. audit/skeptical_audit.md
  1867. final/final_report.md
  1868. README.md
  1869. synthesis_update_status.md
  1870.  
  1871. Stage design:
  1872.  
  1873. 1. source-cards
  1874. - one card per source family;
  1875. - asks what the source uniquely reveals, hides, contradicts, and biases;
  1876. - includes candidate primary-source moments.
  1877.  
  1878. 2. primary-moments
  1879. - reviewable index of high-value source moments;
  1880. - may be deterministic fallback extracted from source-card JSONL blocks if
  1881. a model run fails.
  1882.  
  1883. 3. claim-ledger
  1884. - normalized JSONL claims with source refs, confidence, and risk.
  1885.  
  1886. 4. contrasts
  1887. - compares source families and relationship/platform contexts.
  1888.  
  1889. 5. theme-dossiers
  1890. - cross-source dossiers for major themes such as work/agency, family,
  1891. friendship, attachment/intimacy, conflict/repair, intellectual style,
  1892. development over time, and private-vs-interpersonal self.
  1893.  
  1894. 6. audit
  1895. - skeptical attack on the emerging analysis.
  1896.  
  1897. 7. final-report
  1898. - long-form synthesis downstream of all previous stages.
  1899.  
  1900. Rules:
  1901.  
  1902. - Save every prompt before API submission.
  1903. - Skip existing outputs unless --force is passed.
  1904. - Do not overwrite a good stage output with an incomplete/failed model response.
  1905. - Track response IDs and raw responses separately.
  1906. - Use max request/body size guards.
  1907. - Use active canonical source-family cards, not old per-person message slices,
  1908. unless explicitly enabled for a special rerun.
  1909. - Keep relationship/platform contrasts and theme dossiers marked with their
  1910. run vintage if only part of the pipeline is refreshed.
  1911.  
  1912. Pipeline: psychological and interpretive analysis
  1913. =================================================
  1914.  
  1915. Purpose
  1916. -------
  1917.  
  1918. Build deeper formulation layers from evidence, while keeping them explicitly
  1919. separate from primary sources.
  1920.  
  1921. GPT-style psychoanalysis harness
  1922. --------------------------------
  1923.  
  1924. Commands:
  1925.  
  1926. python3 project/scripts/gpt55_psychoanalysis.py prepare
  1927. python3 project/scripts/gpt55_psychoanalysis.py plan --stage all
  1928. python3 project/scripts/gpt55_psychoanalysis.py run --stage relationships
  1929. python3 project/scripts/gpt55_psychoanalysis.py run --stage self_dossiers
  1930. python3 project/scripts/gpt55_psychoanalysis.py run --stage ledgers
  1931. python3 project/scripts/gpt55_psychoanalysis.py run --stage audit
  1932. python3 project/scripts/gpt55_psychoanalysis.py run --stage final
  1933. python3 project/scripts/gpt55_psychoanalysis.py extract-ledgers
  1934. python3 project/scripts/gpt55_psychoanalysis.py status
  1935.  
  1936. Outputs:
  1937.  
  1938. project/outputs/psychoanalysis/
  1939. input_manifest.json
  1940. run_manifest.json
  1941. inputs/
  1942. relationship_cohort.md
  1943. incomplete_relationships.md
  1944. relationship_packs/
  1945. relationship_dossiers/
  1946. self_dossiers/
  1947. ledgers/
  1948. consolidated_ledgers.md
  1949. extraction_summary.md
  1950. audit/skeptical_audit.md
  1951. final/final_report.md
  1952. literature/
  1953. RESUME.md
  1954. STATUS.md
  1955.  
  1956. Rules:
  1957.  
  1958. - Use prior synthesis as index/foil, not final authority.
  1959. - Build relationship packs from top relationships plus evidence summaries.
  1960. - Build self/theme dossiers across a fixed set of themes.
  1961. - Extract structured ledgers from narrative dossiers.
  1962. - Run a skeptical audit.
  1963. - Mark source-verification queues as open unless manually reviewed.
  1964.  
  1965. Fable/large-context review-first harness
  1966. ----------------------------------------
  1967.  
  1968. Commands:
  1969.  
  1970. python3 project/scripts/claude_fable_psychoanalysis.py prepare
  1971. python3 project/scripts/claude_fable_psychoanalysis.py build-intro
  1972. python3 project/scripts/claude_fable_psychoanalysis.py build-context
  1973. python3 project/scripts/claude_fable_psychoanalysis.py plan
  1974. python3 project/scripts/claude_fable_psychoanalysis.py test-stack --reprepare
  1975. python3 project/scripts/claude_fable_psychoanalysis.py status
  1976.  
  1977. Real model stages only after review:
  1978.  
  1979. python3 project/scripts/claude_fable_psychoanalysis.py run-profiles --provider <provider> --model <model> --confirm-cost --allow-large-context
  1980. python3 project/scripts/claude_fable_psychoanalysis.py discover-questions --provider <provider> --model <model> --confirm-cost --allow-large-context
  1981. python3 project/scripts/claude_fable_psychoanalysis.py run-question-round --round 1 --provider <provider> --model <model> --confirm-cost --allow-large-context
  1982. python3 project/scripts/claude_fable_psychoanalysis.py review-round --round 1 --provider <provider> --model <model> --confirm-cost --allow-large-context
  1983. python3 project/scripts/claude_fable_psychoanalysis.py synthesize --provider <provider> --model <model> --confirm-cost --allow-large-context
  1984.  
  1985. Rules:
  1986.  
  1987. - Default provider should be mock.
  1988. - Non-mock calls require --confirm-cost.
  1989. - Large contexts require --allow-large-context.
  1990. - Existing generated analyses are excluded from source packs by default.
  1991. - Orientation/profile material is tagged not-evidence.
  1992. - RAG leaves vector retrieval off unless explicitly enabled.
  1993. - API keys are environment variables, not project files.
  1994.  
  1995. Open-ended question pipeline
  1996. ----------------------------
  1997.  
  1998. Commands:
  1999.  
  2000. python3 project/scripts/open_ended_questions.py generate
  2001. python3 project/scripts/open_ended_questions.py answer-batches
  2002. python3 project/scripts/open_ended_questions.py synthesize
  2003. python3 project/scripts/open_ended_questions.py status
  2004.  
  2005. Outputs:
  2006.  
  2007. project/outputs/open_ended_questions/
  2008. question_bank.jsonl
  2009. question_bank.md
  2010. answers/
  2011. all_answers.md
  2012. answer_synthesis.md
  2013. prompts/
  2014. raw_responses/
  2015. STATUS.md
  2016.  
  2017. Purpose:
  2018.  
  2019. - Generate empirical/open questions from existing analysis.
  2020. - Answer them in batches using source-aware context.
  2021. - Synthesize what remains unresolved.
  2022.  
  2023. Introspective question pipeline
  2024. -------------------------------
  2025.  
  2026. Commands:
  2027.  
  2028. python3 project/scripts/introspective_questions.py extract
  2029. python3 project/scripts/introspective_questions.py pack-batches
  2030. python3 project/scripts/introspective_questions.py run-batches
  2031. python3 project/scripts/introspective_questions.py run-final
  2032. python3 project/scripts/introspective_questions.py status
  2033.  
  2034. Outputs:
  2035.  
  2036. project/outputs/introspective_questions/
  2037. candidates/
  2038. candidate_summary.json
  2039. candidate_pack.md
  2040. all_question_candidates.jsonl
  2041. dedup_question_candidates.jsonl
  2042. analysis/
  2043. question_cluster_ledger.jsonl
  2044. introspective_question_report.md
  2045. prompts/
  2046. raw_responses/
  2047. STATUS.md
  2048.  
  2049. Purpose:
  2050.  
  2051. - Extract question-like utterances from messages, writing, voice transcripts,
  2052. and prior AI chats.
  2053. - Deduplicate and cluster them.
  2054. - Produce a report on recurring self-questions and unresolved concerns.
  2055.  
  2056. Final conclusions layer
  2057. -----------------------
  2058.  
  2059. Inputs:
  2060.  
  2061. - psychoanalysis final report and skeptical audit;
  2062. - psychoanalysis consolidated ledgers;
  2063. - introspective question report and ledger;
  2064. - open-ended answer synthesis and answer index;
  2065. - work-review/consistency review outputs.
  2066.  
  2067. Outputs:
  2068.  
  2069. project/outputs/final_conclusions/
  2070. final_conclusions_report.md
  2071. final_claims.jsonl
  2072. relationship_takeaways.jsonl
  2073. operating_rules.jsonl
  2074. unresolved_questions.jsonl
  2075. ledger_parse_summary.json
  2076. final_conclusions.prompt.md
  2077. final_conclusions_final.json
  2078. STATUS.md
  2079.  
  2080. Rules:
  2081.  
  2082. - Final conclusions are high-level, model-generated interpretation.
  2083. - Keep structured ledgers parseable.
  2084. - Keep unresolved questions explicit.
  2085. - Do not collapse uncertainty into advice.
  2086.  
  2087. Pipeline: RAG and search
  2088. ========================
  2089.  
  2090. Purpose
  2091. -------
  2092.  
  2093. Build a local retrieval index across raw personal roots and curated generated
  2094. evidence layers. Use it for search, chat, source expansion, and evidence
  2095. review.
  2096.  
  2097. Builder:
  2098.  
  2099. project/scripts/personalos_rag.py
  2100.  
  2101. Primary index:
  2102.  
  2103. project/outputs/rag/personalos_fts.sqlite
  2104.  
  2105. Vector files:
  2106.  
  2107. project/outputs/rag/personalos_fts_<model>_<dimensions>.vectors.npy
  2108. project/outputs/rag/personalos_fts_<model>_<dimensions>.vectors.json
  2109.  
  2110. Default scan roots:
  2111.  
  2112. - personal/
  2113. - project/docs/
  2114. - selected project/outputs subdirectories:
  2115. chronology, synthesis, context_pack, current_state, daily_activity_index,
  2116. final_conclusions, life_moment_examples, hierarchical_summaries, instagram,
  2117. introspective_questions, location_index, media_captioning selected outputs,
  2118. media_face_clustering selected outputs, open_ended_questions, psychoanalysis,
  2119. relationship_graph, voice, work_review, birthdays
  2120. - project/pipelines/merged_message_histories/outputs/
  2121.  
  2122. Default exclusions:
  2123.  
  2124. - prompts unless deliberately indexed for audit;
  2125. - raw model responses;
  2126. - response IDs;
  2127. - errors;
  2128. - batch internals;
  2129. - cached media binaries;
  2130. - broad duplicate acquisition tables that do not make useful chunks;
  2131. - external symlink targets;
  2132. - prior AI chats unless user asks about prior AI/chat behavior.
  2133.  
  2134. Index commands:
  2135.  
  2136. python3 project/scripts/personalos_rag.py index --verbose --include-legacy-globs
  2137. python3 project/scripts/personalos_rag.py stats
  2138. python3 project/scripts/personalos_rag.py refresh-source --path-contains <path_fragment>
  2139.  
  2140. Embedding cost-safe commands:
  2141.  
  2142. python3 project/scripts/personalos_rag.py estimate-embed --missing-only
  2143. python3 project/scripts/personalos_rag.py estimate-embed --missing-only --embedding-scope all
  2144. python3 project/scripts/personalos_rag.py embed --limit 100 --batch-size 128 --concurrency 4 --verbose
  2145. python3 project/scripts/personalos_rag.py embed --embedding-scope all --batch-size 128 --concurrency 8 --max-usd <budget> --verbose
  2146. python3 project/scripts/personalos_rag.py build-vector-index
  2147.  
  2148. Safety:
  2149.  
  2150. - estimate-embed and embed default to high-salience/default scope.
  2151. - Unbounded paid embedding runs fail unless given --limit, --max-usd, or
  2152. --yes-full-embed.
  2153. - Preserve existing embeddings during rebuild when chunk hashes match.
  2154. - Vector search must score all currently embedded chunks matching selected
  2155. filters/scope, not only a tiny FTS candidate slice.
  2156.  
  2157. Search commands:
  2158.  
  2159. python3 project/scripts/personalos_rag.py search "query"
  2160. python3 project/scripts/personalos_rag.py vector-search "query"
  2161. python3 project/scripts/personalos_rag.py hybrid-search "query"
  2162. python3 project/scripts/personalos_rag.py grep "exact phrase"
  2163. python3 project/scripts/personalos_rag.py open <chunk_id>
  2164.  
  2165. Retrieval modes:
  2166.  
  2167. - evidence: multi-lane retrieval with exact variants, FTS, source-family lanes,
  2168. vector lane, fusion, and grading.
  2169. - hybrid: FTS plus vector with reciprocal-rank fusion.
  2170. - fts: SQLite FTS5.
  2171. - vector: direct semantic search.
  2172. - exact: substring grep.
  2173.  
  2174. Embedding salience:
  2175.  
  2176. - default: raw high-signal personal material and curated evidence.
  2177. - selective: useful expanded layers such as daily activity, location, message
  2178. histories, synthesis, psychoanalysis, finance/health, broad media captions.
  2179. - index_only: orientation/structural layers such as context packs, relationship
  2180. graph, summary aliases, face metadata, profile seeds, project docs.
  2181. - exclude: prior AI chats and other material excluded unless explicitly asked.
  2182.  
  2183. Evidence roles:
  2184.  
  2185. - primary: raw/local evidence.
  2186. - derived_index: source-aware indexes/summaries.
  2187. - analysis: model-generated analysis artifacts.
  2188. - orientation: context packs or profile material.
  2189. - prior_chat: previous AI chats.
  2190.  
  2191. Formatting and chunking:
  2192.  
  2193. - Embed source-aware readable text, not raw JSON when a deterministic formatter
  2194. can preserve meaning better.
  2195. - Preserve line breaks in chat transcripts.
  2196. - Split chat-like sources around conversation boundaries and large time gaps.
  2197. - Preserve stable local line ranges and chunk expansion handles.
  2198. - Format media captions, media text, face records, chronology rows, finance
  2199. rows, location rows, relationship graph rows, and generic JSONL/CSV as
  2200. readable evidence blocks.
  2201. - Strip raw response envelopes, request IDs, usage blobs, and batch metadata
  2202. from RAG-facing text.
  2203.  
  2204. Regression:
  2205.  
  2206. python3 project/scripts/eval_personalos_chat_retrieval.py
  2207. python3 project/scripts/eval_personalos_chat_retrieval.py --local-checks-only --include-local-vector-check
  2208.  
  2209. Check:
  2210.  
  2211. - expected paths for representative queries;
  2212. - evidence role assignment;
  2213. - line-preserving previews;
  2214. - duplicate exclusion for seen chunks;
  2215. - prior AI chat exclusion;
  2216. - vector matrix row counts and scoring scope.
  2217.  
  2218. Chat system
  2219. ===========
  2220.  
  2221. Purpose
  2222. -------
  2223.  
  2224. Provide a local web chat over the PersonalOS evidence index without injecting
  2225. the entire archive every turn.
  2226.  
  2227. Server:
  2228.  
  2229. project/scripts/personalos_chat_server.py
  2230.  
  2231. Normal use is through unified app server. Standalone debug command:
  2232.  
  2233. python3 project/scripts/personalos_chat_server.py --port 8780
  2234.  
  2235. Default chat flow:
  2236.  
  2237. 1. First turn seeds the medium context pack.
  2238. 2. Default rag_mode is tool: model receives retrieval/open tools and should
  2239. retrieve for substantive archive/advice/chronology/relationship/work/health
  2240. questions unless retrieval is clearly unnecessary.
  2241. 3. The model writes purpose-built retrieval queries rather than passing the raw
  2242. user prompt as the only query.
  2243. 4. Retrieval uses embedding-backed evidence retrieval over high-salience scope
  2244. by default.
  2245. 5. Inject a retrieval digest with evidence roles, source mix, warnings,
  2246. previews, and expansion handles.
  2247. 6. The model answers from retrieved evidence or states uncertainty.
  2248.  
  2249. Context modes:
  2250.  
  2251. - primer: minimal orientation.
  2252. - context_pack: default medium pack.
  2253. - full: full v4 pack for explicit deep-context sessions.
  2254.  
  2255. Follow-up behavior:
  2256.  
  2257. - Exclude chunks already injected earlier in the same session unless explicitly
  2258. opened for expansion.
  2259. - Keep chat sessions under personal/conversations/ai_chats/.
  2260.  
  2261. Dashboard and local app
  2262. =======================
  2263.  
  2264. Purpose
  2265. -------
  2266.  
  2267. Provide a local browser interface over generated aggregate data plus opt-in raw
  2268. review, search, chat, media review, and relationship graph.
  2269.  
  2270. Static app:
  2271.  
  2272. project/sites/personalos_app/
  2273. index.html
  2274. app.js
  2275. styles.css
  2276. data/dashboard_data.json
  2277. data/daily_activity_index.json
  2278. vendor/
  2279. globe/
  2280. uber-map/
  2281. presence-map/
  2282. final-conclusions/
  2283.  
  2284. Regenerate data:
  2285.  
  2286. python3 project/scripts/build_current_state_dashboard.py
  2287.  
  2288. Serve:
  2289.  
  2290. python3 project/scripts/personalos_app_server.py --port 8765
  2291.  
  2292. With auth:
  2293.  
  2294. PERSONALOS_AUTH_PASSWORD='<strong password>' \
  2295. python3 project/scripts/personalos_app_server.py --host 127.0.0.1 --port 8765
  2296.  
  2297. Prefer password file for persistent launch:
  2298.  
  2299. PERSONALOS_AUTH_PASSWORD_FILE=<0600_password_file> \
  2300. python3 project/scripts/personalos_app_server.py --host 127.0.0.1 --port 8765
  2301.  
  2302. Single-instance behavior:
  2303.  
  2304. - The server should be single-instance guarded by default.
  2305. - Re-running should print the existing URL and exit.
  2306. - Use --allow-multiple and a different port only when explicitly needed.
  2307.  
  2308. Mounted routes:
  2309.  
  2310. - /: dashboard.
  2311. - /chat/: local chat.
  2312. - /search/: RAG search UI.
  2313. - /media-captions/: media caption review.
  2314. - /relationship-graph/: graph viewer.
  2315. - /api/raw-stream: raw stream API.
  2316. - /api/raw-media/photo/<id>: local photo derivative fetch.
  2317. - /api/raw-email-body: on-demand email body expansion from mbox offsets.
  2318.  
  2319. Sensitive APIs:
  2320.  
  2321. - Raw-content endpoints must be loopback-only by default.
  2322. - Use --allow-remote-sensitive-api only for an explicitly trusted local network
  2323. debugging session.
  2324.  
  2325. Dashboard data builder
  2326. ----------------------
  2327.  
  2328. Builder:
  2329.  
  2330. project/scripts/build_current_state_dashboard.py
  2331.  
  2332. Inputs:
  2333.  
  2334. - filesystem metadata;
  2335. - source counts;
  2336. - pipeline status files;
  2337. - aggregate JSON/CSV/Markdown status artifacts;
  2338. - synthesis token/source metadata;
  2339. - message analytics;
  2340. - daily activity index;
  2341. - RAG stats;
  2342. - chronology/travel/location artifacts;
  2343. - finance summaries;
  2344. - media summaries.
  2345.  
  2346. Outputs:
  2347.  
  2348. project/outputs/current_state/dashboard_data.json
  2349. project/outputs/current_state/daily_activity_index_app.json
  2350. project/outputs/current_state/README.md
  2351. project/sites/personalos_app/data/dashboard_data.json
  2352. project/sites/personalos_app/data/daily_activity_index.json
  2353.  
  2354. Rules:
  2355.  
  2356. - Metadata-only by default.
  2357. - Store counts, labels, paths, statuses, token estimates, and aggregate metrics.
  2358. - Do not embed raw message bodies or source document bodies.
  2359. - Validate generated payload before writing dashboard copies.
  2360.  
  2361. Dashboard views
  2362. ---------------
  2363.  
  2364. Recommended sections:
  2365.  
  2366. - overview/source coverage;
  2367. - messages: distribution, timeline, platforms, people, email, Instagram;
  2368. - finance: overview, peaks, bank, PayPal, monthly combined ledger;
  2369. - chronology: overview, candidate datapoints, corrected trip timeline,
  2370. every-day timeline, movements, movement globe, presence map;
  2371. - raw stream: opt-in chronological scroll over source rows;
  2372. - raw plain: day-separated reader with content/source toggles;
  2373. - search: mounted RAG search;
  2374. - chat: mounted RAG chat;
  2375. - media captions: review/search surface;
  2376. - relationship graph: full graph viewer and compact relationship manager;
  2377. - final conclusions: structured high-level interpretation view.
  2378.  
  2379. Raw stream index
  2380. ----------------
  2381.  
  2382. The app consumes:
  2383.  
  2384. project/outputs/raw_stream/raw_stream_index.sqlite
  2385.  
  2386. It should contain chronological rows across messages, photos, calendar rows,
  2387. email headers, and other reviewable source rows, with enough offsets/paths to
  2388. load source content on demand.
  2389.  
  2390. Important gap to address in a clean recreation:
  2391.  
  2392. - The current app consumes this SQLite index, and daily activity can read it,
  2393. but the standalone builder is not clearly documented in the script index.
  2394. A clean implementation should add:
  2395.  
  2396. project/scripts/build_raw_stream_index.py
  2397.  
  2398. It should read normalized source artifacts and write:
  2399.  
  2400. raw_stream_index.sqlite
  2401. raw_stream_summary.json
  2402. README.md
  2403.  
  2404. Required fields:
  2405.  
  2406. stream_id
  2407. timestamp_utc
  2408. timestamp_local
  2409. date_local
  2410. source_family
  2411. source_type
  2412. source_path
  2413. source_locator
  2414. file_key
  2415. byte_start
  2416. byte_end
  2417. line_start
  2418. line_end
  2419. media_id
  2420. conversation_id
  2421. sender_label
  2422. is_from_me
  2423. title
  2424. preview
  2425. has_body
  2426. sensitivity
  2427.  
  2428. Search UI
  2429. ---------
  2430.  
  2431. Standalone fallback:
  2432.  
  2433. python3 project/scripts/personalos_search_server.py --host 127.0.0.1 --port 8770
  2434.  
  2435. Modes:
  2436.  
  2437. - semantic vector;
  2438. - hybrid;
  2439. - keyword FTS;
  2440. - exact.
  2441.  
  2442. Browser semantic search sends query text to the embedding API. Keyword and exact
  2443. stay local.
  2444.  
  2445. Relationship graph viewer
  2446. -------------------------
  2447.  
  2448. Static viewer:
  2449.  
  2450. project/sites/relationship_graph/
  2451.  
  2452. Canonical data:
  2453.  
  2454. project/outputs/relationship_graph/relationship_graph.json
  2455.  
  2456. Mounted URL:
  2457.  
  2458. http://127.0.0.1:8765/relationship-graph/
  2459.  
  2460. Rules:
  2461.  
  2462. - Uses metadata-only graph JSON.
  2463. - Vendor graph renderer locally.
  2464. - Do not load message bodies.
  2465.  
  2466. Chronology globe, ride map, and presence map
  2467. --------------------------------------------
  2468.  
  2469. Movement globe:
  2470.  
  2471. project/sites/personalos_app/globe/
  2472.  
  2473. Inputs:
  2474.  
  2475. - dashboard_data.json;
  2476. - travel movement events;
  2477. - approximate track geometry;
  2478. - optional vendored Earth/land assets and Three.js.
  2479.  
  2480. Rules:
  2481.  
  2482. - Full-bleed scene or dense embedded panel.
  2483. - Verify nonblank desktop/mobile canvas.
  2484. - Use approximate tracks with confidence labels.
  2485.  
  2486. Ride map:
  2487.  
  2488. project/sites/personalos_app/uber-map/
  2489.  
  2490. Inputs:
  2491.  
  2492. - redacted rides visualizer data;
  2493. - optional flight legs;
  2494. - optional OSM road-route cache.
  2495.  
  2496. Commands:
  2497.  
  2498. python3 project/scripts/build_uber_ride_visualizer_data.py
  2499. python3 project/scripts/build_uber_road_route_cache.py
  2500. python3 project/scripts/render_uber_route_qa_contact_sheet.py
  2501.  
  2502. Rules:
  2503.  
  2504. - Redacted rounded endpoints only.
  2505. - OSM routes are approximate recomputations, not actual telemetry.
  2506.  
  2507. Presence map:
  2508.  
  2509. project/sites/personalos_app/presence-map/
  2510.  
  2511. Command:
  2512.  
  2513. python3 project/scripts/build_presence_map_data.py
  2514.  
  2515. Inputs:
  2516.  
  2517. - rides;
  2518. - flight legs;
  2519. - photos/location overlays;
  2520. - endpoint overlays.
  2521.  
  2522. Privacy and security tooling
  2523. ============================
  2524.  
  2525. Secret scanner
  2526. --------------
  2527.  
  2528. Script:
  2529.  
  2530. project/scripts/scan_generated_artifact_secrets.py
  2531.  
  2532. Default command:
  2533.  
  2534. python3 project/scripts/scan_generated_artifact_secrets.py
  2535.  
  2536. Specific artifact:
  2537.  
  2538. python3 project/scripts/scan_generated_artifact_secrets.py project/outputs/path/to/artifact
  2539.  
  2540. Default scope:
  2541.  
  2542. - project/outputs;
  2543. - project/tmp;
  2544. - pipeline-owned outputs directories.
  2545.  
  2546. Default exclusions:
  2547.  
  2548. - personal/raw;
  2549. - personal/conversations;
  2550. - acquisition archives such as large mbox files unless explicitly targeted.
  2551.  
  2552. Flags:
  2553.  
  2554. --include-raw-source
  2555. --include-acquisition-archives
  2556. --report <path>
  2557. --no-status
  2558.  
  2559. Rule families:
  2560.  
  2561. - private key blocks;
  2562. - OpenAI/Anthropic/GitHub/AWS/Google/Slack/Stripe tokens;
  2563. - JWTs;
  2564. - sensitive variable assignments.
  2565.  
  2566. Behavior:
  2567.  
  2568. - Report only path, line, rule ID, severity, and description.
  2569. - Do not print matched secret value.
  2570. - Write redacted report under project/outputs/privacy/.
  2571. - Update STATUS.md unless disabled.
  2572. - A nonzero result blocks external movement until remediated, rotated, or
  2573. explicitly false-positive reviewed and rescanned.
  2574.  
  2575. Backup and change tracking
  2576. --------------------------
  2577.  
  2578. Recommended:
  2579.  
  2580. - encrypted local/cloud backup for raw archives;
  2581. - restore test documentation;
  2582. - narrow git repository for code/docs only, if any;
  2583. - .gitignore raw archives, conversations, RAG indexes, large generated packs,
  2584. credentials, model responses, and local cache files.
  2585.  
  2586. Do not initialize full-workspace git over raw sources and large generated
  2587. private outputs unless you have a strict ignore policy.
  2588.  
  2589. Deployment/proxy layer
  2590. ======================
  2591.  
  2592. There is an optional deploy shim for a remote proxy:
  2593.  
  2594. project/deploy/personalos_vercel_proxy/
  2595. package.json
  2596. vercel.json
  2597. api/proxy.js
  2598.  
  2599. Purpose:
  2600.  
  2601. - Relay HTTP requests from a deployed endpoint to a configured PersonalOS origin
  2602. URL.
  2603.  
  2604. Rules:
  2605.  
  2606. - Do not commit provider-generated .vercel metadata.
  2607. - Do not commit project IDs or organization IDs.
  2608. - Configure upstream via environment variable such as PERSONALOS_ORIGIN_URL.
  2609. - Ensure the origin server itself has strong authentication.
  2610. - Proxy responses should set no-store cache headers.
  2611. - Use this only if you understand the privacy risk of making any PersonalOS
  2612. surface reachable over the network.
  2613.  
  2614. Miscellaneous local utilities
  2615. =============================
  2616.  
  2617. Discord export helper
  2618. ---------------------
  2619.  
  2620. The workspace can vendor a local copy of DiscordChatExporter or an equivalent
  2621. tool under:
  2622.  
  2623. project/tools/discordchatexporter/
  2624.  
  2625. Use it to create Discord JSON exports for DMs, group DMs, and selected server
  2626. snippets. Keep tool binaries out of RAG indexing. Keep raw exported JSON under
  2627. personal/raw/messages/discord_dce/ and normalize into merged-message histories
  2628. from there.
  2629.  
  2630. Browser path blocker
  2631. --------------------
  2632.  
  2633. The workspace can include a small local focus utility:
  2634.  
  2635. project/scripts/x-home-blocker/
  2636.  
  2637. Purpose:
  2638.  
  2639. - Block one distracting browser path such as a social feed home URL without
  2640. blocking the entire host or normal deep links.
  2641.  
  2642. Implementation:
  2643.  
  2644. - Browser enterprise policy for Chrome/Firefox URL blocklists.
  2645. - Optional Chrome extension for single-page-app navigation where URL changes
  2646. happen without a full reload.
  2647. - macOS configuration profile alternative if desired.
  2648.  
  2649. Rules:
  2650.  
  2651. - Use placeholder install commands in docs:
  2652.  
  2653. sudo <workspace>/project/scripts/x-home-blocker/install.sh
  2654. sudo <workspace>/project/scripts/x-home-blocker/uninstall.sh
  2655.  
  2656. - Do not hard-code local usernames in the utility README.
  2657. - Keep browser-control utilities separate from PersonalOS evidence pipelines.
  2658.  
  2659. Testing and verification
  2660. ========================
  2661.  
  2662. Local tests:
  2663.  
  2664. python3 -m unittest discover project/tests
  2665.  
  2666. Current test focus:
  2667.  
  2668. - app server raw stream route behavior;
  2669. - search server behavior.
  2670.  
  2671. Add tests for:
  2672.  
  2673. - path helper canonicalization;
  2674. - message event schema validation;
  2675. - identity override resolution;
  2676. - chronology candidate filter behavior;
  2677. - RAG formatter output for each source family;
  2678. - secret scanner redaction;
  2679. - dashboard payload validation;
  2680. - raw API loopback/auth behavior.
  2681.  
  2682. Pipeline validation checklist:
  2683.  
  2684. - JSONL parse checks for every JSONL output.
  2685. - Required schema field checks.
  2686. - Duplicate ID checks.
  2687. - Missing timestamp checks.
  2688. - Output count parity with source manifests.
  2689. - No raw message bodies in metadata-only reports.
  2690. - No raw credentials in generated artifacts.
  2691. - Model response status complete before overwriting final stage outputs.
  2692. - Empty/error files reviewed.
  2693. - Dashboard JSON validates before copying to site.
  2694. - RAG embedding counts match chunk counts for selected scope.
  2695. - Vector manifest row order matches vector matrix row order.
  2696. - Browser screenshots for dashboard/globe/maps/media views after UI changes.
  2697.  
  2698. Recreate-from-zero runbook
  2699. ==========================
  2700.  
  2701. Phase 0: create workspace
  2702. -------------------------
  2703.  
  2704. 1. Create <workspace>.
  2705. 2. Add README.md, AGENTS.md, personal/, project/.
  2706. 3. Add project/scripts/personalos_paths.py.
  2707. 4. Add project/docs/PRIVACY_SECURITY_CHECKLIST.md.
  2708. 5. Add project/docs/LEGACY_PATHS.md if migrating older paths.
  2709. 6. Add project/config/subject.json.
  2710. 7. Add project/config/identity_seed_events.jsonl.
  2711.  
  2712. Phase 1: collect raw exports
  2713. ----------------------------
  2714.  
  2715. 1. Put writing exports under personal/raw/writing.
  2716. 2. Put message exports under personal/raw/messages.
  2717. 3. Put voice exports/transcripts under personal/raw/voice.
  2718. 4. Put calendar/mail/photos/location exports under relevant raw folders or
  2719. chronology acquisition folders.
  2720. 5. Put finance exports under personal/raw/finance.
  2721. 6. Put health exports under personal/raw/health only if desired.
  2722. 7. Do not normalize in place.
  2723.  
  2724. Phase 2: normalize platform-specific sources
  2725. --------------------------------------------
  2726.  
  2727. 1. Export/normalize iMessage or equivalent local chat DB.
  2728. 2. Export/normalize Signal.
  2729. 3. Export/normalize Discord with account metadata when available.
  2730. 4. Export/normalize Snapchat chat/activity.
  2731. 5. Run Instagram normalizer.
  2732. 6. Export/normalize Twitter/X direct/group messages and public account data.
  2733. 7. Build voice manifest and repeat-artifact triage.
  2734. 8. Extract finance and rideshare/location derived redacted outputs.
  2735. 9. Export calendar/photos metadata.
  2736. 10. Index mail mbox and curate evidence.
  2737.  
  2738. Phase 3: build message graph
  2739. ----------------------------
  2740.  
  2741. 1. Run merged message histories.
  2742. 2. Review identity queues.
  2743. 3. Edit identity_overrides.csv.
  2744. 4. Rerun merged histories.
  2745. 5. Run activity reports.
  2746. 6. Run salience windows.
  2747. 7. Build relationship graph.
  2748. 8. Build friend mention index if desired.
  2749.  
  2750. Phase 4: build chronology/location/activity
  2751. -------------------------------------------
  2752.  
  2753. 1. Build chronology seed.
  2754. 2. Build chronology candidate pack.
  2755. 3. Write initial chronology draft from candidate pack.
  2756. 4. Integrate mail/finance/location/social addenda.
  2757. 5. Build major trip/travel movement ledgers if location data supports it.
  2758. 6. Build daily location model.
  2759. 7. Build daily activity index.
  2760. 8. Build common location index, naming, resolution, and activity layers.
  2761.  
  2762. Phase 5: build media layers
  2763. ---------------------------
  2764.  
  2765. 1. Build media caption manifest.
  2766. 2. Run small sample caption QA.
  2767. 3. Run full batch captioning with budget controls.
  2768. 4. Collect/retry/index captions.
  2769. 5. Run text/OCR pass if needed.
  2770. 6. Build important media text.
  2771. 7. Run emotional/life-moment tag layer.
  2772. 8. Run face candidate/detect/cluster/index if needed.
  2773.  
  2774. Phase 6: build life moments, context packs, synthesis
  2775. -----------------------------------------------------
  2776.  
  2777. 1. Extract life-moment examples.
  2778. 2. Build source tagging for notes/voice.
  2779. 3. Build docs/notes/voice context pack.
  2780. 4. Build full context pack v4.
  2781. 5. Build medium default context pack.
  2782. 6. Run synthesis plan.
  2783. 7. Run synthesis source cards.
  2784. 8. Run primary moments, claim ledger, contrasts, theme dossiers, audit, and
  2785. final report.
  2786. 9. Update synthesis README/status.
  2787.  
  2788. Phase 7: build interpretive layers
  2789. ----------------------------------
  2790.  
  2791. 1. Run psychoanalysis prepare/plan.
  2792. 2. Run relationship dossiers.
  2793. 3. Run self/theme dossiers.
  2794. 4. Extract/consolidate ledgers.
  2795. 5. Run skeptical audit.
  2796. 6. Run final report.
  2797. 7. Run introspective questions.
  2798. 8. Run open-ended questions.
  2799. 9. Run final conclusions.
  2800. 10. Keep all interpretation clearly downstream of evidence.
  2801.  
  2802. Phase 8: build RAG, dashboard, and app
  2803. --------------------------------------
  2804.  
  2805. 1. Build raw stream index.
  2806. 2. Build RAG index.
  2807. 3. Estimate embeddings.
  2808. 4. Embed with limit/budget first.
  2809. 5. Embed full selected scope only with explicit confirmation/budget.
  2810. 6. Build vector index.
  2811. 7. Run retrieval regression checks.
  2812. 8. Build dashboard data.
  2813. 9. Serve unified app locally.
  2814. 10. Verify dashboard/search/chat/media/graph/globe/map views.
  2815.  
  2816. Phase 9: pre-export review
  2817. --------------------------
  2818.  
  2819. 1. Run secret scan.
  2820. 2. Review generated artifacts for personal identifiers and sensitive facts.
  2821. 3. Create purpose-specific redacted artifact if sharing.
  2822. 4. Do not share canonical generated outputs directly.
  2823.  
  2824. Recommended status docs
  2825. =======================
  2826.  
  2827. Every major output family should have README.md or STATUS.md:
  2828.  
  2829. project/outputs/README.md
  2830. project/outputs/synthesis/README.md
  2831. project/outputs/synthesis/synthesis_update_status.md
  2832. project/outputs/chronology/STATUS.md
  2833. project/outputs/daily_activity_index/README.md
  2834. project/outputs/instagram/STATUS.md
  2835. project/outputs/life_moment_examples/STATUS.md
  2836. project/outputs/rag/README.md
  2837. project/outputs/current_state/README.md
  2838. project/outputs/relationship_graph/STATUS.md
  2839. project/outputs/relationship_chunking/STATUS.md
  2840. project/outputs/voice/STATUS.md
  2841. project/outputs/media_captioning/RUN_STATUS.md
  2842. project/outputs/media_face_clustering/STATUS.md
  2843. project/outputs/location_index/README.md
  2844. project/outputs/privacy/STATUS.md
  2845. project/pipelines/merged_message_histories/STATUS.md
  2846. project/outputs/psychoanalysis/STATUS.md
  2847. project/outputs/final_conclusions/STATUS.md
  2848.  
  2849. Status docs should include:
  2850.  
  2851. - generated_at/updated_at;
  2852. - command used;
  2853. - input roots;
  2854. - output paths;
  2855. - counts;
  2856. - validation checks;
  2857. - known gaps/caveats;
  2858. - rerun instructions;
  2859. - privacy notes.
  2860.  
  2861. Output directory catalog
  2862. ========================
  2863.  
  2864. Use these cross-project output families:
  2865.  
  2866. project/outputs/ad_hoc/
  2867. One-off outputs that do not yet deserve a stable pipeline family.
  2868.  
  2869. project/outputs/birthdays/
  2870. Birthday/evidence review artifacts.
  2871.  
  2872. project/outputs/chronology/
  2873. Seed/candidate events, chronology drafts, travel/location movement,
  2874. acquisition outputs, visual timelines, addenda.
  2875.  
  2876. project/outputs/context_pack/
  2877. Full and medium source-aware context packs plus stats/audits.
  2878.  
  2879. project/outputs/current_state/
  2880. Dashboard snapshot and current state reports.
  2881.  
  2882. project/outputs/daily_activity_index/
  2883. Day-by-day evidence packets, model labels, final ledgers.
  2884.  
  2885. project/outputs/final_conclusions/
  2886. Final integrated report and structured ledgers.
  2887.  
  2888. project/outputs/hierarchical_summaries/
  2889. Summary nodes, edges, aliases, life-moment candidates.
  2890.  
  2891. project/outputs/instagram/
  2892. Derived Instagram DMs/public/attention/chronology/identity outputs.
  2893.  
  2894. project/outputs/introspective_questions/
  2895. Extracted self-question candidates, batches, final report.
  2896.  
  2897. project/outputs/life_moment_examples/
  2898. Extracted life-moment examples and source packs.
  2899.  
  2900. project/outputs/location_index/
  2901. Common places, resolution, activity, naming.
  2902.  
  2903. project/outputs/media_captioning/
  2904. Captions, text pass, important text, tag layers, SQLite.
  2905.  
  2906. project/outputs/media_face_clustering/
  2907. Face candidates, detections, clusters, SQLite.
  2908.  
  2909. project/outputs/open_ended_questions/
  2910. Question bank, answers, synthesis.
  2911.  
  2912. project/outputs/privacy/
  2913. Secret-scan reports and status.
  2914.  
  2915. project/outputs/psychoanalysis/
  2916. Deep formulation reports, ledgers, audit, final.
  2917.  
  2918. project/outputs/psychoanalysis_fable/
  2919. Review-first large-context source packs and staged prompts.
  2920.  
  2921. project/outputs/rag/
  2922. SQLite FTS/vector retrieval index and vector matrix.
  2923.  
  2924. project/outputs/raw_stream/
  2925. Raw stream SQLite index used by local app APIs.
  2926.  
  2927. project/outputs/relationship_chunking/
  2928. Model-ranked relationship chunks and compact subchunk packs.
  2929.  
  2930. project/outputs/relationship_graph/
  2931. Metadata-only graph, closeness rankings, friend mentions.
  2932.  
  2933. project/outputs/source_tagging/
  2934. Source ranking/tagging outputs for notes and voice.
  2935.  
  2936. project/outputs/synthesis/
  2937. Contrastive synthesis stages.
  2938.  
  2939. project/outputs/voice/
  2940. Voice manifest, transcript triage, cleanup candidates.
  2941.  
  2942. project/outputs/work_review/
  2943. Consistency review and project self-audit outputs.
  2944.  
  2945. Known implementation improvements for a clean clone
  2946. ===================================================
  2947.  
  2948. 1. Parameterize subject-specific filenames and identity facts.
  2949. Older scripts may use fixed <subject_slug> filenames or embedded seed facts.
  2950. Move those into project/config.
  2951.  
  2952. 2. Add a documented raw-stream builder.
  2953. The app consumes raw_stream_index.sqlite; make its builder explicit.
  2954.  
  2955. 3. Standardize CLI behavior.
  2956. Every script should have argparse --help that never mutates outputs.
  2957.  
  2958. 4. Centralize model/provider config.
  2959. Put model defaults, reasoning defaults, max output tokens, and cost guards in
  2960. config files or shared helpers.
  2961.  
  2962. 5. Add a unified validation command.
  2963. A top-level project/scripts/validate_personalos.py should run JSONL checks,
  2964. schema checks, secret scan, RAG stats, dashboard payload validation, and
  2965. selected unit tests.
  2966.  
  2967. 6. Move large paid-run manifests to explicit budgeted workflows.
  2968. Any full embedding, media captioning, synthesis, or formulation run should
  2969. fail closed without limit, budget, or confirmation.
  2970.  
  2971. 7. Add source-family privacy levels.
  2972. Each source family should carry default exposure, RAG salience, dashboard
  2973. visibility, raw drilldown permission, and export policy.
  2974.  
  2975. 8. Add schema versioning.
  2976. Every durable JSONL/CSV output should include schema version in manifest or
  2977. row metadata.
  2978.  
  2979. 9. Improve redaction/export tooling.
  2980. Build a separate redacted-export generator rather than manually editing
  2981. canonical private artifacts.
  2982.  
  2983. 10. Keep visual QA artifacts but avoid indexing them.
  2984. Screenshots and QA images are useful locally but should not become RAG
  2985. evidence unless deliberately selected.
  2986.  
  2987. Minimal viable PersonalOS
  2988. =========================
  2989.  
  2990. If recreating from scratch with limited time, build this subset first:
  2991.  
  2992. 1. canonical folder layout and path helper;
  2993. 2. raw writing export;
  2994. 3. at least one message-platform normalizer;
  2995. 4. merged message histories;
  2996. 5. identity review/overrides;
  2997. 6. chronology seed/candidate pack;
  2998. 7. context pack medium;
  2999. 8. RAG index with FTS first;
  3000. 9. dashboard data and unified local server;
  3001. 10. secret scanner.
  3002.  
  3003. Then add:
  3004.  
  3005. 11. embeddings/vector search;
  3006. 12. synthesis;
  3007. 13. daily activity;
  3008. 14. location/media layers;
  3009. 15. psychological/open-ended/final-conclusion layers.
  3010.  
  3011. Definition of done
  3012. ==================
  3013.  
  3014. A rebuilt PersonalOS is functionally complete when:
  3015.  
  3016. - Raw exports are preserved under personal/raw.
  3017. - Canonical scripts use path helpers and subject config.
  3018. - Message events, identities, conversations, and top histories build cleanly.
  3019. - Chronology seed and candidate packs build with source refs.
  3020. - Context pack full and medium variants build with stats and coverage audits.
  3021. - RAG index builds, embeds under budget controls, and passes retrieval checks.
  3022. - Dashboard data builds without raw source bodies.
  3023. - Unified local app serves dashboard, search, chat, raw review, media review,
  3024. and graph routes.
  3025. - Privacy scanner runs and blocks external export on findings.
  3026. - Major generated families have README/STATUS docs.
  3027. - High-level analysis artifacts separate evidence from interpretation.
  3028. - The user can answer: "where did this claim come from?" for any important
  3029. claim by following local source references back to primary evidence.
  3030.  
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!