S&DP JSON Manticore

Architecture Explorer

Desain fleksibel untuk perubahan bisnis: JSON sebagai write model evolutif, Manticore sebagai read/search projection, dan tabel typed untuk integritas transaksi serta komisi.

Keputusan Arsitektur

Primary Database

Source of truth untuk write, versioning, relationship, idempotency, policy, dan ledger.

Strong consistency

JSON Document

Profil, role, capability, requirement, dan konfigurasi sparse yang dapat berkembang tanpa DDL rutin.

Flexible schema

Manticore Search

Projection terdenormalisasi untuk pencarian, filter, CMS, dan dashboard. Disposable dan dapat di-reindex.

Eventually consistent

External Services

Masterdata, AWB, dan Billing tetap menjadi pemilik data masing-masing. S&DP menyimpan external ID.

No authoritative duplication

OOP ke Data Model

Encapsulation

Ownership per service dan akses melalui API/event.

Inheritance

Role taxonomy dan default capability, bukan table-per-subclass.

Polymorphism

Behavior dipilih berdasarkan capability, workflow, dan policy.

Composition

Satu party dapat menerima banyak role dan capability tanpa menggandakan identity.

ERD v3.0.0 (Counter-Design Aurora MySQL 8.0)

v3.0.0 Reviewed Draft

v3.0.0 — Snowflake 64-bit Identifier Migration: all internal PK/FK/cross-reference IDs are application-generated signed BIGINT Snowflakes for direct Manticore document-ID compatibility. Layout: 41-bit milliseconds since 2025-01-01, 10-bit worker, 12-bit sequence; valid through 2039-09-07. JSON APIs use decimal strings and JavaScript BigInt, never Number. ERP financial scope is constrained by the current PRD/BRD evidence shown in the traceability matrix.

erDiagram ACCOUNT_TYPE_LOOKUP ||--o{ ACCOUNT : classifies ACCOUNT_STATUS_LOOKUP ||--o{ ACCOUNT : status RELATIONSHIP_TYPE_LOOKUP ||--o{ ACCOUNT_RELATIONSHIP : categorizes ACTIVITY_TYPE_LOOKUP ||--o{ ACTIVITY_EVENT : types ACTIVITY_TYPE_LOOKUP ||--o{ OPERATIONAL_TRANSACTION : types POLICY_STATUS_LOOKUP ||--o{ POLICY_DOCUMENT : status LEDGER_STATUS_LOOKUP ||--o{ LEDGER_ENTRY : status LEDGER_ENTRY_TYPE_LOOKUP ||--o{ LEDGER_ENTRY : entry_type ACCOUNT ||--o{ ACCOUNT : created_by ACCOUNT ||--o{ ACCOUNT_RELATIONSHIP : source ACCOUNT ||--o{ ACCOUNT_RELATIONSHIP : target ACCOUNT ||--o{ ACTIVITY_EVENT : validates_actor ACCOUNT ||--o{ OPERATIONAL_TRANSACTION : validates_actor ACCOUNT ||--o{ LEDGER_ENTRY : validates_beneficiary ACTIVITY_EVENT ||--o{ LEDGER_ENTRY : validates_activity POLICY_DOCUMENT ||--o{ LEDGER_ENTRY : validates_policy OUTBOX_EVENT ||--o{ PROJECTION_RECEIPT : projects ACCOUNT { bigint account_id PK string account_type FK string external_account_id string subledger_code string cost_center_code bigint created_by_account_id FK integer schema_version json document string status FK boolean is_active bigint version datetime deleted_at datetime created_at datetime updated_at } ACCOUNT_RELATIONSHIP { bigint relationship_id PK bigint source_account_id FK bigint target_account_id FK string relationship_type FK datetime valid_from datetime valid_to string status FK boolean is_active json metadata datetime deleted_at datetime created_at datetime updated_at } OPERATIONAL_TRANSACTION { bigint transaction_id PK bigint partition_key PK boolean is_active PK datetime partition_created_time bigint actor_account_id string external_transaction_id string transaction_type FK string branch_code integer financial_period numeric amount string currency string status FK json facts datetime occurred_at datetime settled_at datetime deleted_at datetime created_at datetime updated_at } ACTIVITY_EVENT { bigint activity_id PK bigint partition_key PK boolean is_active PK datetime partition_created_time bigint actor_account_id string activity_type string external_transaction_id string idempotency_key UK json facts datetime occurred_at datetime recorded_at datetime deleted_at datetime created_at datetime updated_at } POLICY_DOCUMENT { bigint policy_id PK string policy_code integer policy_version json definition datetime valid_from datetime valid_to string status FK datetime created_at datetime updated_at } LEDGER_ENTRY { bigint ledger_entry_id PK bigint partition_key PK boolean is_active PK datetime partition_created_time bigint activity_id bigint policy_id FK binary32 accrual_dedup_key UK bigint beneficiary_account_id FK string subledger_code string debit_account_code string credit_account_code integer financial_period string ap_voucher_number bigint journal_batch_id numeric amount string currency string entry_type FK string status FK string billing_external_id UK json metadata datetime settled_at datetime deleted_at datetime created_at datetime updated_at } OUTBOX_EVENT { bigint event_id PK bigint partition_key PK boolean is_active PK datetime partition_created_time string aggregate_type bigint aggregate_id binary32 outbox_dedup_key UK bigint aggregate_version string event_type json payload datetime created_at datetime updated_at datetime available_at datetime published_at integer attempt_count string last_error_code string last_error_ref } PROJECTION_RECEIPT { string projection_name PK bigint event_id bigint aggregate_id bigint aggregate_version datetime processed_at } ACCOUNT_TYPE_LOOKUP { string code PK string description } ACCOUNT_STATUS_LOOKUP { string code PK string description } RELATIONSHIP_TYPE_LOOKUP { string code PK string description } ACTIVITY_TYPE_LOOKUP { string code PK string description } POLICY_STATUS_LOOKUP { string code PK string description } LEDGER_STATUS_LOOKUP { string code PK string description } LEDGER_ENTRY_TYPE_LOOKUP { string code PK string description }

Design Summary

Identity

Signed 64-bit Snowflake IDs; typed account relationships; external business IDs remain strings.

Integrity

Foreign keys on non-partitioned tables; application validation documented where Aurora partitioning forbids FKs.

Operations

Idempotency, optimistic versions, transactional outbox, CDC timestamps, and keyset/batch bounds.

Retention

deleted_at is the only soft-delete truth; is_active is lifecycle/partition state only.

SQL Antipattern & Best-Practice Compliance Matrix

Audit scope: current schemas.sql, scenario-queries.sql, and partition-operations.sql. Sources: Bill Karwin's SQL Antipatterns taxonomy, AWS Aurora/MySQL guidance, internal DBE Rule-of-Thumb, and internal portable partition SOP. “Conditional” means the design is safe only when the stated application/runbook control is enforced.

Rule / riskSourceStatusEvidence & control
Jaywalking / multi-valued attributesSQL AntipatternsPassAccount hierarchy uses typed rows in account_relationship; JSON arrays are bounded profile/capability documents, not relational FK lists.
Polymorphic associationSQL AntipatternsDocumented exceptionCore domain links are typed. outbox_event.aggregate_type + aggregate_id remains an intentional polymorphic integration envelope; producers enforce a closed aggregate-type registry and validate ownership.
EAV / metadata tribblesSQL AntipatternsPassStable invariants are typed; sparse profile/policy payloads use bounded versioned JSON. Search fields are projected to Manticore, avoiding JSON scans on Aurora.
Naive tree / hierarchySQL AntipatternsPassMP-to-mitra relations are adjacency edges with validity and status; no comma-separated ancestry path.
Missing PK/FK, duplicate rowsSQL Antipatterns + MySQLDocumented exceptionNon-partitioned tables use PK/FK. Partitioned tables include partition columns in every PK/unique key; Aurora's partition/FK restriction requires same-service validation plus dedup keys.
Random/oversized identifier mismatchManticore + client interoperabilityPassApplication-generated signed BIGINT Snowflakes map directly to Manticore document IDs; JSON transports decimal strings.
Rounding / money errorsSQL Antipatterns + PRDPassAmounts use DECIMAL(19,4), never float; ledger pins the policy version and immutable calculation facts.
Implicit columns / ambiguous joinsDBE RoTPassNo SELECT *; examples use explicit projection and explicit INNER JOIN.
Database wall-clock predicateDBE RoTPassUTC cutoffs are application-bound literals/parameters; no NOW() or CURRENT_TIMESTAMP() in WHERE.
Unbounded reads / oversized INDBE RoTPassRelay walkthrough uses an ordered LIMIT 10 workset, within the maximum batch/IN bound of 100; cursor pagination is required and broad reports are offloaded.
Soft-delete state duplicationDBE conventionPassdeleted_at alone means deletion. is_active means ongoing/final lifecycle and never means deleted.
CDC precisionDBE / DEHPasscreated_at/updated_at use microseconds; updated_at has ON UPDATE CURRENT_TIMESTAMP(6) and an ETL index.
Partition portability and pruningPartition SOPPassLIST COLUMNS(partition_key,is_active), YYYYMM, explicit active/inactive pairs, no catch-all, and predicate examples include both columns.
Future partition / retention safetyPartition SOPPassRunbook creates both next-month partitions before opening; archives first and drops only the inactive partition.
Partition row movementPartition SOPPassSettlement and outbox publish examples atomically bind updated_at, recompute partition_key, change is_active, and use the previous composite locator.
Outbox concurrency / poison payloadsMessaging best practicePassFOR UPDATE SKIP LOCKED LIMIT 100, dedup key, aggregate version, receipt acknowledgement; only error code/reference stored, not raw logs.

Review conclusion: executable SQL gaps found by independent static review were corrected: explicit seed columns, lookup CDC fields/indexes, single deletion state, policy validity, bounded outbox workset, and atomic partition movement. Remaining exceptions are documented: partitioned-table FKs and the polymorphic outbox envelope. Application controls still require implementation verification.

ERD ↔ PRD/BRD Use-Case Traceability

Source calibration: the authoritative S&DP PRD is ClickUp doc 8crmjdc-130258, current page 8crmjdc-3278538 (predecessor 8crmjdc-3073858). The supplied workspace contains no separately identified S&DP BRD; therefore BRD coverage below is marked Not supplied, never inferred. Rows marked Partial show the bounded v3 ERD support and name the missing product aggregate.

Use case / sourceActors & systemERD mappingCoverage
PRD — public self-registration, referral/induk, unique agent identityCalon Agen, MP, S&DPaccount; account_relationship(REFERRED_BY / MANAGED_BY); outbox_eventCovered
PRD — CMS registration by Sales/RM and creator attributionSales/RM, S&DPaccount.created_by_account_id; role/profile JSON; activity_eventCovered
PRD — RN document review → S&P location approval → advanced-data activationRetail Network, S&P, candidateaccount.status/version; bounded verification JSON; activity_event; outbox_eventPartial — approver attribution and state transition are executable; dedicated approval-decision/reason aggregate is not modeled
PRD — active agent sync to Masterdata Customer/Partner/Gerai PartnerS&DP, Masterdata Core, OPSexternal_account_id; account role JSON; outbox_event/projection_receiptCovered as integration; Masterdata owns replicas
PRD — receive/scan existing AWB and idempotently mark DROP PARTNERAgent, AWB Serviceoperational_transaction; activity_event; external AWB ID; idempotency keyCovered
PRD — create resi and retain immutable commission inputsAgent, AWB/Tariff/SPKoperational_transaction typed money + JSON facts; outbox_eventPartial — S&DP stores transaction reference/facts; sender, recipient, package and tariff remain owning-service data
PRD — pickup/drop-off handover, cutoff and proofAgent, SPK/Hubactivity_event(AWB_HANDED_OVER); transaction facts; external attachment reference; outboxPartial — schedule/evidence lifecycle is external; no local handover aggregate
PRD — configure effective commission/bonus schemeRN, Head Commercial, MPpolicy_document version, validity, status and JSON definition; audit eventCovered for calculation policy
PRD — threshold approval of commission/bonus schemeRN, Head CommercialPolicy status/version + approval activity_eventPartial — decision/reason/threshold snapshot needs a dedicated approval aggregate before implementation
PRD — commission estimate/final, pending/ready, reconciliation and correctionAgent, S&DP, Financeactivity_event + pinned policy_document + ledger_entry; reversal as a new ledger rowCovered
PRD — AP settlement / payout and Finance referencesBilling, ERP AP, FinanceLedger subledger/cost/GL/period; billing_external_id; AP voucher; journal batch; settlement stateCovered as AP integration; AP voucher and journal batch are exercised
PRD — Manticore search/list/dashboard projectionCMS, Agent Dashboardoutbox_eventprojection_receipt → Manticore; authoritative detail remains Aurora/ownerCovered
PRD — role-specific menus, Owner/Admin provisioning, wallet/rekeningOwner, Admin, CMS rolesAccount role/capability JSON and external IAM/payment references onlyPartial — no credential, RBAC policy, wallet or bank-account aggregate in this ERD
PRD — KPI, tiering, penalty, claim chargingAgent, CMS, Claim/FinanceImmutable operational/activity facts can feed derived projectionsPartial/out of bounded v3 scope — no authoritative KPI/tier/claim aggregate
BRD use casesBusiness stakeholdersNo source mapping possibleNot supplied — attach the S&DP BRD doc/page ID to complete bidirectional traceability

Gap decision: this matrix does not invent missing tables. Approval history, handover evidence, wallet, KPI/tier, and claim aggregates require confirmed ownership/lifecycle from source requirements before extending the ERD.

v3.0.0 Data Flow

1. Client / CMS
register or update agent
2. S&DP API
validate Masterdata, AWB, and ERP references
3. Snowflake Generator
allocate signed int64 IDs
4. Aurora MySQL
account + relationship + outbox atomically
5. Outbox Relay
upsert accepted sourceVersion
6. Manticore
search/list projection only

Commission → ERP AP Settlement

1. AWB Service
authoritative logistics event ID
2. Activity & Policy
idempotent event + pinned policy version
3. Commission Ledger
subledger, cost dimension, GL debit/credit, financial period
4. Billing / ERP AP
settlement callback + AP voucher / journal batch reference

Read Rules

Search & CMS

Read Manticore

Text search, filters, cursor pagination, and non-authoritative dashboard projection.

Authoritative Detail

Read Aurora / owning service

Hydrate external identity, AWB, billing, and ERP data from the owning service when needed.

Approval & Eligibility

Never decide from search index

Evaluate Aurora state and versioned policy.

Money & Settlement

Never decide from search index

Use typed ledger fields, pinned policy, ERP AP voucher, and Billing callback state.

v3.0.0 Data Ownership

DataSource of TruthStored by S&DP v3.0.0
Customer/account identity, PIC, contactMasterdataexternal_account_id + bounded profile JSON
Address, region, attachmentsMasterdata / AttachmentExternal IDs + S&DP verification status only
Agent role, hierarchy, lifecycleS&DPaccount + account_relationship
AWB, sender, recipient, delivery statusAWB Serviceexternal_transaction_id + immutable commission facts
Commission policy and accrual/reversalS&DPVersioned policy_document + typed ledger_entry
ERP AP vendor/subledger & cost center mappingERP / Finance Mastersubledger_code, cost_center_code, branch_code reference snapshots
ERP GL account mappingERP / Finance Masterdebit_account_code, credit_account_code, financial_period snapshot used by accrual
Invoice, payment, settlementBilling / ERP APbilling_external_id, ap_voucher_number, journal_batch_id, settlement state
Internal 64-bit identifiersS&DP application / shared Snowflake allocatorSigned BIGINT; same value used as Manticore document ID
Search projectionDerivedManticore; rebuildable and non-authoritative

v3.0.0 Zero-DDL Change Test

Tambah MOBILE_AGENT

Tambah role, capability, document requirement, workflow, dan policy sebagai data.

Zero DDL

Komisi tier baru

Buat policy version baru dengan validity period. Ledger lama tetap menunjuk versi lama.

Zero DDL

Atribut profil baru

Tambah field JSON, naikkan schema version, validasi dengan JSON Schema, lalu update projection.

Zero DDL primary*

Multi-currency ledger

Jika belum tersedia sebagai invariant typed, lakukan migration dan update accounting rules.

DDL layak

Relationship bisnis baru

Tambah relationship type dan metadata selama constraint existing cukup.

Zero DDL

GL account mapping baru

Tambah pemetaan debit/credit account code melalui policy/configuration yang tervalidasi; kolom typed v3.0.0 tetap.

Zero DDL

ERP dimension fundamental baru

Dimensi yang belum memiliki invariant typed tetap memerlukan review model dan migration; jangan dipaksakan ke JSON.

DDL layak

* Perubahan searchable attribute mungkin tetap memerlukan perubahan konfigurasi/index Manticore. Zero-DDL database tidak berarti zero-governance.

User Journeys

Alur tulis selalu selesai di AWS Aurora MySQL 8.0 terlebih dahulu. Manticore diproyeksikan asinkron dan tidak dipakai untuk keputusan approval atau uang.

sequenceDiagram actor Applicant participant API as "S&DP API" participant MD as Masterdata participant DB as AWS Aurora MySQL 8.0 participant Relay as Outbox Relay participant MS as Manticore Applicant->>API: Self-register with MP-001 API->>MD: Validate external IDs MD-->>API: Valid API->>DB: Account, referral, outbox in one transaction DB-->>API: PENDING version 1 API-->>Applicant: Accepted Relay->>DB: Claim unpublished event Relay->>MS: Upsert newer sourceVersion Relay->>DB: Receipt and publish acknowledgement

Registration

Self-service stores a typed REFERRED_BY edge. Validated ERP vendor and cost-center references populate subledger_code and cost_center_code when applicable.

Touches: account, account_relationship, outbox_event

Verification

Document status lives in bounded JSON; approval audit is an idempotent typed activity.

Touches: account, activity_event, outbox_event

AWB & Commission

AWB activity is idempotent. Accrual pins activity/policy and records subledger, branch/cost dimension, financial period, and GL debit/credit mappings.

Touches: operational_transaction, activity_event, policy_document, ledger_entry

ERP AP Settlement & Projection

Billing callback settles the accrual and stores ERP AP voucher / journal batch references. Projection receipts make consumer processing auditable.

Touches: ledger_entry, outbox_event, projection_receipt

See USER-JOURNEYS.md for six complete sequence diagrams and exact columns touched.

v3.0.0 Partition & Query Operations

The complete executable runbook is partition-operations.sql: application computes partition_key=YYYYMM(updated_at), DBE pre-creates one _active and one _inactive LIST COLUMNS partition per month, and retention archives then drops only inactive partitions.

SQL Examples


Snippets are abbreviated for reading. The transactionally complete, rerunnable examples are in scenario-queries.sql; DDL and helper functions are in schemas.sql.

v3.0.0 Raw Schema & Sample Query

Read-only, exact release artifacts embedded in this explorer. Choose the Aurora MySQL DDL, executable walkthrough, or partition runbook. Use the copy button to copy the displayed artifact.

Production Guardrails

Transactional Outbox

Hindari dual-write DB + Manticore pada request yang sama.

Version Ordering

Projection membawa sourceVersion; event lama tidak boleh menimpa event baru.

Idempotency

Deduplicate berdasarkan event ID atau aggregate ID + version.

Schema Contract

JSON memakai schema_version, JSON Schema, validation, dan migration strategy.

Replay & Reindex

Manticore harus dapat dibangun ulang sepenuhnya dari primary DB.

Safe Policy DSL

Gunakan decision table/DSL terbatas, bukan arbitrary SQL atau JavaScript.

ERP Reference Validation

Validate subledger_code, cost center/branch, GL mappings, and open financial_period against ERP/Finance Master before writing the accrual.

ERP Scope Boundary

Only AP subledger, financial dimensions, accrual GL mapping, AP voucher, and journal batch references are in scope; no generic ERP modules.