S&DP JSON Manticore

Architecture Explorer

Desain fleksibel untuk perubahan bisnis: JSON sebagai write model evolutif, Manticore sebagai read/search projection, dan tabel typed untuk integritas transaksi serta komisi.

Keputusan Arsitektur

Primary Database

Source of truth untuk write, versioning, relationship, idempotency, policy, dan ledger.

Strong consistency

JSON Document

Profil, role, capability, requirement, dan konfigurasi sparse yang dapat berkembang tanpa DDL rutin.

Flexible schema

Manticore Search

Projection terdenormalisasi untuk pencarian, filter, CMS, dan dashboard. Disposable dan dapat di-reindex.

Eventually consistent

External Services

Masterdata, AWB, dan Billing tetap menjadi pemilik data masing-masing. S&DP menyimpan external ID.

No authoritative duplication

OOP ke Data Model

Encapsulation

Ownership per service dan akses melalui API/event.

Inheritance

Role taxonomy dan default capability, bukan table-per-subclass.

Polymorphism

Behavior dipilih berdasarkan capability, workflow, dan policy.

Composition

Satu party dapat menerima banyak role dan capability tanpa menggandakan identity.

ERD v2.4.0 (Counter-Design Aurora MySQL 8.0)

v2.4.0 Production Spec

v2.4.0 — ERP Subledger & Financial Integration: incorporates PRD/BRD-scoped ERP financial dimensions into the S&DP domain. account gains subledger_code and cost_center_code; operational_transaction carries branch_code and financial_period; ledger_entry integrates double-entry GL accounts (debit_account_code, credit_account_code), subledger_code, and ERP Accounts Payable Voucher (ap_voucher_number).

erDiagram ENTITY_TYPE_LOOKUP ||--o{ ACCOUNT : classifies ENTITY_STATUS_LOOKUP ||--o{ ACCOUNT : status RELATIONSHIP_TYPE_LOOKUP ||--o{ ACCOUNT_RELATIONSHIP : categorizes ACTIVITY_TYPE_LOOKUP ||--o{ ACTIVITY_EVENT : types ACTIVITY_TYPE_LOOKUP ||--o{ OPERATIONAL_TRANSACTION : types POLICY_STATUS_LOOKUP ||--o{ POLICY_DOCUMENT : status LEDGER_STATUS_LOOKUP ||--o{ LEDGER_ENTRY : status LEDGER_ENTRY_TYPE_LOOKUP ||--o{ LEDGER_ENTRY : entry_type ACCOUNT ||--o{ ACCOUNT : created_by ACCOUNT ||--o{ ACCOUNT_RELATIONSHIP : source ACCOUNT ||--o{ ACCOUNT_RELATIONSHIP : target ACCOUNT ||--o{ ACTIVITY_EVENT : validates_actor ACCOUNT ||--o{ OPERATIONAL_TRANSACTION : validates_actor ACCOUNT ||--o{ LEDGER_ENTRY : validates_beneficiary ACTIVITY_EVENT ||--o{ LEDGER_ENTRY : validates_activity POLICY_DOCUMENT ||--o{ LEDGER_ENTRY : validates_policy OUTBOX_EVENT ||--o{ PROJECTION_RECEIPT : projects ACCOUNT { binary16 account_id PK string account_type FK string external_account_id binary16 created_by_account_id FK integer schema_version json document string status FK boolean is_active bigint version datetime deleted_at datetime created_at datetime updated_at } ACCOUNT_RELATIONSHIP { binary16 relationship_id PK binary16 source_account_id FK binary16 target_account_id FK string relationship_type FK datetime valid_from datetime valid_to string status FK boolean is_active json metadata datetime deleted_at datetime created_at datetime updated_at } OPERATIONAL_TRANSACTION { binary16 transaction_id PK bigint partition_key PK boolean is_active PK datetime partition_created_time binary16 actor_account_id string external_transaction_id string transaction_type FK numeric amount string currency string status FK json facts datetime occurred_at datetime settled_at datetime deleted_at datetime created_at datetime updated_at } ACTIVITY_EVENT { binary16 activity_id PK bigint partition_key PK boolean is_active PK datetime partition_created_time binary16 actor_account_id string activity_type string external_transaction_id string idempotency_key UK json facts datetime occurred_at datetime recorded_at datetime deleted_at datetime created_at datetime updated_at } POLICY_DOCUMENT { binary16 policy_id PK string policy_code integer policy_version json definition datetime valid_from datetime valid_to string status FK datetime created_at datetime updated_at } LEDGER_ENTRY { binary16 ledger_entry_id PK bigint partition_key PK boolean is_active PK datetime partition_created_time binary16 activity_id binary16 policy_id FK binary16 beneficiary_account_id FK numeric amount string currency string entry_type FK string status FK string billing_external_id UK json metadata datetime settled_at datetime deleted_at datetime created_at datetime updated_at } OUTBOX_EVENT { binary16 event_id PK bigint partition_key PK boolean is_active PK datetime partition_created_time string aggregate_type binary16 aggregate_id binary32 outbox_dedup_key UK bigint aggregate_version string event_type json payload datetime created_at datetime updated_at datetime available_at datetime published_at integer attempt_count string last_error_code string last_error_ref } PROJECTION_RECEIPT { string projection_name PK binary16 event_id binary16 aggregate_id binary32 outbox_dedup_key UK bigint aggregate_version datetime processed_at } ENTITY_TYPE_LOOKUP { string code PK string description } ENTITY_STATUS_LOOKUP { string code PK string description } RELATIONSHIP_TYPE_LOOKUP { string code PK string description } ACTIVITY_TYPE_LOOKUP { string code PK string description } POLICY_STATUS_LOOKUP { string code PK string description } LEDGER_STATUS_LOOKUP { string code PK string description } LEDGER_ENTRY_TYPE_LOOKUP { string code PK string description }

Perbandingan Antipattern vs Counter-Design

1. Primary Key Strategy

Antipattern: Random UUIDv4 (gen_random_uuid())

Counter-Design: Time-Ordered UUIDv7 (gen_ordered_uuid() / UUID_TO_BIN(UUID(), 1))

Eliminates B-Tree Index Fragmentation

2. Domain Status & Types

Antipattern: Hardcoded CHECK (status IN (...))

Counter-Design: Zero-DDL Lookup Tables

No AccessExclusiveLock on ADD

3. Creator Attribution

Antipattern: Jaywalking / JSON inline createdBy

Counter-Design: Typed FK created_by_account_id

Engine-Enforced FK Integrity

4. Single Source of Truth

Antipattern: Duplicate fields in JSON & Metadata

Counter-Design: Canonical JSON Document + typed query fields

Prevents Update Anomalies

5. Outbox Concurrency

Antipattern: Read version without lock

Counter-Design: SKIP LOCKED relay + optimistic aggregate version

Safe Under High Concurrent Writes

6. Outbox Projection Relay

Antipattern: Unconditional CTE UPDATE candidate

Counter-Design: Strict INNER JOIN on accepted receipts

Prevents False Positive published_at

7. Operational Table Partitioning

Antipattern: Unpartitioned transaction tables with full table scans

Counter-Design: LIST COLUMNS (partition_key, is_active) active/inactive pairs

Optimized Monthly Scans & Settled Filtering

8. DEH Snowflake ETL Sync

Antipattern: Missing CDC watermarks requiring heavy full extractions

Counter-Design: Native ON UPDATE CURRENT_TIMESTAMP(6) CDC Watermarks

Low-Latency CDC Ingestion to Snowflake

9. Soft Delete Strategy

Antipattern: Hard deletes losing audit trails & state history

Counter-Design: Universal deleted_at; composite indexes (MySQL has no partial indexes)

Audit-Compliant Soft Deletes

Semantic Versioning Comparison: ERD v2.3.0 vs ERD v2.4.0

Two latest versions only: v2.3.0 achieved DB Rule-of-Thumb compliance and canonical deleted_at soft deletes. v2.4.0 extends the model with targeted ERP subledger and financial dimensions strictly aligned with S&DP PRD/BRD requirements.

Release-level ERP integration delta

Concernv2.3.0v2.4.0PRD / BRD Business Reason
Party Subledger IdentityGeneric external_account_id onlysubledger_code & cost_center_codeMaps Mitra/Partner accounts directly to ERP Accounts Payable Subledger vendors and regional cost centers.
Operational DimensionBasic transaction timestampsbranch_code & financial_period (YYYYMM)Attributes logistics movements to branch cost centers for ERP period closing and regional financial reporting.
Double-Entry AccountingSingle amount & statusdebit_account_code & credit_account_codeEnforces double-entry journal integrity (e.g. Debit 6100-Commission-Expense / Credit 2100-Partner-Payable).
AP Settlement VoucherBasic billing_external_idap_voucher_number & journal_batch_idLinks commission settlements directly to ERP Accounts Payable Vouchers (APV) and GL Journal Batches.

v2.4.0 ERP Financial Contract

ERP Subledger & GL Mapping

  • Every Mitra/Partner account carries an ERP Subledger Vendor Code (subledger_code).
  • Every commission accrual generates double-entry GL account codes (debit_account_code / credit_account_code).
  • Financial period (financial_period) is locked to YYYYMM for accounting period closing.

Accounts Payable Settlement Sync

  • Settlement callbacks write the ERP AP Voucher Number (ap_voucher_number).
  • Journal batch postings (journal_batch_id) stream incrementally to DEH Snowflake / ERP GL Data Lake.

Write → Projection → Read

1. Client
create/update agent
2. S&DP API
validate external IDs
3. Primary DB
document + relation + outbox dalam satu transaksi
4. Outbox Relay
enrich dan upsert versi terbaru
5. Manticore
search/list projection

Aturan Read

Search & CMS

Baca Manticore

Text search, faceting, filter, pagination, dan dashboard.

Detail Authoritative

Baca primary/service owner

Candidate ID dari Manticore dapat di-hydrate bila perlu.

Approval & Eligibility

Jangan percaya index

Evaluasi primary DB dan policy engine.

Commission & Settlement

Jangan percaya index

Gunakan activity, policy version, dan ledger typed.

Data Ownership

DataSource of TruthYang disimpan S&DP
Customer/account, PIC, kontakMasterdatacustomer_account_id
Alamat & wilayahMasterdataaddress_id
File dokumenAttachment/MasterdataID + status verifikasi S&DP
Agent role & hierarchyS&DPJSON document + typed relationship
AWB, sender, recipient, statusAWB ServiceID transaksi + immutable facts untuk kalkulasi
Policy komisiS&DPVersioned policy JSON
Accrual/reversal komisiS&DPTyped immutable ledger
Invoice/payment/settlementBillingbilling_external_id
Search projectionDerivedManticore; rebuildable

Uji Perubahan Bisnis

Tambah MOBILE_AGENT

Tambah role, capability, document requirement, workflow, dan policy sebagai data.

Zero DDL

Komisi tier baru

Buat policy version baru dengan validity period. Ledger lama tetap menunjuk versi lama.

Zero DDL

Atribut profil baru

Tambah field JSON, naikkan schema version, validasi dengan JSON Schema, lalu update projection.

Zero DDL primary*

Multi-currency ledger

Jika belum tersedia sebagai invariant typed, lakukan migration dan update accounting rules.

DDL layak

Relationship bisnis baru

Tambah relationship type dan metadata selama constraint existing cukup.

Zero DDL

Double-entry accounting

Konsep fundamental baru; jangan dipaksakan ke JSON.

DDL wajib

* Perubahan searchable attribute mungkin tetap memerlukan perubahan konfigurasi/index Manticore. Zero-DDL database tidak berarti zero-governance.

User Journeys

Alur tulis selalu selesai di AWS Aurora MySQL 8.0 terlebih dahulu. Manticore diproyeksikan asinkron dan tidak dipakai untuk keputusan approval atau uang.

sequenceDiagram actor Applicant participant API as "S&DP API" participant MD as Masterdata participant DB as AWS Aurora MySQL 8.0 participant Relay as Outbox Relay participant MS as Manticore Applicant->>API: Self-register with MP-001 API->>MD: Validate external IDs MD-->>API: Valid API->>DB: Account, referral, outbox in one transaction DB-->>API: PENDING version 1 API-->>Applicant: Accepted Relay->>DB: Claim unpublished event Relay->>MS: Upsert newer sourceVersion Relay->>DB: Receipt and publish acknowledgement

Registration

Self-service stores a typed REFERRED_BY edge. CMS registration stores only the external operator ID.

Touches: account, account_relationship, outbox_event

Verification

Document status lives in bounded JSON; approval audit is an idempotent typed activity.

Touches: account, activity_event, outbox_event

AWB & Commission

Create, receive, and handover use unique keys. Commission pins the activity and policy version.

Touches: activity_event, policy_document, ledger_entry

Settlement & Projection

Billing callback ID is unique. Projection receipts make consumer processing auditable.

Touches: ledger_entry, outbox_event, projection_receipt

See USER-JOURNEYS.md for six complete sequence diagrams and exact columns touched.

v2.3.0 Partition & Query Operations

The complete executable runbook is partition-operations.sql: application computes partition_key=YYYYMM(updated_at), DBE pre-creates one _active and one _inactive LIST COLUMNS partition per month, and retention archives then drops only inactive partitions.

SQL Examples


Snippets are abbreviated for reading. The transactionally complete, rerunnable examples are in scenario-queries.sql; DDL and helper functions are in schemas.sql.

Production Guardrails

Transactional Outbox

Hindari dual-write DB + Manticore pada request yang sama.

Version Ordering

Projection membawa sourceVersion; event lama tidak boleh menimpa event baru.

Idempotency

Deduplicate berdasarkan event ID atau aggregate ID + version.

Schema Contract

JSON memakai schema_version, JSON Schema, validation, dan migration strategy.

Replay & Reindex

Manticore harus dapat dibangun ulang sepenuhnya dari primary DB.

Safe Policy DSL

Gunakan decision table/DSL terbatas, bukan arbitrary SQL atau JavaScript.