S&DP JSON Manticore

Architecture Explorer

Desain fleksibel untuk perubahan bisnis: JSON sebagai write model evolutif, Manticore sebagai read/search projection, dan tabel typed untuk integritas transaksi serta komisi.

Keputusan Arsitektur

Primary Database

Source of truth untuk write, versioning, relationship, idempotency, policy, dan ledger.

Strong consistency

JSON Document

Profil, role, capability, requirement, dan konfigurasi sparse yang dapat berkembang tanpa DDL rutin.

Flexible schema

Manticore Search

Projection terdenormalisasi untuk pencarian, filter, CMS, dan dashboard. Disposable dan dapat di-reindex.

Eventually consistent

External Services

Masterdata, AWB, dan Billing tetap menjadi pemilik data masing-masing. S&DP menyimpan external ID.

No authoritative duplication

OOP ke Data Model

Encapsulation

Ownership per service dan akses melalui API/event.

Inheritance

Role taxonomy dan default capability, bukan table-per-subclass.

Polymorphism

Behavior dipilih berdasarkan capability, workflow, dan policy.

Composition

Satu party dapat menerima banyak role dan capability tanpa menggandakan identity.

ERD v1.0.0 (Initial Baseline)

v1.0.0 Legacy Baseline
erDiagram ENTITY_DOCUMENT { uuid entity_id PK string entity_type string external_party_id int schema_version json document bigint version string status } ENTITY_RELATIONSHIP { uuid relationship_id PK uuid source_entity_id FK uuid target_entity_id FK string relationship_type json metadata } ACTIVITY_EVENT { uuid activity_id PK uuid actor_entity_id FK string activity_type string external_transaction_id string idempotency_key UK json facts } POLICY_DOCUMENT { uuid policy_id PK string policy_code int policy_version json definition string status } LEDGER_ENTRY { uuid ledger_entry_id PK uuid activity_id FK uuid policy_id FK decimal amount string currency string entry_type string status } OUTBOX_EVENT { uuid event_id PK string aggregate_type uuid aggregate_id bigint aggregate_version string event_type json payload } ENTITY_DOCUMENT ||--o{ ENTITY_RELATIONSHIP : source ENTITY_DOCUMENT ||--o{ ENTITY_RELATIONSHIP : target ENTITY_DOCUMENT ||--o{ ACTIVITY_EVENT : performs ACTIVITY_EVENT ||--o{ LEDGER_ENTRY : produces POLICY_DOCUMENT ||--o{ LEDGER_ENTRY : calculates ENTITY_DOCUMENT ||--o{ OUTBOX_EVENT : publishes

JSON tidak dipakai untuk menggantikan constraint. Uang, relationship penting, idempotency, validity period, dan lifecycle transaksi tetap typed.

ERD v2.2.0 (Counter-Design Aurora MySQL 8.0)

v2.2.0 Production Spec

v2.2.0 — Portable Partitioning SOP alignment: PARTITION BY LIST COLUMNS (partition_key, is_active) across Aurora MySQL, MySQL, MariaDB, and OceanBase. Every partitioned table uses application-maintained partition_key BIGINT (YYYYMM from updated_at), is_active, and partition_created_time; each month is pre-created as an _active and _inactive pair. Inactive pairs are archive-then-drop retention targets.

erDiagram ENTITY_TYPE_LOOKUP ||--o{ ACCOUNT : classifies ENTITY_STATUS_LOOKUP ||--o{ ACCOUNT : status RELATIONSHIP_TYPE_LOOKUP ||--o{ ACCOUNT_RELATIONSHIP : categorizes ACTIVITY_TYPE_LOOKUP ||--o{ ACTIVITY_EVENT : types ACTIVITY_TYPE_LOOKUP ||--o{ OPERATIONAL_TRANSACTION : types POLICY_STATUS_LOOKUP ||--o{ POLICY_DOCUMENT : status LEDGER_STATUS_LOOKUP ||--o{ LEDGER_ENTRY : status LEDGER_ENTRY_TYPE_LOOKUP ||--o{ LEDGER_ENTRY : entry_type ACCOUNT ||--o{ ACCOUNT : created_by ACCOUNT ||--o{ ACCOUNT_RELATIONSHIP : source ACCOUNT ||--o{ ACCOUNT_RELATIONSHIP : target ACCOUNT ||--o{ ACTIVITY_EVENT : validates_actor ACCOUNT ||--o{ OPERATIONAL_TRANSACTION : validates_actor ACCOUNT ||--o{ LEDGER_ENTRY : validates_beneficiary ACTIVITY_EVENT ||--o{ LEDGER_ENTRY : validates_activity POLICY_DOCUMENT ||--o{ LEDGER_ENTRY : validates_policy OUTBOX_EVENT ||--o{ PROJECTION_RECEIPT : projects ACCOUNT { binary16 account_id PK string account_type FK string external_account_id binary16 created_by_account_id FK integer schema_version json document string status FK boolean is_active bigint version datetime deleted_at datetime created_at datetime updated_at } ACCOUNT_RELATIONSHIP { binary16 relationship_id PK binary16 source_account_id FK binary16 target_account_id FK string relationship_type FK datetime valid_from datetime valid_to string status FK boolean is_active json metadata datetime deleted_at datetime created_at datetime updated_at } OPERATIONAL_TRANSACTION { binary16 transaction_id PK bigint partition_key PK boolean is_active PK datetime partition_created_time binary16 actor_account_id string external_transaction_id string transaction_type FK numeric amount string currency string status FK json facts datetime occurred_at datetime settled_at datetime deleted_at datetime created_at datetime updated_at } ACTIVITY_EVENT { binary16 activity_id PK bigint partition_key PK boolean is_active PK datetime partition_created_time binary16 actor_account_id string activity_type string external_transaction_id string idempotency_key UK json facts datetime occurred_at datetime recorded_at datetime deleted_at datetime created_at datetime updated_at } POLICY_DOCUMENT { binary16 policy_id PK string policy_code integer policy_version json definition datetime valid_from datetime valid_to string status FK datetime created_at datetime updated_at } LEDGER_ENTRY { binary16 ledger_entry_id PK bigint partition_key PK boolean is_active PK datetime partition_created_time binary16 activity_id binary16 policy_id FK binary16 beneficiary_account_id FK numeric amount string currency string entry_type FK string status FK string billing_external_id UK json metadata datetime settled_at datetime deleted_at datetime created_at datetime updated_at } OUTBOX_EVENT { binary16 event_id PK bigint partition_key PK boolean is_active PK datetime partition_created_time string aggregate_type binary16 aggregate_id bigint aggregate_version string event_type json payload datetime created_at datetime updated_at datetime available_at datetime published_at integer attempt_count } PROJECTION_RECEIPT { string projection_name PK binary16 event_id binary16 aggregate_id bigint aggregate_version datetime processed_at } ENTITY_TYPE_LOOKUP { string code PK string description } ENTITY_STATUS_LOOKUP { string code PK string description } RELATIONSHIP_TYPE_LOOKUP { string code PK string description } ACTIVITY_TYPE_LOOKUP { string code PK string description } POLICY_STATUS_LOOKUP { string code PK string description } LEDGER_STATUS_LOOKUP { string code PK string description } LEDGER_ENTRY_TYPE_LOOKUP { string code PK string description }

Perbandingan Antipattern vs Counter-Design

1. Primary Key Strategy

Antipattern: Random UUIDv4 (gen_random_uuid())

Counter-Design: Time-Ordered UUIDv7 (gen_ordered_uuid() / UUID_TO_BIN(UUID(), 1))

Eliminates B-Tree Index Fragmentation

2. Domain Status & Types

Antipattern: Hardcoded CHECK (status IN (...))

Counter-Design: Zero-DDL Lookup Tables

No AccessExclusiveLock on ADD

3. Creator Attribution

Antipattern: Jaywalking / JSON inline createdBy

Counter-Design: Typed FK created_by_account_id

Engine-Enforced FK Integrity

4. Single Source of Truth

Antipattern: Duplicate fields in JSON & Metadata

Counter-Design: Canonical JSON Document + typed query fields

Prevents Update Anomalies

5. Outbox Concurrency

Antipattern: Read version without lock

Counter-Design: SKIP LOCKED relay + optimistic aggregate version

Safe Under High Concurrent Writes

6. Outbox Projection Relay

Antipattern: Unconditional CTE UPDATE candidate

Counter-Design: Strict INNER JOIN on accepted receipts

Prevents False Positive published_at

7. Operational Table Partitioning

Antipattern: Unpartitioned transaction tables with full table scans

Counter-Design: LIST COLUMNS (partition_key, is_active) active/inactive pairs

Optimized Monthly Scans & Settled Filtering

8. DEH Snowflake ETL Sync

Antipattern: Missing CDC watermarks requiring heavy full extractions

Counter-Design: Native ON UPDATE CURRENT_TIMESTAMP(6) CDC Watermarks

Low-Latency CDC Ingestion to Snowflake

9. Soft Delete Strategy

Antipattern: Hard deletes losing audit trails & state history

Counter-Design: Universal deleted_at; composite indexes (MySQL has no partial indexes)

Audit-Compliant Soft Deletes

Semantic Versioning Comparison: ERD v2.1.0 vs ERD v2.2.0

Two latest versions only: v2.1.0 introduced the Account Party model and Operational Transaction aggregate, but used non-portable RANGE partitioning and incomplete partition-key design. v2.2.0 is a backward-compatible partition refinement driven by the draft DBE SOP: LIST COLUMNS (partition_key, is_active), BIGINT keys, active/inactive monthly pairs, and explicit application ownership.

Release-level partition delta

Concernv2.1.0v2.2.0Reason
Partition methodRANGE (partition_key)LIST COLUMNS (partition_key, is_active)Portable common denominator: OceanBase, MySQL, MariaDB.
Partition keyINT YYYYMMBIGINT YYYYMM, app-derived from updated_atMatches SOP and permits controlled row movement after update.
Lifecycleis_active was indexed onlyis_active is a partition dimensionRetain/drop inactive data without affecting current transactions.
Key rulePK omitted is_activeEvery PK/unique key includes both partition columnsRequired for MySQL/MariaDB partitioned unique keys.
Period operationsStatic partitions plus catch-allPre-create monthly _active/_inactive pairs; no catch-allMissing periods fail fast; automation can create and retire periods predictably.
RetentionNo executable safety processArchive → evidence → drop only pYYYYMM_inactivePartition-level removal preserves active rows.
Referential integrityMixed FK expectationsService-side validation on partitioned tables; DB FKs on reference tablesMySQL/MariaDB disallow FKs on partitioned InnoDB tables.

v2.2.0 Operational Contract

On every application write

  1. Set updated_at in UTC.
  2. Derive partition_key = YYYYMM(updated_at).
  3. Set is_active=1 until final settlement, otherwise 0.
  4. Submit both previous and new partition tuple in the guarded update predicate.

DBE automation

  1. Pre-create two partitions per table for the next period.
  2. Inspect INFORMATION_SCHEMA.PARTITIONS and pruning plans.
  3. Archive an eligible inactive period.
  4. Record evidence, then DROP PARTITION pYYYYMM_inactive.

Write → Projection → Read

1. Client
create/update agent
2. S&DP API
validate external IDs
3. Primary DB
document + relation + outbox dalam satu transaksi
4. Outbox Relay
enrich dan upsert versi terbaru
5. Manticore
search/list projection

Aturan Read

Search & CMS

Baca Manticore

Text search, faceting, filter, pagination, dan dashboard.

Detail Authoritative

Baca primary/service owner

Candidate ID dari Manticore dapat di-hydrate bila perlu.

Approval & Eligibility

Jangan percaya index

Evaluasi primary DB dan policy engine.

Commission & Settlement

Jangan percaya index

Gunakan activity, policy version, dan ledger typed.

Data Ownership

DataSource of TruthYang disimpan S&DP
Customer/account, PIC, kontakMasterdatacustomer_account_id
Alamat & wilayahMasterdataaddress_id
File dokumenAttachment/MasterdataID + status verifikasi S&DP
Agent role & hierarchyS&DPJSON document + typed relationship
AWB, sender, recipient, statusAWB ServiceID transaksi + immutable facts untuk kalkulasi
Policy komisiS&DPVersioned policy JSON
Accrual/reversal komisiS&DPTyped immutable ledger
Invoice/payment/settlementBillingbilling_external_id
Search projectionDerivedManticore; rebuildable

Uji Perubahan Bisnis

Tambah MOBILE_AGENT

Tambah role, capability, document requirement, workflow, dan policy sebagai data.

Zero DDL

Komisi tier baru

Buat policy version baru dengan validity period. Ledger lama tetap menunjuk versi lama.

Zero DDL

Atribut profil baru

Tambah field JSON, naikkan schema version, validasi dengan JSON Schema, lalu update projection.

Zero DDL primary*

Multi-currency ledger

Jika belum tersedia sebagai invariant typed, lakukan migration dan update accounting rules.

DDL layak

Relationship bisnis baru

Tambah relationship type dan metadata selama constraint existing cukup.

Zero DDL

Double-entry accounting

Konsep fundamental baru; jangan dipaksakan ke JSON.

DDL wajib

* Perubahan searchable attribute mungkin tetap memerlukan perubahan konfigurasi/index Manticore. Zero-DDL database tidak berarti zero-governance.

User Journeys

Alur tulis selalu selesai di AWS Aurora MySQL 8.0 terlebih dahulu. Manticore diproyeksikan asinkron dan tidak dipakai untuk keputusan approval atau uang.

sequenceDiagram actor Applicant participant API as "S&DP API" participant MD as Masterdata participant DB as AWS Aurora MySQL 8.0 participant Relay as Outbox Relay participant MS as Manticore Applicant->>API: Self-register with MP-001 API->>MD: Validate external IDs MD-->>API: Valid API->>DB: Account, referral, outbox in one transaction DB-->>API: PENDING version 1 API-->>Applicant: Accepted Relay->>DB: Claim unpublished event Relay->>MS: Upsert newer sourceVersion Relay->>DB: Receipt and publish acknowledgement

Registration

Self-service stores a typed REFERRED_BY edge. CMS registration stores only the external operator ID.

Touches: account, account_relationship, outbox_event

Verification

Document status lives in bounded JSON; approval audit is an idempotent typed activity.

Touches: account, activity_event, outbox_event

AWB & Commission

Create, receive, and handover use unique keys. Commission pins the activity and policy version.

Touches: activity_event, policy_document, ledger_entry

Settlement & Projection

Billing callback ID is unique. Projection receipts make consumer processing auditable.

Touches: ledger_entry, outbox_event, projection_receipt

See USER-JOURNEYS.md for six complete sequence diagrams and exact columns touched.

v2.2.0 Partition Operations

The complete executable runbook is partition-operations.sql: application computes partition_key=YYYYMM(updated_at), DBE pre-creates one _active and one _inactive LIST COLUMNS partition per month, and retention archives then drops only inactive partitions.

SQL Examples


Snippets are abbreviated for reading. The transactionally complete, rerunnable examples are in scenario-queries.sql; DDL and helper functions are in schemas.sql.

Production Guardrails

Transactional Outbox

Hindari dual-write DB + Manticore pada request yang sama.

Version Ordering

Projection membawa sourceVersion; event lama tidak boleh menimpa event baru.

Idempotency

Deduplicate berdasarkan event ID atau aggregate ID + version.

Schema Contract

JSON memakai schema_version, JSON Schema, validation, dan migration strategy.

Replay & Reindex

Manticore harus dapat dibangun ulang sepenuhnya dari primary DB.

Safe Policy DSL

Gunakan decision table/DSL terbatas, bukan arbitrary SQL atau JavaScript.