Designing data platforms: examples

Design documents designing-data-platforms wrote, one per evaluation scenario, as the agent wrote them. Each opens on its decision; the sidebar walks through the architecture, the pipeline, the build order, the reasons and the decisions it needs from you.

• New platform on AWS

  > We're a ~150-person e-commerce company on AWS.

  a new platform: read replicas first, a warehouse over a lakehouse and why, one tool per layer, what to verify before building

• Ingestion

  > Design ingestion into our warehouse (Postgres-based today, may move to a > lakehouse later) for three sources, in ingestion-design.md: > > - orders in a PostgreSQL service DB (40M rows, ~200k changes/day).

  ingestion only: the current job recorded first, then CDC with a version guard, dead letters only for the stream (the batch sources fail the run), deletes as tombstones

• Change to a lakehouse

  > We have a lakehouse on AWS: Iceberg tables on S3, Glue catalog, Spark + > dbt for transforms, Trino for queries, Superset for BI.

  a change on the recorded platform, new parts marked: reverse ETL into the app's database, loads visible whole, protected customer data

• Sensitive data on BigQuery

  > A clinic-booking platform (GCP, BigQuery as warehouse) wants analytics.

  per-column decisions, card numbers kept out, keyed pseudonyms, erasure through every layer

• Existing platform, from a repo

  > Product wants order status changes to show on the operations dashboard within 5 minutes; today they see > yesterday's data.

  the platform read from the repo and recorded with the file each part came from, then the change drawn on it