Muhammad Ikram Let's talk

Why Every Query You Run Today Stands on Fifty Years of Broken Databases

1970: an IBM paper ends a decade of duplicate spreadsheets. 2007: Amazon builds Dynamo because one server can't hold a billion carts. 2020: a warehouse startup raises $3.4B for finally letting storage and compute stop fighting. Six decades of "permanent" fixes, none permanent.

Read the slides as text

Slide 1

SYSTEMS ARCHITECTURE LOG
01 / 09
E N G I N E E R I N G B E H I N D B I G S Y S T E M S
Why Every Query You Run Today Stands on
Fifty Years of Broken Databases
From punch-card files that duplicated every record six times to
warehouses that scan a petabyte in seconds. Here is the exact
chain of failures that forced each redesign.
NIGHT OWL
SWIPE TO TRACE THE TIMELINE
6
Major architecture shifts since
1960
1000x
Growth in queryable data volume
per warehouse
$3.4B
Raised in Snowflake's 2020 IPO,
proof of the cloud model

Slide 2

SYSTEMS ARCHITECTURE LOG
02 / 09
S T A G E 0 1 · T H E F I L I N G C A B I N E T P R O B L E M
Every Program Kept Its Own Copy of the
Truth
Before 1970, applications read and wrote directly to flat files.
Payroll, inventory, and billing each stored the same customer
record separately, with no shared engine enforcing
consistency.
Why it broke down
A single address change had to be updated in every file that
referenced the customer. Miss one, and payroll shipped a check
to the wrong address while billing sent the invoice correctly.
There was no query language. Retrieving a report meant a
programmer writing a custom batch job, compiling it, and running
it overnight against tape or disk.
NIGHT OWL
STAGE 01 OF 06
FLAT FILES
1960s
RELATIONAL
1970
CLIENT-SERVER
1979-95
WEB SCALE
2006-07
NOSQL
2008-09
CLOUD DW
2010-20
0
Built-in mechanisms for data consistency across files
N+1
Copies of the same record, one per consuming
program

Slide 3

SYSTEMS ARCHITECTURE LOG
03 / 09
S T A G E 0 2 · T H E R E L A T I O N A L B R E A K T H R O U G H
One Paper Ended a Decade of Duplicate
Data
In 1970, IBM researcher Edgar Codd published a model that
stored data in tables linked by shared keys instead of
hardcoded file structures, letting one fact live in exactly one
place.
Tables
Data organized in rows and columns,
independent of any single program
Keys
Records linked by shared identifiers
instead of duplicated fields
SQL
A shared query language, standardized
by ANSI in 1986
The relational model separated how data is stored from how
data is queried. Any application could ask a question through
SQL without knowing the physical file layout underneath it.
NIGHT OWL
STAGE 02 OF 06
FLAT FILES
1960s
RELATIONAL
1970
CLIENT-SERVER
1979-95
WEB SCALE
2006-07
NOSQL
2008-09
CLOUD DW
2010-20

Slide 4

SYSTEMS ARCHITECTURE LOG
04 / 09
S T A G E 0 3 · T H E C O M M E R C I A L R A C E
The Model Won the Argument, Vendors
Fought for the Market
Oracle shipped the first commercial SQL database in 1979.
Over the next fifteen years, IBM, Sybase, and eventually a free
alternative turned the relational model into standard
infrastructure.
OracleDB
IBM DB2
MySQL
PostgreSQL
The ceiling nobody discussed yet
Every one of these systems ran on a single, larger server. Scaling
meant buying more powerful hardware, not adding more
machines. That approach had a hard limit the moment traffic
became global.
NIGHT OWL
STAGE 03 OF 06
FLAT FILES
1960s
RELATIONAL
1970
CLIENT-SERVER
1979-95
WEB SCALE
2006-07
NOSQL
2008-09
CLOUD DW
2010-20

Slide 5

SYSTEMS ARCHITECTURE LOG
05 / 09
S T A G E 0 4 · T H E W E B - S C A L E W A L L
One Server Cannot Hold a Billion Users
By the mid 2000s, Google and Amazon were running services
no single relational database could shard cleanly. Both
companies published the papers that redefined the field.
2 Landmark papers, in back-to-
back years, that founded the
distributed database era
NIGHT OWL
STAGE 04 OF 06
FLAT FILES
1960s
RELATIONAL
1970
CLIENT-SERVER
1979-95
WEB SCALE
2006-07
NOSQL
2008-09
CLOUD DW
2010-20
2006
Google publishes the Bigtable paper for structured
storage across thousands of machines
2007
Amazon publishes the Dynamo paper on always-
available, partitioned storage

Slide 6

SYSTEMS ARCHITECTURE LOG
06 / 09
S T A G E 0 5 · C O N S I S T E N C Y B E C O M E S A C H O I C E
Teams Started Trading Consistency for
Uptime
Facebook open sourced Cassandra in 2008. MongoDB
followed in 2009. Both broke from strict relational rules on
purpose, accepting the CAP theorem trade-off instead of
ignoring it.
The trade-off, stated plainly
A distributed system cannot guarantee perfect consistency, full
availability, and network partition tolerance all at once. NoSQL
databases picked availability and partition tolerance, and let
consistency catch up moments later.
Cassandra
MongoDB
DynamoDB
NIGHT OWL
STAGE 05 OF 06
FLAT FILES
1960s
RELATIONAL
1970
CLIENT-SERVER
1979-95
WEB SCALE
2006-07
NOSQL
2008-09
CLOUD DW
2010-20

Slide 7

SYSTEMS ARCHITECTURE LOG
07 / 09
S T A G E 0 6 · S T O R A G E M E E T S C O M P U T E , T H E N S P L I T S
The Real Unlock Was Separating Storage
From Compute
Google BigQuery launched in 2010, Amazon Redshift in 2012,
and Snowflake in 2014. Each let a company store unlimited
data cheaply, then pay for processing power only during a
query.
Store
Cheap, elastic object storage holding
raw data at any volume
Scale
Compute clusters spin up on demand
for the duration of a query
Bill
Pay for the seconds of compute used,
not for idle servers
BigQuery
Redshift
Snowflake
NIGHT OWL
STAGE 06 OF 06
FLAT FILES
1960s
RELATIONAL
1970
CLIENT-SERVER
1979-95
WEB SCALE
2006-07
NOSQL
2008-09
CLOUD DW
2010-20

Slide 8

SYSTEMS ARCHITECTURE LOG
08 / 09
P R O O F P O I N T · S E P T E M B E R 1 6 , 2 0 2 0
Wall Street Confirmed What Engineers
Already Knew
Snowflake went public and raised roughly three point four
billion dollars, the largest software IPO in history at the time,
built entirely on the separated storage and compute model.
$3.4B
$3.4B Raised in a single day,
validating cloud data
warehouses as core
enterprise infrastructure
rather than an experiment
NIGHT OWL
DATA-ANCHORED CASE STUDY
2012
Year Snowflake was founded, two
years before public launch
2014
Year the platform launched
commercially to customers
2020
Year of the IPO that reset
expectations for data
infrastructure valuations

Slide 9

SYSTEMS ARCHITECTURE LOG
09 / 09
W H E R E I T S T A N D S T O D A Y
Six Decades of Fixes, Compressed Into One
Lesson
Databricks open sourced Delta Lake in 2019, merging the
flexibility of a data lake with the reliability guarantees of a
warehouse. Every generation solved the previous generation's
bottleneck rather than starting over.
The pattern behind every shift
Flat files broke on duplication. Relational databases broke on
server limits. Distributed NoSQL broke on analytical complexity.
Cloud warehouses solved cost and scale together, and
lakehouses are now closing the remaining gap between raw data
and governed data.
The architecture a company chooses today is rarely permanent.
It is the best trade-off available until the next bottleneck defines
the next generation.
NIGHT OWL
END OF BREAKDOWN

More insights

Keep reading

All posts
Carousel

AI & Tech Explained 1 min read

AI Pilots Impress. Only Production Moves Revenue.

A common trap: the AI demo wins the room, then stalls. Gartner predicted at least 30% of GenAI projects would be dropped after proof of concept. The growth lesson: budget the data, owner, integration and run cost before the demo.

AI & Tech Explained 1 min read

Stripe, Adyen or Checkout.com?

Stripe, Adyen or Checkout.com? The right payment provider depends on where your business is going—not just the brand name. There’s no universal winner.…