Database Paper Browser

Back to papers

Realtime Data Processing at Facebook

Summary: Realtime data processing at Facebook handles hundreds of GB/s across pipelines, guided by five design decisions on usability, performance, fault tolerance, scalability, and correctness. Seconds-latency with a persistent message bus enables Puma, Swift, and Stylus, delivering scalable, fault-tolerant streaming and valuable design lessons. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
5204
Venue
SIGMOD
Year
2016
Pagerank
0.00011140777
Overall Rank
1,613 | 88.79%
DOI
10.1145/2882903.2904441

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 23 of 23 citing papers.

Rank Citing Paper Year Venue Pagerank
569 Optimizing Space Amplification in RocksDB 2017 CIDR 0.00019924098
1,610 MyRocks: LSM-Tree Database Storage Engine Serving Facebook's Social Graph 2020 VLDB 0.00011148094
2,798 Chucky: A Succinct Cuckoo Filter for LSM-Tree 2021 SIGMOD 8.1080111e-05
3,544 Rosetta: A Robust Space-Time Optimized Range Filter for Key-Value Stores 2020 SIGMOD 6.9898874e-05
3,965 Spooky: Granulating LSM-Tree Compactions Correctly 2022 VLDB 6.5820028e-05
5,286 StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance 2023 VLDB 5.5838392e-05
6,436 Providing Streaming Joins as a Service at Facebook 2018 VLDB 5.0636254e-05
6,629 A Holistic View of Stream Partitioning Costs 2017 VLDB 4.9880986e-05
6,715 Shared Foundations: Modernizing Meta's Data Lakehouse 2023 CIDR 4.9509939e-05
7,995 BP-tree: Overcoming the Point-Range Operation Tradeoff for In-Memory B-trees 2023 VLDB 4.6109825e-05
8,009 CAMAL: Optimizing LSM-trees via Active Learning 2024 SIGMOD 4.6066863e-05
9,317 Are Joins over LSM-trees Ready? Take RocksDB as an Example 2025 VLDB 4.3556432e-05
9,318 Disaggregated State Management in Apache Flink® 2.0 2025 VLDB 4.3556432e-05
9,363 BonsaiKV: Towards Fast, Scalable, and Persistent Key-Value Stores with Tiered, Heterogeneous Memory System 2024 VLDB 4.3503444e-05
10,021 Hourglass: An Adaptive Range Filter with Lightweight Hybrid Encoding 2026 SIGMOD 4.1945683e-05
10,191 PartitionKV: Redesigning LSM-tree KV Stores on NVMs with Adaptive Partitioning for Reducing Write Stalls and Amplification 2026 SIGMOD 4.1945683e-05
10,419 Unified Lineage System: Tracking Data Provenance at Scale 2025 SIGMOD 4.1945683e-05
10,643 Keigo: Co-designing Log-Structured Merge Key-Value Stores with a Non-Volatile, Concurrency-aware Storage Hierarchy 2025 VLDB 4.1945683e-05
10,766 Scribe: How Meta transports terabytes per second in real time 2025 VLDB 4.1945683e-05
11,485 Real-time Data Infrastructure at Uber 2021 SIGMOD 4.1945683e-05
11,625 InvaliDB: Scalable Push-Based Real-Time Queries on Top of Pull-Based Databases (Extended) 2020 VLDB 4.1945683e-05
11,805 CarStream: An Industrial System of Big Data Processing for Internet-of-Vehicles 2017 VLDB 4.1945683e-05
11,807 Upsortable: Programming Top-K Queries Over Data Streams 2017 VLDB 4.1945683e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 11 of 11 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
288 Storm @Twitter 2014 SIGMOD 0.00028939871
314 MillWheel: Fault-Tolerant Stream Processing at Internet Scale 2013 VLDB 0.00028084774
538 The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing 2015 VLDB 0.00020678804
824 Twitter Heron: Stream Processing at Scale 2015 SIGMOD 0.0001623129
1,226 Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management 2013 SIGMOD 0.00013180799
1,286 Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams 2013 SIGMOD 0.0001282373
1,487 Scuba: Diving into Data at Facebook 2013 VLDB 0.00011701099
1,794 Summingbird: A Framework for Integrating Batch and Online MapReduce Computations 2014 VLDB 0.00010532024
2,264 S-Store: Streaming Meets Transaction Processing 2015 VLDB 9.1575142e-05
2,658 Data Warehousing and Analytics Infrastructure at Facebook 2010 SIGMOD 8.3607429e-05
5,120 Fast Database Restarts at Facebook 2014 SIGMOD 5.6803959e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

Overall Rank Paper Year Venue Pagerank
1,286 Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams 2013 SIGMOD 0.0001282373
7,660 Scalable Delivery of Stream Query Result 2009 VLDB 4.6862657e-05
6,131 Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture 2013 SIGMOD 5.1956688e-05
2,658 Data Warehousing and Analytics Infrastructure at Facebook 2010 SIGMOD 8.3607429e-05
11,485 Real-time Data Infrastructure at Uber 2021 SIGMOD 4.1945683e-05
9,504 Supporting Scalable Analytics with Latency Constraints 2015 VLDB 4.3341665e-05
2,706 Design, Implementation, and Evaluation of the Linear Road Benchmark on the Stream Processing Core 2006 SIGMOD 8.2673299e-05
1,499 Apache Hadoop Goes Realtime at Facebook 2011 SIGMOD 0.00011675192
6,850 Petabyte Scale Databases and Storage Systems at Facebook 2013 SIGMOD 4.9085019e-05
6,436 Providing Streaming Joins as a Service at Facebook 2018 VLDB 5.0636254e-05