Database Paper Browser

Back to papers

Realtime Data Processing at Facebook

Summary: Realtime data processing at Facebook handles hundreds of GB/s across pipelines, guided by five design decisions on usability, performance, fault tolerance, scalability, and correctness. Seconds-latency with a persistent message bus enables Puma, Swift, and Stylus, delivering scalable, fault-tolerant streaming and valuable design lessons. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
5205
Venue
SIGMOD
Year
2016
Pagerank
0.00011133818
Overall Rank
1,616 | 88.78%
DOI
10.1145/2882903.2904441

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 23 of 23 citing papers.

Rank Citing Paper Year Venue Pagerank
568 Optimizing Space Amplification in RocksDB 2017 CIDR 0.00019932335
1,614 MyRocks: LSM-Tree Database Storage Engine Serving Facebook's Social Graph 2020 VLDB 0.00011137963
2,797 Chucky: A Succinct Cuckoo Filter for LSM-Tree 2021 SIGMOD 8.1116755e-05
3,545 Rosetta: A Robust Space-Time Optimized Range Filter for Key-Value Stores 2020 SIGMOD 6.9831585e-05
3,970 Spooky: Granulating LSM-Tree Compactions Correctly 2022 VLDB 6.5756727e-05
5,292 StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance 2023 VLDB 5.5784769e-05
6,430 Providing Streaming Joins as a Service at Facebook 2018 VLDB 5.0587634e-05
6,628 A Holistic View of Stream Partitioning Costs 2017 VLDB 4.9833118e-05
6,716 Shared Foundations: Modernizing Meta's Data Lakehouse 2023 CIDR 4.9462431e-05
7,990 BP-tree: Overcoming the Point-Range Operation Tradeoff for In-Memory B-trees 2023 VLDB 4.6086117e-05
8,011 CAMAL: Optimizing LSM-trees via Active Learning 2024 SIGMOD 4.6022693e-05
9,322 Are Joins over LSM-trees Ready? Take RocksDB as an Example 2025 VLDB 4.351469e-05
9,323 Disaggregated State Management in Apache Flink® 2.0 2025 VLDB 4.351469e-05
9,371 BonsaiKV: Towards Fast, Scalable, and Persistent Key-Value Stores with Tiered, Heterogeneous Memory System 2024 VLDB 4.3461752e-05
10,021 Hourglass: An Adaptive Range Filter with Lightweight Hybrid Encoding 2026 SIGMOD 4.1905499e-05
10,191 PartitionKV: Redesigning LSM-tree KV Stores on NVMs with Adaptive Partitioning for Reducing Write Stalls and Amplification 2026 SIGMOD 4.1905499e-05
10,429 Unified Lineage System: Tracking Data Provenance at Scale 2025 SIGMOD 4.1905499e-05
10,651 Keigo: Co-designing Log-Structured Merge Key-Value Stores with a Non-Volatile, Concurrency-aware Storage Hierarchy 2025 VLDB 4.1905499e-05
10,772 Scribe: How Meta transports terabytes per second in real time 2025 VLDB 4.1905499e-05
11,489 Real-time Data Infrastructure at Uber 2021 SIGMOD 4.1905499e-05
11,629 InvaliDB: Scalable Push-Based Real-Time Queries on Top of Pull-Based Databases (Extended) 2020 VLDB 4.1905499e-05
11,813 CarStream: An Industrial System of Big Data Processing for Internet-of-Vehicles 2017 VLDB 4.1905499e-05
11,815 Upsortable: Programming Top-K Queries Over Data Streams 2017 VLDB 4.1905499e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 11 of 11 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
287 Storm @Twitter 2014 SIGMOD 0.00028917909
314 MillWheel: Fault-Tolerant Stream Processing at Internet Scale 2013 VLDB 0.00028059664
536 The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing 2015 VLDB 0.00020651621
822 Twitter Heron: Stream Processing at Scale 2015 SIGMOD 0.00016226851
1,226 Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management 2013 SIGMOD 0.00013168869
1,284 Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams 2013 SIGMOD 0.00012820565
1,488 Scuba: Diving into Data at Facebook 2013 VLDB 0.00011690191
1,793 Summingbird: A Framework for Integrating Batch and Online MapReduce Computations 2014 VLDB 0.00010522029
2,270 S-Store: Streaming Meets Transaction Processing 2015 VLDB 9.149662e-05
2,658 Data Warehousing and Analytics Infrastructure at Facebook 2010 SIGMOD 8.3634079e-05
5,121 Fast Database Restarts at Facebook 2014 SIGMOD 5.675304e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

Overall Rank Paper Year Venue Pagerank
1,284 Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams 2013 SIGMOD 0.00012820565
7,658 Scalable Delivery of Stream Query Result 2009 VLDB 4.6817213e-05
6,135 Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture 2013 SIGMOD 5.1908958e-05
2,658 Data Warehousing and Analytics Infrastructure at Facebook 2010 SIGMOD 8.3634079e-05
11,489 Real-time Data Infrastructure at Uber 2021 SIGMOD 4.1905499e-05
9,505 Supporting Scalable Analytics with Latency Constraints 2015 VLDB 4.3300131e-05
2,710 Design, Implementation, and Evaluation of the Linear Road Benchmark on the Stream Processing Core 2006 SIGMOD 8.2580301e-05
1,497 Apache Hadoop Goes Realtime at Facebook 2011 SIGMOD 0.00011668393
6,843 Petabyte Scale Databases and Storage Systems at Facebook 2013 SIGMOD 4.9046389e-05
6,430 Providing Streaming Joins as a Service at Facebook 2018 VLDB 5.0587634e-05