DBScholar

Back to papers

Realtime Data Processing at Facebook

Summary: Realtime data processing at Facebook handles hundreds of GB/s across pipelines, guided by five design decisions on usability, performance, fault tolerance, scalability, and correctness. Seconds-latency with a persistent message bus enables Puma, Swift, and Stylus, delivering scalable, fault-tolerant streaming and valuable design lessons. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
h034a41a45c71db61
Venue
SIGMOD
Year
2016
Pagerank
0.00011065568
Overall Rank
1,310 | 91.23%
DOI
10.1145/2882903.2904441

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@inproceedings{chen_sigmod16,
        title = {{Realtime Data Processing at Facebook}},
        author = {Chen, Guoqiang Jerry and Wiener, Janet L. and Iyer, Shridhar and Jaiswal, Anshul and Lei, Ran and Simha, Nikhil and Wang, Wei and Wilfong, Kevin and Williamson, Tim and Yilmaz, Serhat},
        series = {{SIGMOD} '16},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/2882903.2904441},
        url = {https://dl.acm.org/doi/10.1145/2882903.2904441},
        year = {2016}
}

Incoming Citations (Sorted by Pagerank)

Showing 25 of 25 citing papers.

Rank Citing Paper Year Venue Pagerank
438 Optimizing Space Amplification in RocksDB 2017 CIDR 0.00018259155
1,147 MyRocks: LSM-Tree Database Storage Engine Serving Facebook's Social Graph 2020 VLDB 0.00011777851
1,991 Chucky: A Succinct Cuckoo Filter for LSM-Tree 2021 SIGMOD 9.2264312e-05
2,667 Rosetta: A Robust Space-Time Optimized Range Filter for Key-Value Stores 2020 SIGMOD 8.1324472e-05
2,882 Spooky: Granulating LSM-Tree Compactions Correctly 2022 VLDB 7.894356e-05
5,952 StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance 2023 VLDB 5.9182633e-05
6,240 Providing Streaming Joins as a Service at Facebook 2018 VLDB 5.8234435e-05
6,290 Shared Foundations: Modernizing Meta's Data Lakehouse 2023 CIDR 5.8051881e-05
6,553 A Holistic View of Stream Partitioning Costs 2017 VLDB 5.7306258e-05
7,049 BP-tree: Overcoming the Point-Range Operation Tradeoff for In-Memory B-trees 2023 VLDB 5.5967284e-05
7,925 CAMAL: Optimizing LSM-trees via Active Learning 2024 SIGMOD 5.4086685e-05
8,728 Upsortable: Programming Top-K Queries Over Data Streams 2017 VLDB 5.2705676e-05
8,930 InvaliDB: Scalable Push-Based Real-Time Queries on Top of Pull-Based Databases (Extended) 2020 VLDB 5.2361195e-05
9,678 Hourglass: An Adaptive Range Filter with Lightweight Hybrid Encoding 2026 SIGMOD 5.1258854e-05
9,681 Are Joins over LSM-trees Ready? Take RocksDB as an Example 2025 VLDB 5.1258854e-05
9,683 Disaggregated State Management in Apache Flink 2.0 2025 VLDB 5.1258854e-05
9,726 BonsaiKV: Towards Fast, Scalable, and Persistent Key-Value Stores with Tiered, Heterogeneous Memory System 2024 VLDB 5.1210873e-05
10,739 PartitionKV: Redesigning LSM-tree KV Stores on NVMs with Adaptive Partitioning for Reducing Write Stalls and Amplification 2026 SIGMOD 4.9605343e-05
10,835 Kirin: Efficient In-Storage Learned Compaction for LSM-Trees via System-Algorithm Co-Design 2026 VLDB 4.9605343e-05
11,106 The Dataflow Model Revisited Or: That Feeling When You Realize Every Problem You've Been Solving Is a Database Problem 2026 VLDB 4.9605343e-05
11,203 Unified Lineage System: Tracking Data Provenance at Scale 2025 SIGMOD 4.9605343e-05
11,356 Keigo: Co-designing Log-Structured Merge Key-Value Stores with a Non-Volatile, Concurrency-aware Storage Hierarchy 2025 VLDB 4.9605343e-05
11,431 Scribe: How Meta transports terabytes per second in real time 2025 VLDB 4.9605343e-05
12,040 Real-time Data Infrastructure at Uber 2021 SIGMOD 4.9605343e-05
12,353 CarStream: An Industrial System of Big Data Processing for Internet-of-Vehicles 2017 VLDB 4.9605343e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 11 of 11 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
219 MillWheel: Fault-Tolerant Stream Processing at Internet Scale 2013 VLDB 0.0002429959
233 Storm @Twitter 2014 SIGMOD 0.00023754096
328 The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing 2015 VLDB 0.00020889528
607 Twitter Heron: Stream Processing at Scale 2015 SIGMOD 0.00015577054
973 Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management 2013 SIGMOD 0.00012734154
1,178 Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams 2013 SIGMOD 0.00011605998
1,379 Scuba: Diving into Data at Facebook 2013 VLDB 0.00010836332
1,640 Summingbird: A Framework for Integrating Batch and Online MapReduce Computations 2014 VLDB 9.9825117e-05
1,951 S-Store: Streaming Meets Transaction Processing 2015 VLDB 9.302451e-05
2,243 Data Warehousing and Analytics Infrastructure at Facebook 2010 SIGMOD 8.767428e-05
5,154 Fast Database Restarts at Facebook 2014 SIGMOD 6.2372301e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 1,178 Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams 2013 SIGMOD
2 8,850 Scalable Delivery of Stream Query Result 2009 VLDB
3 5,842 Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture 2013 SIGMOD
4 2,243 Data Warehousing and Analytics Infrastructure at Facebook 2010 SIGMOD
5 12,040 Real-time Data Infrastructure at Uber 2021 SIGMOD
6 9,857 Supporting Scalable Analytics with Latency Constraints 2015 VLDB
7 2,453 Design, Implementation, and Evaluation of the Linear Road Benchmark on the Stream Processing Core 2006 SIGMOD
8 1,308 Apache Hadoop Goes Realtime at Facebook 2011 SIGMOD
9 6,814 Petabyte Scale Databases and Storage Systems at Facebook 2013 SIGMOD
10 6,240 Providing Streaming Joins as a Service at Facebook 2018 VLDB