DBScholar

Back to papers

Realtime Data Processing at Facebook

Summary: Realtime data processing at Facebook handles hundreds of GB/s across pipelines, guided by five design decisions on usability, performance, fault tolerance, scalability, and correctness. Seconds-latency with a persistent message bus enables Puma, Swift, and Stylus, delivering scalable, fault-tolerant streaming and valuable design lessons. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
5266
Venue
SIGMOD
Year
2016
Pagerank
0.00010943767
Overall Rank
1,389 | 90.48%
DOI
10.1145/2882903.2904441

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@inproceedings{chen_sigmod16,
        title = {{Realtime Data Processing at Facebook}},
        author = {Chen, Guoqiang Jerry and Wiener, Janet L. and Iyer, Shridhar and Jaiswal, Anshul and Lei, Ran and Simha, Nikhil and Wang, Wei and Wilfong, Kevin and Williamson, Tim and Yilmaz, Serhat},
        series = {{SIGMOD} '16},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/2882903.2904441},
        url = {https://dl.acm.org/doi/10.1145/2882903.2904441},
        year = {2016}
}

Incoming Citations (Sorted by Pagerank)

Showing 23 of 23 citing papers.

Rank Citing Paper Year Venue Pagerank
472 Optimizing Space Amplification in RocksDB 2017 CIDR 0.000179044
1,213 MyRocks: LSM-Tree Database Storage Engine Serving Facebook's Social Graph 2020 VLDB 0.00011646797
1,954 Chucky: A Succinct Cuckoo Filter for LSM-Tree 2021 SIGMOD 9.4208165e-05
2,698 Rosetta: A Robust Space-Time Optimized Range Filter for Key-Value Stores 2020 SIGMOD 8.2450522e-05
3,054 Spooky: Granulating LSM-Tree Compactions Correctly 2022 VLDB 7.8090808e-05
5,821 StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance 2023 VLDB 6.07598e-05
6,108 Providing Streaming Joins as a Service at Facebook 2018 VLDB 5.9730511e-05
6,407 Shared Foundations: Modernizing Meta's Data Lakehouse 2023 CIDR 5.8850639e-05
6,410 A Holistic View of Stream Partitioning Costs 2017 VLDB 5.884019e-05
7,189 BP-tree: Overcoming the Point-Range Operation Tradeoff for In-Memory B-trees 2023 VLDB 5.6775326e-05
7,757 CAMAL: Optimizing LSM-trees via Active Learning 2024 SIGMOD 5.5508469e-05
9,463 Hourglass: An Adaptive Range Filter with Lightweight Hybrid Encoding 2026 SIGMOD 5.2634238e-05
9,465 Are Joins over LSM-trees Ready? Take RocksDB as an Example 2025 VLDB 5.2634238e-05
9,468 Disaggregated State Management in Apache Flink 2.0 2025 VLDB 5.2634238e-05
9,507 BonsaiKV: Towards Fast, Scalable, and Persistent Key-Value Stores with Tiered, Heterogeneous Memory System 2024 VLDB 5.258497e-05
10,480 PartitionKV: Redesigning LSM-tree KV Stores on NVMs with Adaptive Partitioning for Reducing Write Stalls and Amplification 2026 SIGMOD 5.093636e-05
10,701 Unified Lineage System: Tracking Data Provenance at Scale 2025 SIGMOD 5.093636e-05
10,895 Keigo: Co-designing Log-Structured Merge Key-Value Stores with a Non-Volatile, Concurrency-aware Storage Hierarchy 2025 VLDB 5.093636e-05
10,996 Scribe: How Meta transports terabytes per second in real time 2025 VLDB 5.093636e-05
11,683 Real-time Data Infrastructure at Uber 2021 SIGMOD 5.093636e-05
11,820 InvaliDB: Scalable Push-Based Real-Time Queries on Top of Pull-Based Databases (Extended) 2020 VLDB 5.093636e-05
12,008 CarStream: An Industrial System of Big Data Processing for Internet-of-Vehicles 2017 VLDB 5.093636e-05
12,010 Upsortable: Programming Top-K Queries Over Data Streams 2017 VLDB 5.093636e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 11 of 11 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
220 Storm @Twitter 2014 SIGMOD 0.00024244587
224 MillWheel: Fault-Tolerant Stream Processing at Internet Scale 2013 VLDB 0.00024130894
361 The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing 2015 VLDB 0.00020138717
612 Twitter Heron: Stream Processing at Scale 2015 SIGMOD 0.0001573018
983 Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management 2013 SIGMOD 0.0001283214
1,171 Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams 2013 SIGMOD 0.00011826434
1,352 Scuba: Diving into Data at Facebook 2013 VLDB 0.00011064595
1,639 Summingbird: A Framework for Integrating Batch and Online MapReduce Computations 2014 VLDB 0.00010155021
1,915 S-Store: Streaming Meets Transaction Processing 2015 VLDB 9.4884706e-05
2,191 Data Warehousing and Analytics Infrastructure at Facebook 2010 SIGMOD 8.9780154e-05
5,019 Fast Database Restarts at Facebook 2014 SIGMOD 6.3987706e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 1,171 Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams 2013 SIGMOD
2 8,663 Scalable Delivery of Stream Query Result 2009 VLDB
3 5,748 Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture 2013 SIGMOD
4 2,191 Data Warehousing and Analytics Infrastructure at Facebook 2010 SIGMOD
5 11,683 Real-time Data Infrastructure at Uber 2021 SIGMOD
6 9,640 Supporting Scalable Analytics with Latency Constraints 2015 VLDB
7 2,393 Design, Implementation, and Evaluation of the Linear Road Benchmark on the Stream Processing Core 2006 SIGMOD
8 1,324 Apache Hadoop Goes Realtime at Facebook 2011 SIGMOD
9 6,676 Petabyte Scale Databases and Storage Systems at Facebook 2013 SIGMOD
10 6,108 Providing Streaming Joins as a Service at Facebook 2018 VLDB