Large-Scale Machine Learning at Twitter
Summary: Case study of integrating ML into Twitter's Hadoop/Pig analytics stack. Pig extensions enable supervised learning with online SGD and ensembles; ML tasks (sampling, feature generation, training, testing) run as Pig loaders/UDFs, making ML a first-class Pig script in production. (summarized by gpt-5-nano on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Jimmy Lin
- 2. Alek Kolcz
Incoming Citations (Sorted by Pagerank)
Showing 9 of 9 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 1,407 | Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML | 2014 | VLDB | 0.00012163413 |
| 1,793 | Summingbird: A Framework for Integrating Batch and Online MapReduce Computations | 2014 | VLDB | 0.00010522029 |
| 4,563 | The Unified Logging Infrastructure for Data Analytics at Twitter | 2012 | VLDB | 6.0752535e-05 |
| 4,859 | The "Big Data" Ecosystem at LinkedIn | 2013 | SIGMOD | 5.8680444e-05 |
| 4,889 | GraphJet: Real-Time Content Recommendations at Twitter | 2016 | VLDB | 5.8480413e-05 |
| 5,265 | Probabilistic Demand Forecasting at Scale | 2017 | VLDB | 5.5950627e-05 |
| 6,135 | Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture | 2013 | SIGMOD | 5.1908958e-05 |
| 8,931 | Tripartite Graph Clustering for Dynamic Sentiment Analysis on Social Media | 2014 | SIGMOD | 4.4229886e-05 |
| 12,109 | Optimization Strategies for A/B Testing on HADOOP | 2013 | VLDB | 4.1905499e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 7 of 7 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 3 | Pig Latin: A Not-So-Foreign Language for Data Processing | 2008 | SIGMOD | 0.0024217964 |
| 158 | HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads | 2009 | VLDB | 0.00040401371 |
| 169 | MAD Skills: New Analysis Practices for Big Data | 2009 | VLDB | 0.00038967371 |
| 330 | An Architecture for Parallel Topic Models | 2010 | VLDB | 0.00027271063 |
| 776 | Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience | 2009 | VLDB | 0.00016765694 |
| 2,340 | Efficient Processing of Data Warehousing Queries in a Split Execution Environment | 2011 | SIGMOD | 9.001663e-05 |
| 3,120 | Llama: Leveraging Columnar Storage for Scalable Join Processing in the MapReduce Framework | 2011 | SIGMOD | 7.533894e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| Overall Rank | Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 2,476 | A Platform for Scalable One-Pass Analytics using MapReduce | 2011 | SIGMOD | 8.6907971e-05 |
| 9,505 | Supporting Scalable Analytics with Latency Constraints | 2015 | VLDB | 4.3300131e-05 |
| 1,407 | Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML | 2014 | VLDB | 0.00012163413 |
| 6,135 | Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture | 2013 | SIGMOD | 5.1908958e-05 |
| 4,913 | Machine Learning for Big Data | 2013 | SIGMOD | 5.8320287e-05 |
| 2,658 | Data Warehousing and Analytics Infrastructure at Facebook | 2010 | SIGMOD | 8.3634079e-05 |
| 3 | Pig Latin: A Not-So-Foreign Language for Data Processing | 2008 | SIGMOD | 0.0024217964 |
| 4,563 | The Unified Logging Infrastructure for Data Analytics at Twitter | 2012 | VLDB | 6.0752535e-05 |
| 776 | Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience | 2009 | VLDB | 0.00016765694 |
| 4,859 | The "Big Data" Ecosystem at LinkedIn | 2013 | SIGMOD | 5.8680444e-05 |