Large-Scale Machine Learning at Twitter

Summary: Case study of integrating ML into Twitter's Hadoop/Pig analytics stack. Pig extensions enable supervised learning with online SGD and ensembles; ML tasks (sampling, feature generation, training, testing) run as Pig loaders/UDFs, making ML a first-class Pig script in production. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 4665
Venue: SIGMOD
Year: 2012
Pagerank: 7.1764857e-05
Overall Rank: 3,714 | 74.52%
DOI: 10.1145/2213836.2213958

Incoming Non-self Citations Over Time

Authors

1. Jimmy Lin (Twitter)
2. Alek Kolcz (Twitter)

BibTeX Citation

@inproceedings{lin_sigmod12,
        title = {{Large-Scale Machine Learning at Twitter}},
        author = {Lin, Jimmy and Kolcz, Alek},
        series = {{SIGMOD} '12},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/2213836.2213958},
        url = {https://dl.acm.org/doi/10.1145/2213836.2213958},
        year = {2012}
}

Incoming Citations (Sorted by Pagerank)

Showing 9 of 9 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,079	Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML	2014	VLDB	0.00012258469
1,639	Summingbird: A Framework for Integrating Batch and Online MapReduce Computations	2014	VLDB	0.00010155021
4,169	The Unified Logging Infrastructure for Data Analytics at Twitter	2012	VLDB	6.8561463e-05
4,254	GraphJet: Real-Time Content Recommendations at Twitter	2016	VLDB	6.8024576e-05
4,595	The "Big Data" Ecosystem at LinkedIn	2013	SIGMOD	6.6137084e-05
5,182	Probabilistic Demand Forecasting at Scale	2017	VLDB	6.3287692e-05
5,748	Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture	2013	SIGMOD	6.1014813e-05
9,083	Tripartite Graph Clustering for Dynamic Sentiment Analysis on Social Media	2014	SIGMOD	5.3251649e-05
12,298	Optimization Strategies for A/B Testing on HADOOP	2013	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 7 of 7 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
6	Pig Latin: A Not-So-Foreign Language for Data Processing	2008	SIGMOD	0.0010686205
120	HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads	2009	VLDB	0.00031680027
155	MAD Skills: New Analysis Practices for Big Data	2009	VLDB	0.00028713176
452	An Architecture for Parallel Topic Models	2010	VLDB	0.00018146809
642	Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience	2009	VLDB	0.00015395331
2,159	Efficient Processing of Data Warehousing Queries in a Split Execution Environment	2011	SIGMOD	9.061086e-05
2,942	Llama: Leveraging Columnar Storage for Scalable Join Processing in the MapReduce Framework	2011	SIGMOD	7.9358593e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	9,640	Supporting Scalable Analytics with Latency Constraints	2015	VLDB
2	2,265	A Platform for Scalable One-Pass Analytics using MapReduce	2011	SIGMOD
3	1,079	Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML	2014	VLDB
4	5,748	Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture	2013	SIGMOD
5	4,859	Machine Learning for Big Data	2013	SIGMOD
6	2,191	Data Warehousing and Analytics Infrastructure at Facebook	2010	SIGMOD
7	6	Pig Latin: A Not-So-Foreign Language for Data Processing	2008	SIGMOD
8	4,169	The Unified Logging Infrastructure for Data Analytics at Twitter	2012	VLDB
9	642	Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience	2009	VLDB
10	4,595	The "Big Data" Ecosystem at LinkedIn	2013	SIGMOD