A Caltech Library Service

Reliable and Efficient Long-Term Social Media Monitoring

Cao, Jian and Adams-Cohen, Nicholas and Alvarez, R. Michael (2020) Reliable and Efficient Long-Term Social Media Monitoring. . (Unpublished)

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Social media data is now widely used by many academic researchers. However, long-term social media data collection projects, which most typically involve collecting data from public-use APIs, often encounter issues when relying on local-area network servers (LANs) to collect high-volume streaming social media data over long periods of time. In this technical report, we present a cloud-based data collection, pre-processing, and archiving infrastructure, and argue that this system mitigates or resolves the problems most typically encountered when running social media data collection projects on LANs at minimal cloud-computing costs. We show how this approach works in different cloud computing architectures, and how to adapt the method to collect streaming data from other social media platforms.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper
Adams-Cohen, Nicholas0000-0003-2251-1744
Alvarez, R. Michael0000-0002-8113-4451
Alternate Title:Reliable and Efficient Long-Term Twitter Monitoring
Additional Information:We thank the John Randolph Haynes and Dora Haynes Foundation for supporting some of this research. We received support for our use of Google Cloud Platform through Google’s COVID-19 research program. We also thank Anima Anandkumar and Anqi Liu for their work with us on related projects.
Funding AgencyGrant Number
John Randolph Haynes and Dora Haynes FoundationUNSPECIFIED
Google Cloud PlatformUNSPECIFIED
Subject Keywords:Social media, Cloud computing, Twitter, Time series
Record Number:CaltechAUTHORS:20210510-141337458
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:109048
Deposited By: George Porter
Deposited On:10 May 2021 21:24
Last Modified:10 May 2021 21:24

Repository Staff Only: item control page