CaltechAUTHORS
  A Caltech Library Service

Reliable and Efficient Long-Term Social Media Monitoring

Cao, Jian and Adams-Cohen, Nicholas and Alvarez, R. Michael (2020) Reliable and Efficient Long-Term Social Media Monitoring. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20210510-141337458

[img] PDF - Submitted Version
See Usage Policy.

178kB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20210510-141337458

Abstract

Social media data is now widely used by many academic researchers. However, long-term social media data collection projects, which most typically involve collecting data from public-use APIs, often encounter issues when relying on local-area network servers (LANs) to collect high-volume streaming social media data over long periods of time. In this technical report, we present a cloud-based data collection, pre-processing, and archiving infrastructure, and argue that this system mitigates or resolves the problems most typically encountered when running social media data collection projects on LANs at minimal cloud-computing costs. We show how this approach works in different cloud computing architectures, and how to adapt the method to collect streaming data from other social media platforms.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/2005.02442arXivDiscussion Paper
ORCID:
AuthorORCID
Adams-Cohen, Nicholas0000-0003-2251-1744
Alvarez, R. Michael0000-0002-8113-4451
Alternate Title:Reliable and Efficient Long-Term Twitter Monitoring
Additional Information:We thank the John Randolph Haynes and Dora Haynes Foundation for supporting some of this research. We received support for our use of Google Cloud Platform through Google’s COVID-19 research program. We also thank Anima Anandkumar and Anqi Liu for their work with us on related projects.
Funders:
Funding AgencyGrant Number
John Randolph Haynes and Dora Haynes FoundationUNSPECIFIED
Google Cloud PlatformUNSPECIFIED
Subject Keywords:Social media, Cloud computing, Twitter, Time series
Record Number:CaltechAUTHORS:20210510-141337458
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20210510-141337458
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:109048
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:10 May 2021 21:24
Last Modified:10 May 2021 21:24

Repository Staff Only: item control page