A Caltech Library Service

Moving from Data-Constrained to Data-Enabled Research: Experiences and Challenges in Collecting, Validating and Analyzing Large-Scale e-Commerce Data

Bapna, Ravi and Goes, Paulo and Gopal, Ram and Marsden, James R. (2006) Moving from Data-Constrained to Data-Enabled Research: Experiences and Challenges in Collecting, Validating and Analyzing Large-Scale e-Commerce Data. Statistical Science, 21 (2). pp. 116-130. ISSN 0883-4237. doi:10.1214/088342306000000231.

PDF - Published Version
See Usage Policy.


Use this Persistent URL to link to this item:


Widespread e-commerce activity on the Internet has led to new opportunities to collect vast amounts of micro-level market and nonmarket data. In this paper we share our experiences in collecting, validating, storing and analyzing large Internet-based data sets in the area of online auctions, music file sharing and online retailer pricing. We demonstrate how such data can advance knowledge by facilitating sharper and more extensive tests of existing theories and by offering observational underpinnings for the development of new theories. Just as experimental economics pushed the frontiers of economic thought by enabling the testing of numerous theories of economic behavior in the environment of a controlled laboratory, we believe that observing, often over extended periods of time, real-world agents participating in market and nonmarket activity on the Internet can lead us to develop and test a variety of new theories. Internet data gathering is not controlled experimentation. We cannot randomly assign participants to treatments or determine event orderings. Internet data gathering does offer potentially large data sets with repeated observation of individual choices and action. In addition, the automated data collection holds promise for greatly reduced cost per observation. Our methods rely on technological advances in automated data collection agents. Significant challenges remain in developing appropriate sampling techniques integrating data from heterogeneous sources in a variety of formats, constructing generalizable processes and understanding legal constraints. Despite these challenges, the early evidence from those who have harvested and analyzed large amounts of e-commerce data points toward a significant leap in our ability to understand the functioning of electronic commerce.

Item Type:Article
Related URLs:
URLURL TypeDescription
Additional Information:© 2006 Institute of Mathematical Statistics. The research reported here has been supported by the Center for Internet Data Research and Intelligence Services (CIDRIS), Department of Operations and Information Management (OPIM), University of Connecticut. CIDRIS has been funded in part by a 2004 University of Connecticut Provost Grant. More information about CIDRIS is available at The authors would like to thank Rob Garfinkel, Kim Marsden and Dave Pingry for their helpful suggestions in improving the manuscript.
Funding AgencyGrant Number
University of ConnecticutUNSPECIFIED
Subject Keywords:Large-scale, Internet data, web crawling agents, online auctions, music file sharing.
Other Numbering System:
Other Numbering System NameOther Numbering System ID
Zentralblatt MATH05191854
Issue or Number:2
Record Number:CaltechAUTHORS:20100713-080413320
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:19012
Deposited By: Ruth Sustaita
Deposited On:13 Jul 2010 16:05
Last Modified:08 Nov 2021 23:49

Repository Staff Only: item control page