Data mining a large digital sky survey: from the challenges to the scientific results

Abstract

The analysis and an efficient scientific exploration of the digital Palomar observatory sky survey represents a major technical challenge. The input data set consists of 3 Terabytes of pixel information, and contains a few billion sources. We describe some of the specific scientific problems posed by the data, including searches for distant quasars and clusters of galaxies, and the data-mining techniques we are exploring in addressing them Machine- assisted discovery methods may become essential for the analysis of such multi-Terabyte data sets. New and future approaches involve unsupervised classification and clustering analysis in the Giga-object data space, including various Bayesian techniques. In addition to the searches for known types of objects in this database, these techniques may also offer the possibility of discovering previously unknown, rare types of astronomical objects.