Self-tuning techniques for large scale cluster analysis on textual data collections

This paper proposes PASTA (PArameter-free Solutions for Textual Analysis), a large scale engine providing strategies to automatically tune the algorithm parameters for the whole text clustering process. A data weighting strategy (e.g., TF-IDF) and a transformation method of input data (e.g., LSI) is explored before performing the cluster analysis to reduce sparseness, and make the overall analysis problem more eectively tractable. PASTA includes auto-selection strategies to o-load the end-user from parameter tuning and achieve a good quality of the clustering results. PASTA's current implementation runs on Apache Spark, a state-of-the-art distributed computing framework. As a case study, PASTA has been validated on three collections of Wikipedia documents. The experimental results show the eectiveness and the eciency of the proposed solution in analyzing collections of documents without tuning algorithm parameters and in discovering cohesive and well-separated groups of documents.