### Abstract

This paper establishes a negative result for clustering: above a certain ratio of random noise to nonrandom information, it is impossible for a large class of cost functions to distinguish between two partitions of a data set. In particular, it is shown that as the dimension increases, the ability to distinguish an accurate partitioning from an inaccurate one is lost unless the informative components are both sufficiently numerous and sufficiently informative. We examine squared error cost functions in detail. More generally, it is seen that the VC-dimension is an essential hypothesis for the class of cost functions to satisfy for an impossibility proof to be feasible. Separately, we provide bounds on the probabilistic behavior of cost functions that show how rapidly the ability to distinguish two clusterings decays. In two examples, one simulated and one with genomic data, bounds on the ability of squared-error and other cost functions to distinguish between two partitions are computed. Thus, one should not rely on clustering results alone for high dimensional low sample size data and one should do feature selection.

Original language | English (US) |
---|---|

Pages (from-to) | 30-53 |

Number of pages | 24 |

Journal | Statistical Analysis and Data Mining |

Volume | 4 |

Issue number | 1 |

DOIs | |

State | Published - Feb 2011 |

### Keywords

- Clustering impossibility
- Cost function
- High dimensions
- VC-dimension

### ASJC Scopus subject areas

- Analysis
- Information Systems
- Computer Science Applications

## Fingerprint Dive into the research topics of 'On the limits of clustering in high dimensions via cost functions'. Together they form a unique fingerprint.

## Cite this

*Statistical Analysis and Data Mining*,

*4*(1), 30-53. https://doi.org/10.1002/sam.10095