New CDL clustering index handles non-convex data without extra tuning

A new paper on arXiv introduces the Central Description Length (CDL) clustering validation index, designed to overcome a long-standing limitation of internal clustering metrics. Traditional indices like the silhouette score or Davies–Bouldin index rely on Euclidean compactness and separation, making them biased toward convex, spherical clusters. They fail on non-convex, irregular, or variable-density data without expensive kernel transformations or alternative distance measures. CDL addresses this by computing a probabilistic upper bound on the description length of the unobservable true cluster centers, using only observed within-cluster compactness, estimated centers, and covariances. This bound condenses intra-cluster compactness and centroid displacement into a single computable quantity—no ground truth labels are required.

On synthetic benchmarks with non-convex and arbitrary shape clusters, CDL-CVI selected the reference number of clusters more often and achieved higher Adjusted Rand Index (ARI) values than conventional CVIs without any kernel preprocessing stage. On image benchmarks (MNIST, CIFAR-10, STL-10) clustered from frozen unsupervised embeddings, CDL returned cluster numbers close to the true class counts across K-means, DBSCAN, and spectral clustering. The approach avoids the extra tuning and computational cost of kernel methods, making it a practical drop-in replacement for real-world unsupervised pipelines in sensor data, image segmentation, and process monitoring.