ddo.aroughidea

Introduction

High-dimensional data is common in fields such as machine learning, bioinformatics, and natural language processing. However, visualizing and analyzing such data directly is difficult. Dimensionality reduction techniques aim to simplify this data by projecting it into lower-dimensional spaces (typically 2D or 3D), while preserving essential structure and relationships.

This set of experiments compares four widely used techniques—PCA, t-SNE, UMAP, and MDS—highlighting their similarities, differences, and typical use cases in the context of visualization and exploratory data analysis.

Summary

Descriptions

1. PCA (Principal Component Analysis)
Type: Linear
Goal: Reduce dimensionality by projecting data onto orthogonal axes that maximize variance.
Strengths: Fast, scalable, globally interpretable, suitable for preprocessing.
Limitations: Cannot capture nonlinear relationships.
Use Cases: Initial data exploration, feature decorrelation, input for other algorithms.

2. t-SNE (t-distributed Stochastic Neighbor Embedding)
Type: Nonlinear, probabilistic
Goal: Capture local relationships in a low-dimensional map.
Strengths: Excellent for visualizing clusters and local structures.
Limitations: Does not preserve global distances, high computational cost, no generalization to new data.
Use Cases: Visualizing embeddings, cluster inspection, exploratory analysis.

3. UMAP (Uniform Manifold Approximation and Projection)
Type: Nonlinear, manifold learning
Goal: Preserve topological and geometric structure at both local and global scales.
Strengths: Captures both local and some global structure, faster and more scalable than t-SNE, supports transforming new data points.
Limitations: Sensitive to hyperparameters, stochastic results unless seeded.
Use Cases: Large-scale data visualization, preprocessing for clustering/classification, embedding for interactive dashboards.

4. MDS (Multidimensional Scaling)
Type: Linear or nonlinear (depending on variant)
Goal: Preserve pairwise distances between all points in the low-dimensional embedding.
Strengths: Maintains global geometric structure; straightforward interpretation of distances.
Limitations: Computationally intensive for large datasets; sensitive to noise and poor local separation.
Use Cases: Visualizing similarity matrices, testing distance metrics, applications with meaningful global distances.

Technique Comparison Table

What Is a Permutation?

A permutation is an ordered arrangement of a set of distinct items. In data analysis, permutations commonly arise in problems involving rankings, orderings, or any scenario where the position of elements matters. The space of all permutations of n elements is called the symmetric group S_n.

Interpretation of Permutations

Permutations can be interpreted as discrete points in a structured, non-Euclidean space. Each permutation represents a specific ranking or sequencing, and no arithmetic mean exists in the traditional Euclidean sense. Common use cases include ranked-choice voting, preference modeling, and genome orderings.

Distance Metrics for Permutations

To compare permutations, specialized distance functions are used that reflect changes in ordering. These include:

• Kendall tau distance: Counts the number of pairwise disagreements (inversions).
• Spearman footrule: Sum of absolute differences in positions.
• Cayley distance: Minimum number of transpositions needed to transform one permutation into another.
• Hamming distance: Number of positions at which two permutations differ.

Why Euclidean Distance Is Inappropriate

Euclidean distance assumes data lies in a continuous vector space, where differences between numeric values are meaningful in terms of spatial geometry. In permutation space:

• There is no canonical embedding into ℝⁿ where Euclidean distance reflects rank change.
• Arithmetic operations such as averaging do not preserve permutation structure.
• Symmetric group topology does not align with Euclidean assumptions.

As a result, using Euclidean distance can yield misleading clustering outcomes. Specialized permutation distances are necessary to maintain meaningful similarity relationships.

Use in Clustering

When clustering permutations, use distance metrics that respect the permutation structure (e.g., Kendall tau, Spearman footrule). These metrics support hierarchical and k-medoids clustering, but are generally not suitable for centroid-based algorithms like k-means, which rely on Euclidean space properties.