One Single Hub Text Breaks CLIP: Identifying Vulnerabilities in Cross-Modal Encoders Via Hubness

The hubness problem, in which hub embeddings are close to many unrelated examples, occurs often in high-dimensional embedding spaces and may pose a practical threat for purposes such as information retrieval and automatic evaluation metrics. In particular, since cross-modal similarity between tex...