I was recently experimenting with the linear separability of embeddings, and showed a graph which indicated the embeddings were substantially more linearly separable than the quantized embeddings. One person commented that this didn't make sense, and the quantized vectors should be as separable as the non-quantized vectors. That didn't seem right to me, but on the spot I couldn't think of a trivial counter example. In this blog, I present two.

The first, and most trivial case, is when you have more classes than quantized vectors. If you are quantizing your vectors to a set of N vectors, and you have N + 1 classes, then naturally two classes have to have overlap in this quantized space. However, such a reduced quantized space isn't as common in machine learning problems.

For the second example, consider the XOR problem. This is known to be not linearly separable. So in the case where we have 2 classes and 2 quantization vectors, then if class 1 maps to (0, 1) and (1, 0) and class 2 maps to (1, 1) and (0, 0) in quantized space, but the non-quantized vectors are separable then we have shown our goal. Given these four quantization vectors, consider the following embedding points, class 1: (0.75, -2), (-2, 0.8), class 2: (-0.75, 0.25), (0.8, 0.8), these embed to (1, 0), (0, 1), (0, 0), (1, 1). Clearly these embeddings are not linearly separable, but with f(x, y) = -0.225x - 0.37y - 0.1, the non-quantized vectors are.

Changelog

November 29, 2025: Published initial version.