Transformation of the RGB color cube

I assume you are talking about transformation in color space. One of the ways to transform a color space is color matrix. See http://en.wikipedia.org/w/index.php?title=Color_matrix&redirect=no[^].

The edge and the corner really present a problem.

You should understand that when a non-linear operator is applied to anything, part of information can be lost. If happens, for example, in all kinds of cut-off filter. For example, if you try to cut sine wave by amplitude, this is the case of such loss due to non-linearity to the filter. Well, pretty remote example. More close example is this: if you brighten the image, it can be a linear transform, but if you overdo it, part of pixels with different colors gets the same color. For example, in 8-bits-per-pixel gray image some pixels initially showing different shades of gray will get the value of 255. It means the loss of information, and the maximum value of byte act like a specific digital non-linear filter causing loss of some information. Another mode subtle kind of non-linearity is presented by digital discretization. For example, color grain in the picture is boosted in most color transforms just because, for example, the calculated pixel value is supposed to be 128.32142309, but it is rounded to 128.

Are you getting the picture? The color space but its nature is not a linear space as it is understood in mathematics. To be a linear space it has to be infinite. The non-linearity is introduced in digital word by discretization of color values and limitation of the ranges of values.

You can get the idea of color space here: http://en.wikipedia.org/wiki/Color_space[^], see also http://en.wikipedia.org/wiki/Color_theory[^].

Color spaces are created to model human color perception (which is in fact individual in different people) and are not of rectangular/cubical shape. In computer presentation, all pixel values are confined in some color cube, and the boundary of the cube present non-linearity. All values are limited between 0 and 2⁸−1 0 and or 0 and 2¹⁶−1. In wide-gamut digital cameras some 12 bits are used (not sure about so called "HDR" sensors, but they are not actually "HDR"). Here is how color transformation is done: you formally apply the matrix to pixel values, and when the resulting values go beyond the said range, they are just cut by maximum or minimum value. As simple as that.

To see it, take any photo editor and perform "strong" color transform — some parts of the picture will "burn out". Nothing else.

—SA