It just composes a 32-bit value from the three components r, g, and b. Here is it step by step. First let's take the outer parentheses and cast to COLORREF away:
(((BYTE)(r)|((WORD)((BYTE)(g))<<8))|(((DWORD)(BYTE)(b))<<16))
The r value is just reduced to 8-bits by the cast to BYTE and copied into the lower 8 bits:
(BYTE)(r)
The green value is also masked by its lower 8 bits, then cast into 16-bits (which is unnecessary) and shifted 8 bits to the left so that the green component now occupies bits 8...15:
((WORD)((BYTE)(g))<<8)
Both components are bitwise or-ed together:
((BYTE)(r)|((WORD)((BYTE)(g))<<8))
Finally the blue values is masked to 8 bits, cast up to 32-bit (again unnecessary) and shifted 16 bits to the left such that the blue component occupies now bits 16...23:
(((DWORD)(BYTE)(b))<<16)
Then it's combined to the red and green combined value that we did in the previous step. So we end up with
red: bits 0...7
green: bits 8...15
blue: bits 16...23
Easy, wasn't it.