Why does function createpolygonrgn() fail?

your code is very simple minded, and it is bad.

Quote:

i use Timer which redraws the digits every 100 milliseconds.

A simple test can divide by 10 the workload of redraw:

if (LastTime != NewTime) {
    redrawClock();
    LastTime = NewTime;
}

Rather than rebuilding the 2 same polygons every times

int trLen = min(rect.right, rect.bottom) *0.5;  // ◣ length of this triangle (there will be 4 of them)
bool bHorizontal = rect.right > rect.bottom; // should we rotate the thing?

// make clipping rgn that will make rgn look like a dash in digit clock
POINT pointsVert[] = { trLen, 0,   trLen *2, trLen,    trLen *2, rect.bottom -trLen,
        trLen, rect.bottom,    0, rect.bottom -trLen,    0, trLen};
POINT pointsHorz[] = { trLen, 0,   rect.right -trLen, 0,    rect.right, trLen,
        rect.right -trLen, rect.bottom,    trLen, rect.bottom,    0, trLen };

POINT *ptArray = bHorizontal ? pointsHorz : pointsVert;

wgeb you know that only 1 will be useful

int trLen = min(rect.right, rect.bottom) *0.5;  // ◣ length of this triangle (there will be 4 of them)
// make clipping rgn that will make rgn look like a dash in digit clock
if (rect.right > rect.bottom) { // should we rotate the thing?
    POINT pointsHorz[] = { trLen, 0,   rect.right -trLen, 0,    rect.right, trLen,  rect.right -trLen, rect.bottom,    trLen, rect.bottom,    0, trLen };
    POINT *ptArray = pointsHorz;
}
else {
    POINT pointsVert[] = { trLen, 0,   trLen *2, trLen,    trLen *2, rect.bottom -trLen, trLen, rect.bottom,    0, rect.bottom -trLen,    0, trLen};
    POINT *ptArray = pointsVert;
}

This rebuild only the useful polygon.
You can also the 2 polygons once for all and keep then in persistent variables between calls.

Your problem is here:

FillRgn(hdc, hrgnClip, CreateSolidBrush(color));

in creating a brush and NOT deleting it.

Asign it to a var use it and than delete it later, or if the color is constant you need to create it only once.

Your error case is typical for not release GUI objects correctly ;-)

Since you are using C++, I recommend writing a class to handle this for you. Here's one possibility :

template< typename T >
class handleobj
{
public:
   handleobj( T h )
   {
       m_handle = h;
   }

   virtual ~handleobj()
   {
       if( m_prevObj )  // restore previous object, for fonts and such
       {
          ::SelectObject( m_hDC, m_prevObj );
       }
       if( m_handle )
       {
          ::DeleteObject( m_handle );
          m_handle = nullptr;
       }
   }

   operator T()    { return m_handle; }

public:
    HDC m_hDC     = { nullptr };
    T   m_handle  = { nullptr };
    T   m_prevObj = { nullptr };   // to support selection
};

// usage example :

   // the brush object will delete itself automatically
   handleobj< HBRUSH > solidBrush( CreateSolidBrush( color ) );
   FillRgn(hdc, hrgnClip, solidBrush );

I have not tried this in operation but it will compile. If you want, you can add methods to the class that will select a GDI object and save the DC and previous object. The destructor of the class is ready to deal with that. It's also ready if you never do that. Another option is to make derived classes for things like fonts that create them and select them into a DC. There are a lot of possibilities.

You might think that's a lot of work to destroy a brush handle BUT once you have this and use it you will never have to remember to destroy a brush again because this class will do it for you automatically. It fact, it can be used with any GDI object that uses DeleteObject so deleting GDI objects will be automatic.