If you mean "how do I detect a self-intersecting polygon?" such as an hourglass:
X------>X
^ /
\ /
\ /
\
/ \
/ \
v \
X------>X
Then it's non-trivial. But...several algorithms exist, including Bentley-Ottmann which is described here:
http://geomalgorithms.com/a09-_intersect-3.html[
^]
You may be able to find a suitable C++ or C# implementation via Google - I haven't looked - but it might be easier to write your own, the pseudocode examples look reasonably simple to follow.
[edit]Drew arrow head on wrong end... :doh: [/edit]