Natural Feature Tracking using optical flow

See more:

C#

Visual-Studio

VS2010

Emgu-CV

, +

This is what i want to do :

http://www.youtube.com/watch?v=0IVjxO63Fec[^]

so far, i managed to do the "optical flow" part. I calculated coordinates of the feature points on the current captured video frame given their coordinates on the previous frame by using OpenCV methods : cvGoodFeaturesToTrack and cvCalcOpticalFlowPyrLK. I'm using EmguCV (.Net wrapper to the OpenCV).

now what i need to do is placing the 3D object and move/rotate it according to the optical flow. how can i use these points to do that ?