GDI+ is using matrices internally to sort out your transformations. So you need to be careful which order you do them in, for example if you had a sphere which you moved and then rotated it would be like the earth moving around the sun, if you rotated then moved it would be like the earth staying still but spinning around it's own axis.
So, if you want to rotate the image around some arbitrary point, you will need to shift the picture so that your rotation point is at (0,0) [which should by default be the centre of the image] then rotate by the desired amount, and then move it again to where you need it to be.
If your image was 100x100 for example the default centre would be (50,50) if you wanted to rotate it around (75,75) then move the picture 250 pixels to the right you would have to do this:
e.Graphics.TranslateTransform(-25,-25);
e.Graphics.RotateTransform(45);
e.Graphics.TranslateTransform(250 +25,25);
e.Graphics.ScaleTransform(0.25f, 0.25f);
e.Graphics.DrawImage( myImage, 0, 0 );
EDIT: Even easier that translating then rotating, I've just noticed that you could use
e.Graphics.Transform.RotateAt(PointF point, float angle)
which means you could skip the first translate and you won't have to compensate for it later.
EDIT: As for checking if the mouse cursor is within the rotated rectangle you're going to need to do a little more work. All you need to do is move the mouse position into the rectangles object space, so if you rotated the image by 45 degrees then you need to rotate the mouse position by -45 degrees around the same point. Even easier, you're using a matrix already so when you change the matrix to render your image, store it so that you can use is later. Then your check can become something like:
Matrix InverseMat;
void OnPaint(...)
{
...
InverseMat = e.Graphics.Transform;
InverseMat.Inverse();
...
}
bool RectContainsPoint(Rectangle rect, Point MousePos)
{
Point[] pts = { MousePos };
InverseMat.TransformPoints(pts);
return rect.Contains(pts[0]);
}