Algodoo

I can cut up the box and the polygon with the knife tool and each smaller piece still detects my cursor by changing color, but not so with the circle.

It's actually kinda fun to do that!

How does your code detect the line intersections and where does the ray start? this method seems more reliable.

Q: How does your code detect the line intersections and where does the ray start?
A: It's Kilinich's code. It works as follows:

count := 0
foreach side in polygon:
. . . . if ray_intersects_segment(P,side) then
. . . . . . . . count = count + 1
if is_odd(count) then
. . . . . . . . return inside
. . . . else
. . . . . . . . return outside

See Ray-casting algorithm

Kilinich calculates the mouse position relative to the non-rotated polygon. He then subtracts the new mouse position from each line segment end point. If the new endpoints are above and below zero, and the new line segment is to the right of zero, then add one to the count. If the final count is odd then inside polygon.

Q: Where does the ray start?
A: The ray starts from zero (which is the mouse position after translation).

P.S. I just noticed I did a similar scene 4 years ago with Mouseover Code Injector. My approach at the time was to divide the polygon into triangles made from the polygon center and line segment end points and then check to see if the mouse position was inside any triangle.

Now the circle can be chopped up and each piece detects the cursor.

Here's a caveat: If you chop up those geometries into many smaller pieces, the computer lag will increase in proportion to how many peaces you make. So, don't over-do it!