Checking if a point is included in a quad (2D)

Let P(i) = (x(i), y(i)) be the 4 points defining your quad, with i going from 0 to 3. Let M(x, y) be the point that we want to find out if it is contained in the quad or not.

For the i-th side of the quad, calculate the directional vector of the side as follows : u(i) = P(i + 1 % 3) - P(i)

Then, calculate the cartesian equation for the i-th side. If that equation is a(i)x + b(i)y + c(i) = 0, then u(i) = (-b(i), a(i)). From that, we calculate :

a(i) = u(i).y
b(i) = -u(i).x
c(i) = -(a(i) * x(i) + b(i) * y(i))
because we know the point P(i) = (x(i), y(i)) is actually on the side, so we can use it to find the cartesian equation.

Then, apply the cartesian equation to the point M :

result = a(i) * M.x + b(i) * M.y + c(i)

If result = 0, the point M is right on the side. If result is non-zero, the point M's position, relative to the side, is given by the sign of the result.

To avoid having to know if a negative or positive result means left or right, apply this method with two sides at once, and just test if the results of the cartesian equations of the two sides have opposite signs. If yes, that means the point is between the two sides. Then, do the same for the two remaining sides.