That is a complicated puzzle to solve.
If I read the question correctly, the pendulum rudder would need to be in front of the vane mast?
At first glance, a system could be set up that way, but I'd have to study that setup to see if the geometry would actually steer the boat.
It would mean that the pendulum rudder can not be swung out of the water when at rest, as it would be between mounting tubes.
But the distance between the pendulum rudder and vane mast can be made "at any length"