My concern would be diseases.
But how do they get in and out of your duck's yard? Is there no fencing? If no fencing then do your ducks go down to the pond also? Or maybe there is but they fly over it? Can you take some pictures of the geese and ducks when they visit - are they all domestic breeds?