In some cases it's better to leave the golf balls, some birds get upset if they get removed as it makes the nest seem unsafe.
Birds that are going to go broody will do so regardless if there's anything in the box. That said, I only leave 1 golf ball per nest box now, but that's more to reduce...