I think you're undervaluing the biological element of attraction and sex and overvaluing the cultural element. Your argument seems to imply that if biology plays a role, then it's transphobic.