I think that you need to photograph what you respond to. If there's no emotional connection to the work, then no matter how good it is technically, it won't succeed. It will be lifeless. That said, during your learning period, at least, you should experience photographing at least men and women (and if you're fortunate enough, someone of ambiguous or trans-gendered), to learn the differences and nuances of each. You may find that your expected preference is NOT what works best for you artistically after all - photographing men is different than photographing women, and photographing trans people is a whole other ball of wax entirely, as it demands you throw out your preconceptions of gender roles and identity.
I have male friends who are heterosexual as the day is long, but they won't photograph female nudes because for them there is too much sexual tension. They are interested in shooting male nudes because they can work with the model without that tension. I've worked mostly with male nudes, but I have also worked with female nudes back in the day when I was learning the ropes, and I will probably again as I have a project that really needs female models for some roles.
In short, don't limit yourself for the sake of limiting yourself- do it when and because it makes sense for your vision. Don't be afraid to break that limit if it starts to limit your vision, rather than guide it.
. that was a rather strange attempt at creating a semantic distinction where none existed, was needed, or wanted. Too bad Zoe wasn't still here when he came along- those two would have had quite the popcorn-popping displays I bet...