Why are clothes categorized in stores by genders instead of sizes?

My woman body isn't physically incapable of wearing clothes intended for men, but it is incapable of fitting well into something that is too small or too big.

Arranging clothing by sizes makes so much more sense. Gendering clothing is very stupid.