This legend is found in the Kojiki, the oldest "history book" from Japan. The original text makes the reason quite plain: because Izanami is a woman.
告其妹曰「女人先言、不良」 . . . 「因女先言而不良」
He then told his sister, "The woman speaking first is bad" . . . [they copulated anyway and then consulted the gods, who advised:] . . . "It was bad because the woman spoke first.
As is apparent by the use of Chinese, Kojiki was written at a time when Japan was importing Chinese ideas wholescale. One aspect of the Confucian philosophy was the concept "男尊女卑", lit. "male superiority, female inferiority". Since Kojiki was commissioned by the Imperial family for propaganda purposes, it has been theorised that this passage reflected the Yamato Court's then ongoing effort to implement Chinese legalism.
(Ancient Japan was not originally patriarchal; hence, the theorised need for ideological propaganda to justify the reforms being adopted at the time.)
There are various theoretical frameworks to rationalise this belief. For instance, under the yin and yang, duality men are held to be yang while women are yin. Yet, according to Dong Zhongsu's theorem:
All things rise and rests with yang; the year begins and ends with yang . . . therefore, yang is noble and ying is lowly.
Regardless, it's pretty obvious that Yannis' initial observation applies: it's blatant chauvinism.