Loki's insults in the poem Lokasenna aren't necessarily truthful. The Norse, like other medieval people including the Scots and the Inuit, had a traditon of insult contests called flyting that served to show off verbal cleverness and channel agression. There are many examples of this in Norse sagas, where people insult each other in ways that aren't even physically possible.
We know two important things about Njord: 1) he was sent as a hostage to the Aesir to end the war between them and the Vanir gods, and 2) he married a giantess (Skadi) to keep peace between the gods and giants, and it ended when she left him.
There has been a lot of discussion about whether Njord may also have been sent as a hostage to the giants, but if you take the two facts above and add to it the story of how the giantess Gjalp nearly drowned Thor by pissing into a river he was trying to cross, you can see how Loki was inspired.
The fact that in the story of Njord and Skadi the sea-god seems particularly passive, not even getting a say in being married off to the giantess, may also have contributed. After all, an insult doesn't have to be true, it just has to be funny and likely to stick, which means it should have some basis in a person's behaviour or appearance.