Well, let's start with the obvious: where the jotuns in fact, giants?
(I will be using "jotuns" for the group throughout, to have avoid confusion).
This seems likely, on at least some level; etymologically, the word means "(great) eater", so it seems like that they were always associated with large size.
Now, the source which probably is best for size comparisons between giants and gods is the story in Gylfaginning about Thor's journey to Útgarða-Loki, which he does in company of (ordinary) Loki and Tjalfi and his human servants Þjálfi and Röskva. On their way, they meet up with the jotun Skrymir, which is much, much larger than them: they in fact mistake his gauntlet for a house. This indicates that jotuns other than Ymir can be really gigantic. It also does indicate that there is no significant difference in size between humans and gods.
When they arrive at Útgarða-Loki's castle, it is also enourmous. Even Thor can not open the gate, but they manage to squeeze in through the trellis. However, when they meet the inhabitants, it is only noted that they are "big enough". Finally, as Thor is asked to wrestle with Útgarða-Loki's "grandmother", she can not have appeared to be so much larger than him that it would be impossible.
It should be noted that Útgarða-Loki is a master of illusion, so that while the sizes of the giants where probably not surprising to the party, we can not take the size of anything else for granted.
As for other pairings of gods and giants, we should first note that all the Aesir are descended from jotuns, many born by jotun mothers (including Thor). Loki is in fact seen as a jotun. A few notable examples are Njord and Skaldi, Frey and Gerðr, Thor and Járnsaxa, and Odin and Gunnlöð.
Sources
- Snorri, Gylfaginning, in translation by Karl G. Johansson and Mats Malm
- SAOB, for the etymology of giant.
- Gro Steinsland, Fornnordisk religion, for double checking details.