There is here and there some confusions in your question.
The Nephilims
What your are talking about in your question are the "Nephilims". Let's read the Bible (I will use the NIV as a reference text).
6-1: When human beings began to increase in number on the earth and
daughters were born to them, the sons of God saw that the daughters
of humans were beautiful, and they married any of them they chose
6-4: The Nephilim were on the earth in those days—and also afterward—when
the sons of God went to the daughters of humans and had children by
them. They were the heroes of old, men of renown.
Notce that: the text imply the "sons of god" (not fallen angels) got children with "female human beings" (clearly). The text does not imply the resulting offsprings ARE giants in size.
The "Son of gods" is bene heloim in Hebrew, bene Heloim does not actually refer to "angels", fact is in modern judaism it does NOT at ALL. A certain number of late Hebrews/early christians commentors translated Bene heloim by fallen angel. Reason is their resulting offsrpings, the "Nephilims", is probably in Hebrew coming from "N-PH-L[naphal]" to fall.
Later on in the Numbers we find that:
13-3 We saw the Nephilim there (the descendants of Anak come from the
Nephilim). We seemed like grasshoppers in our own eyes, and we looked
the same to them."
Nephilim is translated either by "Nephilim" or "giants". It comes from the hebraic "hannəppilim" which has no direct translation.
The Bene Heloim
We find references on the "Sons of god" in biblical passages:
Job 1: 6 One day the angels[Bene Heloim] came to present themselves
before the Lord, and Satan also came with them.
Job 2-1 On another day the angels[Bene Heloim] came to present themselves before the
Lord, and Satan also came with them to present himself before him.
We see here the "Sons of gods" linked with Satan (the adversary).
Luke 20 :36 and they can no longer die; for they are like the angels.
They are God’s children [remind Luke is written in Greek], since they are children of the resurrection
Here, and everywhere, Jesus refers to the son of gods as human beings, without a doubt, they are "like angels", which fairly clearly imply they are NOT angels, and certainly not a fallen specie of some angels.
So the notions of the meaning of the bene heloim is not quite easy to understand. The NIV clearly use "angels", systematically. You can also find, depending the translatin, Sons of God. Heavenly beings.
You will find those "play with the translation" in most language. The sense to give to "bene heloim" is totally unclear.
Let's add that Job, which i quite used here totally know a term in Hebrew for "angel", as shown here:
4-18 If God places no trust in his servants, if he charges his angels [ube malakaw]
with error,
Kalm malak, malakaw in Hebrew meaning sending a messenger, in Greek an "Angelos". Here are your angels.
Conclusion
The Nephilims are giants: the Bible never explicitly mentions them as... Contrary to Goliath which is NOT a Nephilim, he is a Rephaim, common foes of those time Hebrews, and clearly depicted as a giant (His height was six cubits and a span ==> around 3m, when the Bible is mentioning giants, the Bible states their heights...). So EVEN if they was giants, this is not particulary a trait from them.
the Nephilims are the offspring of fallen angels: the term sons of gods in the Genesis is quite unclear; they could be angels, fallen angels, or humans one thing is clear! they are male... because they fall in love with the daughters of men. Fact is, they could perfectly be extraterrestrials or visitors from the future.
the Nephilims are highly praised (They were the heroes of old, men of renown) it seems pretty strange to praise the sons of random falty fallen angels of your God and seemingly survived the Flood as the rest of mankind (in those days—and also afterward)
As a lot of things here and there (Example: Excalibur) there is a traditional common culture which is linking the Nephilims as the giant offsprings of fallen angels and women. Except the Bible, while not stating this interpretation as false, neither specifically hint it as exact. The translation of Nephilim as giants and sons of god as angels is "personal".
the only thing I can do so, is debunking the extraordinary false conception.