The last Ice age glaciers were gone at least by 5000 BCE. This is about 500 years before Proto-Indo-European is usually dated, which is about as far we can trace any shred of mythological knowledge, but which AFAIK does not include any Frost giants. Thus, no connection there.
Next step is the speakers of Proto-German. They are thought to have lived in southern Scandinavia and northern Germany, from about 500 BCE. They could most likely have come into contact with glaciers; there are today several in Norway, even in the southern part of the country. Proto-Germanic had a word for glacier, so they must have known of them.
But did these speakers of Proto-German have any concept of Frost Giants? That, we can not know. The first reliable knowledge of Norse mythology comes from Tacitus Germania, which mentions a few gods identifiable with deities in the Norse pantheon. Archeological finds with images that can be clearly identified with them are younger. However, this does not really tell us anything about giants: they could be older or younger than the Aesir and Vanir.
We can reconstruct several Proto-Germanic words for "giant", such as for jotun, which comes from "eater". The word for "Frost Giant" was "hrimþurs", where "hrim" meant "frost", and "þurs" is a word of more obscure origin but which means giant or monster. Again, there is no reason to connect it to glaciers.
The first sure mention of these Frost giants I've found is in Snorri's Edda from the 11th century CE. There, they are not in any way connected to glaciers. They are, however, mentioned to be evil. In general, though, most of the Norse stories we have show how giants and gods fight with each other, not with humans (at least not until Ragnarök).
To summarise: there is no direct reason, based on the myths or etymology, to suppose any connection between frost giants and glaciers. Glaciers, in general, does not reveal especially fertile lands - it is still far too cold next to them for the Norse to have had any use of the land for anything except maybe grazing. They also retreat rather slowly. However, there was (and still is) an ongoing process by which new, fertile land was revealed: the Plost-glacial rebound. This is especially noticeable in the areas around Stockholm and Uppsala in Sweden. However, I know of no myth that accounts for it.
The date of the Ice age, as well as the archeological finds of pcitures of Norse gods, I found in Sveriges Historia, 13000 f Kr -600 e Kr. The dating of PIE in Ola Wikander, Ett träd med vida grenar. The etymologies as per links in the text, as well as Svenska Akademiens Ordbok.