There still remains much controversy on this question. In fact, there are books written devoted to just this narrative. It's frankly impossible to answer, however, I've outlined some of the theories.
On the surface, god prefers blood sacrifices. Surely we must have, at least at some layer, some etiological myth that explains why the ancient Hebrews used lambs, bulls, and birds for sacrifices instead of more mundane objects like vegetables.
Others have noted that it may preserve a tradition of backlash against the introduction of agriculture in the region. Abel's offerings would be the offerings of a semi-nomadic shepherd, while Cain's is indicative of a farmer.
Some like Frazer thinks this is a fertility narrative, and God's rejection has nothing to do with the original story except to serve as a device for moving the narrative. The real story is the slaughter of Abel, whose blood "fertilizes" the earth. This is apparent in the Greek translation (which actually may precede the current, Masoretic text of the Old Testament, or at least is a translation of an earlier Hebrew text), which has Cain and Abel in a field, and—in both Greek and Hebrew—the earth "opened her mouth to receive [Abel's] blood," which offers evidence of fertilizing the field. As Hooke notes, the Mark of Cain is symbolic of a larger plan of Yahweh's, that this was meant to happen, probably so not as to condemn all farmers.
Hooke also points out some Near Eastern parallels, though they're rather tenuous. One is the Babylonian New Year Festival, in which after slaughtering a lamb, two priests smear blood on the walls of a shrine, "after which they are obliged to flee into the...desert until the Festival is over, because they are defiled by their ritual act." Instead of a lamb, though, the ancient Hebrews may have sacrificed a human, though this again seems far-fetched to me.
The ancient commentators, writing long afterward, would not have thought of this. Indeed, most, like Philo or Didymus or even Hellenistic Jewish writings, state or imply that Cain's offerings were rejected because Cain himself was evil, and thus warranted rejection.
Some starting points for further research:
J. D. Frazer 1918. Folklore in the Old Testament: Studies in Comparative Religion, Legend, and Law. MacMillan & Co.
S. H. Hooke 1939. "Cain and Abel," in Folklore vol. 50 no. 1 pp. 58–65.
G. D. Hornblower 1944. "Cain and Abel: The Choice of Kind of Sacrifice," in Man vol. 44 pp. 45–46.
E. A. Speiser 1964. Genesis: Introduction, Translation, and Notes. Anchor Bible Commentary. Doubleday.
D. Bergant 1992. The Collegeville Bible Commentary: Old Testament. Liturgical Press.
J. Byron 2011. *Cain and Abel in Text and Tradition: Jewish and Christian Interpretations of the First Sibling Rivalry." Brill.