tl;dr Mob rule.
Penelope didn't have a choice. She may have been Queen of Ithaca, but she had little actual power. All men loyal to Odysseus had followed him to Troy, she simply had no way of forcing the suitors to leave the palace. And of course she feared that antagonizing the suitors in any way would put Telemachus' life in danger.
Nevertheless, Telemachus did try to order the suitors out of the palace, in front of the Ithacan assembly:
Telemachus took this speech as of good omen and rose at once, for he was bursting with what he had to say. He stood in the middle of the assembly and the good herald Pisenor brought him his staff. Then, turning to Aegyptius, "Sir," said he, "it is I, as you will shortly learn, who have convened you, for it is I who am the most aggrieved. I have not got wind of any host approaching about which I would warn you, nor is there any matter of public moment on which I would speak. My grieveance is purely personal, and turns on two great misfortunes which have fallen upon my house. The first of these is the loss of my excellent father, who was chief among all you here present, and was like a father to every one of you; the second is much more serious, and ere long will be the utter ruin of my estate. The sons of all the chief men among you are pestering my mother to marry them against her will. They are afraid to go to her father Icarius, asking him to choose the one he likes best, and to provide marriage gifts for his daughter, but day by day they keep hanging about my father's house, sacrificing our oxen, sheep, and fat goats for their banquets, and never giving so much as a thought to the quantity of wine they drink. No estate can stand such recklessness; we have now no Ulysses to ward off harm from our doors, and I cannot hold my own against them. I shall never all my days be as good a man as he was, still I would indeed defend myself if I had power to do so, for I cannot stand such treatment any longer; my house is being disgraced and ruined. Have respect, therefore, to your own consciences and to public opinion. Fear, too, the wrath of heaven, lest the gods should be displeased and turn upon you. I pray you by Jove and Themis, who is the beginning and the end of councils, [do not] hold back, my friends, and leave me singlehanded- unless it be that my brave father Ulysses did some wrong to the Achaeans which you would now avenge on me, by aiding and abetting these suitors. Moreover, if I am to be eaten out of house and home at all, I had rather you did the eating yourselves, for I could then take action against you to some purpose, and serve you with notices from house to house till I got paid in full, whereas now I have no remedy."
With this Telemachus dashed his staff to the ground and burst into tears. Every one was very sorry for him, but they all sat still and no one ventured to make him an angry answer, save only Antinous, who spoke thus:
"Telemachus, insolent braggart that you are, how dare you try to throw the blame upon us suitors? It is your mother's fault not ours, for she is a very artful woman. This three years past, and close on four, she has been driving us out of our minds, by encouraging each one of us, and sending him messages without meaning one word of what she says. And then there was that other trick she played us. She set up a great tambour frame in her room, and began to work on an enormous piece of fine needlework. 'Sweet hearts,' said she, 'Ulysses is indeed dead, still do not press me to marry again immediately, wait- for I would not have skill in needlework perish unrecorded- till I have completed a pall for the hero Laertes, to be in readiness against the time when death shall take him. He is very rich, and the women of the place will talk if he is laid out without a pall.'
Source: The Odyssey, Book II, translated by Samuel Butler
Telemachus insolence was not only ignored, but it also instigated a plot amongst the suitors to kill him as he returned from his visits to Nestor and Menelaus.