Any religion can tell you that the righteous certainly have meaning in God's eyes, wherever they may sojourn. The example of Joseph in Egypt comes to mind. An appropriate Talmudic quote on this is, "Even for the sake of one righteous person, the world can continue to exist, as it says "A righteous man is the foundation of the world (Proverbs 10.25)" (Yoma 38b)
But in Judaism, even the non-righteous can gain merit from the performance of a mitzvah, and this merit is potent enough to help others throughout the world as well. "If this person does one mitzvah, he is praiseworthy, for he has caused himself and the entire world to be on the balance of merit." (Kiddushin 40b)
How can Olmert attack such a basic concept of the Jewish faith and get away with it?! How can he have any significant following in this?! We are the people who believe, "All Jews are responsible, one for another." (Shevuos 39a)
But even for those Olmert followers who reject Judaism itself, and only believe that the righteous and otherwise great people deserve attention, even such people as these, according to this twisted belief, deserve zero acknowledgment.
We are not discussing allocating a preserved historic site for destruction, for the sake of progress. Matters of ecological preservation alone properly belong to the people living in whichever particular country that such a issue arises in. An exception to the rule is if there would be a risk of some sort to other nations from the loss of the ecological site, such as the Rain Forests of the Amazon.
In this case we are discussing uprooting free access, that is, access that is free of bodily danger, of those to whom it belongs, specifically, those Jewish people who are religious or care about religion. This situation is comparable to a person without an artistic bone in his body trying to destroy every piece of art that he finds because in his view all that art ever does is wastes people's time. A modicum of humility, wisdom and civility would call for allowing the art to flourish in the hands of artists and those who appreciate art, and let the anti-artistic people attend to matters they prefer and understand. Therefore this matter represents a breakdown in humility, wisdom and civility by those who would deny the Jews of the Diaspora from having a say in the matter.
But one thing is clear. Even secular people should value Human Rights, and the physical city of Jerusalem as explained before is intrinsic to the spirituality upon which the Jewish faith rests. So if you do not believe in Judaism, then don't you dare try to alter the practice of its adherents by changing the status of the seat of their religion without their consent!