The person who swears by the altar is really using the altar and also everything on the altar.
You are blind! Which is greater: the gift or the altar that makes the gift holy?
And the person who swears by the Temple is really using the Temple and also everything in the Temple.
People always use the name of someone greater than themselves when they swear. The oath proves that what they say is true, and this ends all arguing.