A real love for others will chase those worries away. The thought of being punished is what makes us afraid. It shows we have not really learned to love.
We should be grateful we were given a kingdom that cannot be shaken. And in this kingdom we please God by worshiping him and by showing him great honor and respect.