A real love for others will chase those worries away. The thought of being punished is what makes us afraid. It shows that we have not really learned to love.
We should be grateful that we were given a kingdom that cannot be shaken. And in this kingdom we please God by worshiping him and by showing him great honor and respect.