A real love for others will chase those worries away. The thought of being punished is what makes us afraid. It shows that we have not really learnt to love.
We should be grateful that we were given a kingdom that cannot be shaken. And in this kingdom we please God by worshipping him and by showing him great honour and respect.