If your fellow-believer is hurt by you over matters of food, then you're no longer behaving in a loving way. Don't destroy someone for whom Christ died by the food you choose to eat.
But you're not following your sinful nature, but the Spirit—if it's true that the Spirit of God is living in you. For those that don't have the Spirit of Christ in them don't belong to him.