The yeast of the Pharisees was their over caution, to the neglect of the heart of God, in the Law. Following the Babylonian Exile, when Israel reflected on the reason for their exile, they realised that they had failed in keeping the Law of Moses and therefore had suffered the exile. So in order to not suffer another such process they went overboard in their corrective actions. They concentrated on actions and not the heart of the act on worship. Their main aim being no repetitions of exiles.
Therefore, in todays exile the Pharisees are torn between accepting Jesus as Christ and their fear of exile.
James is stressing on faith as replacement for signs. Faith is a product of relationship, and where there is a relationship with God, our hearts are balanced and we know what is right and wrong. There is no pride in wealth and there is no shame in poverty, because the only wealth is Jesus, and His continued presence in our life.
When God is first in our lives, everything falls in order, and there is no fear of exile or punishment. There is only assurance.