Remove button does not work if validation fails first.

I use field collection on a specific content type.
And on hook_node_validate function, I check whether input values are correct or not.
The problem goes like this.

1. If I click remove button without saving node form, It works perfectly even though input values are wrong.
2. But If I type some wrong values on field collection field, and click save button on node form, hook_node_validate sets errors on the form as expected.
3. After that If I click remove button, then the item is not removed and It shows earlier form error that says you have wrong values. I think despite the fact that the values are wrong, remove button should work properly because validation is not required for remove.

Is there anyone facing same problem or solved the issue?