James Buchan said he wanted to be able to "look my family in the eye and say, 'that's not who I am'" after his former party announced plans to scrap indefinite leave to remain.