You have a fiduciary duty as a director to act in the best interests of the company. Not your interests, not his interests, not the shareholders' interests (individually or as a group); the company's interests.
You must also separate the other guy's performance as an employee from his performance as a director. If he is not performing as an employee then the correct course of action is to sack him as an employee, not as a director; and vice-versa. If he turns up to the board meetings and considers the information he is given as a director then he is not "performing poorly" in that role.
However, provided you follow the law and your company's constitution I have no doubt this would be legal. You need to consult a lawyer before you do anything - it can't be the company's lawyer because he does have a conflict of interest.
Notwithstanding its legality, it's a stupid idea. It doesn't fix the fundamental problem: you are in business with someone you don't want to be in business with. You need to find a way to not be in business with him and to start that process by ticking him off is a bad move.
You need to reach agreement on how to end your relationship: just shoving him out while he is still a shareholder will have him looking for the first opportunity to sue you and the company the instant it underperforms.