Witness demeanor is absolutely relevant, both to evaluate the credibility of a witness and in a matter such as a child custody case, to evaluate the merits of what constitutes the "best interests of the child" which hinges, in part, on the interpersonal social skills of a parent in dealing with the parent's children.
Before I start investing time and energy into acting classes . . .
Generally speaking, I would not encourage you to take acting classes. There is nothing that you can learn in a few weeks or months that will fool a judge. Instead, your inept acting effort will only make you look dishonest because, you are being dishonest.
If you are "hyper-rational and put too much emphasis towards factual accuracy rather than feelings and emotions of another person", you are better advised to focus on how your personality can have positive aspects, rather than trying to hide who you are in reality.
For example, you can emphasize how you have a detailed child care plan, are on top of the children's medical and educational needs, are stable in your work life, and may be able to interact productively with any of the children who share your tendencies in a way that a more neurotypical person might not.
In a family law case, the marriage itself, if the couple was married, is assumed to be a lost cause. And, pretty much nobody in a family law court has exemplary interpersonal skills. If they did, their relationships wouldn't have fallen apart, or at least, they would have worked out mutually agreeable parenting arrangements without court assistance.
You are better off acknowledging that you are not perfect and showing the court that you have good coping mechanisms than to fake being the person that you believe that the court wants you to be. Someone who has personality and temperament issues who doesn't admit that those are issues is a much bigger concern than someone who has issues but is aware of those issues and demonstrates a conscious attempt to work around or cope with those issues.