The agreement doesn't have to be notarized under either the law of India or the law of Japan (unless there are some highly unusual facts in this case not mentioned in the question). In this situation, the notarization merely provides one additional means to prove that the agreement was actually signed by the parties whose signatures appear on the agreement, rather than being forged.
It is not possible to determine, without knowing more about the agreement, whether the arbitration agreement would be valid. The validity of the arbitration agreement would probably be determined under the law of Japan since that is the specified venue, unless the agreement provided otherwise.
Japan allows disputes over some agreements to be resolved by arbitration, but not every possible kind of dispute. For example, Japan prohibits arbitration in domestic relations and adoption cases. India likewise has certain kinds of disputes for which arbitration is prohibited (for example, rent control). A subject matter that cannot legally be arbitrated in India might also render the arbitration agreement void despite a Japanese forum.
Both India and Japan do, however, permit arbitration of international "commercial" disputes, which normally would involve business to business contracts.