It's against the Minor Offense Act to carry pepper spray if you don't have a good reason.
A 2009 issue of Tokyoweekender offers this column:
Are there any substances that are legal to carry in the West (e.g. Mace or pepper spray), but are illegal to carry in Japan, even if you can buy them here?
And the answer is...
Article 1-2 of the Minor Offense Act states that “a person who secretly carries around, without justifiable reason, knives, iron rods, or items that are used to harm another person’s life or body” shall be fined or punished by temporary imprisonment. If a woman carries pepper spray with a rational purpose such as to defend herself, such act shall be justified. However, it still may be punishable under the Minor Offense Act if she carries it without any necessity.
So, what is this Minor Offense Act? Luckily there is an English tranlation. So Article 1, paragraph 2...
軽犯罪法1条柱書
「左の各号の一に該当する者は、これを拘留又は科料に処する。」
2号「正当な理由がなくて刃物、鉄棒その他人の生命を害し、又は人の身体に重大な害を加えるのに使用されるような器具を隠して携帯していた者」
Article 1.
Any person who falls under any of the following items shall be punished by detention or fine.
(2) A person who carries a knife, iron bar, or any other instrument that can be used to injure another person’s life or cause serious bodily harm without a justifiable reason
Pepper spray can cause temporary to permanent blindness, cause pulmonary problems, and in general, is an awful substance to even inhale if not getting it into the face directly. So... The default is, that carrying pepper spray is illegal if your carrying doesn't qualify for a justifiable reason. It is up to you to prove that your reason to carry it is justified. However, what is a justifiable reason?
Well, as Tokyoweekender said: if you are not able to defend yourself and you are in a bad area, or if you are stalked, or if you go into the mountains where there truly are wild animals, you might be understood justified. However, you gamble there: you'd need to convince the police that you have justification, and the Japanese police does not like people carrying any such items at all. It's not upon the Japanese police to prove you didn't have a reason, it's on you to prove you have justification for the exception!
In the case that was discussed in OP's link, someone that was regularly transporting large amounts of money was under suit for carrying the pepper spray hidden on them. One of the main factors in the assessment was the fact that the accountant was transporting large amounts of cash money, which made it socially acceptable that he might need to protect himself (and the money) with the spray. The accountant then accidentally put the spray in his pocket when cycling and was met by police, resulting in the suit. The judge ruled, as far as I understand it, that on the job with money the spray was ok. However, the judge in the very case also noted that self-defense isn't a justification in itself, and that in large crowds there couldn't be a justification.
およそ護身用であれば、催涙スプレーの携帯が一般的に「正当な理由」を充たすと判断するものではない
If it is for self-defense, we [the court] do not judge that carrying pepper spray generally satisfies "just cause".
また、これといった必要性もないのに、人の多数集まる場所などで催涙スプレーを隠匿携帯する行為は、一般的には「正当な理由」がないと判断されることが多いと考える
In addition, I [the judge] think that it is often judged that there is no "justifiable reason" for the act of concealing and carrying pepper spray in a place where many people gather, even though there is no particular need to do so.
Akin to the laws regulating swords - where you need to prove you want to collect swords before buying them - you'd best inquire with the police before and ask for a certification that you require it for a specific reason - such as to defend against bears on a hiking trail or to protect the valuable goods you transport for work.