I should note that court rulings are almost necessarily retrospective, and with respect to their rulings on law, retroactive - they always judge what the law was at the time the act(s) occurred, generally disregarding what the law says now if it differs.
If someone did an act which was not generally considered to be illegal at the time, but which the court later ruled to be illegal, would he be liable or go to jail?
If there were no laws against it at the time, courts generally cannot make a finding of guilt against a defendant. An exception apparently applies to war crimes and crimes against humanity - see the Nuremburg trials for more.
For example, if Oracle wins the Oracle-Google copyright lawsuit and Google would have to pay Oracle for damages, would Google have to pay the same damages as if the Supreme Court ruled that API was copyrightable before Google's copyright violation?
This is... different. This is the court deciding whether the parties are guilty based on the law at the time that the acts occurred. It's a fine and somewhat arguable distinction, but the difference here is that rather than a court making new law, it is interpreting the laws enacted by the legislature. Of course, in practice, there mightn't be a difference.
In answer to your question, yes - damages are likely to be statutory as well as based on actual loss as calculated from the time of the act.