A state generally cannot be tried in court because of sovereign immunity and state immunity. The first prevents citizens from suing their own states (or state's agents) over state actions within the scope of the state's functions. The second prevents states (and their agents) from being sued in foreign courts for the reason that it would infringe on the sovereignty of the state being sued. There are courts which have international jurisdiction, but I'm not aware of any that would be able to claim jurisdiction over the Khashoggi murder. Additionally, to the best of my knowledge you can't actually prosecute a state as a whole for crimes; you have to prosecute individuals, though I can't tell you why that is, specifically. However, the United States has previously indicted agents of foreign governments who acted in a way which was harmful to the united states, as well as indicting people for crimes which had no direct relation to government interests. By extension, the US could, hypothetically, indict anyone who they believed was responsible for the murder of Khashoggi because he was a US resident and has american children. However, it may be impossible for them to indict crown prince Mohammed bin Salman due to his specific position. There is also the problem of putting the accused on trial. The agents who were involved in the murder of Khashoggi, to my knowledge were all Saudis, and Saudi Arabia does not have any extradition treaties, so it would be difficult to bring them to the US to be prosecuted, though in theory it's possible, for example that they could be arrested if they travelled to a country with which the US has an extradition treaty and the US requested their arrest by that country.
In terms of court action by Khasgoggi's wife, there isn't all that much that she could do. She wouldn't be able to sue the US for not acting because one generally cannot compel the government to do anything that it is not required to do by law, and chasing after the foreign murderers of peoples family members is not one of those things. She could try to sue the perpetrators in civil court, but that would require her to find a court that could claim jurisdiction over the matter, which would likely be difficult if not impossible, and even if she succeeded in that, and won her case, she would likely be unable to enforce any judgement that was handed down. Alternatively, she could perhaps try to sue from within Saudi Arabia, perhaps as a wrongful death lawsuit, but I am not familliar with Saudi law so it's possible she wouldn't even have the ability to bring a suit there. Outside of the legal system, though, if she was able to gather enough public support she could probably pressure politicians to take some kind of action under threat of not being reelected, or she could go to the UN and ask for support of somekind, though that would not be likely to get her very far. She would not be able to sue in the International Court of Justice because she is an individual and not a state.
There is one viable option that the US government could take, and that would be to issue sanctions against Saudi Arabia which would be lifted when a certain condition was met; for example, if certain people involved in the plot were convicted and punished within Saudi Arabia, or extradited to the US to be tried. This would put economic pressure on the Saudi Government to take action on the issue, though it relies on the US being able to do sufficient economic damage through the sanctions.
As for your points about spies and terrorists: it depends. In wartime it is generally permitted to execute enemy spies, but in peacetime it would vary by jurisdiction. For terrorists in foreign countries that are not warzones, the government of the country that authorized the assassination would shield their operatives from liability within the country, but the country where the assassination took place would likely attempt to prosecute the assassins, assuming they were caught and barring some kind of agreement to the contrary.
