Anyone can be arrested, as long as the police (and in some cases, anyone) have probable cause to do so. This generally includes two points:
- You are in the process of committing a criminal act
- The police have probable cause, generally through evidence, that you are in the process of, about to, or have committed a criminal act. This usually requires obtaining a warrant for the arrest.
However, it's important not to conflate searches with arrest. Arrests are when you are being taken into custody for the reasons mentioned above. Searches are when your personal belongings (i.e. property), are searched to collect evidence.
It is not up to police to convict someone. This is the job of the judicial system. There's a bit of terminology here: convictions basically mean being found guilty of an offence. Police don't have the ability to convict someone of an offence. They do have the ability to arrest, and charge with an offence. To charge someone with a crime basically means to accuse them of having done something.
Now let's get to the actual crime. Since you haven't mentioned a specific jurisdiction, I'm going to use Canadian cases and law, but in general, it should apply worldwide.
If the police have reason to believe that you are engaging in unlawful hacking-related behaviour, then they have the authority to arrest you. They simply need to have a reasonable belief that you are engaged in the crime. Something such as connecting you with an IP address and connecting that you were online at the time is enough.
If the police need to find out more information (which they generally do, to investigate further into the matter) - they can obtain a search warrant afterwards. With this, they can legally search into your computer, and investigate. Any evidence that they collect can be used in court.
I've recently done some research into cyber crimes. One rather infamous case involved the attack on multiple large scale websites, including Dell, Yahoo, and Amazon, in 2000. This sparked a large scale investigations, between the RCMP (Royal Canadian Mounted Police) and the FBI. The attacks were claimed by some to cause nearly 1.7 billion dollars of damage. The hacker was charged with 58 different offences. You can imply from the linked article that he was arrested before his computer was searched.