Say some neighborhood kid was playing baseball and accidentally hit the ball into my yard. Theoretically, if I own the house or my lease permits, I have the right to control access to my yard and, therefore, access to the ball, even if I don't own the ball. If someone retrieves the ball without my permission, they are trespassing.
This is a general common law answer; specifics in California may differ.
Yes, they are committing the tort of trespass and you could sue them for whatever damage they did to your property; probably nothing.
On the other hand, if you keep the ball, you are committing the tort of detinue (wrongful detention of another person's goods when the owner has requested their return) and they can sue you for whatever damage you have done to them; the cost of a new ball probably.
No, it isn't; give it back.