Is it illegal to copy a picture of someone else, found on a public site, and use it on your own site?
Broadly yes, this is copyright infringement, even if you can see the picture on a "publicly accessible" website, that doesn't give you the right to use it. You need permission from the copyright holder (which may not be the same as the website owner).
If by "public site" you mean a site that shares stock pictures for free (with copyright release), then it would be legal, but I'm guessing you mean any picture from a site you can access. This would not be a legal use.