I was filming at a County Social Service office, they clearly pointed out there dissatisfaction of the camera and called the Sheriffs, the Social Service Administrator told me to leave because I was filming and had me arrested for trespassing.
Generally, if someone asks you to leave their property you have to leave*. Just because a place is owned by the public, doesn't mean anyone can go there any time they wish. Military bases, firehouses, and jails are owned by the public, but many of these have limited access to the public.
It may be open to the general public, but that does not mean restrictions cannot be put into place, either on times, or activities, or individuals. For example, public parks often have time and activity restrictions; schools have the power to restrict individuals from their premises, either specifically or by general category.
As a general point of law, the owner of any property, or their agent, can order anyone without the right to stay (e.g. not a co-owner or tenant), and that person must depart, otherwise that person is tresspassing.
The Social Service Administrator is almost certainly an agent of the controlling entity that owns the property. Thus their demand that you leave the premises is enforceable, unless you have a non-revokable right to be in that space.
*As user Justaguy points out there are some exceptions. Most notably, police can some times enter a property uninvited or against the owner's wishes (such as under emergency circumstances or with a warrant).
As discussed, just because the government owns a building, it does not make it public for the purposes of First Amendment rules automatically. Generally, the Government, when acting as an employer and a place of economic business, has the same rights as those of private employers and places of business, including the rules of conduct for customers while on property as well as employees (if you get a check from Uncle Sam, there are things you cannot say while doing that work... and those rules are different between agencies of the government (typically political appointees are allowed to express more political speech than your career civil servant, but even then, it's not as a non-government citizen... and the appointee is much easier to fire at will than the civil servant (almost always for cause even in states with at will employment laws. It's also compliant with for cause state laws for a specific state... though Feds tend to be more strict on what is a fire able offense than the state laws.).
Because Social Security is acting as a buisness, you do not have a right to film them in their own facitlity like you would when recording a cop dealing with the public on the street. This means that two party consent is in play (I believe California is a Two Party Consent state) and the office worker explicitly told you they didn't consent.
And while it's not a concern to Social Security's services, there's another reason most federal places of business don't allow recording on the premises. Aside from D.C., often times Federal office buildings house regional branches of multiple federal offices in the same building. It's not uncommon to find the local FBI building sharing a few floors with the DEA or the IRS having a Secret Service annex. I've actually seen a EPA field office on an Army base because that's where they could put it.
The Alfred P. Murrah Building (aka the one that was targeted by Timothy McVeigh in the 1995 Oklahoma City Bombing) contained offices for Social Security, U.S. Department of Housing and Urban Development, Secret Service, Department of Veterans Affairs vocational rehabilitation counseling center, the Drug Enforcement Administration (D.E.A.), and the Bureau of Alcohol, Tobacco, and Firearms (A.T.F.), as well as military recruitment offices (I know there was a Marine recruitment office, but I can't tell if other branches had offices there) and a daycare center for children of employees of these agencies.
There's some good reasons why the Government does not allow cameras in buildings like this. Of the 6 federal agencies in the building, three have employees who might be chekcing in from undercover investigations at any given moment, another is recruiting for the military, and may have people coming in who will be given access to covert operations, and a third is dealing with people who have come back from war with some serious problems both physical and mental. All of them have valid reasons to not want some person coming off the streets And many of those concerns are very very valid concerns about their safety and the safety of people important to them (from their families, co-works, up to the President of the United States (Secret Service doesn't have the luxury of choosing to like the guy or not) if they are specifically identified in a federal law enforcement office.
The VA, Housing, and Social Security are all dealing with people who really would prefer not to be on some random person's film out of dignity. The people in the VA are in pain and maybe don't want people gawking at them, and the other two agency frequently deal with people who may not want the social stigma that can be associated with needing those benefits.
And then we get to the daycare, which I shouldn't have to tell you, would have problems with someone with a camera wandering around enough with out realizing that the kids playing there are all young children of the employees with the above concerns. And less you think I'm trying to scare you by threatening the children, I again remind you, I'm discussing a building that was blown up by a man with a vendetta against the federal government driven by a botched law enforcement raid that ended up killing American Children and specifically targeted the building because ATF and DEA, though killing people who protect the president and recruits and injured veterans of the military were added bonuses... and a parking spot right by the day care center was icing on the cake (his words, not mine. He viewed the daycare as a valid target specifically because of the death of the children during the Waco Siege). And he knew this because he could enter and exit the building freely and did some scouting. He didn't bomb it just because he had a problem with the government... he had specific problems with the government and wanted to send a message... and they certainly received it.
There's a very very valid reason why they don't want you coming in with a camera. Whatever you think you were doing with it there, and whatever point you think you wanted to prove... I can assure you, it's not worth the risk you brought into the lives of people you who do good work, you may even say are heroes, who you would never intend to put in harms way.