If a trespasser openly and notoriously, exclusively and continuously possesses your property by building a fence on your land for the right time period, they automatically own the land. It still takes a court proceeding to record the passing of title (the trespasser has to prove in court that it is legally theirs). The trespasser would also have to establish that the recent survey was correct (survey errors do exist): was there an earlier survey in connection with the fence that established different boundaries?
If (as it turns out) this has become his property, he abstractly has title to it, but only you and he know about it. The trespasser may have an interest in officially changing the property description, because it will officially increase the size of his lot and thus the value of the house+land. This also will increase their tax burden (while decreasing yours). The county has no knowledge of the fence: they go off of the official record, which says that you own that wedge. You also may have an interest in changing the property description, primarily to reduce your tax bite. There could also be issues with your resale of the property, since a mortgage company may require a survey of the property. Whether or not that is bad is hard to say: the consequence could be that the buyer is alerted to the fact that the lot is smaller than advertised and so on; in the current market I doubt anyone would care. If the fence goes away and you start using the land, then it will officially revert to you after a while.
If you catch the party and complain within 10 years, you may recover the property (RCW 7.28.010). The limitations statute says that
The period prescribed for the commencement of actions shall be as
follows: Within ten years: (1) For actions for the recovery of real
property, or for the recovery of the possession thereof; and no action
shall be maintained for such recovery unless it appears that the
plaintiff, his or her ancestor, predecessor or grantor was seized or
possessed of the premises in question within ten years before the
commencement of the action.
That ship has (apparently) long since sailed. There is a different law pertaining to "Adverse possession under title deducible of record" which shortens the limit to 7 years, which is even less useful to the original owner. RCW 7.28.070 also shortens the time limit for an adverse possession case, to 7 years:
Every person in actual, open and notorious possession of lands..who
shall for seven successive years continue in possession, and shall
also during said time pay all taxes legally assessed on such lands or
tenements, shall be ... the legal owner of said lands
There is another (more recent) tax-related provision, RCW 7.28.083.
(1) A party who prevails against the holder of record title at the
time an action asserting title to real property by adverse possession
was filed, or against a subsequent purchaser from such holder, may be
required to:
(a) Reimburse such holder or purchaser for part or all of any taxes or
assessments levied on the real property during the period the
prevailing party was in possession of the real property in question
and which are proven by competent evidence to have been paid by such
holder or purchaser;
This does not require them to have paid taxes, it say that the victor in the dispute may nevertheless be ordered to reimburse taxes paid by the other party (assuming the other party has paid the tax). So there is some chance of getting the taxes back. The reimbursement is at the court's discretion (continuing that section):
(2) If the court orders reimbursement for taxes or assessments paid or
payment of taxes or assessments due under subsection (1) of this
section, the court shall determine how to allocate taxes or
assessments between the property acquired by adverse possession and
the property retained by the title holder. In making its
determination, the court shall consider all the facts and shall order
such reimbursement or payment as appears equitable and just.
One should also pay attention to the last provision in that statute:
(3) The prevailing party in an action asserting title to real property
by adverse possession may request the court to award costs and
reasonable attorneys' fees. The court may award all or a portion of
costs and reasonable attorneys' fees to the prevailing party if, after
considering all the facts, the court determines such an award is
equitable and just.
That means that the victor can request the loser to pay his attorney's fees. This is delicate math, balancing the chance of recovering some paid taxes vs. paying the other guy's costs. You could try calling the assessor to find out how much the decrease in lot size might net you (the land vs. improvement proportion of taxes is all over the map in KC, easily ranging from 60% to 250% depending on year).
One additional feature of adverse possession is that it must be "hostile", i.e. without permission. If a neighbor builds on your land, you can explicitly give them revocable permission (to avoid "no you didn't" arguments, explicit and revocable written permission, signed by the neighbor, would bar an adverse possession claim).
This raises an interesting question, to which I don't know the answer. Suppose the prior owner gave permission to the fence builder, and did not demand the removal of the fence when he sold the property or right after the neighbor sold his property (there was only on act of granting permission). Does the clock start from your acquisition of the property (whereupon the element of hostility is satisfied)? Or does it start from the point where they acquired the property and were in hostile possession of the land (I would bet a quarter that that's the answer).
If (or, given that) the fence was moved further onto your property more recently, there is a chance to recover the newly-taken piece of land. If you grant them revocable permission to build a fence on your property, you would not be subject to an adverse possession taking for the newly-taken land. If at some point you tell them to tear down the fence and they refuse, you can sue them and the court will (almost certainly) order the removal of the fence. The neighbor might then initiate an action to quiet title on the originally-taken piece of land, so you'd be back to where you were 4 years ago.
From a practical perspective, this is well-worth the small amount of money involved to consult with an attorney to get legal advice. The legal matter probably will not go away quickly, and they may be presently inclined to settle in a manner more in your favor.