Suppose, for example, that the Senate is considering the passage of a law. A Senator has decided to open a filibuster. After a half hour, the majority leader tires of hearing this Senator speak. He motions for an end to the filibuster, but is only able to get 51% of the vote. According to Senate rules, I believe, you need a 2/3 vote to end a filibuster. Over said Senator's objections, the majority leader begins the voting process and the bill gets 51% yeas.
Generally not. Federal court uses a principle known as the enrolled bill rule -- in deference to the coequal status of the three branches of government, the "enrolled bill" (the thing printed on fancy paper that actually went to the President for signature) is irrebuttable evidence that the law was properly passed. The courts cannot deal with inquiries into whether legislative process was followed; it's the legislature's job to decide what the right process is. They can't even look into whether the same text passed both houses -- as a matter of law, the enrolled bill is conclusive evidence that it did.
Senate rules are enforceable in the Senate. But the Senate is the body in charge of enforcing them, not the courts.
- The senate can change their rules at any time by a majority vote.
- 60 (3/5) votes are required to end a filibuster (with the exceptions of federal judge nominees (1/2) and rule change motions (2/3)).
So, for your hypothetical to occur, there needs to be an intermediate step where a majority agrees to change the rules to allow a simple majority to cloture the filibuster.
Generally speaking, the senate has resisted changing the 60-vote filibuster rule (with the above mentioned exceptions) because the supermajority requirement protects the interests of the minority party.
Every senator understands they can be in the minority depending upon the outcome of the next election every two years. (Even though each senator is elected every six years, elections are held for 1/3 of the senate every two years.)
Therefore, the filibuster rule protects the interests of all the senators. Which is why they haven't eliminated it and are unlikely to change it for the passage of any particular bill. Although, theoretically, it could happen.