In Vaden v. Discover Bank, 556 U.S. 49 (2009), the Supreme Court held:
A federal court may “look through” a §4 petition to determine whether it is predicated on a controversy that “arises under” federal law; in keeping with the well-pleaded complaint rule as amplified in Holmes Group [535 U.S. 826], however, a federal court may not entertain a §4 petition based on the contents of a counterclaim when the whole controversy between the parties does not qualify for federal-court adjudication.
Discover sued Vaden to recover a credit card debt in state court. Vaden filed a counter-claim challenging the charges under state usury laws. Discover filed a claim in federal court seeking to compel arbitration under 9 U.S.C. §4, which provides:
A party aggrieved by the alleged failure … of another to arbitrate under a written agreement for arbitration may petition any United States district court which, save for such agreement, would have jurisdiction … of the subject matter of a suit arising out of the controversy between the parties, for an order directing that such arbitration proceed …
"Look through" analysis gives effect to the bolded words. A dispute about arbitrability itself is not sufficient to attract federal jurisdiction. But the petition does not need to be based on an existing claim in federal court, either – the court can "look through" the petition for a claim that would attract federal jurisdiction.
Although Discover's debt recovery claim was based solely on state law, Discover asserted that the court would have had federal question jurisdiction because §27(a) of the Federal Deposit Insurance Act preempted the state laws supporting Vaden's counterclaim.
The Supreme Court agreed that the district court should "look through" the §4 petition and consider the possible existence of federal jurisdiction, but reversed the court of appeals' finding that Discover's claim "arose under" federal law for the purposes of 28 U.S.C. §1331. The Supreme Court applied the "well-pleaded complaint rule" to Discover's §1331 claim, which required the court to focus on Discover's original (state law) claim, not its response to Vaden's counterclaim.
To answer your question, "look through" analysis does not require the court to consider the "merits" of a federal claim before compelling arbitration. However, it is not sufficient that the dispute "seems to be of federal nature"; the court must conclude that (but for the arbitration agreement) it would have jurisdiction. The disagreement between the Supreme Court and the court of appeals in Vaden, which related to the scope of federal jurisdiction rather than the application of "look through" analysis, demonstrates that disputes about whether jurisdiction would have existed can become very complex.
Thus, in general, a federal court should not decline to compel arbitration of a federal claim because "the case would have been dismissed summarily had the court been asked to rule on it." However, the court must dismiss the petition if the federal claim is so insubstantial that federal jurisdiction is not attracted at all. As the Supreme Court held in Hagans v. Lavine, 415 U.S. 528 (1974):
Over the years, this Court has repeatedly held that the federal courts are without power to entertain claims otherwise within their jurisdiction if they are "so attenuated and unsubstantial as to be absolutely devoid of merit," "wholly insubstantial," "obviously frivolous," "plainly unsubstantial," or "no longer open to discussion."