House Speaker Kevin McCarthy (R-Calif.) on Tuesday said he will block Reps. Adam Schiff (D-Calif.) and Eric Swalwell (D-Calif.) from taking seats on the House Intelligence Committee, in a move that the speaker insists is not political payback for the Democrats’ decision to strip hard-right lawmakers from their committee assignments during the last term.

In a letter addressing Rep. Hakeem Jeffries (D-N.Y.), McCarthy said he appreciated the House Minority Leader’s “loyalty” to his colleagues in trying to reinstate them into the committee but said he “cannot put partisan loyalty ahead of national security.”

Taking a shot at Schiff and Swalwell, McCarthy added he does not “simply recognize” their prior service on the committee as a qualifier for continued membership, adding “integrity matters more.”

McCarthy insisted Tuesday that the decision was “not anything political” and “not similar to what the Democrats did,” referring to the removal of Rep. Marjorie Taylor Greene (R-Ga.) and Rep. Paul Gosar (R-Ariz.) from their committee assignments.