The only time I have had problems with misfires in rifle, I was using the Lapua small rifle primer case (.308), heavy powder charge, and heavy bullet (185grn BTLR Berger). After I pulled the bullet I discovered the primer had fired, however, the powder did not fully ignite. I figured the small primer did not have enough 'flash' to bust through the nearly packed powder to reliably ignite the powder charge.
Also, the Lapua smalll rifle .308 brass has a reduced size flash hole, which I believed to exacerbate the problem.
My solution was to ream the flash hole a tad larger. Seems to have worked since, and as a bonus, it seems to have reduced my ES/SD of that particular load a bit.
With your large primer brass, perhaps the federal primers, known to be a particularly mild flash primer, may be having a similar issue. Maybe a clean up of flash hole, since the hole is already large, or a change in primer brand might resolve the issue. Also, check for cleaning media in there before you load. It likes to block that tiny puka, it seems.

I have had great results with wolf or pmc LR non magnum primers for precision work. And they are cheap to boot.