The E is generally the best choice for Midtown, (unless you want to pay for LIRR.) but the A really isn't slow. The problem is the frequency- you likely will have a long wait for the train.
I take redeye flights into JFK about every other month, and head straight into work. Unless I plan on taking LIRR, I find that taking whichever Airtrain comes first ends up working pretty well for me. I'm a little south of midtown, but end up getting in at about the same time whichever train I take.