I would get the PharmD letter if you think it will be a good letter. As long as they can speak to your desire to do ID fellowship and they have a terminal degree, I think anyone will do. You do need others, obviously, like your PD and maybe a general internist, public health person, critical care or other specialty that can speak to your plans and potential.
For reference, I had 4 letters. 1 from my PD, 1 from my clinic preceptor (general medicine), 1 from a PhD-only, 1 from an ID physician. I interviewed at many of the top programs (Partners, Emory, UWashington, Hopkins, Penn, BIDMC, Vanderbilt, etc.). Having a lot of ID letters is not critical.