I was on the residency selection committee as chief resident, but I never served as PD or APD. However, I did review applications for candidates to select some for interviews.
I would say you should definitely try to get someone you worked with. If the IM letter writer is ghosting you, try and find a different specialty. The "summary letter" that you're IM chief is going to submit is going to be a completely impersonal, super formatted excel file more or less that will not look good. They don't know you and they're not claiming to, it would be like submitting a Dean's letter as an LOR. It's assumed the LOR is written personally for you, by an attending/mentor that has worked with you and effectively is attesting that you are not insane/incompetent. If someone submitted a formatted "summary letter" I would assume they could not find a 3rd letter writer which is suspect. May still get an invite but would definitely hurt their application.
The other people responding may not fully understand you're question, seeing as you had to clarify, but I think I understand the issue. In all fairness it has nothing to do with "Radiology" letters or "IM" letter writers. A "summary letter" written by the Rads PD (if that was an expectation that all med students applying to rads needed a summary letter from the rads PD) would have the same issue.
Sorry to freak you out if you just wanted the reassurance, but id say better to find an actual LOR.