You can definitely ask your doctor about this. That'd be the best thing to do because s/he could have a much better idea of what's going on than we could.
You may try using non-latex polyurethane condoms to see if that helps though. They are specifically made for people allergic to latex. See if those feel better and if they do, maybe all if was is that you are allergic to latex.
Good luck figuring it out.
PM me anytime.