[OMFG!You'reActuallySmart!]

If you're wearing a baggy shirt that "puffs" out, it may look like you have a bigger belly. Alternatively, if the shirt looks unusually large or unfitting, it may look like you're struggling to find a big enough shirt despite not being huge. In both cases, pregnancy is a possible reason that comes to mind, however, I don't go up asking women if they're pregnant anymore. I did that a few times and almost each time I was wrong so I received a glare and tried to make a brief exiting conversation.

If you wear something that is not so baggy or doesn't look like a "pregnancy shirt", people probably will lay off asking you.