are you routinely being self-deprecating about your looks, hair, body? i think a lot of men feel they can't win if they GIVE a compliment and get, the "oh do i look fat in these jeans?" response.
and no i don't think "setting him straight" helps at all!