I've never had an issue removing my physcial sunscreen regardless of what facial cleanser I've used in the past- cream, oil, etc. I was wondering how you know it isn't being removed completely?
Regarding oil cleansers and drying your face out- I found this to be true with many because often they contain alcohol (the drying kind, like denatured alcohol or sometimes listed as simply "alcohol"- Tactcha's cleansing oil is an example), citrus oils (tangerine, orange, lemon, etc- Josie maran and many other brands contain these) which can be drying and irritating to some, and/or mint type extracts (peppermint, menthol, etc). It is possible the ones you try contain one or more of these types of ingredients which can be drying and completely counter-intuitive to what you'd expect from an "oil" facial cleanser.