Let me anywhere near a can of sweetened condensed milk and I'll eat it by the spoonful! What you described sounds exactly right. Maybe you were remembering evaporated milk, which is white and thinner.
"Breathe in, breathe out, move on." Jimmy Buffett