I suspect it has its beginning in something to eat when poor people had little but some hog fat and chickens. That's what it sounds like to me.
Often when you had nothing else you had a speck of flour, something from a hog or chicken to make gravy and some eggs. Pair that with a biscuit, rice or bread of some kind, dumplings or noodles and you have a meal.
I once heard an old old man in New Orleans say it was a great place to live because something free was always going on, and you could live until the end of the month if you had a little rice and gravy.
That will always stick with me. There's not having what you like to eat, and not having anything to eat.