This idiom or expression has to do with:

Being tricked.

When someone says “I was left holding the bag,” they mean something like “They tricked me and left me to take the blame for something I didn’t do.”

Compare to:
Taking the blame.