There are a lot of things you can do. One would be use double face tape.
Another thing (you could use a very small piece of that hook and loop stuff that holds things together. It comes in various strengths (as far as what it will hold). One strength is very light. I've found them to just peel off when not used.

You put one on each side (they are sticky) and then just press together. They are made to be used with material and will not hurt leather. Just peel them off when unwanted.