Gorilla Glue foams and expands as it dries, so it's not suitable for this application. You can use double-stick tape.
I have applied a very thin layer of Weldbond Contact Cement to both surfaces, let dried for a couple of minutes and then pressed together.
You need to have uniform thickness so that the mirror sits parallel to the surface of the mirror holder.
For TLRs, I've used double-stick tape.
For an SLR mirror that moves, you'll need something a bit stronger, I think. However, you might want to try double-stick tape first and see if it does the trick.
One product that I've used is the tape that is used to apply photos to photo albums. It's inexpensive and very easy to use.