If you want to stick with an SLR but more compact I like (and own) the Pentax ME Super. With the aforementioned "pancake 40" 40mm lens it makes a pretty small package, but unfortunately like a lot of Pentax lenses (due to backwards-compatibility) the Pentax-M pancake 40's price keeps going up, so I don't currently own the lens personally. I find the Pentax 28/2.8 and the 50/1.7 are compact enough for me for walking around, although it doesn't fit in a pocket this way. You can use the camera manually or in aperture-priority mode.
The only issues with the ME Super that I see are:
- it needs batteries to operate, even if you want to manually select the shutter speed (without batteries it always fires at 1/125), and it uses a couple of buttons on the top deck to select the shutter speed, which can be a bit fiddly if you're trying to do it quickly, have fat fingers, or if you're shooting outside in the cold
- because of its small size, if you want to use a longer, heavier lens you may run into balance issues
If you want to move to a rangefinder, the Olympus 35 series is likely the most compact fixed-lens variety, which can be used manually or automatically. If you want to get into removable-lens Leica M or compatible lenses, the CL/CLe I believe are the most compact cameras. Similarly in screw-mount, a FED-2 or screw-mount Leica with collapsible 50mm lens are also fairly compact, and are entirely manual.