What 2F has said.
The 320 has a depressed midtones curve, which gives snappy highlights and darker shadow, to the expense of midtones. In contrasty light (e.g. a winter scene) this has the advantage of preserving the shining aspect of whites.
The 400 has a more straight curve, which makes it useful as an all-purpose film, including pushing. It will handle mistakes more gracefully (underexposure, overdevelopment, etc).
If you want to see a quick demonstration, open up any image editor that uses curves, and compare the straight curve with one in which midtones are a little depressed (don't touch the endpoints). You will see your highlights glow nicely, but other tones will get darker.
If you were to use it for studio portrait, you would probably put skin tones on Zone VII (i.e. spot meter reading + 2 stops) instead of the usual Zone VI (reading + 1 stop), to be sure they don't look too dark.
I did the side-by-side comparison test once: I took pictures of snowbanks in which there were various degrees of dirt: clean white, greyish, brown, pitch sooty, etc. On the contact prints, I could notice the difference, but it is in the order of a 10-20% difference. A normal person would have found either print normal.
So the main reason to use 320 is when you know that extra edge will be useful. On the other hand, you won't ruin a photo taken in the so-called "normal" situation by using 320 instead of 400 if you expose and process correctly.