Here are some simple principals for 35mm B&W that I found helpful while learning (some of this has been mentioned already):
great pictures can be made using the box speed and normal development
when the subject is very contrasty (bright light with deep shadows) and the film is exposed at box speed and given normal development negatives may be tricky to print, either the shadows turn solid black or the highlights wash-out all white, reduced development can be used to reduce film contrast (for the whole roll unfortunately) and retain in the print either more of the shadows or more of the highlights using normal grade paper.
Some people find normal development gives negatives that are too contrasty even with normal subject contrast, the printed look is too "hard". In this situation the development can be reduced to lower the contrast and give "softer" prints. Underdevelopment causes some loss of shadow detail, this can be compensated for by increasing exposure, which is done by using a slower film speed. If it is desired to reduce the development by one stop to lower contrast then the film should be over-exposed one stop to retain shadows, this may be thought of as a change of film speed by one stop down.
Some history. Originally film was exposed one sheet (originally a glass plate) at a time and developed by inspection under red light (early film was blue sensitive only), the development was stopped when the highlight density reached the desired quantity judged by eye. The concept of film "speed" was primitive. Development by inspection became impossible with panchromatic film (all B&W film is now panchromatic and has to be developed in darkness). Things became even more complicated when roll films were introduced it became impossible to develop frames individually.
There is a motto for exposure and development that comes from the very early days of photography and is still relevant today "expose for the shadows, develop for the highlights". Highlights create a lot of silver density in the negative, sometimes too much. Shadows create very little silver density, sometimes not enough. Highlights are strongly affected by development duration, shadows less so. The exposure should be enough to capture the shadows. Longer development will emphasize the highlights and should be stopped when they give the desired contrast.
Here is an example of "speed re-rating". Films can be underexposed and over developed to give results that may be useful in some situations. For example in a theater we may chose to use ISO800 film at ISO3200 to get very short shutter times so that motion blur does not ruin the shot. In order to get a useable density on the negative the development has to be increased, a lot, so that the underexposed face highlights are printable. However there is no shadow detail, the exposure was too short to capture shadows. This loss does not matter much in this situation, the print of the face is what matters. This is called "pushing" a film to increase "speed".
Ilford makes a range of developers and publishes helpful information sheets that give details of their use.
Some special film developers are chosen because of characteristics that influence the shape of the film tonal curve (toe slope and shoulder) for special effects that are preferred by some advanced photographers, this may be distracting for those learning the fundamentals of the process.
I believe Rodinal was invented to give increased sharpness and not for its exposure/speed/contrast/tonal curve properties.