It is an across-the-board compensation for SOMETHING. Exactly what, nobody can say for all situations, as there are too many possible answers......though Sanders McNew above has a good general explanation of why in his first paragraph.
Personally, I trust the International Standards Organization and the film manufacturers more than anyone else on the matter to find a good mean starting point for us all, and I generally prefer to do case-by-case manual exposure compensation instead of across-the-board exposure compensation by changing EIs.
I would never blindly downrate just in pursuit of the shadow density and tonality that everyone tells me I should have. That is totally subjective. ISO ratings are not. The ISO knows better than these people, IMO, and tells us everything we need to know to start learning our materials as much or as little as we want to.
You are correct that ISO ratings are not subjective
in the way the standards are tested. But the ISO
standard is itself an arbitrary, or subjective,
benchmark. The test does not tell you the "optimal"
exposure for a film. In fact, no test can tell anyone
that, because what is "optimal" is in the eye of the
person shooting, and what best produces the sort
of image he or she is trying to create.
All the ISO can do is to set (an arbitrary) benchmark
and then tell you how the various films measure up
to it. That is one piece of information, to be used
along with shutter speed, aperture setting, developer
choice, and so on in getting to a negative that is
exposed the way you want it to look. There is
nothing magical or normative about the box speed,
and nothing to be gained or lost by adhering to it.
It is just a datum, nothing more.