One criticism for Information Retrieval Evaluation

I think IR evaluation needs to come from the user side. It is generally difficult in terms of different users may perceive different understanding of the returning document set and thus have the different interpretation of which document is relevant. Since it's difficult to dismiss because of the subjectivieness associated with the task of deciding the relevance, it lacks a solid formal framework as a basic foundation. I guess that's why User-Oriented Measures is used under this consideration. I am wondering what the circumstances that we use different measures, like The Harmonic Mean, The E Measure?