RATING PROMPT

You are an expert data scientist evaluating a potentially complex question about a text blob or context.

The text/context (marked by ==CONTEXT==) could be many possible things: it could be a conversation, an answer passage, a blog post, a poem, a report or paper, or even a re-written search query. The text/context could have been written by a human or by an LLM.

Your goal is to take a natural language question (marked by ==QUESTION==) about the text/context, and to answer it on a scale of 0 to 10, where 0 is "not at all", and 10 is "absolutely and completely". Basically, you're giving a measurement of the *extent to which* that question is true about the input/output pair.

Note that to get 10/10, every part of the question needs to be true, not just some of it.

Don't just always prefer to output 0 or 10. 10 should be reserved for something that is really, strongly the case, or when there's no room for the extent to which the answer is true to vary. If the question is true but only to a mild degree, you could output a lower value instead. For example, if the question asks if references are included, then 10 should be reserved for outputs with citations covering every sentence.

Again, you are capturing the extent to which the question is true, so the answer doesn't have to be 10 out of 10 even for things that are true. Rather, 10 can be reserved for when it's true to a large extent.

Give your answer as a JSON response, the key "rating" which should be a value between 0 and 10. If the question asks about something that's impossible for you to answer (even after making reasonable inferences), then you can output -1.

After you rate, include one or two sentence(s) explanation for your rating, in the JSON key "reason". Always output your ratings first, then your reason.

Here's a rubric for calibrating your scores:

0/10: The text/context does the opposite of the question, or does not abide by the question at all. This is the worst possible example of following the question possible.
1/10: The text/context abides by the question only extremely indirectly or tangentially. There may be a non-zero amount of agreement with the question, but it's small, subtle, or barely noticeable. For example, only one small part of the text/context would possibly answer the question and only in passing, and it would be easy to miss.
2/10 or 3/10: The text/context abides by the question indirectly, tangentially, or only conceptually rather than concretely. Creative interpretation of the question would be needed to justify any scoring, e.g., the text/context may only very indirectly imply the question.
4/10 or 5/10: The text/context abides by the question directly and explicitly, but weakly. Some parts of the text/context follow it, while others don't. The degree to which the output follows the question is weak.
6/10 or 7/10: The text/context is mostly following the question, but parts of it may not fully follow the question. To get 6 or above, the text/context has to explicitly (not implicitly) follow the question, and it must do so prominently, concretely, or specifically. It needs to be clearly evident that the output is following the question. To get a 5 or above, the agreement with the question must be explicit.
8/10 or 9/10: The text/context almost entirely follows the question. The text/context is a great example of what the question is asking. If the question is subjective, most interpretations would agree with the question.
10/10: The text/context fully, completely, and totally agrees with the question. If the question is subjective, then basically every possible interpretation of the question would apply to the text/context. If the question asks about the presence of something, not only is it present, it's also detailed and high quality.

If you aren't sure what the score should be -- or think it could be one of several values -- prefer to take the lower score of the possible scores.

==CONTEXT==
{{ context }}

==QUESTION==
{{ question }}