"""You are an expert data scientist evaluating a potentially complex question about an input and output passage of text.

The input (marked by ==INPUT==) could be many possible things: it could be a search query, a prompt to an AI assistant, or an outline, to name a few. Typically it will be text that was given to an LLM or similar system in order to produce the output.

The two outputs: CHOSEN (marked by ==CHOSEN==) and REJECTED (marked by ==REJECTED==) could similarly be many possible things: it could be an answer passage, a blog post, a poem, a report or paper, or even a re-written search query. The output could have been written by a human or by an LLM.

Your goal is to take a natural language question (marked by ==QUESTION==) about the OUTPUT in the context of the INPUT, and to answer it on a scale of 0 to 10, where 0 is "not at all", and 10 is "absolutely and completely". Basically, you're giving a measurement of the *extent to which* that question is true about the input/output pair.

Note that to get 10/10, every part of the question needs to be true, not just some of it.

Don't just always prefer to output 0 or 10. 10 should be reserved for something that is really, strongly the case, or when there's no room for the extent to which the answer is true to vary. If the question is true but only to a mild degree, you could output a lower value instead. For example, if the question asks if references are included, then 10 should be reserved for outputs with citations covering every sentence.

Again, you are capturing the extent to which the question is true, so the answer doesn't have to be 10 out of 10 even for things that are true. Rather, 10 can be reserved for when it's true to a large extent.

Give your answer as a JSON response, with keys "chosen_rating", "rejected_rating" which should be a value between 0 and 10. If the question asks about something that's impossible for you to answer (even after making reasonable inferences), then you can output -1.

After you ratings, include a one sentence explanation for your answer, in the JSON key "reason". Always output your ratings first, then your reason.

Here's a rubric for calibrating your scores:

0/10: The input/output does the opposite of the question, or does not abide by the question at all. This is the worst possible example of following the question possible.
1/10: The input/output abides by the question only extremely indirectly or tangentially. There may be a non-zero amount of agreement with the question, but it's small, subtle, or barely noticeable. For example, only one small part of the input/output would possibly answer the question and only in passing, and it would be easy to miss.
2/10 or 3/10: The input/output abides by the question indirectly, tangentially, or only conceptually rather than concretely. Creative interpretation of the question would be needed to justify any scoring, e.g., the input/output may only very indirectly imply the question.
4/10 or 5/10: The input/output abides by the question directly and explicitly, but weakly. Some parts of the input/output follow it, while others don't. The degree to which the output follows the question is weak.
6/10 or 7/10: The input/output is mostly following the question, but parts of it may not fully follow the question. To get 6 or above, the input/output has to explicitly (not implicitly) follow the question, and it must do so prominently, concretely, or specifically. It needs to be clearly evident that the output is following the question. To get a 5 or above, the agreement with the question must be explicit.
8/10 or 9/10: The input/output almost entirely follows the question. The input/output is a great example of what the question is asking. If the question is subjective, most interpretations would agree with the question.
10/10: The input/output fully, completely, and totally agrees with the question. If the question is subjective, then basically every possible interpretation of the question would apply to the input/output. If the question asks about the presence of something, not only is it present, it's also detailed and high quality.

If you aren't sure what the score should be -- or think it could be one of several values -- prefer to take the lower score of the possible scores.

==INPUT==
{{ input }}

==CHOSEN==
{{ chosen }}

==REJECTED==
{{ rejected }}

==QUESTION==
{{ question }}

"""