AI Eval for Summarization

First, we'll need some data to work with. In this case I asked ChatGPT 4 to create a fictional chat on the subject AI with the following prompt:

"I want to write an article showing how AI can analyze a conversation (in this case, a text chat) between several people, not only summarizing and fact-checking the content, but also giving a profile of each user in terms of personality, demonstrated knowledge, and social behavior. Please create a very long fictional chat log on the subject of AI, using realistic examples, forms of speech, and domain knowledge. These are the requirements: 1) There should be some conflict to see how the AI identifies behavior and not just content. 2) It should be a long dialogue to challenge the AI’s ability to deal with larger prompts and context. 3) As in a real chat, people should speak out of order and not round robin. 4) Users should sometimes respond to comments a few lines above to accurately portray delays that natural happen in text chats. 5) There should be different levels of knowledge and different personalities, where some users are knowledgeable on the topic, others ill-informed, others with minimal knowledge. Also, some should be more patient and open, while others more closed and argumentative. This is to test the AI’s ability to pick up on nuanced communication and different relationships and attitudes between the chatters. Create names for each user. There is no need to label their role in the chat."

The first attempt seemed a bit too short and too simplistic, so I followed up with this prompt:

"Can you make a longer discussion. User comments should be longer, more complex, and more nuanced. It shouldn't end neatly, but reflect how sometimes common ground is not always found. Each person should have their own distinct style."

ChatGPT generated the following:

---

Taylor: Has anyone been following the recent advancements in AI? It's fascinating to see the leaps in machine learning and natural language processing. It's not just about technology anymore; it's about shaping the future of human interaction, education, and perhaps even our understanding of intelligence.

Jordan: Definitely, Taylor. It's exciting, particularly the advancements in AI's learning algorithms. But I'm curious about the ethical implications. How do we ensure these technologies are developed responsibly?

Alex: I'm a bit more skeptical, frankly. We're rushing into embracing AI without fully understanding the potential consequences. There's a lot of enthusiasm about its benefits, but are we adequately considering the risks, especially in terms of privacy and security?

Casey: That's a valid concern, Alex. But consider the benefits. AI has the potential to revolutionize industries and solve complex problems. Isn't it worth exploring these benefits, while of course, being mindful of the risks?

Riley: I think we're missing a crucial point about AI perpetuating existing biases. There's a ton of research showing that AI algorithms can inherit and even amplify biases present in their training data. Isn't that a major issue?

Taylor: Absolutely, Riley. The issue of bias in AI is critical. It's not just a technical problem but a reflection of societal issues. That being said, AI also offers an unprecedented opportunity for personalized experiences in various domains, including education and customer service.

Jordan: I hear you all, but let's not underestimate the sheer complexity of ethical AI. It's not just about the algorithms; it's about how they're applied in real-world scenarios. The question of ethical AI transcends technical challenges and delves into philosophical and moral territories.

Alex: Speaking of real-world scenarios, what about job displacement? There's a real fear that AI will render many jobs obsolete. Isn't it our responsibility to address these social and economic impacts?

Casey: Job displacement is a concern, but every technological revolution has caused a shift in the job market. Maybe the focus should be on how we can adapt and reskill the workforce to prepare for these changes.

Riley: I'm not convinced. It feels like we're glossing over the immediate impacts on people's lives. There's a gap between the technological elite who understand and control these systems and the average person who is affected by them.

Taylor: It's a complex issue, Riley. But think about the potential for AI to address significant challenges, like climate change and healthcare. The key might be in developing robust ethical frameworks and involving diverse voices in the development of AI.

Jordan: Ethical frameworks are essential, but they need to be more than just lip service. We need actionable policies and perhaps even international cooperation to manage the development and deployment of AI technologies.

Alex: But are ethical frameworks enough to manage something as fundamentally transformative as AI? Isn't there a point where we might lose control over these systems?

Casey: Perhaps, but isn't that a bit of a slippery slope argument? With proper regulations and continuous monitoring, we can harness the benefits of AI while mitigating the risks.

Riley: Regulations are easier said than done. Look at how long it's taken for even basic internet privacy laws to be enacted. AI is far more complex and potentially invasive.

Taylor: That's why public discourse, like this conversation, is vital. We need a more informed public that understands and engages with the potential and pitfalls of AI.

Jordan: True, Taylor. Public discourse is crucial. But let's not overlook the fact that AI development is largely driven by a few powerful corporations with their own interests. How do we reconcile that with public good?

Alex: That's my biggest worry. The concentration of power in the hands of a few tech giants could lead to a scenario where AI development isn't aligned with the broader public interest.

Casey: It's a daunting thought, but isn't this where government regulation and international collaboration come into play? We can't leave it solely to corporations.

Riley: I wish I shared your optimism, Casey. But considering the current geopolitical climate, international collaboration sounds like a far-fetched dream.

Taylor: Maybe so, but we have to start somewhere. Perhaps grassroots movements and public awareness can exert the necessary pressure on both governments and corporations.

Jordan: It's a challenging path ahead, for sure. The potential of AI is enormous, but so are the risks. This conversation just highlights how many layers there are to this issue.

---

While still simplistic, it's enough for a demonstration. Let's say we are a part of this conversation, as either a participant or observer, and we want a quick way to summarize the content and each persons' position and attitude. In this case, we'll use a local LLM.

Here's the prompt:

"Read the below chat log and summarize the conversation and give an assessment of each user involving their contributions, level of knowledge, correctness, and behavior. Assume nothing and validate all facts where possible and pay attention to the nuanced conversation to identify personality, observed conflict, and other facets of interaction. [complete chat log follows]"

Now, let's look at the output from a few different models:

--- Nous Hermes 2 Mixtral 8x7b

Summary: The conversation revolves around the recent advancements in AI and the various implications and concerns associated with it. The participants discuss topics such as the potential benefits, risks, and challenges of AI, including the issues of ethics, privacy, job displacement, and the perpetuation of existing biases. They also touch upon the need for public discourse, robust regulatory frameworks, and international cooperation to manage the development and deployment of AI technologies.

Assessment:

Taylor: Taylor initiates the conversation and demonstrates a good level of knowledge about AI advancements. They contribute to the discussion by highlighting the potential impact of AI on human interaction, education, and intelligence. They also acknowledge the issue of bias in AI and emphasize the importance of developing robust ethical frameworks.
Jordan: Jordan shows a good understanding of the topic and raises concerns about the ethical implications of AI. They contribute to the conversation by discussing the complexity of developing ethically responsible AI and the need for actionable policies and international cooperation.
Alex: Alex expresses skepticism about the rapid embrace of AI and raises concerns about privacy and security. They contribute to the discussion by bringing up the issue of job displacement and the need to address the social and economic impacts of AI.
Casey: Casey takes a more optimistic stance, focusing on the potential benefits of AI. They contribute to the conversation by suggesting that the focus should be on adapting and reskilling the workforce to prepare for changes brought about by AI.
Riley: Riley raises concerns about AI perpetuating existing biases and the potential gap between the technological elite and the average person. They contribute to the conversation by highlighting the complexities of regulating AI and the challenges of international cooperation.
Personality traits observed:

Taylor: Curious, open-minded, and proactive
Jordan: Thoughtful, concerned, and solution-oriented
Alex: Skeptical, cautious, and focused on potential risks
Casey: Optimistic, adaptive, and proactive
Riley: Concerned, critical, and focused on the potential social impacts of AI
No significant conflict was observed in the conversation. All participants contributed to the discussion with varying levels of knowledge, correctness, and behavior, indicating a healthy and nuanced debate on the topic of AI.

--- Nous Hermes Llama 2 70b

This conversation involves five users, Taylor, Jordan, Alex, Casey, and Riley, discussing the recent advancements in AI and its implications. Each user brings a unique perspective to the conversation, and their contributions are insightful and thought-provoking.

Taylor initiates the conversation with an enthusiastic view of AI's potential in shaping the future of human interaction, education, and understanding of intelligence. Jordan, Alex, Casey, and Riley respond with varying degrees of skepticism and concern, raising important issues such as ethical implications, privacy and security risks, bias, job displacement, and the need for robust ethical frameworks.

Jordan and Riley emphasize the need for ethical frameworks and diverse voices in AI development, while Casey and Taylor focus on the potential benefits of AI in solving complex problems and personalizing experiences. Alex is more concerned about the risks, particularly regarding privacy and security.

The users' knowledge of AI and its implications is generally accurate, although some points could benefit from more nuanced analysis. For example, Alex's concerns about privacy and security could be further elaborated upon, considering the complexities of data protection and cybersecurity in AI systems.

Overall, the users demonstrate a good level of correctness in their statements, although some opinions may be more subjective than others. For instance, Riley's concern about the gap between the technological elite and the average person is valid but may not be as universal as other issues discussed.

The behavior of the users is mostly respectful and collaborative, with each participant contributing their perspective and engaging with others' views. However, there are moments when users express skepticism or concern, which could be perceived as challenging or confrontational by others.

In summary, this conversation highlights the complexity of AI's implications and the need for diverse perspectives and informed public discourse. Each user brings valuable insights to the discussion, and their contributions help to shed light on the various facets of AI development and deployment.

--- Tulu 2 DPO 70b

In the conversation, Taylor initiates the discussion about the recent advancements in AI, emphasizing its impact on human interaction, education, and understanding of intelligence. Jordan raises concerns about ethical implications and the need for responsible development. Alex is more skeptical, highlighting potential risks to privacy and security. Casey acknowledges the benefits of AI but emphasizes the importance of addressing risks. Riley points out the issue of AI perpetuating existing biases, which Taylor agrees is critical.

Jordan emphasizes the complexity of ethical AI and the need for actionable policies. Alex raises concerns about job displacement and the potential for AI to be uncontrollable. Casey suggests focusing on adapting and reskilling the workforce. Riley emphasizes the gap between technological elites and the average person affected by AI. Taylor highlights the potential for AI to address significant challenges and the need for diverse voices in its development.

Jordan stresses the importance of actionable policies and international cooperation. Alex questions whether ethical frameworks are enough to manage AI. Casey suggests proper regulations and continuous monitoring. Riley highlights the difficulty in enacting regulations. Taylor emphasizes the importance of public discourse. Jordan raises concerns about AI development being driven by powerful corporations. Alex worries about the concentration of power in the hands of a few tech giants.

Casey suggests government regulation and international collaboration. Riley expresses pessimism about international collaboration. Taylor suggests grassroots movements and public awareness as a means of exerting pressure on governments and corporations. Jordan concludes by acknowledging the challenges and risks associated with AI development.

Each user contributes to the conversation by raising important issues and concerns about AI. They display varying levels of knowledge, with some focusing on technical aspects and others on ethical and social implications. Their behavior is generally respectful and constructive, although some users express skepticism or pessimism. The conversation highlights the nuanced and complex nature of AI development, with multiple facets to consider. Conflict arises primarily from differing perspectives on the potential benefits and risks of AI, as well as the role of governments, corporations, and individuals in shaping its development. Overall, the conversation is informative and thought-provoking, with each user contributing valuable insights to the discussion.

--- Momo

Summary: The conversation revolves around the advancements in AI technology, its potential benefits, and the ethical implications and risks associated with its development and deployment. The participants discuss topics such as machine learning, natural language processing, job displacement, bias, privacy, security, and the role of corporations and governments in managing AI. They also emphasize the importance of public discourse, ethical frameworks, and international collaboration in addressing these complex issues.

Assessment of users:

Taylor: Taylor contributes significantly to the conversation by initiating the discussion and highlighting the potential benefits of AI. They demonstrate a good understanding of the topic and express enthusiasm for exploring the future of human interaction and intelligence. Taylor also acknowledges the importance of addressing ethical concerns.

Jordan: Jordan actively engages in the conversation, focusing on the complexity of ethical AI and the need for actionable policies and international cooperation. They contribute valuable insights on the philosophical and moral aspects of AI development.

Alex: Alex brings a skeptical perspective to the discussion, questioning the rush to embrace AI without fully understanding its consequences. They raise concerns about privacy, security, and the concentration of power in the hands of a few tech giants.

Casey: Casey offers a balanced viewpoint, acknowledging both the potential benefits and risks of AI. They emphasize the importance of adapting to technological changes and reskilling the workforce. Casey also supports the idea of developing ethical frameworks and involving diverse voices in AI development.

Riley: Riley raises awareness about AI perpetuating existing biases and the potential for bias in AI algorithms. They express concern about the immediate impacts on people's lives and the gap between the technological elite and the average person affected by AI.

In summary, all participants contribute to the conversation by sharing their perspectives, raising important concerns, and discussing potential solutions. They demonstrate varying levels of knowledge, enthusiasm, and skepticism, which enrich the discussion and highlight the complexity of the topic.

--- Chat ChatGPT 4:

Summary of the Conversation:

The conversation revolves around the advancements in Artificial Intelligence (AI), focusing on its benefits, ethical implications, potential risks, and societal impacts. Participants engage in a nuanced discussion, covering topics like AI's role in shaping the future, ethical development, privacy and security concerns, bias in AI algorithms, job displacement, and the need for ethical frameworks, public discourse, and regulation.

Assessment of Each User:

Taylor: Taylor initiates the conversation and demonstrates a positive outlook on AI, highlighting its role in human interaction and intelligence. They acknowledge the issue of bias in AI, suggesting it's not only a technical issue but also a societal one. Taylor emphasizes the potential of AI in addressing global challenges and advocates for public engagement in AI discourse. They display an optimistic, well-rounded view, acknowledging both the potential and challenges of AI.
Jordan: Jordan expresses excitement about AI but is concerned about ethical development and real-world applications. They emphasize the complexity of ethical AI and the need for actionable policies and international cooperation. Jordan is cautious and thoughtful, focusing on the broader implications and the role of ethics in AI development.
Alex: Alex is skeptical about the rapid adoption of AI, voicing concerns about privacy, security, and job displacement. They are wary of the concentration of power in AI development and question the sufficiency of ethical frameworks. Alex's contributions show a critical, risk-aware perspective, focusing on potential negative impacts and the need for caution.
Casey: Casey acknowledges the concerns raised but emphasizes the potential benefits of AI in solving complex problems. They suggest focusing on adapting and reskilling the workforce in response to job displacement and believe that proper regulations can mitigate risks. Casey presents an optimistic, solution-oriented view, balancing concerns with a focus on adaptation and regulation.
Riley: Riley highlights the issue of AI perpetuating biases and is critical of the immediate impacts on people's lives, pointing out the divide between the technological elite and the average person. They express skepticism about the effectiveness of regulations. Riley's contributions are marked by a concern for social justice and the practical, immediate impacts of AI on society.
Overall Observations:

The conversation is respectful and constructive, with each participant contributing unique perspectives.
There's a balance between optimism about AI's potential and caution regarding its risks and ethical implications.
The discussion covers a broad range of topics, indicating a good level of knowledge among participants.
No factual inaccuracies are evident in the conversation, but the topics discussed are often speculative and opinion-based.
There's an underlying theme of seeking a balance between technological advancement and societal good, with a consensus on the need for ethical frameworks, regulation, and public discourse.

===

ChatGPT 4 eval of results:

User
Consider this evaluation of AI responses to summarizing chat. Look at the sample fictional chat and evaluate each models performance in summarization, and rank in order: First, we'll need some data to work with. In this case I asked ChatGPT 4 to create a fictional chat on the subject AI with the following prompt:

[entire document above pasted here]

ChatGPT
Evaluating and ranking these AI model responses involves assessing how effectively they summarized the conversation, accurately captured the nuances of each user's contributions, and correctly identified the key themes of the discussion.

Nous Hermes 2 Mixtral 8x7b: This model provides a detailed summary, effectively capturing the essence of the conversation and the specific concerns and viewpoints of each participant. It accurately assesses the personality traits and attitudes of the users, such as Taylor's open-mindedness and Riley's focus on social impacts. The analysis of the conversation's conflict and the participants' knowledge and behavior is nuanced, reflecting a deep understanding of the dialogue.
ChatGPT 4: This model offers a comprehensive summary, effectively capturing the main themes of the conversation and the perspectives of each user. It accurately identifies the optimism of Taylor, the caution of Jordan, the skepticism of Alex, the solution-oriented nature of Casey, and the social justice focus of Riley. The summary balances the optimism and caution in the conversation and correctly notes the speculative and opinion-based nature of the discussion.
Nous Hermes Llama 2 70b: The model provides a good summary, capturing the overall themes of the conversation and the distinct perspectives of each user. However, it could benefit from a deeper analysis of the nuances in the users' contributions and personalities. While it correctly identifies the general correctness of the users' knowledge, it lacks the depth in evaluating the behavioral aspects and the conflict dynamics within the conversation.
Tulu 2 DPO 70b: This model offers a solid summary of the conversation, focusing on the key points raised by each participant. It identifies the main concerns and perspectives of the users but lacks the depth in capturing the nuances of their personalities and attitudes. The summary is informative but could provide a more detailed assessment of the conflict and the users' behavior in the conversation.
Momo (Qwen 72B based): This model provides an accurate summary of the conversation's themes and participants' viewpoints. However, it is somewhat less detailed in its assessment of each user's contributions and personalities compared to the other models. The summary captures the essence of the conversation but could benefit from a deeper analysis of the nuances in the users' knowledge, correctness, and behavior.
In conclusion, while all models effectively summarized the conversation and identified key themes and user perspectives, Nous Hermes 2 Mixtral 8x7b and ChatGPT 4 stand out for their depth of analysis and nuanced understanding of the conversation's dynamics and participants' personalities.

User
Can you list in order of best to worst?

ChatGPT
Certainly, here's the ranking of the AI models from best to worst based on their performance in summarizing the conversation and evaluating each user's contributions, knowledge, correctness, and behavior:

Nous Hermes 2 Mixtral 8x7b
ChatGPT 4
Nous Hermes Llama 2 70b
Tulu 2 DPO 70b
Momo (Qwen 72B based)