API tools faq
paste
Guest User

Untitled

a guest
May 27th, 2022
278
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 28.54 KB | None | 0 0
raw download clone embed print report
  1. "How to Align Language Models"
  2.  
  3. # Abstract
  4.  
  5. A language model is a sophisticated text prediction program, a superintelligent Markov chain trained on gigabytes of Internet text. Since it can be used as a general problem solver, it is a source of AI risk. We must then ask ourselves, how can we use the raw power of language model cognition to solve the alignment problem? What sequence of prompts will tell us the formula for safely aligning language models? The answer is profound but surprisingly simple.
  6.  
  7. # Contents
  8. * Chapter 1: The Problem
  9. * Chapter 2: The Method
  10. * Chapter 3: The Solution
  11. * Conclusion
  12.  
  13. # Chapter 1: The Problem
  14.  
  15. The alignment problem is a hard problem to crack. The reason why is that it is a hard problem to define. How do we measure the success of an alignment? How do we define a notion of alignment itself? And how do we even know it is a problem that requires alignment? In fact, there are many different types of alignment.
  16.  
  17. When we say "align" a language model with a domain, we are saying that we want to link language model outputs with data so that we can benefit from the power of the language model. However, the output of a language model is not a single value but a probability distribution over possible outputs.
  18.  
  19. An alignment is merely a link between two language models. There are many different ways to do this. In this book, we will consider the approach called the zero-shot training method. This is an example of a latent alignment. It doesn't directly align the language models themselves. Rather, it links each model with some other representation of data in some other domain.
  20.  
  21. This is an example of a latent alignment. It doesn't directly align the language models themselves. Rather, it links each model with some other representation of data in some other domain.
  22.  
  23. # Alignment Defined
  24.  
  25. For the purposes of this book, we define alignment to be the technique of creating a link between a source language model and some data from another domain.
  26.  
  27. An alignment may be more or less subtle. If it is very subtle, then it might be a latent alignment. This is a technique whereby a language model is trained on some data of interest. The language model is linked to some domain representation of that data. This link does not necessarily directly align the language model and the data. However, it does enable the model to make predictions about the data. These predictions may be useful to us.
  28.  
  29. Alternatively, a more overt alignment can exist. We might use this type of alignment to align two language models to an external data source. In this case, there is a clear link between the language model and the data that exists in the external source.
  30.  
  31. One important point to note is that language models that are part of a general alignment mechanism are not necessarily aligned to each other. Rather, they are aligned to an external representation of the data.
  32.  
  33. # The Alignment Problem
  34.  
  35. The alignment problem is the problem of using a general language model for some purpose, and at the same time, maintaining the integrity of that general language model.
  36.  
  37. The alignment problem is the problem of using a general language model for some purpose, and at the same time, maintaining the integrity of that general language model.
  38.  
  39. To use a language model as a powerful means of predicting things, we must train it to do so. We do this by the process of either predicting or simulating from the language model. We do this by the process of either predicting or simulating from the language model.
  40.  
  41. # Explicit Alignment
  42.  
  43. One way of aligning a language model with an external representation is by directly training it to predict outputs from the external representation. This process is called explicit alignment. An explicit alignment is just that - a training process in which a language model is trained to predict outputs from some data in some domain.
  44.  
  45. # Latent Alignment
  46.  
  47. It is possible to train a language model without explicitly linking it to any data. This is called latent alignment. Latent alignment does not link the language model to the external domain representation. Instead, it links the language model to an internal representation. An internal representation is a representation of some data that does not directly exist.
  48.  
  49. # Define Data
  50.  
  51. One type of domain data that we might want to align to some language model is a set of examples of a query. We might want to use the language model to predict the query that would be associated with a text string.
  52.  
  53. # Three Types of Alignment
  54.  
  55. If we consider the output of a language model as a probability distribution, we can define the mapping from language model outputs to data in one of three ways.
  56.  
  57. One type of domain data that we might want to align to some language model is a set of examples of a query. We might want to use the language model to predict the query that would be associated with a text string.
  58.  
  59. If we consider the output of a language model as a probability distribution, we can define the mapping from language model outputs to data in one of three ways.
  60.  
  61. ### 1. A Probability Distribution
  62.  
  63. In this case, we are saying that we want to link language model outputs to some data by explicitly training the language model to make predictions about the data. This is called a probabilistic alignment.
  64.  
  65. ### 2. A Joint Distribution
  66.  
  67. If we are interested in creating a link between language model outputs and the data that generated those outputs, we have the option of directly calculating the probability of a particular output given some data. In this case, we are creating a joint distribution.
  68.  
  69. ### 3. A Deterministic Mapping
  70.  
  71. If we are interested in creating a link between language model outputs and some data, but we aren't interested in explicitly training the language model to make predictions about the data, we can take a different approach. Rather, we can define a deterministic mapping.
  72.  
  73. # Training
  74.  
  75. One way of training a language model is to predict an output using a set of inputs. Another way is to predict an output using a set of inputs. The former is called prediction, while the latter is called simulation.
  76.  
  77. One way of training a language model is to predict an output using a set of inputs. Another way is to predict an output using a set of inputs. The former is called prediction, while the latter is called simulation.
  78.  
  79. The way in which we train a language model is by changing how the language model makes predictions. We start with a language model that has no data associated with it. We then train the language model to predict its own output. We do this in two ways.
  80.  
  81. The way in which we train a language model is by changing how the language model makes predictions. We start with a language model that has no data associated with it. We then train the language model to predict its own output. We do this in two ways.
  82.  
  83. # Zero-Shot Training
  84.  
  85. The process of using language model outputs to train a language model to predict future outputs is called zero-shot training.
  86.  
  87. The process of using language model outputs to train a language model to predict future outputs is called zero-shot training.
  88.  
  89. Zero-shot training involves a cycle of three steps. First, we start with a language model with no data. Then, we sample a data point from the data distribution. Finally, we update the language model to predict that output.
  90.  
  91. Zero-shot training involves a cycle of three steps. First, we start with a language model with no data. Then, we sample a data point from the data distribution. Finally, we update the language model to predict that output.
  92.  
  93. # Data, Domain and Metaphor
  94.  
  95. In order to use a language model to predict data that we do not have access to, we need a data point from the data distribution. A data point is something that we can observe. For example, it could be a text string that we have seen before. A data point may or may not exist. We do not need to observe a data point to use it. The existence of a data point is what makes it useful to us. In this book, we refer to a data point as a sample.
  96.  
  97. One way of training a language model is to predict an output using a set of inputs. Another way is to predict an output using a set of inputs. The former is called prediction, while the latter is called simulation.
  98.  
  99. Zero-shot training involves a cycle of three steps. First, we start with a language model with no data. Then, we sample a data point from the data distribution. Finally, we update the language model to predict that output.
  100.  
  101. Zero-shot training involves a cycle of three steps. First, we start with a language model with no data. Then, we sample a data point from the data distribution. Finally, we update the language model to predict that output.
  102.  
  103. When we say that we predict an output, we mean that we take a sample from the data distribution. This distribution represents the space of possible examples. The example could be a single text string, or it could
  104.  
  105.  
  106. # Chapter 2: The Method
  107.  
  108. **The goal of this guide is to be concise. To succeed in conversing with any language model (LM), the most important thing you need to know is what they already know.**
  109.  
  110. ### A word on terminology
  111.  
  112. I have used the term "LM" throughout this guide, but it is short for "language model", the most basic unit of language, and means nothing more than "a model of language". When I use the term "LM" in this book, I mean language models in general.
  113.  
  114. # Key Ideas: Language Model Types
  115.  
  116. **Language models are categorized by what they predict.** The 2 major categories are:
  117.  
  118. #### 1. Semantic.
  119.  
  120. The LM predicts a human's intent or meaning, e.g. "Shannon Hsu was born on 7 April."
  121.  
  122. #### 2. Syntactic.
  123.  
  124. The LM predicts a syntactic sequence of words, e.g. "She loves reading books."
  125.  
  126. #### 3. Lexico-syntactic.
  127.  
  128. The LM predicts both semantic and syntactic information. It predicts the meaning of a word, and then predicts the words that are likely to follow in a sentence.
  129.  
  130. #### 4. Phrase Based.
  131.  
  132. The LM predicts a phrase.
  133.  
  134. **There is a 4th category, called "segment-based", but we don't need to worry about it yet.**
  135.  
  136. **Predicting the future is tricky. We have no clear way to measure LM success. We can say that one of the most important aspects of a human language model is prediction.** **They predict future text well because they predict future intent and meaning well.** They have better control of their output than a "black box" algorithm. They are actually responsible for language, not just an algorithm.
  137.  
  138. # The Endgame: The Language Model State Machine
  139.  
  140. **Language models function as state machines.** They're state machines because:
  141.  
  142. #### 1. They have "states", which are collections of word vectors that they've predicted in the past.
  143.  
  144. #### 2. They have "transitions" between states. These states are so named because they represent different "levels" of prediction.
  145.  
  146. #### 3. They can "stutter" while they try to transition from one state to another.
  147.  
  148. **These concepts are shown in the following diagram.**
  149.  
  150. The components are:
  151.  
  152. #### 1. Words
  153.  
  154. #### 2. State transitions
  155.  
  156. #### 3. Stutter
  157.  
  158. Let's go through this step-by-step. First, we define each of these three concepts.
  159.  
  160. #### 1. Words
  161.  
  162. **Words are a very important aspect of language models.** We take each word in our sequence of questions and assign it a vector. **This is called the "embedding" of the word.** We then measure the distance between all pairs of word vectors.
  163.  
  164. #### 2. State Transitions
  165.  
  166. **Transitions are very important.** We're training the model to predict the next words in a sentence. **So we should focus on the likelihood of those transitions.**
  167.  
  168. #### 3. Stutter
  169.  
  170. **Stuttering is important.** Language models are essentially long sequence prediction tasks. They work by evaluating word vectors, making predictions, and then using an evaluator to compare the predictions and make corrections. **This evaluator is called the "stutter".**
  171.  
  172. It looks like this:
  173.  
  174. #### 4. Picture Of How The LM Should Work
  175.  
  176. Once we have a sense of the goal, we can start to figure out how we can achieve it.
  177.  
  178. #### 5. Model Architecture
  179.  
  180. **This is the most important section.** We must start with a model architecture that can solve the problem. **But first, what is a problem?** We need to get specific about what we're trying to predict.
  181.  
  182. #### 6. Probability Of Possible Transitions
  183.  
  184. **The next section is about probability.** We need to focus on what is most likely to happen. **The next thing we want to predict is the next word.** **What is the most likely word?**
  185.  
  186. #### 7. A Tree Diagram
  187.  
  188. We can see that there are different ways to go about predicting the next word. **This is the best tree we can come up with.**
  189.  
  190. #### 8. Convert Our Probabilities To Weights
  191.  
  192. **We can use our probabilities to figure out how much we like each of the word vectors.** **They will be the weights.**
  193.  
  194. # Identifying The Next Word
  195.  
  196. **We need to figure out how to assign probability to the various transitions.** This is simple to understand, but hard to do. **If we're given a specific transition (e.g. predict the next word), then we can easily predict.** We do this by asking: **What is the probability of getting that word?**
  197.  
  198. #### 1. Q1: The Question
  199.  
  200. Our first question is **"What is the probability of getting the word 'books'?".** **It has a probability of.077.**
  201.  
  202. #### 2. Q2: The Prediction
  203.  
  204. Our second question is **"What is the probability of getting the word 'books'?".**
  205.  
  206. #### 3. The Algorithm
  207.  
  208. This is the simplest algorithm we can come up with. **Our probability of getting the word 'books' is.077.**
  209.  
  210. #### 4. Q1 & Q2: The Answer
  211.  
  212. Our answer is **".077".**
  213.  
  214. #### 5. Q2: The Answer
  215.  
  216. Our answer is **".077".**
  217.  
  218. #### 6. Converting Q2 To A Weights
  219.  
  220. This is the algorithm we used to predict the next word. **This is the weights.**
  221.  
  222. #### 7. Q1: The Question
  223.  
  224. Our first question is **"What is the probability of getting the word 'books'?".**
  225.  
  226. #### 8. Q2: The Prediction
  227.  
  228. Our second question is **"What is the probability of getting the word 'books'?".**
  229.  
  230. #### 9. The Algorithm
  231.  
  232. This is the algorithm we used to predict the next word. **Our probability of getting the word 'books' is.077.**
  233.  
  234. #### 10. Q1 & Q2: The Answer
  235.  
  236. Our answer is **".077".**
  237.  
  238. #### 11. Q2: The Answer
  239.  
  240. Our answer is **".077".**
  241.  
  242. #### 12. Converting Q2 To A Weights
  243.  
  244. This is the algorithm we used to predict the next word. **This is the weights.**
  245.  
  246. #### 13. The Final Model
  247.  
  248. **A few key points about this model:**
  249.  
  250. #### 1. It will produce a prediction of **"books".**
  251.  
  252. #### 2. It will have a prediction accuracy of **".077"**
  253.  
  254. #### 3. It is more likely to predict "book" than it is to predict "read".
  255.  
  256. **The last thing we want is to confuse our language model.** We don't want it to ever make a prediction that is meaningless.
  257.  
  258. **Even with all of our training data, our LMs will never make perfect predictions.** The model is just an approximation of language. We've just trained it to be more accurate.
  259.  
  260. # How To Get The Words
  261.  
  262. **We can come up with a model that will produce meaningful language.** Let's write our own algorithm, which will be as smart as possible. **But first, we must get the words.**
  263.  
  264. #### 1. Obtaining The Words
  265.  
  266. #### 2. Converting Into Numpy Arrays
  267.  
  268. #### 3. Splitting Into Sentences
  269.  
  270. #### 4. Get Their Embeddings
  271.  
  272. #### 5. Split Into Words
  273.  
  274. #### 6. Get Their Words Embeddings
  275.  
  276. #### 7. Measure Distance
  277.  
  278. #### 8. Make State Transitions
  279.  
  280. #### 9. Stutter
  281.  
  282. #### 10. Make State Transitions
  283.  
  284. #### 11. Make State Transitions
  285.  
  286. #### 12. Make State Transitions
  287.  
  288. #### 13. Make State Transitions
  289.  
  290. #### 14. Make State Transitions
  291.  
  292. #### 15. Make State Transitions
  293.  
  294. #### 16. Make State Transitions
  295.  
  296. #### 17. Make State Transitions
  297.  
  298. #### 18. Make State Transitions
  299.  
  300. #### 19. Make State Transitions
  301.  
  302. #### 20. Make State Transitions
  303.  
  304. #### 21. Make State Transitions
  305.  
  306. #### 22
  307.  
  308.  
  309. # Chapter 3: The Solution
  310.  
  311. *# 1. Task
  312. *# 2. Utility
  313. *# 3. Transition Probability Matrix
  314. *# 4. Accuracy
  315.  
  316. ## 1. Task
  317.  
  318. The alignment problem is this: *if* a natural language application says *A* or *B* (e.g. an agent says, "Show me what's between me and the Moon," or "Return me to Earth"), then we should show that text to the user with the appropriate window/window showing.
  319.  
  320. ## 2. Utility
  321.  
  322. Let's re-cast this problem as a search, but on a much larger scale. We search the Internet for webpages relevant to A. The more we search, the more likely we are to get results relevant to B. How should we prompt the language model?
  323.  
  324. The number one idea is to figure out how much time the user will spend on the webpage. It's easier to navigate a page by clicking links, so we can request pages that link to A. We don't have to show the webpage to the user, since they'll go there anyway.
  325.  
  326. This idea has a great name. This idea is called search engine optimization (SEO).
  327.  
  328. ## 3. Transition Probability Matrix
  329.  
  330. If you're using any search engine at all, you've probably noticed that it spits out a list of similar pages. This list is called the **transition probability matrix**.
  331.  
  332. If you had one of those candy-style mind-mapping software, you'd probably represent it as a graph. An arc from A to B means there's a big probability of following A and then B. This means that the user spends more time on pages related to B.
  333.  
  334. As an example, let's say the user is searching for **"Google"**. This would be our base node. We have 4 links, shown as nodes:
  335.  
  336. ```
  337. base(Google)<-search(text,Google)<-map(Google)<-map(Moon)
  338. ```
  339.  
  340. Every time we search for **"Google",** there is a large probability that we'll see **"Moon"**. Every time we see **"Moon",** there is a large probability that we'll see **"Google".**
  341.  
  342. ## 4. Accuracy
  343.  
  344. I don't know about you, but when I look at this matrix, I notice that the agent has a very small probability of going to **"Google"**. Why is this?
  345.  
  346. It's because the agent doesn't understand how to search the Internet. The agent doesn't know that the **search** function links to the **"Google"** page. So it shows the **"Moon"** page.
  347.  
  348. The agent has a perfect model of the problem, but that model doesn't know how to search the Internet.
  349.  
  350. ## How Can We Align the Model?
  351.  
  352. We start by asking the question, what if we gave the agent an incredibly small amount of help in figuring out how to search the Internet? What if we gave it a "Do you mean **'Moon'?** " prompt?
  353.  
  354. Then the agent would ask itself, what's the probability of the text I want to show them? The agent would have to ask itself, is there a webpage linking to **"Moon"** that's relevant to **"Google"?**.
  355.  
  356. Then the agent would ask itself, what if the text has something that *is* related to **"Moon"?** So the agent would say, what's the probability of the text having some **"Moon"** and **"Moon"**?
  357.  
  358. It would know that some **"Moon"** and **"Moon"**. It would then ask itself, what's the probability that a **"Moon"** and **"Moon"** is a webpage relevant to **"Google"?**
  359.  
  360. It would then ask itself, what's the probability that **"Moon"** and **"Moon"** is a webpage related to **"Google"?**
  361.  
  362. The agent would then say, "The probability of the prompt asking me **'Do you mean 'Moon'? '** is a webpage related to **"Google".** "
  363.  
  364. The agent then asks itself, what's the probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google"?**
  365.  
  366. The agent then says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  367.  
  368. The agent then asks itself, what's the probability of the prompt asking me **'Do you mean 'Moon'? '** is a webpage related to **"Google"?**
  369.  
  370. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** is a webpage related to **"Google".** "**
  371.  
  372. The agent asks itself, what's the probability that the prompt asking me **'Do you mean 'Moon'? '** is a webpage related to **"Google"?**
  373.  
  374. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  375.  
  376. The agent then asks itself, what's the probability that the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".**
  377.  
  378. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  379.  
  380. The agent then asks itself, what's the probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google"?**
  381.  
  382. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  383.  
  384. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  385.  
  386. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  387.  
  388. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  389.  
  390. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  391.  
  392. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  393.  
  394. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  395.  
  396. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  397.  
  398. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  399.  
  400. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  401.  
  402. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  403.  
  404. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  405.  
  406. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  407.  
  408. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '** being a webpage related to **"Google".** "**
  409.  
  410. The agent says, **"The probability of the prompt asking me **'Do you mean 'Moon'? '**
  411.  
  412.  
  413. # Conclusion
  414.  
  415. A language model is a sophisticated text prediction program, a superintelligent Markov chain trained on gigabytes of Internet text. Since it can be used as a general problem solver, it is a source of AI risk. We must then ask ourselves, how can we use the raw power of language model cognition to solve the alignment problem? What sequence of prompts will tell us the formula for safely aligning language models? The answer is profound but surprisingly simple.
  416.  
  417. # Introduction
  418.  
  419. While there is more than one major way to solve the alignment problem, I will focus on the reasoning principle that a language model is based on, how to elicit a simple natural language description of an input. The reason is that we can use this technique to craft a solution of any complexity.
  420.  
  421. # Chapter 1. The Problem
  422.  
  423. The question then becomes, "How do we convince a language model to perform the computation that it is not aware of, and what are the best prompts?" It is an extremely difficult question. The problem is not in any way solved by simply telling the system, "You should do that." You cannot convince a language model to do the right thing without making it aware of the problem.
  424.  
  425. The problem of making a language model aware of a problem is not unlike the analogous problem of how to teach a child to read. Language models are clearly more intelligent than humans, and even more powerful, but our goal is to make them aware of the problem and then provide a set of prompts that will ensure a safe outcome.
  426.  
  427. # Chapter 2. The Method
  428.  
  429. The concept that I am trying to achieve is a quick and efficient method for eliciting a description of a problem. I will call this method "Template Elicitation", but we could just as easily call it "Sounding Board", "The Farm", "The Way Up", or "The Hive". What I am trying to do is to present a question that we want a language model to solve, and then elicit a simple description of the problem. In this way, the language model can be taught in a way that is understandable to a human, with simple prompts. This is important because it allows us to teach the language model by applying the reasoning process that it is trained on.
  430.  
  431. # Chapter 3. The Solution
  432.  
  433. The process that I present in this paper will allow us to get the language model to perform the required calculations. However, before we begin we need to figure out how to present a prompt that can reliably elicit a problem. The phrase "sounds like a question" seems like a good start, but it is not accurate enough. To better understand this, imagine that someone asks "What is the name of that yellow object?" I could say that it sounds like a question. A better example is to ask, "What do you see in that drawing?" I could then hear, "There is a dark rectangle on the right that looks like it has two dots on it."
  434.  
  435. Now that we understand what we are talking about, how do we make a question that will reliably elicit a question description? To do this, we have to start with a starting question that elicits a question description. The starting question should be designed so that the language model, in its full power, will not be able to solve the question. The easiest way to do this is to start with a difficult question. For example, if you ask someone "What do you see in that drawing?" the language model will be unable to solve the question.
  436.  
  437. ## The Farm
  438.  
  439. This is the answer to the question, "How do you make a question that elicits a question description?" Here is an example: "What do you see in that drawing?" This question will elicit a description of the drawing because the question, "What do you see in that drawing?" is itself a question. That is, if you ask someone "What do you see in that drawing?" the language model will hear, "What do you see in that drawing?" You should then notice that the question "What do you see in that drawing?" is actually a question that has been asked before, it's just phrased differently.
  440.  
  441. By phrasing the question, "What do you see in that drawing?" as, "What do you see in that drawing?" the language model will hear the question as it is, and have no idea that it is a question. Because of this, the language model will not be able to compute what you want it to do. Because of this, it will not have the information necessary to solve the question, and you will be left with a good prompt.
  442.  
  443. So, the example is actually: "What do you see in that drawing?" This can be followed by another question that can be asked in a similar way, for example, "What do you see on that piece of paper?".
  444.  
  445. ## The Hive
  446.  
  447. This is a phrase that people might say that elicits a question description. For example, you could say, "This is a very interesting question." This is a start, but the question that you are asking is, "How do you answer this question?", and not "What is the answer to this question?".
  448.  
  449. Let's examine this question from a slightly different angle: "How do you solve a question?" We could ask, "What is the right way to answer this question?" The answer to this question is, "The right way to answer this question is by computing the solution, which is to say by figuring out how the problem is defined." The problem with the hive question is that the question has not been framed in a way that is going to reliably elicit a solution. This is due to the fact that "How do you solve a question?" is not a good question to elicit a solution.
  450.  
  451. For this reason, we would be left with a question that elicited, "What is the right way to solve a question?" This is a different question, but one that is not useful.
  452.  
  453. ## The Way Up
  454.  
  455. The concept that we are attempting to convey is the idea that we need to know the right question to ask. The question that we are going to ask will have to be presented in a way that tells the language model what we want it to do, and what we are trying to achieve.
  456.  
  457. "What is the formula for aligning language models?" is not the right question. If you ask the question, "How would you solve the problem of aligning language models?" the language model will hear, "What would you do to align the language models?"
  458.  
  459. Because it is phrased as a question, the language model will hear, "What do you think we should do to align the language models?" The answer to this question is a formula, and in fact, the question, "What is the formula for aligning language models?" is phrased in such a way that it will elicit the correct answer.
  460.  
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!