API tools faq
paste
Advertisement
S1R_L

[DRAFT] A more holistic formulation of rationality

S1R_L
Aug 25th, 2017
137
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
HTML 34.94 KB | None | 0 0
raw download clone embed print report
  1. <p>This is a work in progress, and I'll be continuously editing it as I receive feedback. The idea expressed herein is still a fledgling, and has not yet been developed to any appreciable level. I made mistakes in the writing of this article, and I leave all my mistakes here (correcting them later on). I came here to get feedback, criticism, counter examples, etc. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  2. <p>&nbsp;</p>
  3. <h1>What Is Rational? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</h1>
  4. <p style="margin: 0px 0px 0.357143em; padding: 0px; font-size: 14px; line-height: 1.42857em; color: #222222; font-family: verdana, arial, helvetica, sans-serif;">Eliezer defines rationality as such: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  5. <blockquote>
  6. <p style="margin: 0px 0px 0.357143em; padding: 0px; font-size: 14px; line-height: 1.42857em; color: #222222; font-family: verdana, arial, helvetica, sans-serif;"><strong>Instrumental rationality:</strong>&nbsp;systematically achieving your values.</p>
  7. <p style="margin: 0px 0px 0.357143em; padding: 0px; font-size: 14px; line-height: 1.42857em; color: #222222; font-family: verdana, arial, helvetica, sans-serif;">....</p>
  8. <p style="margin: 0px 0px 0.357143em; padding: 0px; font-size: 14px; line-height: 1.42857em; color: #222222; font-family: verdana, arial, helvetica, sans-serif;"><span style="font-family: verdana, arial, helvetica, sans-serif; color: #222222;">Instrumental rationality, on the other hand, is about steering reality&mdash; sending the future where you want it to go. It&rsquo;s the art of choosing actions that lead to outcomes ranked higher in your preferences. I sometimes call this &ldquo;winning.&rdquo; &nbsp;</span><a style="vertical-align: super;" href="/r/DragonGod-drafts/lw/pcw/a_more_holistic_formulation_of_rationality/#Ref_1" target="_self">[1]</a></p>
  9. </blockquote>
  10. <p style="margin: 0px 0px 0.357143em; padding: 0px; font-size: 14px; line-height: 1.42857em; color: #222222; font-family: verdana, arial, helvetica, sans-serif;">Extrapolating from the above definition, we can conclude that an act is rational, if it causes you to achieve your goals/win. The issue with this definition is that we cannot evaluate the rationality of an act, until after observing the consequences of that action. We cannot determine if an act is rational without first carrying out the act. This is not a very useful definition, as one may want to use the rationality of an act as a guide.</p>
  11. <p style="margin: 0px 0px 0.357143em; padding: 0px; font-size: 14px; line-height: 1.42857em; color: #222222; font-family: verdana, arial, helvetica, sans-serif;">&nbsp;<br style="margin-left: 0px; margin-right: 0px; margin-bottom: 0px;" />Another definition of rationality is the one used in AI when talking about rational agents: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  12. <blockquote>
  13. <p style="margin: 0px 0px 0.357143em; padding: 0px; font-size: 14px; line-height: 1.42857em; color: #222222; font-family: verdana, arial, helvetica, sans-serif;"><span style="font-family: verdana, arial, helvetica, sans-serif; color: #222222;">For each possible percept sequence, &nbsp;a rational agent should select an action that is expected to maximize its performance measure, given the evidence provided by the percept sequence and whatever built-in &nbsp;knowledge the agent has.&nbsp;</span><a style="vertical-align: super;" href="/r/DragonGod-drafts/lw/pcw/a_more_holistic_formulation_of_rationality/#Ref_2">[2]</a></p>
  14. </blockquote>
  15. <p style="margin: 0.357143em 0px 0px; padding: 0px; font-size: 14px; line-height: 1.42857em; color: #222222; font-family: verdana, arial, helvetica, sans-serif;"><span style="font-family: verdana, arial, helvetica, sans-serif; color: #222222;">A precept sequence is basically the sequence of all perceptions the agent as had from inception to the moment of action. The above definition is useful, but I don't think it is without issue; what is rational for two different agents A and B, with the exact same goals, in the exact same circumstances differs. Suppose A intends to cross a road, and A checks both sides of the road, ensures it's clear and then attempts to cross. However, a meteorite strikes at that exact moment, and A is killed. A is not irrational for attempting to cross the road, giving that t hey did not know of the meteorite (and thus could not have accounted for it). Suppose B has more knowledge than A, and thus knows that there is substantial delay between meteor strikes in the vicinity, and then crosses after A and safely crosses. We cannot reasonably say B is more rational than A.</span><br style="margin-left: 0px; margin-right: 0px; margin-top: 0px;" /><span style="font-family: verdana, arial, helvetica, sans-serif; color: #222222;">&nbsp;</span><br style="margin-left: 0px; margin-right: 0px;" /></p>
  16. <p><span style="font-family: verdana, arial, helvetica, sans-serif; color: #222222;"><span style="font-size: 14px;">The above scenario doesn't break our intuitions of what is rational, but what about in other scenarios? What about the gambler who knows not of the gambler's fallacy, and believes that because the die hasn't rolled an odd number for the past n turns, that it would definitely roll odd this time (after all, the probability of not rolling odd&nbsp;<img title="n \text{ times is } 2^{-n}" src="http://www.codecogs.com/png.latex?n \text{ times is } 2^{-n}" alt="" align="bottom" />).</span></span><span style="font-family: verdana, arial, helvetica, sans-serif; color: #222222;"><span style="font-size: 14px;">&nbsp;Are they then rational for betting the majority of their fund on the die rolling odd? Letting what's rational depend on the knowledge of the agent involved, leads to a very broad (and possibly useless) notion of rationality. It may lead to what I call "folk rationality" (doing what you think would lead to success). Barring a few exceptions (extremes of emotion, compromised mental states, etc), most humans are folk rational. However, this folk rationality isn't what I refer to when I say "rational".</span></span></p>
  17. <p><span style="font-family: verdana, arial, helvetica, sans-serif; color: #222222;"><span style="font-size: 14px;">&nbsp;</span></span><br style="margin-left: 0px; margin-right: 0px; margin-bottom: 0px;" /><span style="font-family: verdana, arial, helvetica, sans-serif; color: #222222;"><span style="font-size: 14px;">How then do we define what is rational to avoid the two issues I highlighted above? <sup><a href="/r/DragonGod-drafts/lw/pcw/a_more_holistic_formulation_of_rationality/#Ref_3">[3]</a></sup></span></span></p>
  18. <p>&nbsp;</p>
  19. <hr />
  20. <h1>Introduction. &nbsp; &nbsp;&nbsp;</h1>
  21. <p>I think I have an idea for how to define rationality. First of all, I would start by redefining an agent. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  22. <blockquote>
  23. <p>An agent is an entity which exists in an environment, and possesses sensors through which it perceives its environment (receives precepts from it) and actuators through which it can effect changes in its environment. Furthermore, an agent is defined in terms of its percept sequence, it's inbuilt knowledge, it's current percepts, it's goals and/or preference, and it's decision algorithm(s). For any decision problem, an agent seeks to bring about outcomes in its environment that rank higher in its preference. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  24. </blockquote>
  25. <p>&nbsp;</p>
  26. <p>I would also explain what I mean by "decision problem". &nbsp; &nbsp;</p>
  27. <blockquote>
  28. <p>A decision problem is a choice between at least two options that lead to at least two different outcomes. A decision problem is defined in terms of its options, the information the agent attempting that problem possesses about the problem, and its configuration (exact environmental representation of the problem). &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  29. </blockquote>
  30. <p>&nbsp;</p>
  31. <p>An instance of a problem is a problem that is exactly the same as another problem, except in the configuration (e.g in betting on a coin toss, the scenario where the coin lands heads, and the coin lands tails are the only two instances of the coin toss problem (ignoring other unreasonable considerations)). &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  32. <p>&nbsp;</p>
  33. <p>Next, I would explain what is meant by preference: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  34. <blockquote>
  35. <p>In economics and other social sciences, preference is the ordering of alternatives based on their relative utility. <a href="/r/DragonGod-drafts/lw/pcw/a_more_holistic_formulation_of_rationality/#Ref_4"><sup>[4]</sup></a><span style="white-space: pre;"> </span></p>
  36. </blockquote>
  37. <p>&nbsp;</p>
  38. <p>The preference of an agent can be thought of as a function mapping an outcome O<sub>i</sub> to an ordinal number (1st, 2nd, 3rd, etc) X<sub>i</sub>. X<sub>i</sub> &lt; X<sub>j</sub> implies that O<sub>i</sub> grants the agent more relative utility than O<sub>j</sub>, or alternatively that the agent would prefer O<sub>i</sub> to O<sub>j</sub>. If two outcomes grant the agent the same relative utility, then we say they occupy the same level in the preference ordering.&nbsp;</p>
  39. <p>&nbsp;</p>
  40. <p>My definitions may be wrong/not how the word is used, but I ask that you please bear with me. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  41. <p>&nbsp;</p>
  42. <p>For any decision problem P_i, consider any agent A. A has a certain precept sequence, current precept, preference, actuators, sensors, inbuilt knowledge, and decision algorithm(s). Consider the set of all agents which are the same as A in all aspects save the decision algorithm(s). This set is the Agent space of A, and is denoted by A*. A<sub>i</sub> is a member of A*. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  43. <p>&nbsp;</p>
  44. <p>Yudkowsky's concept of rationality as systematised winning is a concept I agree with, and the spirit is something I want to retain in my new definition. So I will state a principle. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  45. <p>The fundamental principle of rationality: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  46. <blockquote>
  47. <p><a href="https://www.google.com.ng/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=5&amp;cad=rja&amp;uact=8&amp;ved=0ahUKEwj3t-_0wfPVAhVFbBoKHXiDCA4QFghEMAQ&amp;url=http%3A%2F%2Flesswrong.com%2Flw%2Fnc%2Fnewcombs_problem_and_regret_of_rationality%2F&amp;usg=AFQjCNEyJYo199q8lmSeV5FAowQcIRnQfw">Rational agents win</a>. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  48. </blockquote>
  49. <p>&nbsp;</p>
  50. <p>Or to be more specific: &nbsp; &nbsp; &nbsp; &nbsp;</p>
  51. <blockquote>
  52. <p>For any well defined decision problem P<sub>i</sub>, and a given rational agent A<sub>r</sub> (a member of A*). There does not exist another agent A<sub>j</sub> (A member of A*) who consistently outperforms A<sub>r</sub> on P<sub>i</sub>. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  53. </blockquote>
  54. <p>&nbsp;</p>
  55. <p>Rational agents always win.&nbsp;</p>
  56. <p>&nbsp;</p>
  57. <p>At this junction, I think it is prudent for me to explain what I mean by "consistently" and "outperform". &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  58. <p>&nbsp;</p>
  59. <p>Recall that I mentioned "instances of a problem": &nbsp; &nbsp;&nbsp;</p>
  60. <blockquote>
  61. <p>An agent A<sub>i</sub> consistently outperforms another agent A<sub>j</sub> on problem P<sub>i</sub>, if across all possible instances of P<sub>i</sub>, A<sub>i</sub> outperforms A<sub>j</sub> more times than A<sub>j</sub> outperforms A<sub>i</sub>. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  62. </blockquote>
  63. <p>&nbsp;</p>
  64. <p>Now for "outperform": &nbsp; &nbsp; &nbsp;</p>
  65. <blockquote>
  66. <p>An agent A<sub>i</sub> outperforms another agent A<sub>j</sub> on a particular instance of a particular problem P<sub>i</sub>, if A<sub>i</sub>'s choice for that instance of P<sub>i</sub> leads to an outcome higher in their (mutual) preference than A<sub>j</sub>'s choice for that Pi. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  67. </blockquote>
  68. <p>&nbsp;</p>
  69. <p>Using the fundamental principle of rationality, I can arrive at a definition for a rational action: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  70. <blockquote>
  71. <p>For a particular problem Pi, the rational choice is the choice that Ar makes for Pi. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  72. </blockquote>
  73. <p>&nbsp;</p>
  74. <p>Or: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  75. <blockquote>
  76. <p>A choice is rational if there does not exist another decision algorithm that consistently outperforms the decision algorithm that produced that choice. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  77. </blockquote>
  78. <p>&nbsp;</p>
  79. <p>The conception of rationality I identified above may not be the most aesthetically pleasing, but it has some nice properties: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  80. <p>&nbsp;</p>
  81. <ol>
  82. <li>An action does not have to be executed before we can determine if the action was rational or not (Eliezer's definition). &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</li>
  83. <li>The rational choice for all members of A* on any Pi is the same; this rules out the phenomenon of folk rationality (Russel and Norvig's Definition). &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</li>
  84. <li>I suspect that this definition is something that can be programmed into an AI, making it useful as well. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</li>
  85. <li>It gives me <a href="/lw/h8/tsuyoku_naritai_i_want_to_become_stronger/">something to aspire towards</a>&mdash;the platinum standard of the rational agent.</li>
  86. <li>It captures what I conceive when I think of an action as "rational". &nbsp; &nbsp; &nbsp; &nbsp;</li>
  87. </ol>
  88. <p>&nbsp;</p>
  89. <p>&nbsp;</p>
  90. <p>I would like to make one last distinction, that is when a choice is <em>right</em>. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  91. <blockquote>
  92. <p>For any agent A<sub>i</sub> in A*, and a particular instance of a particular problem P<sub>i</sub>, a choice is right, if the outcome of that choice is not ranked lower than the outcome of any other choices. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  93. </blockquote>
  94. <p>Bear in mind, that it is often unknown which instance of the decision problem the agent is dealing with.</p>
  95. <p>The person who bought the lottery ticket and won, made the <em>right </em>choice, but <strong>not </strong>the <em>rational </em>choice. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  96. <p>&nbsp;</p>
  97. <h1>Issues &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</h1>
  98. <p>1. Outperforming only considers the preference for each instance, what about a case, where Ai outperforms Aj on 4 instances, but on the one instance in which Aj outperforms Ai, the payoff gained by Aj by realising the outcome is greater than the payoffs gained by Ai in realising all the other outcomes combined. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  99. <p>&nbsp;</p>
  100. <p>2. Only considers the different instances, and doesn't consider the probability of any given instance. &nbsp; &nbsp;</p>
  101. <p>&nbsp;</p>
  102. <p>&nbsp; &nbsp; &nbsp; &nbsp;To some extent, information about probabilities counts as provided information, and we can assume both agents have access to such information. However, if what is given isn't probabilities themselves, but information that allows one agent (perhaps due to superior decision algorithms) the relevant probabilities? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  103. <p>&nbsp;</p>
  104. <p>&nbsp; &nbsp; &nbsp; &nbsp; However, when considering consistent out performance, it may be necessary to consider the probabilities of each of the instances occurring. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  105. <p>&nbsp;</p>
  106. <p>&nbsp;</p>
  107. <p>&nbsp;</p>
  108. <p>#Resolution of the Issues. &nbsp; &nbsp; &nbsp;&nbsp;</p>
  109. <p>&nbsp;</p>
  110. <p>##Problem 1 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  111. <p>&nbsp;</p>
  112. <p>In problem 1, the problem arises as we are no longer merely dealing with an agent who possesses an ordinal preference, but a cardinal preference (the ranking of the outcomes in the preference is based on the value of some real number assigned to each outcome). For this problem to arise, the agent must have a utility function, which maps each outcome to a utility. A utility function maps each Oi to some ui. ui &gt; uj implies Xi &lt; Xj.</p>
  113. <p>&nbsp;</p>
  114. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  115. <p>&nbsp;</p>
  116. <p>In such a case, it may be prudent to reconsider our rational agent. The earlier definition of a rational agent provided, merely seeks a way to compare the aggregate performance of all members of A\* across all instances of Pi, and the agent who doesn't perform worse than any other agent in A\* is defined as the rational agent. Due to the absence of a utility function, the best measure we have to compare the aggregate performance between two agents, is the number of times each agent's choice produced an outcome ranked higher in the preference ordering than the outcome produced by the other agents. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  117. <p>&nbsp;</p>
  118. <p>&nbsp;</p>
  119. <p>&nbsp;</p>
  120. <p>Outperforming is transitive. If Ai outperforms Aj, and Aj outperforms Ak, then Ai outperforms Ak. I shall demonstrate this below: &nbsp;</p>
  121. <p>&nbsp;</p>
  122. <p>Assume the problem in question has n instances. For any problem Pi, the set of all instances of Pi is Si. \#Si = n. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  123. <p>&nbsp;</p>
  124. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  125. <p>&nbsp;</p>
  126. <p>First consider Aj and Ak: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  127. <p>&nbsp;</p>
  128. <p>Aj outperforms Ak on e instances, the set of these outcomes is E. \#E = e. &nbsp; &nbsp; &nbsp; &nbsp;</p>
  129. <p>&nbsp;</p>
  130. <p>Aj and Ak have the same performance (they produce the same outcome, or outcomes on the same level in the preference) on f instances, the set of these outcomes is F. \#F = f. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  131. <p>&nbsp;</p>
  132. <p>Ak outperforms Aj on g instances, the set of these outcomes is G. \#G = g. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  133. <p>&nbsp;</p>
  134. <p>It follows that:&nbsp;</p>
  135. <p>&nbsp;</p>
  136. <p>e + f + g = n. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  137. <p>&nbsp;</p>
  138. <p>e &gt; g. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  139. <p>&nbsp;</p>
  140. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  141. <p>&nbsp;</p>
  142. <p>Now consider Ai and Aj: &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  143. <p>&nbsp;</p>
  144. <p>Ai outperforms Aj on b instances, the set of these outcomes is B. \#B = . &nbsp; &nbsp; &nbsp; &nbsp;</p>
  145. <p>&nbsp;</p>
  146. <p>Ai and Aj have the same performance on c instances, the set of these outcomes is C. \#C = C. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  147. <p>&nbsp;</p>
  148. <p>Aj outperforms Ai on d instances, the set of these outcomes is D. \#D = d. &nbsp; &nbsp;</p>
  149. <p>&nbsp;</p>
  150. <p>It follows that:&nbsp;</p>
  151. <p>&nbsp;</p>
  152. <p>b + c + d = n. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  153. <p>&nbsp;</p>
  154. <p>b &gt; d. &nbsp; &nbsp; &nbsp; &nbsp;</p>
  155. <p>&nbsp;</p>
  156. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  157. <p>&nbsp;</p>
  158. <p>Finally, we shall consider Ai and Ak. &nbsp; &nbsp;</p>
  159. <p>&nbsp;</p>
  160. <p>Ai outperforms Ak on p instances, the set of these outcomes is P. \#P = p. &nbsp; &nbsp; &nbsp; &nbsp;</p>
  161. <p>&nbsp;</p>
  162. <p>Ai and Ak have the same performance on q instances, the set of these outcomes is Q. \#Q = q. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  163. <p>&nbsp;</p>
  164. <p>Ak outperforms Ai on r instances, the set of these outcomes is R. \#R = r. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  165. <p>&nbsp;</p>
  166. <p>It follows that: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  167. <p>&nbsp;</p>
  168. <p>p + q + r = n. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  169. <p>&nbsp;</p>
  170. <p>I assert that p &gt; r (transitivity). &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  171. <p>&nbsp;</p>
  172. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  173. <p>&nbsp;</p>
  174. <p>Before I attempt to proof that p &gt; r, I think it is prudent I make a few elaborations. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  175. <p>&nbsp;</p>
  176. <p>&nbsp;</p>
  177. <p>&nbsp;</p>
  178. <p>To prove transitivity I shall employ the means of proof by contradiction. &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  179. <p>&nbsp;</p>
  180. <p>Assume Ai does not outperform Ak. &nbsp; &nbsp; &nbsp;</p>
  181. <p>&nbsp;</p>
  182. <p>This implies p &lt;= r. &nbsp; &nbsp; &nbsp;</p>
  183. <p>&nbsp;</p>
  184. <p>The above condition if it is true, will definitely hold when p is as small as possible and r is as large as possible. If we maximise p and minimise r, and p &gt; r, then there is no scenario in which p &lt;= r. (The proof of this is left as an excercise for the curious reader). For those who do not grok this and do not care to prove it themselves, maximising p and minimising r would occur when q is 0. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  185. <p>&nbsp;</p>
  186. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  187. <p>&nbsp;</p>
  188. <p>The sets P, Q, and R are determined by the sets B, C, D, E, F and G. Particularly: &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  189. <p>&nbsp;</p>
  190. <p>B (intersection) E (subset of) P &nbsp; &nbsp;&nbsp;</p>
  191. <p>&nbsp;</p>
  192. <p>B (intersection) F (subset of) P &nbsp; &nbsp;&nbsp;</p>
  193. <p>&nbsp;</p>
  194. <p>B (intersection) G is distributed over P, R and Q. &nbsp; &nbsp;&nbsp;</p>
  195. <p>&nbsp;</p>
  196. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  197. <p>&nbsp;</p>
  198. <p>C (intersection) E (subset of) P. &nbsp; &nbsp;&nbsp;</p>
  199. <p>&nbsp;</p>
  200. <p>C (intersection) F (subset of) Q. &nbsp; &nbsp;&nbsp;</p>
  201. <p>&nbsp;</p>
  202. <p>C (intersection) G (subset of) R. &nbsp; &nbsp;&nbsp;</p>
  203. <p>&nbsp;</p>
  204. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  205. <p>&nbsp;</p>
  206. <p>D (intersection) E is distributed over P, R and Q. &nbsp; &nbsp;&nbsp;</p>
  207. <p>&nbsp;</p>
  208. <p>D (intersection) F (subset of) R. &nbsp; &nbsp;&nbsp;</p>
  209. <p>&nbsp;</p>
  210. <p>D (intersection) G (subset of) R. &nbsp; &nbsp;&nbsp;</p>
  211. <p>&nbsp;</p>
  212. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  213. <p>&nbsp;</p>
  214. <p>If we aim to maximise Q, then we can rewrite the above as: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  215. <p>&nbsp;</p>
  216. <p>B (intersection) E (subset of) P. &nbsp; &nbsp;&nbsp;</p>
  217. <p>&nbsp;</p>
  218. <p>B (intersection) F (subset of) P. &nbsp; &nbsp;&nbsp;</p>
  219. <p>&nbsp;</p>
  220. <p>B (intersection) G (subset of) R. &nbsp; &nbsp;&nbsp;</p>
  221. <p>&nbsp;</p>
  222. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  223. <p>&nbsp;</p>
  224. <p>C (intersection) E (subset of) P. &nbsp; &nbsp;&nbsp;</p>
  225. <p>&nbsp;</p>
  226. <p>C (intersection) F (subset of) Q. &nbsp; &nbsp;&nbsp;</p>
  227. <p>&nbsp;</p>
  228. <p>C (intersection) G (subset of) R. &nbsp; &nbsp;&nbsp;</p>
  229. <p>&nbsp;</p>
  230. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  231. <p>&nbsp;</p>
  232. <p>D (intersection) E (subset of) R. &nbsp; &nbsp;&nbsp;</p>
  233. <p>&nbsp;</p>
  234. <p>D (intersection) F (subset of) R. &nbsp; &nbsp;&nbsp;</p>
  235. <p>&nbsp;</p>
  236. <p>D (intersection) G (subset of) R. &nbsp; &nbsp;&nbsp;</p>
  237. <p>&nbsp;</p>
  238. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  239. <p>&nbsp;</p>
  240. <p>It can be seen from the above formulation, that G (subset of) R. As I said earlier, when we maximise r and minimise p, q would be 0. Thus, p + r = n. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  241. <p>&nbsp;</p>
  242. <p>p = n - r. &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  243. <p>&nbsp;</p>
  244. <p>R = G (union) D. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  245. <p>&nbsp;</p>
  246. <p>r = g + d - D (intersection) G. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  247. <p>&nbsp;</p>
  248. <p>It follows that r is maximum when D (intersection) G = 0. Since nothing prevents this, it is possible, and maximising r, we get: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  249. <p>&nbsp;</p>
  250. <p>r = g + d. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  251. <p>&nbsp;</p>
  252. <p>Maximising r occurs when we maximise g and d. Since r and p are complementary, maximisng r is equivalent to minimising p. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  253. <p>&nbsp;</p>
  254. <p>By definition, g &lt; e. Therefore, maximum g is floor(n-1/2). &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  255. <p>&nbsp;</p>
  256. <p>By definition d &lt; b. Therefore, maximum f is floor(n-1/2). &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  257. <p>&nbsp;</p>
  258. <p>Maximum r is floor(n-1/2) + floor(n-1/2) = 2 \* floor(n-1/2) = floor(n-1) = n-1. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  259. <p>&nbsp;</p>
  260. <p>Therefore maximum r is n-1, and minimum p is 1. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  261. <p>&nbsp;</p>
  262. <p>&amp;nbsp; &nbsp; &nbsp;&nbsp;</p>
  263. <p>&nbsp;</p>
  264. <p>&nbsp;</p>
  265. <p>&nbsp;</p>
  266. <p>Consider a case where n = 3. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  267. <p>&nbsp;</p>
  268. <p>We have problem instance 1, 2 and 3. &nbsp; &nbsp; &nbsp;&nbsp;</p>
  269. <p>&nbsp;</p>
  270. <p>Each instance has 3 outcomes x, y and z. The preference is x &gt; y &gt; z. The payoff matrix is: &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  271. <p>&nbsp;</p>
  272. <p>&nbsp;1 2 3</p>
  273. <p>&nbsp;</p>
  274. <p>Ai x &nbsp;z y</p>
  275. <p>&nbsp;</p>
  276. <p>Aj y x z &nbsp;</p>
  277. <p>&nbsp;</p>
  278. <p>Ak z y x</p>
  279. <p>&nbsp;</p>
  280. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  281. <p>&nbsp;</p>
  282. <p>The above demonstrates the problem of deciding who is more rational merely by looking at which agent outperforms the other agent on more instances. With the lack of transitivity using that definition, creating a ranking of A\* is going to be much more difficult than it would have been if (more rational than) was transitive. Now I can bite the bullet, accept the flaws of this model of rationality, and concoct a more complex ordering mechanism and use it to rank A\*. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  283. <p>&nbsp;</p>
  284. <p>Yet, even if I did, I would not be resolving all the problems with this model. The model might produce a counterintuitive result in a case in which Pi has v instances and w outcomes (v &gt; w). To illustrate, consider two agents Ai and Aj, and a Pi which has 2 instances 1 and 2, but 3 outcomes x, y and z. The preference is x &gt; y &gt; z. The payoff matrix is: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  285. <p>&nbsp;</p>
  286. <p>&nbsp;1 2</p>
  287. <p>&nbsp;</p>
  288. <p>Ai y &nbsp;z &nbsp; &nbsp;&nbsp;</p>
  289. <p>&nbsp;</p>
  290. <p>Aj z &nbsp;x &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  291. <p>&nbsp;</p>
  292. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  293. <p>&nbsp;</p>
  294. <p>My concept of rationality proclaims that Ai and Aj are equally rational; this runs counter to my intuition. Intuitively, I think that Aj is more rational than Ai. Thus, no matter how I spin it, the concept of (more rational than) as outperforming on more instances, doesn't cut it, and as such a better implementation is needed (hopefully, this one would be transitive as well). &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  295. <p>&nbsp;</p>
  296. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  297. <p>&nbsp;</p>
  298. <p>I shall forego a specific implementation for now, and use this opportunity to define (implementation agnostic) a rational agent: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  299. <p>&nbsp;</p>
  300. <p>&gt; A rational agent Ar (a member of A\*) is an agent whose aggregate performance is not worse than the aggregate performance of any other agent that is a member of A/*. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  301. <p>&nbsp;</p>
  302. <p>&nbsp;</p>
  303. <p>&nbsp;</p>
  304. <p>It seems "consistently outperforming" is a poor implementation of "aggregate performance". There are other implementations of aggregate performance (and I would later describe my preferred implementation), but an issue with the implementation is not an issue with the concept itself. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  305. <p>&nbsp;</p>
  306. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  307. <p>&nbsp;</p>
  308. <p>I would state a few guidelines for any implementation of "aggregate performance". &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  309. <p>&nbsp;</p>
  310. <p>1. It should be transitive: Ai (more rational than) Aj and Aj (more rational than) Ak implies Ai (more rational than) Ak. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  311. <p>&nbsp;</p>
  312. <p>2. It should cover all cases: there shouldn't be a counter example where Ai intuitively has a better performance than Aj, but the particular measure of aggregate performance we use does not reflect this. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  313. <p>&nbsp;</p>
  314. <p>3. It should be as simple as possible: ideally, the simplest possible implementation of "aggregate performance" that satisfies the above two criteria, will be the implementation chosen. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  315. <p>&nbsp;</p>
  316. <p>&nbsp;</p>
  317. <p>&nbsp;</p>
  318. <p>Now, that I have listed the properties of a good implementation of aggregate performance, I will describe an implementation that I think satisifies those properties (except maybe property 3, due to my inability to conceive of a simpler implementation). &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  319. <p>&nbsp;</p>
  320. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  321. <p>&nbsp;</p>
  322. <p>Recall that each outcome Oi is mapped to some Xi. &nbsp; &nbsp; &nbsp; &nbsp;</p>
  323. <p>&nbsp;</p>
  324. <p>Let M\* = sum(Xi), i.e. the sum of all the different ranks of each outcome in the preference. &nbsp; &nbsp; &nbsp;&nbsp;</p>
  325. <p>&nbsp;</p>
  326. <p>Let the set of all outcomes for Ai be Ti. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  327. <p>&nbsp;</p>
  328. <p>For each Ai, Mi = sum(Xi) for all Oi in Ti) i.e. the sum of the ranks of all the outcomes for Ai. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  329. <p>&nbsp;</p>
  330. <p>Vi = M\* - Mi. &nbsp; &nbsp; &nbsp;</p>
  331. <p>&nbsp;</p>
  332. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  333. <p>&nbsp;</p>
  334. <p>The following properties hold: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  335. <p>&nbsp;</p>
  336. <p>1. Vi &gt; Vj implies Ai (more rational than) Aj. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  337. <p>&nbsp;</p>
  338. <p>2. Vi &gt; Vj, and Vj &gt; Vk implies Vi &gt; Vk. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  339. <p>&nbsp;</p>
  340. <p>3. The higher Vi is, the more rational Ai is. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  341. <p>&nbsp;</p>
  342. <p>&nbsp;</p>
  343. <p>&nbsp;</p>
  344. <p>Using Vi, we can rank the members of A\* according to their rationality. Let the set of all Vi be V\*. Each Vi maps to some Wi. Vi &gt; Vj implies Wi &lt; Wj. max(V\*) maps to 1. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  345. <p>&nbsp;</p>
  346. <p>Ar has Wr of 1. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  347. <p>&nbsp;</p>
  348. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  349. <p>&nbsp;</p>
  350. <p>Now, to revisit the case of when the agent has a utility function. The implementation of aggregate performance in this case is pretty straightforward. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  351. <p>&nbsp;</p>
  352. <p>Let Ti be the set of all outcomes in Ai. &nbsp; &nbsp; &nbsp;</p>
  353. <p>&nbsp;</p>
  354. <p>For each Ai, Mi = sum(ui) for all Oi in Ti i.e. the sum of the utilities for all the outcomes for Ai. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  355. <p>&nbsp;</p>
  356. <p>Vi = Mi. &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  357. <p>&nbsp;</p>
  358. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  359. <p>&nbsp;</p>
  360. <p>##Problem 2. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  361. <p>&nbsp;</p>
  362. <p>The definition of a rational agent so provided does not have any provision for accounting for the probabilities of the different problem instances. I shall outline a procedure for dealing with such cases here, preserving out notion of a rational agent. The reader may be interested to know, that the above examples implicitly assume that the probabilities of the different problem instances are equal (i.e a uniform distribution over the problem instances). Giving no information on the probabilities of the different instances, assuming a uniform distribution is no less reasonable than any other apriori distribution. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  363. <p>&nbsp;</p>
  364. <p>&nbsp;</p>
  365. <p>&nbsp;</p>
  366. <p>It seems prudent to define probability here: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  367. <p>&nbsp;</p>
  368. <p>&gt; When dealing with experiments that are random and well-defined in a purely theoretical setting (like tossing a fair coin), probabilities can be numerically described by the number of desired outcomes divided by the total number of all outcomes. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  369. <p>&nbsp;</p>
  370. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  371. <p>&nbsp;</p>
  372. <p>The most popular version of objective probability is frequentist probability, which claims that the probability of a random event denotes the relative frequency of occurrence of an experiment's outcome, when repeating the experiment. This interpretation considers probability to be the relative frequency "in the long run" of outcomes.</p>
  373. <p>&nbsp;</p>
  374. <p>&nbsp;</p>
  375. <p>&nbsp;</p>
  376. <p>The frequentist definition of probability is what I shall use in constructing the concept of rationality for when probabilities are assigned to at least one instance of the problem. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  377. <p>&nbsp;</p>
  378. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  379. <p>&nbsp;</p>
  380. <p>For a given decision problem Pi, let each instance of Pi be represented as some I^j. Pi is the set of all I^j. The probability of I^j (Prj) is the relative frequency of I^j in Pi. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  381. <p>&nbsp;</p>
  382. <p>Prj = freq(I^j)/n. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  383. <p>&nbsp;</p>
  384. <p>freq(I^j) = Prj*n. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  385. <p>&nbsp;</p>
  386. <p>&amp;nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  387. <p>&nbsp;</p>
  388. <p>Using the frequentist notion of probability, all probabililites are rational numbers. This makes things convenient. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</p>
  389. <p>&nbsp;</p>
  390. <p>n = product of denominators. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  391. <p>&nbsp;</p>
  392. <h1>References. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</h1>
  393. <p><a name="Ref_1">[1]:</a> Eliezer Yudkowsky *Rationality: From* ***A****I to* ***Z****ombies*, p 7(45), Machine Intelligence Research Institute Berkeley, CA. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  394. <p><a name="Ref_2">[2]:</a> Russel &amp; Norvig *Artificial Intelligence: A Modern Approach (3rd Edition)*, p 58, Pearson Education NJ. &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  395. <p><a name="Ref_3">[3]:</a>&nbsp;http://lesswrong.com/r/discussion/lw/pco/what_is_rational/</p>
  396. <p><a name="Ref_3">[4]:</a> https://en.wikipedia.org/wiki/Preference_(economics) &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>
  397. <p>[]: https://en.wikipedia.org/wiki/Probability &nbsp;</p>
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!