a guest Jul 17th, 2019 61 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. # Why messages?
  2. At the very lowest level of abstraction, streams are a natural representation for data traveling on a wire. They represent the continuous flow of bits (or symbols) streaming over a communications channel.
  4. However, as soon as data enters a computer, a CPU must handle it, which involves a call to a handler. So, in a computer, a call that handles a block of data is the lowest level primitive. Streams, as currently defined in libp2p, are a higher level abstraction consisting of some state and an interface of calls. These streams have a number of complex properties that depend on underlying transports, and may or may not be desirable:
  5. * Statefulness
  6. * In-order delivery (of bytes)
  7. * Reliability
  8. * Flow control (optionally)
  9. * Congestion control (optionally)
  11. A stream inherently has the first three properties, and often the rest. However, in practice it is often useful to mix and match these and not have all of them at once. They can instead be provided modularly, through a message abstraction.
  13. In this definition, a *message* is an abstraction that lives purely inside a libp2p node. It may or may not correspond to something similar on the wire, like an IP packet. It is better to think of it as a function call that passes network data around.
  15. # Why this proposal?
  16. The goal of this proposal is to define a single interface that unifies messages and streams. The basic building block is a *message* object, which is a block of bytes with some metadata attached.
  18. The rationale for unifying these concepts is that there are many other potentially useful behaviors that are in between unreliable messages and full streams; for example, an ordered message transport, a reliable message transport (ordered or unordered), etc.
  20. # Message structure
  21. A *message*, as conceptually described above, consists simply of an array of bytes. In addition to the bytes themselves, any message also has an associated *context* (distinct from a golang context), which define additional properties of how messages are handled.
  23. Context can be:
  24. 1. Ephemeral context, which applies only to an individual message. An example would be the source and destination multiaddrs on a UDP transport, and more generally includes data that was parsed out of message headers at a lower level abstraction. This context would typically be garbage collected by a language runtime unless a message handler explicitly stores it somewhere.
  25. 2. Stateful context, which is provided by a pointer to a stateful object that implements a context interface. For example, a TCP "message" would point to a TCP connection object that implements the context interface.
  27. Contexts are allowed to inherit from each other, such that the message can be said to have one single "context" that contains the ephemeral part itself and inherits from the stateful part.
  29. In either case, the context would consist at least partially of a set of key/value pairs. Here are some useful keys that may be part of a context:
  30. * Connection that the data is part of (e.g. for TCP)
  31. * Source multiaddr
  32. * Destination multiaddr
  33. * MTU (maximum message size)
  34. * In order flag
  35. * Reliability
  36. * Flow control flag
  37. * Congestion control flag
  38. * Encryption/authentication flag
  39. * Transport allowed to merge/split messages flag
  41. ## Raw UDP transport
  42. For example, a minimal, native UDP transport would output messages with an ephemeral context containing just the source and destination multiaddrs, which inherits from a singleton stateful context belonging to the transport itself that describes the behavior of UDP:
  43. * In order flag clear
  44. * Reliability flag clear
  45. * Flow control flag clear
  46. * Congestion control flag clear
  47. * Encryption/authentication flag clear
  48. * Merge/split flag clear
  50. The same native UDP transport would accept messages to be sent out after verifying that they have the correct context (none of these flags set, plus a destination multiaddr).
  52. ## Raw TCP transport
  53. A minimal, native TCP transport would output messages with no ephemeral context, just a stateful context that defines the particular connection:
  54. * Connection object pointer
  55. * Source multiaddr
  56. * Destination multiaddr
  58. This would then inherit from a singleton stateful context that defines the behavior of TCP:
  59. * Unlimited MTU
  60. * In order flag set
  61. * Reliability flag set
  62. * Flow control flag set
  63. * Congestion control flag set
  64. * Encryption/authentication flag clear
  65. * Merge/split flag set
  67. In addition, the TCP transport would emit events (not messages) to indicate when connections are opened or closed. The transport would also have methods to create outgoing connections. To send data, send a message that co to the TCP transport that points to an existing connection.
  69. # The router
  70. This concept works well with the idea of a reentrant router. Each time a transport emits a message, the router determines where to forward it next. The receiving transport then examines the context to handle it appropriately.
  72. # The "dialer"
  73. In current libp2p, the dialer determines how to open a connection to a peer ID. The logic is rather ad-hoc.
  75. In this model, the "dialer" (in quotes because that is probably not the right name anymore) would handle both messages and connections. In the message case, it would directly use the message's context to determine what transport to send it to, where most transports would wrap the message and forward it to another transport. In the connection case, establishing a connection would require a providing context that defines what properties are needed (like flow control or encryption), which would determine whether the connection goes directly to a physical transport like TCP or through a transport like secio that wraps the connection and passes it back to the "dialer".
  77. The process of determining which transport gets a message next would be mediated by a multistream/multigram module. I would hope that it could typically go through a fast path that makes assumptions about what the receiver can handle, and then backtracks to a slow path if the receiver cannot handle what we choose.
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand