Advertisement
Guest User

Untitled

a guest
Aug 23rd, 2019
94
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 2.89 KB | None | 0 0
  1. This program reads an export reference graph (i.e. a graph representing the
  2. runtime dependencies of a set of derivations) created by Nix and groups them
  3. in a way that is likely to match the grouping for other derivation sets with
  4. overlapping dependencies.
  5.  
  6. This is used to determine which derivations to include in which layers of a
  7. container image.
  8.  
  9. # Inputs
  10.  
  11. * a graph of Nix runtime dependencies, generated via exportReferenceGraph
  12. * a file containing absolute popularity values of packages in the
  13. Nix package set (in the form of a direct reference count)
  14. * a maximum number of layers to allocate for the image (the "layer budget")
  15.  
  16. # Algorithm
  17.  
  18. It works by first creating a (directed) dependency tree:
  19.  
  20. ```
  21. img (root node)
  22. ├───> A ─────┐
  23. │ v
  24. ├───> B ───> E
  25. │ ^
  26. ├───> C ─────┘
  27. │ │
  28. │ v
  29. └───> D ───> F
  30. └────> G
  31. ```
  32.  
  33. Each node (i.e. package) is then visited to determine how important
  34. it is to separate this node into its own layer, specifically:
  35.  
  36. 1. Is the node within a certain threshold percentile of absolute
  37. popularity within all of nixpkgs? (e.g. `glibc`, `openssl`)
  38.  
  39. 2. Is the node's runtime closure above a threshold size? (e.g. 100MB)
  40.  
  41. In either case, a bit is flipped for this node representing each
  42. condition and an edge to it is inserted directly from the image
  43. root, if it does not already exist.
  44.  
  45. For the rest of the example we assume 'G' is above the threshold
  46. size and 'E' is popular.
  47.  
  48. This tree is then transformed into a dominator tree:
  49.  
  50. ```
  51. img
  52. ├───> A
  53. ├───> B
  54. ├───> C
  55. ├───> E
  56. ├───> D ───> F
  57. └───> G
  58. ```
  59.  
  60. Specifically this means that the paths to A, B, C, E, G, and D
  61. always pass through the root (i.e. are dominated by it), whilst F
  62. is dominated by D (all paths go through it).
  63.  
  64. The top-level subtrees are considered as the initially selected
  65. layers.
  66.  
  67. If the list of layers fits within the layer budget, it is returned.
  68.  
  69. Otherwise layers are merged together in this order:
  70.  
  71. * layers whose root meets neither condition above
  72. * layers whose root is popular
  73. * layers whose root is big
  74. * layers whose root meets both conditions
  75.  
  76. # Threshold values
  77.  
  78. Threshold values for the partitioning conditions mentioned above
  79. have not yet been determined, but we will make a good first guess
  80. based on gut feeling and proceed to measure their impact on cache
  81. hits/misses.
  82.  
  83. # Example
  84.  
  85. Using the logic described above as well as the example presented in
  86. the introduction, this program would create the following layer
  87. groupings (assuming no additional partitioning):
  88.  
  89. Layer budget: 1
  90. Layers: { A, B, C, D, E, F, G }
  91.  
  92. Layer budget: 2
  93. Layers: { G }, { A, B, C, D, E, F }
  94.  
  95. Layer budget: 3
  96. Layers: { G }, { E }, { A, B, C, D, F }
  97.  
  98. Layer budget: 4
  99. Layers: { G }, { E }, { D, F }, { A, B, C }
  100.  
  101. ...
  102.  
  103. Layer budget: 10
  104. Layers: { E }, { D, F }, { A }, { B }, { C }
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement