API tools faq
paste
dfarrell07

odl_cd_planning_with_ed

dfarrell07
Oct 24th, 2016
197
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 10.41 KB | None | 0 0
raw download clone embed print report
  1.  
  2. 1:49:07 PM E<edwarnicke> dfarrell07: zxiiro Shall we build a CI/CD pipeline to OpNFV (ie, autorelease -> packaging in repos)
  3. 1:49:19 PM dfarrell07: zxiiro Because I'm in a mood ;)
  4. 1:59:37 PM D<dfarrell07> Daniel Farrell edwarnicke: /me would love help with that TODO
  5. 1:59:47 PM E<edwarnicke> dfarrell07: OK, lets do this shit then
  6. 1:59:56 PM D<dfarrell07> Daniel Farrell edwarnicke: so, some thoughts..
  7. 2:00:03 PM E<edwarnicke> dfarrell07: Go on...
  8. 2:00:08 PM D<dfarrell07> Daniel Farrell edwarnicke: and also, do you have tooling in mind?
  9. 2:00:44 PM edwarnicke: easy first step is just to set up logic to trigger current rpm repacakge job from autorelease builds, parsing required version info
  10. 2:00:45 PM E<edwarnicke> dfarrell07: I do, two options in fact
  11. 2:00:59 PM dfarrell07: That is an easy first step
  12. 2:01:04 PM D<dfarrell07> Daniel Farrell edwarnicke: plans for longer-term steps involve DRLN, to build per-commit packages like OpenStack https://www.rdoproject.org/documentation/rdo-packaging/#dlrn
  13. 2:01:19 PM E<edwarnicke> dfarrell07: Basically, we build out a job that builds the rpms and is triggered by autorelease and attaches the result to the Jenkins Job
  14. 2:01:35 PM dfarrell07: Next step then is publishing to repos (for which we have a couple of available options)
  15. 2:01:51 PM D<dfarrell07> Daniel Farrell edwarnicke: there are CentOS packaging people having planning talks about this a at OpenStack summit this week, but I'm not there :(
  16. 2:02:15 PM E<edwarnicke> dfarrell07: At first blush dlrn looks overly complicated for our case... but I may be missing something
  17. 2:02:26 PM D<dfarrell07> Daniel Farrell edwarnicke: that may be true
  18. 2:02:48 PM edwarnicke: at least for now, since it's not integrated into CentOS Community Build System yet
  19. 2:03:00 PM edwarnicke: and we can push packages to multiple places
  20. 2:03:08 PM edwarnicke: where did you have in mind for hosting?
  21. 2:03:11 PM repos
  22. 2:03:26 PM E<edwarnicke> dfarrell07: Yeah... so I see two options, one fd.io is using now, one we are considering for the future
  23. 2:03:39 PM dfarrell07: What we are doing now is pushing to nexus yum/apt repose
  24. 2:03:53 PM repos
  25. 2:04:03 PM dfarrell07: What we are considering for the future is packagecloud.io
  26. 2:04:13 PM D— dfarrell07 goes to look
  27. 2:04:32 PM D<dfarrell07> Daniel Farrell edwarnicke: also ;) https://trello.com/c/qE5GZzAZ/345-investigate-package-cloud-io
  28. 2:04:42 PM E<edwarnicke> dfarrell07: LOL
  29. 2:05:57 PM D<dfarrell07> Daniel Farrell edwarnicke: so why that service? Pitch me on it
  30. 2:06:05 PM E<edwarnicke> dfarrell07: Sure
  31. 2:06:15 PM dfarrell07: Basically, nexus yum/apt repos work... barely.
  32. 2:06:20 PM D<dfarrell07> Daniel Farrell lol
  33. 2:06:26 PM E<edwarnicke> dfarrell07: They don't handle packaging and metadata correctly
  34. 2:06:43 PM dfarrell07: They force filenames based on maven semantics, etc
  35. 2:06:53 PM packagecloud.io works like an actual yum/apt repo
  36. 2:07:02 PM (in theory, still need to prove out with it)
  37. 2:08:04 PM D<dfarrell07> Daniel Farrell edwarnicke: okay, ACK at those being problems, especially naming strangeness, and could be nice to have easy cross distro hosting
  38. 2:08:40 PM E<edwarnicke> dfarrell07: But on the flipside... with nexus you can do staging repos (again: haven't tried yet... but in theory very useful)
  39. 2:08:41 PM D<dfarrell07> Daniel Farrell edwarnicke: alternative I see is to use non-commercial hosting provided by distros
  40. 2:08:49 PM E<edwarnicke> dfarrell07: Pointers?
  41. 2:10:24 PM D<dfarrell07> Daniel Farrell edwarnicke: cbs.centos.org/koji/ CentOS, CBS
  42. 2:10:45 PM edwarnicke: https://copr.fedorainfracloud.org/coprs/ Fedora Copr
  43. 2:10:45 PM → pradeeban joined ([email protected])
  44. 2:11:20 PM E<edwarnicke> dfarrell07: Can we push merge by merge to those? and keep those repo segregated from the release packages?
  45. 2:12:25 PM D<dfarrell07> Daniel Farrell edwarnicke: with CBS we have -testing, -candidate-, -staging, -release repo segmentation by tag (it's koji on backend), but need DRLN integration to get push-by-push I think
  46. 2:12:46 PM E<edwarnicke> dfarrell07: Ah... so back to the complexity then
  47. 2:12:53 PM D<dfarrell07> Daniel Farrell edwarnicke: right
  48. 2:13:06 PM edwarnicke: assuming alternatives are simpler ;)
  49. 2:13:19 PM E— edwarnicke would like the record to reflect that not all complexity is bad, and he's not educated enough to know whether its good or bad here
  50. 2:13:31 PM D<dfarrell07> Daniel Farrell lol, noted and mp
  51. 2:13:33 PM np*
  52. 2:13:50 PM E<edwarnicke> dfarrell07: My current bias is to just get something *working* ASAP
  53. 2:13:55 PM D<dfarrell07> Daniel Farrell edwarnicke: +1
  54. 2:14:00 PM E<edwarnicke> dfarrell07: So lets start with getting a job that triggers from autorelease and builds packaging
  55. 2:14:07 PM dfarrell07: Sound good?
  56. 2:14:22 PM ⇐ jamoluhrsen quit ([email protected]) Ping timeout: 256 seconds
  57. 2:14:23 PM E<edwarnicke> dfarrell07: We can then attach the resulting packages to the Jenkins Job as artifacts
  58. 2:14:25 PM D<dfarrell07> Daniel Farrell edwarnicke: yes, and now the main question - who will do it?
  59. 2:14:34 PM E<edwarnicke> And we then have *something* that doesn't involve your manual intervention
  60. 2:14:44 PM dfarrell07: That is always the 64k question
  61. 2:14:53 PM D<dfarrell07> Daniel Farrell I know it needs to be done, but have other prios that are more pressing
  62. 2:15:13 PM edwarnicke: maybe you have some fresh blood you can recruit?
  63. 2:15:17 PM E<edwarnicke> dfarrell07: So my number one issue right now is that I don't understand your rpm build system
  64. 2:15:31 PM dfarrell07: It shows indications of useful complexisty
  65. 2:15:39 PM dfarrell07: But I get lost in there ;)
  66. 2:15:43 PM D<dfarrell07> Daniel Farrell edwarnicke: the relevant part is this job https://jenkins.opendaylight.org/releng/job/packaging-build-rpm-master/
  67. 2:16:05 PM edwarnicke: just need to setup JJB magic to trigger it, passing the appropriate params
  68. 2:16:21 PM E<edwarnicke> dfarrell07: That I could probably do, where's the existing JJB for that job?
  69. 2:16:47 PM D<dfarrell07> Daniel Farrell edwarnicke: https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=tree;f=jjb/packaging;h=9bcd6677f4c1a976f9d785f90bbca7437736233c;hb=refs/heads/master
  70. 2:17:21 PM → pradeeban1 joined ⇐ pradeeban quit
  71. 2:18:01 PM E<edwarnicke> dfarrell07: Looks largely straight forward... how are you constructing versions and releases?
  72. 2:18:53 PM D<dfarrell07> Daniel Farrell edwarnicke: https://wiki.opendaylight.org/view/Integration/Packaging/Versioning#RPMs
  73. 2:19:16 PM E<edwarnicke> dfarrell07: Let me ask the question a little better
  74. 2:19:37 PM dfarrell07: Given an autorelease build... how to we construct a unique and monatomic version/release ?
  75. 2:19:42 PM dfarrell07: version strickes me as simple
  76. 2:19:54 PM dfarrell07: But release strikes me as containing some decisions
  77. 2:20:11 PM D<dfarrell07> Daniel Farrell edwarnicke: yeah, good points
  78. 2:20:21 PM edwarnicke: The RPM version part of the release would be hardest I guess
  79. 2:20:27 PM E<edwarnicke> dfarrell07: I can tell we how we do it in fd.io
  80. 2:20:40 PM D<dfarrell07> Daniel Farrell edwarnicke: we need the prerelease info part of the release field for snapshot/autorelease version numbers
  81. 2:21:00 PM E<edwarnicke> https://git.fd.io/cgit/vpp/tree/build-root/scripts/version#n16
  82. 2:21:21 PM D— dfarrell07 is okay with just always using 0.1.<prerelease> for these jobs, since we need them for CD only for now
  83. 2:21:41 PM ⇐ pradeeban1 quit ([email protected]) Ping timeout: 250 seconds
  84. 2:21:43 PM E<edwarnicke> dfarrell07: Basically, we use git describe + tags to build it out
  85. 2:22:11 PM D<dfarrell07> Daniel Farrell edwarnicke: yes, I've looked at that I think
  86. 2:22:34 PM E<edwarnicke> dfarrell07: So it sounds like you are suggestion we use the autorelease versions (Carbo-RC0, Carbon-RC1, etc) and then simply use a build number for release?
  87. 2:22:48 PM D<dfarrell07> Daniel Farrell edwarnicke: we currently use <datestamp from maven><snap|rel><unique build ID from maven>
  88. 2:22:51 PM edwarnicke: not git stuff
  89. 2:23:10 PM E<edwarnicke> dfarrell07: In the ODL case, I don't think the git stuff makes as much sense
  90. 2:23:15 PM D<dfarrell07> Daniel Farrell edwarnicke: I think using maven make sense, since we're just repackaging it
  91. 2:23:20 PM E<edwarnicke> dfarrell07: Exactly
  92. 2:23:24 PM → pradeeban joined ([email protected])
  93. 2:23:26 PM D<dfarrell07> Daniel Farrell cool
  94. 2:23:58 PM E<edwarnicke> dfarrell07: So your release is hardcoded as a parameter in the job?
  95. 2:24:07 PM dfarrell07: Why not just derive it?
  96. 2:25:36 PM D<dfarrell07> Daniel Farrell edwarnicke: we could add logic to the build scripts to figure out release and version info from the URL to the artifact, making some assumptions
  97. 2:25:51 PM E<edwarnicke> dfarrell07: Which assumptions?
  98. 2:26:21 PM D<dfarrell07> Daniel Farrell edwarnicke: the formatting of the maven build result name
  99. 2:26:30 PM edwarnicke: it has changed a few times
  100. 2:26:34 PM edwarnicke: but I guess is pretty stable now
  101. 2:26:42 PM E<edwarnicke> dfarrell07: Say more (just so I have context)
  102. 2:27:32 PM D<dfarrell07> Daniel Farrell edwarnicke: zxiiro has done refactoring a few times in ODL's lifecycle that changed the location/name of autorelease and snapshot build results
  103. 2:28:02 PM edwarnicke: it shouldn't be a problem
  104. 2:28:25 PM E<edwarnicke> dfarrell07: So it seems to be that we could derive everything from the URL, and simply have autorelease pass in the URL...
  105. 2:28:30 PM D<dfarrell07> Daniel Farrell edwarnicke: so yes, need patches to https://git.opendaylight.org/gerrit/gitweb?p=integration/packaging.git;a=tree;f=rpm;h=45735474b3936b97c9ca84197b180cdcfc2eea4d;hb=refs/heads/master
  106. 2:28:42 PM E<edwarnicke> dfarrell07: Is autorelease publishing to staging repos?
  107. 2:29:20 PM D<dfarrell07> Daniel Farrell edwarnicke: yes https://nexus.opendaylight.org/content/repositories/staging/org/opendaylight/integration/distribution-karaf/
  108. 2:29:34 PM edwarnicke: and yes to first bit about passing URL
  109. 2:30:17 PM → jamoluhrsen joined ⇐ pradeeban quit
  110. 2:30:22 PM E<edwarnicke> dfarrell07: So I think we have our real first step
  111. 2:30:43 PM D<dfarrell07> Daniel Farrell edwarnicke: I'll raise a Trello to track it, but can't work on it right away
  112. 2:42:45 PM edwarnicke: https://trello.com/c/s59ygFhk/365-extract-version-logic-from-url-in-rpm-build-scripts
  113. 2:43:59 PM edwarnicke: I'll ask one of my interns about it
  114. 2:44:18 PM edwarnicke: he was looking for low-hanging TODOs, and mentioned wanting Python, so this seems perfect
  115. New messages since you tabbed out
  116. 2:50:43 PM E<edwarnicke> dfarrell07: Excellent :)
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!