Guest User

Untitled

a guest
Apr 12th, 2024
71
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 2.89 KB | None | 0 0
  1. Announcement:
  2. - Intro to diffusion models for text-to-image and Image Editing
  3. - DreamFusion — the power of pretrained diffusion models for 3D synthesis, and some following works
  4. - InstructNeRF2NeRF — prompt-based editing of 3D models
  5. - APAP — dragging manipulations with diffusion models as priors
  6.  
  7. Plan:
  8. - About
  9. - I'll go over a few things, without focusing too much on one single thing, just jumping over cool ideas.
  10. - This is tough, but we have a whiteboard for any explanations, we can detour at any point
  11. - Context
  12. - reminder: ASK PEOPLE WHETHER THEY KNOW THIS STUFF
  13. - Everyone knows about DALL-E / Midjourney / Stable Diffusion
  14. - There's stuff like ComfyUI and other community efforts for making Stable Diffusion smarter
  15. - Like taking in human pose, or normals, or depth
  16. - There have also been attempts at generating multi-view images of the same object with Stable Diffusion, for gaming assets
  17. - Well, why not use the power of diffusion models for actually creating 3D objects?
  18. - Intro to diffusion
  19. - REMINDER TO ASK WHETHER ANYONE KNOWS ABOUT DIFFUSION OR NEURAL NETWORKS
  20. - The base diagram – forward process and back process
  21. - Funny formula slides
  22. - Btw, all this math is bullshit, researchers intentionally complicate this to make their papers look smarter
  23. - Here's the formula (with all the notation, still looks tough)
  24. - Let's simplify step-by-step
  25. - Take an image
  26. - Add noise
  27. - Ask the model to predict the noise
  28. - Loss is MSE(true_noise, predicted_noise)
  29. - Now let's replace with notation + expand the add_noise/reduce_noise, and we have exactly that
  30. - During sampling, in order to do it over multiple steps, we start from pure noise, denoise, but add it not fully, and so on
  31. - For editing
  32. - Basic idea from SDEdit — just add noise and denoise with another prompt
  33. - InstrucPix2Pix — дообучаем принимать исходную картинку для сохранения геометрии
  34. - For 3D generation
  35. - REMINDER TO ASK ABOUT WHETHER ANYONE KNOWS ABOUT 3D
  36. - Once again NeRFs — WE TRAIN THEM ONE TIME one the scene
  37. - Usually we have photos, pass them on to train the NeRF, get the 3D object like that
  38. - Now what if we take this image from this viewpoint, and instruct "make it more like prompt T", and use that for training
  39. - That is the idea behind DreamFusion (show images)
  40. - Works that do the same but finetuning over Objaverse (MVDream)
  41. - For 3D editing
  42. - Apply the same idea but with a NeRF we already have, and InstructNeRF2NeRF that will preserve the geometry of the object
  43. - This gets us InstructNeRF2NeRF
  44. - For 3D editing with dragging
  45. - For simplicity assume we're working with a mesh
  46. - On it we can define this differential geometry method that minimizes weird angles after moving vertices around, resulting in this nice deformation (ARAP)
  47. - Now let's add a diffusion model on top of that as prior, this becomes APAP
Add Comment
Please, Sign In to add comment