daily pastebin goal
30%
SHARE
TWEET

Untitled

a guest Feb 21st, 2018 48 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. class PublishWorker < BackgrounDRb::Worker::Base
  2.   require 'net/http'
  3.  
  4.   def do_work(args={})
  5.     loop {
  6.       urls_to_publish = PublishQueue.find_all_by_needs_publishing(true)
  7.       urls_to_publish.each do |pubq|
  8.         Net::HTTP.start('publisher.foo.com') do |http|
  9.           response = http.get(pubq.url)
  10.           fail response.code unless response.code == "200"
  11.           File.open("whatever/path/#{pubq.url.sub('/', '_')}.html", 'w+'){|f| f.write(response.body)}
  12.         end
  13.         pubq.needs_publishing = false
  14.         pubq.save
  15.       end  
  16.       sleep args[:sleep]
  17.     }
  18.   end
  19. end
  20. PublishWorker.register
  21.  
  22.  
  23. then call it like:
  24.  
  25. MiddleMan.new_worker :class => :publish_worker, :args => {:sleep => 3.minutes}
  26.  
  27. This assumes you have a PublishQueue model that has these two attributes:
  28.  
  29. url -> this is /foo/bar and does not include the domain
  30. needs_publishing -> this is a boolean value true or false. When you want
  31. to put something in the publish queue just create or find a PublishQueue
  32. object and set its needs_publishing to true or 1.
  33.  
  34. then the worker will just loop with a sleep you can control the interval of
  35. and each time it wakes up it does PublishQueue.find_all_by_needs_publishing(true)
  36. which finds all pending PublishQueue items. then the worker processes each object,
  37. grabs the url does http and rwrites the file somewhere. Then it sets the needs_publishing
  38. to false.
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top