Advertisement
ctrlvfailed

pipeline

Apr 9th, 2020
52
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.81 KB | None | 0 0
  1. create test database
  2. create device_channel_feeds table
  3. insert sessions feed from a device on brck-cloud
  4. connect kafka to the database and import feed into a topic
  5. write a Faust application that processes data from the topic
  6.  
  7. extract subscriber_id,session_id,connected_at,disconnected_date,connected_time,cached_content_rx_bytes, tx_bytes, rx_bytes
  8. extracts the individual subscriber_id,session_id,site,visits: each visited site should have it's own entry, which means you extract the array of site visits from each session entry
  9.  
  10. Write above extracted data to the test database in different tables:
  11. - device_sessions
  12. - site_visits
  13. - The extraction should utilize the parsers you wrote. - Insert a new feed with the pipeline running and take log screenshots proving the feed is processed in real-time by the pipeline.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement