SHARE
TWEET

Untitled

a guest Jul 17th, 2019 76 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. I attempted to use flatbuffers to send lots of data between Python and Rust programs.  This was my first time using flatbuffers.  
  2.  
  3. ##  Challenges
  4. - Compared to, say, protobuf, the flatbuffers documentation is minimal, and finding examples relevant to my particular use case wasn't easy.
  5. - It's not a streaming format in itself. Like protobuf, flatbuffers _can_ be saved to disk or streamed; however, out-of-the-box they don't have streaming support.
  6.  
  7. ## Send length-prefixed chunks
  8. Here's a summary of the solution I used: Wrap an arbitrary number of core messages in a flatbuffer vector within a flatbuffer table.  Flatten the table, measure the size of the flattened buffer, and stream that length prefix followed by the body of the buffer.
  9.  
  10. ### The Writer / Sender
  11. On the writing / sending end:
  12. - Create a basic message type (this could be the location of a player in a game, or the kinematics of a robot, or 6DOF of a drone)
  13. - Wrap an arbitrary number of these messages in a flatbuffer vector, as a field in a flatbuffer table
  14. - Push messages into the vector (and thus into the table)
  15. - Finish the vector, finalize the table, measure the flattened size of the table
  16. - Write the flattened size to disk/stream as a little-endian encoding uint32
  17. - Write the flattened buffer to disk
  18.  
  19. Here's a python sample:
  20. ```
  21. import flatbuffers
  22. import struct
  23. import Simulator.Event
  24. import Simulator.FrameData
  25.  
  26. def generate_events(rising_locations, falling_locations, event_file):
  27.   fbb = flatbuffers.Builder(0)
  28.  
  29.   total_events = len(rising_locations) + len(falling_locations)
  30.   Simulator.FrameData.FrameDataStartEventsVector(fbb, total_events)
  31.  
  32.   # create a series of flatbuf events and insert into vector
  33.   for item in rising_locations:
  34.     x = item[0][0]
  35.     y = item[0][1]
  36.     event = Simulator.Event.CreateEvent(fbb, x, y, 1)
  37.  
  38.   for item in falling_locations:
  39.     x = item[0][0]
  40.     y = item[0][1]
  41.     event = Simulator.Event.CreateEvent(fbb, x, y, 0)
  42.  
  43.   # done creating the vector of events
  44.   all_events = fbb.EndVector(total_events)
  45.  
  46.   # now start the wrapping table
  47.   Simulator.FrameData.FrameDataStart(fbb)
  48.   # add the event vector
  49.   Simulator.FrameData.FrameDataAddEvents(fbb, all_events)
  50.   framed_data = Simulator.FrameData.FrameDataEnd(fbb)
  51.   fbb.Finish(framed_data)
  52.   # grab the flattened data buffer
  53.   buf = fbb.Output()
  54.   flatbuf_len = len(buf)
  55.   print("total events: %d flatbuf size: %d" % (total_events, flatbuf_len))
  56.   # encode the chunk with a LE-encoded length prefix
  57.   event_file.write(struct.pack('<L', flatbuf_len))
  58.   event_file.write(buf)
  59. ```
  60.  
  61. ### The Reader / Receiver
  62. On the reading / receiving end:
  63. - Read the LE-encoded length prefix
  64. - Read exactly length bytes into a buffer
  65. - Use the `get_root` methods to obtain the table root
  66. - Dig into the vector which contains an arbitrary number of messages
  67.  
  68. Here's a rust sample:
  69.  
  70. ```
  71. use flatbuffers;
  72. use std::io::Read;
  73. use std::io::BufReader;
  74. use byteorder::{LittleEndian, ReadBytesExt};
  75.  
  76. #[allow(dead_code, unused_imports)]
  77. mod simulator_flatbuf_generated;
  78. use crate::simulator_flatbuf_generated::simulator;
  79. use crate::simulator_flatbuf_generated::simulator::FrameData;
  80. use crate::simulator_flatbuf_generated::simulator::Event;
  81. ...
  82.  
  83.   let  f = std::fs::File::open("./out/events.dat").unwrap();
  84.   let mut buf_reader = BufReader::new(f);
  85.  
  86.   let mut chunk_len:u32 = 0;
  87.   while {
  88.     chunk_len = buf_reader.read_u32::<LittleEndian>().unwrap_or(0);
  89.     println!("chunk_len: {}", chunk_len);
  90.     chunk_len > 0 } {
  91.     let mut buf:Vec<u8> = vec![0; chunk_len as usize];
  92.  
  93.     buf_reader.read_exact(&mut buf).expect("couldn't read_exact");
  94.     let rooty:FrameData = flatbuffers::get_root::<simulator::FrameData>(&buf);
  95.     let evts = rooty.events().expect("no events");
  96.     println!("events: {}", evts.len());
  97.     // TODO actually use the events
  98.   }
  99. ```
  100.  
  101. ### The Schema
  102. Here's roughly the schema file (`.fbs` file) I used:
  103.  
  104. ```
  105. namespace Simulator;
  106.  
  107. struct Event {
  108.     x: uint16;
  109.     y: uint16;
  110.     rising:uint8;
  111. }
  112.  
  113. table FrameData {
  114.     events: [Event];
  115. }
  116. ```
  117.  
  118.  
  119.  
  120. ## Notes and open questions
  121.  
  122. - There are some methods in the flatbuffers API related to `Prefix` -- it sounds like these might be intended to automatically send length prefix, but there's zero documentation.
  123. - For any flatbuffers samples/examples, the minimal set of things needed to form a useful example is:
  124.   - The schema file (`.fbs` file) used
  125.   - The writer code (the code writing the flatbuffer)
  126.   - The reader code (the code reading the flatbuffer)
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!
 
Top