Upon some inspection, I have a few observations/questions;
Firstly, the not so important bit, on MumbleSource.cs line 166 there's a while loop to get rid of some samples if the buffer's too long. I believe this might be the source of the stutter we're hearing.
Second and the important bit is, this same method I mentioned above, Prepare is called by AudioCore.Process which in turn is called by Microphone.AudioAvailable and BaseAudioSink.Read. However the second call, BaseAudioSink.Read is called within MumbleSink.OnSamplesProcessed which is the handler for AudioSystem.Core.SamplesProcessedEvent, which is invoked within AudioCode.Process. Therefore, the only thing that's triggering AudioCore.Process in the first place is Microphone.AudioAvailable. This means that we are consuming the reception buffers, only when we have input into our own microphone which is, obviously wrong.
Am I missing something here?