Networking, sockets, and UDP Streaming fun! Part II


Yesterday, in my arrogance I spoke about some old UDP audio streaming logic I found online yesterday. I suggested that I could see where the choppiness problems were and fix them. Today I couldn’t help myself. I dove in and ran the code between two Mac computers to hear how it sounded. I was amazed that the code worked at all without any modification! Hi, I’m Cliff. You’re here because you wanna hear how it sounds to break your voice up into tiny little ones and zeroes on one device then reassemble them on another. I’m here because I’m just fortunate enough to be able to do this task. Picking up from yesterday I can proudly announce that I have indeed solved the mystery of choppy audio! It took about 20-30 minutes of copyin/pasting the original code and tweaking it to sound natural. My goal is to give you the answer to the mysterious choppy problem but…! Before I do so, let’s try to understand what the code is actually doing. There are two components, a server and a client.

The Server
We’ll Look first at the server. It starts off in the main() method and prints information about the audio system’s connected devices to the console.

    Mixer.Info minfo[] = AudioSystem.getMixerInfo() ;
    for( int i = 0 ; i < minfo.length ; i++ )
    {
     System.out.println( minfo[i] ) ;    
    }

It then looks to see if the Microphone audio line is supported before proceeding.

if (AudioSystem.isLineSupported(Port.Info.MICROPHONE))

This code merely detects whether there is any sort of “microphone-like” input attached to the computer.

It then opens a target data line for the microphone and starts it.

      DataLine.Info dataLineInfo = new DataLine.Info( TargetDataLine.class , getAudioFormat() ) ;
      TargetDataLine targetDataLine = (TargetDataLine)AudioSystem.getLine( dataLineInfo  ) ;
      targetDataLine.open( getAudioFormat() );
      targetDataLine.start();

These API calls [above] create a special DataLine.Info object instance which describes the format of the audio we wish to capture from the microphone. We call a getAudioFormat() method where the details of the audio format are encoded and returned in a special AudioFormat object instance. Without going into too much detail, understand that audio can come in many shapes and sizes depending on if you want stereo, mono, high fidelity or whatever. The AudioFormat class models things such as sample rate sample size, number of channels, etc. The format is given to the DataLine.Info constructor which creates an instance that we pass to the AudioSystem API via the getLine() method. At this point we have a connection to the microphone which we can open() and start() to capture audio samples.

How Audio works
For those of you who are very green to programming and/or audio I’ll explain briefly how audio, or sound in general works in computer systems. This all may seem pretty complicated but audio is one of the simplest concepts you can learn and is actually pretty cool when you understand it’s basic building blocks. Sound is merely the vibration of matter which our ears detect using little tiny hairs. The speed of the vibration makes the different noises that we hear. The vibrations come in various wave forms with different frequencies. Computers record and create sound using a process called digitalization. This is where frequencies, which are analog in nature, are converted to and from digits. It captures digital audio by using a microphone which is a piece of hardware that measures vibrations, or wave forms in the air. It takes a series of samples of the intensity of the wave at specific intervals and it creates or synthesizes audio by sending recorded samples to a speaker which includes a paper cone that vibrates based on the size of the digits in the sample. In short, the bigger the digits are in the sample the more intense the paper cone vibrates. You can think of the digits 16 as causing a small vibration where the digits 128 would cause a much more intense vibration of the paper cone inside the speaker. If a bunch of samples are sent quickly to cone the vibrations happen quickly, if they are sent slowly then the vibrations occur slowly. The combination of the speed and intensity of the vibrations of the paper creates noise or sound that we hear. I’m over-simplifying and glossing over a lot but that is the basics of sound. The samples and sample speed are the key to sound!

Going back to our example, the guts of our server are below:

      byte tempBuffer[] = new byte[1000] ;
      int cnt = 0 ;
      while( true )
      {
      targetDataLine.read( tempBuffer , 0 , tempBuffer.length );
      sendThruUDP( tempBuffer ) ;
      }

Here we see a byte buffer is created (with a strange not-round length of 1000 instead of 1024) and we enter a loop where we continually pass the buffer between two methods, targetDataLine.read() and sendThruUDP(). The first method reads a block of samples from the microphone, which (as described above) measures the vibrations in the air and writes these samples to the buffer. The second method sends the buffer of samples over UDP to the client.

The Client
We’ll now turn our attention over to the client. The client is a RadioReceiver which extend Thread. As it turns out, this is unnecessary complexity as there is no work being done other than playing back the captured audio samples. We can safely ignore the Thread part and pay attention to the code inside of the run method, which is invoked indirectly by the call to r.start() in the main method below.

    public static void main(String[] args) {
  
    RadioReceiver r = new RadioReceiver() ;
    r.start() ;
  
    }

The run method below declares a byte array which is used in a while loop. Inside the while loop we load the byte array variable with the result from the receiveThruUDP() method. This method attempts to capture sample data sent over the network and return it to the caller. In short the byte array is loaded with the samples captured from the network which were originally captured and sent from the server. We then pass the array of samples to a method called toSpeaker. This method eventually hands the samples to a Java API called SourceDataLine.write(). This Java API will eventually send the samples to the speaker which causes the paper cone to vibrate and recreate the sound on the client. See the run snippet below:

    public void run()
    {
        byte b[] = null ;
        while( true )
        {
           b = receiveThruUDP() ; 
           toSpeaker( b ) ;
        }        
    }

That’s the basics of how the whole operation works. There’s not a terribly large amount of code and I haven’t described the UDP networking pieces at all. I’ll continue this series explaining UDP next. in the mean time, keep looking to see if you too can figure out where things are slowing, **ahem**, chopping up. Keep in mind how I explained the key components of audio, sample intensity and the frequency, or speed of the samples.

Networking, sockets, and UDP Streaming fun!


I’ve been tinkering with robotics lately and itching to put a camera on my Bluebot so I can watch it from my wristwatch as I drive it around. That got me looking for info on how to stream video from one device to another. Hi, I’m Cliff. You’re here because you probably want to have a device spew data out into the sky then receive it from another device using a UDP network connection. I had done UDP video streaming about 4-5 years ago as a prototype for my wife’s salon business and that time I got extremely lucky. She wanted a way to stream video from a webcam over the internet and I didn’t know where to start. I went low level and wrote a UDP video client/server in Java. I’ve since lost the code but it wasn’t good code. (It was a dirty hack where I was exceeding the practical size for each packet… and I don’t think the UDP stuff even worked. I think I converted to TCP at the last minute! There was a time limit for my experiment and I came in literally one second before time was up!)

Flash forward to today where I again have the need to stream video over the network. I found an old abandoned post on a Java forum where someone wrote a choppy Audio UDP client server. I don’t know why but I think this example, as crude as it is, looks pretty cool! The guy who posted the code wanted to know why the audio sounds so choppy. I looked at the code and found the problem in about 30 seconds without compiling, or even trying to run and listen to it! The problem is obvious, so I’m going to post the code here for you to read and ponder. I’ll try to work through a solution in my free time.

The server below attempts to record audio from the microphone and stream it via UDP to a client.

The Server:

public class MicPlayer {
 
    private static final String IP_TO_STREAM_TO   = "localhost" ;
    private static final int PORT_TO_STREAM_TO     = 8888 ;
 
    /** Creates a new instance of MicPlayer */
    public MicPlayer() {
 
    }
 
    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
    Mixer.Info minfo[] = AudioSystem.getMixerInfo() ;
    for( int i = 0 ; i < minfo.length ; i++ )
    {
     System.out.println( minfo[i] ) ;    
    }
 
 
    if (AudioSystem.isLineSupported(Port.Info.MICROPHONE)) {
    try {
 
 
      DataLine.Info dataLineInfo = new DataLine.Info( TargetDataLine.class , getAudioFormat() ) ;
      TargetDataLine targetDataLine = (TargetDataLine)AudioSystem.getLine( dataLineInfo  ) ;
      targetDataLine.open( getAudioFormat() );
      targetDataLine.start();
      byte tempBuffer[] = new byte[1000] ;
      int cnt = 0 ;
      while( true )
      {
      targetDataLine.read( tempBuffer , 0 , tempBuffer.length );
      sendThruUDP( tempBuffer ) ;
      }
 
    }
    catch(Exception e )
    {
    System.out.println(" not correct " ) ;
    System.exit(0) ;
    }
    }
 
 
 
    }
 
 
    public static AudioFormat getAudioFormat(){
    float sampleRate = 8000.0F;
    //8000,11025,16000,22050,44100
    int sampleSizeInBits = 16;
    //8,16
    int channels = 1;
    //1,2
    boolean signed = true;
    //true,false
    boolean bigEndian = false;
    //true,false
    return new AudioFormat( sampleRate, sampleSizeInBits, channels, signed, bigEndian );
    }
 
 
    public static void sendThruUDP( byte soundpacket[] )
    {
       try
       {
       DatagramSocket sock = new DatagramSocket() ; 
       sock.send( new DatagramPacket( soundpacket , soundpacket.length , InetAddress.getByName( IP_TO_STREAM_TO ) , PORT_TO_STREAM_TO ) ) ; 
       sock.close() ;
       }
       catch( Exception e )
       {
       e.printStackTrace() ;
       System.out.println(" Unable to send soundpacket using UDP " ) ;   
       }
 
    }
 
 
}

The client below attempts to receive the UDP packets and push them into the speakers for output:

The client:

public class RadioReceiver extends Thread {
 
    private static final String IP_TO_STREAM_TO   = "localhost" ;
    private static final int PORT_TO_STREAM_TO     = 8888 ;
 
    /** Creates a new instance of RadioReceiver */
    public RadioReceiver() {
    }
 
    public void run()
    {
        byte b[] = null ;
        while( true )
        {
           b = receiveThruUDP() ; 
           toSpeaker( b ) ;
        }        
    }
 
    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
 
    RadioReceiver r = new RadioReceiver() ;
    r.start() ;
 
    }
 
 
    public static byte[] receiveThruUDP()
    {
       try
       {
       DatagramSocket sock = new DatagramSocket(PORT_TO_STREAM_TO) ; 
       byte soundpacket[] = new byte[1000] ;
       DatagramPacket datagram = new DatagramPacket( soundpacket , soundpacket.length , InetAddress.getByName( IP_TO_STREAM_TO ) , PORT_TO_STREAM_TO ) ;
       sock.receive( datagram ) ; 
       sock.close() ;
       return datagram.getData() ; // soundpacket ;
       }
       catch( Exception e )
       {
       System.out.println(" Unable to send soundpacket using UDP " ) ;   
       return null ;
       } 
 
    }
 
 
     public static void toSpeaker( byte soundbytes[] )
     {
 
      try{  
      DataLine.Info dataLineInfo = new DataLine.Info( SourceDataLine.class , getAudioFormat() ) ;
      SourceDataLine sourceDataLine = (SourceDataLine)AudioSystem.getLine( dataLineInfo );
      sourceDataLine.open( getAudioFormat() ) ;
      sourceDataLine.start();
      int cnt = 0;
      sourceDataLine.write( soundbytes , 0, soundbytes.length );
      sourceDataLine.drain() ;
      sourceDataLine.close() ;
      }
      catch(Exception e )
      {
      System.out.println("not working in speakers " ) ;
      }
 
    }
 
 
    public static AudioFormat getAudioFormat()
    {
    float sampleRate = 8000.0F;
    //8000,11025,16000,22050,44100
    int sampleSizeInBits = 16;
    //8,16
    int channels = 1;
    //1,2
    boolean signed = true;
    //true,false
    boolean bigEndian = false;
    //true,false
    return new AudioFormat( sampleRate, sampleSizeInBits, channels, signed, bigEndian );
    }
 
 
}

Again, the complaint was that the audio sounds choppy and nobody on the abandoned thread could answer why. I wish there wasn’t a registration for posting because I so want to answer the obvious but I don’t want to fill out my email and register a user name and etc. Can you see where the problem is? I may be getting ahead of myself so I should actually see if the code compiles and works at all!