Sharing output streams through a JNI interface


Maybe it’s me, but there seems to be a shortage of expert information on the net these days. I dunno about you but I constantly find myself running out of Google links and StackOverflow replies when looking for an answer to some rather valid technical issues. Am I the only one out there trying some of these crazy ideas or is the internet playing a savage game on me? Hi, I’m Cliff, and you’re here because you were waiting for me to post something crazy and far reaching. Maybe you were waiting on the, “How to tether your 8830 to your 1st Gen iPad” article. Maybe you were looking for info on how to install a Linux based firmware on your brand new Samsung 3D television. Whatever it is that brought you here, welcome to my little corner of the net where solutions to obscure technical issues are the norm.

I recently went on a journey into JNI land. I had visited many years ago plugging native MS Excel reporting into legacy RPG/400 programs via ILE wrappers around the POI project. Things were so much smoother back then. Flash forward to recent weeks as I wade through my old C manuals and skip through google links for Makefile tutorials. My project was to take a 3rd party native API, with C source code examples into the modern age. We need to use this API in a serious multi-user environment. To put it in simple yet technical terms, I needed to stream binary data from this API over the internet to a mobile device. Without being too specific and revealing let me give a generic example that colors the picture of what I’ve accomplished.

Say you have this blob of binary data. Go ahead, say it. Now say it once more but this time wait for a co-worker to walk by. Say the binary blob is generated from a native API that costs, oh about $2000 a seat. (The cost is not nearly as important as the seat.) Now say the API is not documented with any examples in the programming language of your choice. (Say that last sentence aloud with a girlie giggle between the 6th and 7th words.) Let’s pretend the only example given uses an old fashioned main function C reading cmd line arguments. Let’s pretend you have clients that need this blob on the other side of the internet. What are your options?

You could call the API using the command line example which writes to the file system. You could then read the generated file back across the internet using your favorite Ruby/Python/PHP programming toy. Then you would have to deal with file write collisions, the overhead of disk access, cleaning up the file after its written, file locks if you’re on Windows, and many other headaches.

You could modify the example to write directly to the internet. Then you have to deal with either implementing HTTP or talking to yet another 3rd party API written in the same language. You’d also have to deal with the language which you only use on the rarest of occasions… like never.

You could opt for the Language compatibility API that comes in the language of your liking with the additional purchase of two other vendor products. (API not sold separately.) You could do a number of different things each with their own drawbacks or you could do what I did… design a native interface.

I used JNI to write binary data from C back to a Java object. My first reaction was, “Gosh! How do I write a C++ output stream like thing to a Java output stream?” My second reaction was, “Don’t I need an input stream instead of an outputstream? Which end of which stream plugs in where?” (I always have that reaction when I deal with I/O streams.) the short answer, which applies equally to pure Java, C++, and Java/C++ is that you need threads to connect two streams… ALWAYS. There’s no safe way to deal with threads so thought a little harder. Because the output was delivered incrementally in my case, I opted to use a callback strategy. In other words, the API used callbacks to hand chunks of binary in array form to a calling application. I simply plugged in my JNI C code to accept these arrays and relay them to the original invoking Java object.

You have to be really careful with JNI native array conversion. There’s a proper way to copy bytes from C to Java and then there’s the whole, “is this a copy or a direct reference?” issue. Also you might get confused with array pinning in later VMs, but that really shouldn’t matter if you only need to materialize the data briefly in C.

In all, the general strategy for sharing binary data (A/V files, images, etc.) from C with Java requires byte arrays. You create a Java byte array in C like this:

const char[] rawData = {0,1,2,3,4,5,6,7,8,9}; //Or get some raw data from somewhere
int dataSize = sizeof(rawData);
printf("Building raw data array copy\n");
jbyteArray rawDataCopy = env->NewByteArray(dataSize);
env->SetByteArrayRegion(rawDataCopy, 0, dataSize, rawData);

And pass it to Java like this:

printf("Finding callback method\n");
//Assumes obj is the Java instance that will receive the raw data via callback
jmethodID aMethodId = env->GetMethodID(env->GetObjectClass(obj),"handleData","([B)V"); 
if(0==aMethodId) throw MyRuntimeException("Method not found error");
printf("Invoking the callback\n");
env->CallVoidMethod(obj,aMethodId, &rawDataCopy);

you would have a Java object that looked something like this:

public class MyDataHandler {
  OutputStream dataStream;
  public MyDataHandler(OutputStream writeTo) { dataStream = writeTo;}
  public void handleData(byte[] incomingData) { dataStream.write(incomingData); }
}

That handler would be passed to C via native method like so:

public class NativeIntegration {
  public native void generateBinaryWithHandler(MyDataHandler handler);

  //Here we assume response is something like a network stream
  public void doCallNativeFunction(ResponseStream response) {
    MyDataHandler handler = new MyDataHandler(response);
    generateBinaryWithHandler(handler);
  }
}

That’s a rather niave example without the necessary consideration of error handling/reporting but it gets the point across.

8 Comments

  1. anon

    I’m not too faimilar with JNI, but what about open a local socket (I think this can be called pipes.)

    http://stackoverflow.com/questions/266913/efficient-data-transfer-from-java-to-c-on-windows

    http://stackoverflow.com/questions/2656494/java-c-communication-via-pipe-on-windows

    Or use this thing from Google.

    http://code.google.com/apis/protocolbuffers/docs/tutorials.html

    ….

    I mention this is because on windows you can use this thing called pipes. You can do it in Unix too. Or you can use boost::asio. But this is mostly for c++ to c++ inter process communication. However, i figure for Java you and do the same thing, inter-process communication.

  2. Yeah a socket is another option. Still you’d have to write code that binds to the socket and performs some sort of crude RPC over it. For quick and dirty you’d probably just pass raw data back and forth from the socket and be done but as soon as your needs grow (within the next hour) you’d find your self needing to configure the C code at run-time which means passing options over the socket which is essentially RPC. In the long run, you’d be exchanging a smashed kneecap for an anvil on the pinky toe. No matter how you look at it there’s low level muckery to be done along with cross language method/function calls.

  3. great article.. thanks

  4. igs

    what, you invented JNI now?

    It was really frustrating to read your intro and get all excited about the solution you’re about to post and then it turns out to be plain JNI. Plain copy C memory to Java memory and loose ages over it. I don’t know about anyone else, but “shared” implied avoiding this, too me.

    You managed to copy binary arrays over JNI. Congratulations, you’re soooo crazy.

    So disappointing.

  5. Nmk

    The article helped me to solve my problem.
    Thanks Cliff.

  6. @igs actually, I invented a programming language called “Cliff” sold it to Dennis Ritchie, who shortened the syntax then later the name to C which created the interface we now refers to as JNI. So, yeah!!! I invented JNI just a year before I invented to internet (and later gave the credit to Al Gore). Honestly, I don’t understand why programmers get so tight in the shorts over the wording used in an article. Actually I do understand, being a programmer and losing my patience on a number of occasions. But loosen up! Clearly the title “Sharing output streams through a JNI interface” says it all.

    I figured out how to use JNI to share output streams from C to Java. Yes, it’s copying memory incrementally from one to another which is how data flows when streamed. In my mind I see it as a virtual stream since data provided by the lower C api comes in chunks which are incrementally pushed to a Java write stream and over the network. Much better than buffering and copying all at once.

  7. Arindam

    Hi Cliff

    I’ve defined a native function as follows:

    JNIEXPORT void JNICALL Java_sandbox_JNITest_populateArray(JNIEnv *env, jclass class, jintArray array) {
    jint *vector = (jint *) malloc(sizeof (jint) * 100);
    int i;
    for(i = 0; i SetIntArrayRegion(env, array, 0, 100, (const jint *) vector);
    }

    and I call it from inside Java using:

    public static void jniTest() {
    int[] array;

    array = new int[100];
    populateArray(array);
    System.out.println(java.util.Arrays.toString(array));
    }

    In the native function, does (*env)->SetIntArrayRegion() write directly to the array passed to the native function without creating any copies of it?

    Help will be much appreciated.

  8. follow to suppose of it, Admission to the net, At that place are thousands of articles you can
    Say on how to come through in blogging.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 250 other followers

%d bloggers like this: