Multiple symbol definitions

I found a way to allow multiple symbol definitions in a static library assembled from a bunch of object (.o) files. The secret sauce is adding “-z muldefs” to the g++ command line. I’m not sure what it tickles in the linker but it made my pains go away. One of these days I’ll spend a weekend playing with “ld” to see what other magic I can come up with.

Sharing output streams through a JNI interface

Maybe it’s me, but there seems to be a shortage of expert information on the net these days. I dunno about you but I constantly find myself running out of Google links and StackOverflow replies when looking for an answer to some rather valid technical issues. Am I the only one out there trying some of these crazy ideas or is the internet playing a savage game on me? Hi, I’m Cliff, and you’re here because you were waiting for me to post something crazy and far reaching. Maybe you were waiting on the, “How to tether your 8830 to your 1st Gen iPad” article. Maybe you were looking for info on how to install a Linux based firmware on your brand new Samsung 3D television. Whatever it is that brought you here, welcome to my little corner of the net where solutions to obscure technical issues are the norm.

I recently went on a journey into JNI land. I had visited many years ago plugging native MS Excel reporting into legacy RPG/400 programs via ILE wrappers around the POI project. Things were so much smoother back then. Flash forward to recent weeks as I wade through my old C manuals and skip through google links for Makefile tutorials. My project was to take a 3rd party native API, with C source code examples into the modern age. We need to use this API in a serious multi-user environment. To put it in simple yet technical terms, I needed to stream binary data from this API over the internet to a mobile device. Without being too specific and revealing let me give a generic example that colors the picture of what I’ve accomplished.

Say you have this blob of binary data. Go ahead, say it. Now say it once more but this time wait for a co-worker to walk by. Say the binary blob is generated from a native API that costs, oh about $2000 a seat. (The cost is not nearly as important as the seat.) Now say the API is not documented with any examples in the programming language of your choice. (Say that last sentence aloud with a girlie giggle between the 6th and 7th words.) Let’s pretend the only example given uses an old fashioned main function C reading cmd line arguments. Let’s pretend you have clients that need this blob on the other side of the internet. What are your options?

You could call the API using the command line example which writes to the file system. You could then read the generated file back across the internet using your favorite Ruby/Python/PHP programming toy. Then you would have to deal with file write collisions, the overhead of disk access, cleaning up the file after its written, file locks if you’re on Windows, and many other headaches.

You could modify the example to write directly to the internet. Then you have to deal with either implementing HTTP or talking to yet another 3rd party API written in the same language. You’d also have to deal with the language which you only use on the rarest of occasions… like never.

You could opt for the Language compatibility API that comes in the language of your liking with the additional purchase of two other vendor products. (API not sold separately.) You could do a number of different things each with their own drawbacks or you could do what I did… design a native interface.

I used JNI to write binary data from C back to a Java object. My first reaction was, “Gosh! How do I write a C++ output stream like thing to a Java output stream?” My second reaction was, “Don’t I need an input stream instead of an outputstream? Which end of which stream plugs in where?” (I always have that reaction when I deal with I/O streams.) the short answer, which applies equally to pure Java, C++, and Java/C++ is that you need threads to connect two streams… ALWAYS. There’s no safe way to deal with threads so thought a little harder. Because the output was delivered incrementally in my case, I opted to use a callback strategy. In other words, the API used callbacks to hand chunks of binary in array form to a calling application. I simply plugged in my JNI C code to accept these arrays and relay them to the original invoking Java object.

You have to be really careful with JNI native array conversion. There’s a proper way to copy bytes from C to Java and then there’s the whole, “is this a copy or a direct reference?” issue. Also you might get confused with array pinning in later VMs, but that really shouldn’t matter if you only need to materialize the data briefly in C.

In all, the general strategy for sharing binary data (A/V files, images, etc.) from C with Java requires byte arrays. You create a Java byte array in C like this:

const char[] rawData = {0,1,2,3,4,5,6,7,8,9}; //Or get some raw data from somewhere
int dataSize = sizeof(rawData);
printf("Building raw data array copy\n");
jbyteArray rawDataCopy = env->NewByteArray(dataSize);
env->SetByteArrayRegion(rawDataCopy, 0, dataSize, rawData);

And pass it to Java like this:

printf("Finding callback method\n");
//Assumes obj is the Java instance that will receive the raw data via callback
jmethodID aMethodId = env->GetMethodID(env->GetObjectClass(obj),"handleData","([B)V"); 
if(0==aMethodId) throw MyRuntimeException("Method not found error");
printf("Invoking the callback\n");
env->CallVoidMethod(obj,aMethodId, &rawDataCopy);

you would have a Java object that looked something like this:

public class MyDataHandler {
  OutputStream dataStream;
  public MyDataHandler(OutputStream writeTo) { dataStream = writeTo;}
  public void handleData(byte[] incomingData) { dataStream.write(incomingData); }

That handler would be passed to C via native method like so:

public class NativeIntegration {
  public native void generateBinaryWithHandler(MyDataHandler handler);

  //Here we assume response is something like a network stream
  public void doCallNativeFunction(ResponseStream response) {
    MyDataHandler handler = new MyDataHandler(response);

That’s a rather niave example without the necessary consideration of error handling/reporting but it gets the point across.

JNI, C++, Idea, and Netbeans looks good

I’ve been trying to get comfortable with C++. I’m finding quite a few native libraries that I need to talk to… programmatically. So I tried GEdit. Then I moved on to Windows and Eclipse. Everytime I want to install an Eclipse plugin I somehow find it easier to just grab an Eclipse bundle. (Plugin conflicts can burn so much clock.) Grabbing the Eclipse CDT, I found myself accidentally expanding and overwriting my Blackberry Eclipse install. (I always expand to C:\eclipse, it’s just a habit.) Frustrated, I decided to boot back into Linux and try Idea. I tried out the C++ plugin under Idea and had some minor trouble because there’s no documentation on it. I then tried KDevelop. Then something came over me. I decided to try out Netbeans. Netbeans…

Once upon a time there was an IDE I kinda liked. It was Forte (based on Netbeans). That was my first Netbeansy experience. Back then I was satisfied but then I found Eclipse and later got hooked on Idea. Netbeans was something I always toyed with here and there. I liked it but it lacked the refactoring support in Idea. Still it was kinda cool.

Fast forward a couple of years and now I’m all about Eclipse and Idea. I use Eclipse only because others force it on me. RIM tools are built for it, most people in my company swear by it. Every time I need to do something slightly different there seems to be an Eclipse bundle for that. It’s pretty full featured only falling behind Idea in a couple of spots. It’s just a pain to configure/setup. Idea has always been my tool of choice. If I could write ObjC in Idea I’d pitch XCode in a minute.

Today I tried Netbeans for the first time since it turned 5. I almost fell out my chair when I instinctively hit Alt+Enter to fix a syntax error. It worked! Just like Idea! Then I started looking at the right click and refactoring menus. I got really happy! Netbeans is now starting to feel like Idea felt back in the day. To be honest, Idea has been giving me lots of trouble these days with plugin “Blame this plugin” error pop-ups and the AWT_TOOLKIT thing that causes the text area to become uneditable. Overall Netbeans is looking much improved! I am totally impressed… I want to write more about both, my experience with Netbeans and my use of the C++ plugin under Idea.