`Which Java`?

I’m a little frustrated because I’ve been hitting some memorable distro-issues throughout the week and revisiting some of my old topics to help me through. Today I wanted to install Charles proxy so I can sniff HTTP traffic and rememberize all of the neat-o stuff I used to know about HTTP. Here’s one thing that used to bug me alot, Java explodes on linux. No matter what distro I tried and would inevitably hit a situation where Java would just blow up in my face. I’ll be running my IDE and it will just vanish. Or I’ll be running some app I just wrote and it craps out with VM error stack trace.That prompted me to see which Java I had installed in Mepis. A naive person would type “java -version” on the command line but after years of experience I learned that not only is that misleading it’s downright dangerous! If Java is officially installed (a process that varies from machine to machine and OS to OS and distro to distro) you have to understand that the system can pick a different version of Java than what you get on the command line. The command line is influenced by the $PATH while there are a number of different ways to invoke Java on your system. Consider hitting a page with an Applet? Java runs out of your Browser. Depending on which browser you run you’ll get a different JVM. Consider what happens when you click a Web Start link. You can have Firefox configured to associate it with one version of Java while your file manager (Windows Explorer, KWin, whatever) can associate it with another. Then there’s the executable jar thing. Once again window manager file associations can point someplace you never intended. Lastly there’s batch files and shell scripts. On Linux your $PATH can be overridden in your .bash_profile and/or your .bashrc while the $PATH assumed by your Window Manager is completely different.

So which Java do you get? On Linux you can literally type `which java` and it will tell you the path to the executable but that doesn’t help much. Try adding “ls -la” in front.

ls -la `which java`

Then you’ll see that this is a symbolic link to the actual (sometimes other sym link to the actual) executable. Keep following sym links until you get tired and you’ll find the true path to Java. This used to be controlled by and update-alternatives -config java command but I just tried that on Mepis and it doesn’t exist. I tried looking for update-alternatives under apt-get install update and got nowhere. I’m spilling all these details here because now I really don’t know `which java`???

How to use rapc from RIM… dirty details!

What’s really going on in rapc, Blackberry’s crazed compile tool? Let me begin by telling you that everything you’ve read until this point is not entirely accurate. There are those that will tell you that you have to run preverify.exe prior to running rapc. Then there are those who will tell you that the bb-ant-tools are sufficient. There are people that complain that the RIM compiler has bugs in it. (This may be true but I’ve so far been able to fix broken builds by using all of the advice I’m including here.) I’ve also found a blog that claims you can build from non-Windows operating systems. I’ve read in some places that either WTK or the JDE must be in the PATH so that rapc can find a preverifier. While some of these tips, tools, tutorials work they all have limits. Eventually you’ll find your application either throws random NPE’s, verification errors, or infinitely reboots the device! Now pay attention!

Now let me explain my project. I have a little bit of dependency injection going on via source generation. Then I also have some external jar dependencies. All of this creates a bit of a challenge for basic rapc invocation. In order to generate my midlet cod file I need to:

  • Run a source generator
  • Include the generated source in my list of things to compile
  • somehow include my external jar dependencies in the compile.
  • Include resources in my compiled output
  • Pick up my suit from the cleaners
  • Get the entire thing to run on my Mac?

While the last bullet is entirely optional and irrelevant, I include it because its the only way to get completely familiar with rapc. That is, if I can meet/greet/beat the challenge of cross platform compilation of Blackberry then I’ll really understand the intracacies of compiling with rapc. Now onto the juice.

How Rapc works… I think…
There are various approaches to compiling Blackberry applications that I know of.

  • Compile your app against the MIDP and CLDC APIs (include either WTK or MPowerplayer or someother party’s midp and CLDC jars on your bootstrap classpath for javac… making sure you use javac1.3 or below), preverify with JDE preverify.exe (found in %JDE_HOME%/bin), jar and finally compile with rapc.
  • Compile your application source files entirely with Rapc.

The first option has problems because there’s the possibility that javac will generate class files that will cause verification errors or (as my team discovered recently) the white screen of death when loaded onto the device. The second option has problems because it’s very tricky and prone to the same errors unless you are very dillegent. It took a lot of head banging and trial/error to come up with these details so read closely.

How do we use rapc? This is the question that still plagues me. There’s little documentation around the nuances you’ll encounter when invoking the rapc compiler from the command line. The rapc compiler is an executable (.exe) file in the JDE install directory living under the bin subfolder. First thing to note is the rapc.exe just delegates everything to rapc.jar. That means the same exact command line parameters you would use with rapc.exe will work with rapc.jar. (Call the jar with the “java -jar” command.) The most important thing to understand is that you cannot mix and match. That is you cannot use the output of javac/WTK preverify to fuel rapc. Furthermore you need to pay strict attention to the version of the JDE you use because it must be the same (or earlier) version of your target platform. For eg., if you intend to target Blackberry devices running a 4.2.2 OS you should use the compiler tools from the JDE version 4.2.2 or earlier. If you don’t pay attention to this step your product will work just fine until it’s time to ship… at which time you’ll experience weird, impossible to track/reproduce on a developer workstation, errors. There are other important tools in the JDE that you’ll need as well. The preverify tool is required (unless you’re using JDE 4.3.0 and targeting OS 4.3+) as well as the SignatureTool.jsr. Finally there is the net_rim_api.jar which lives under the lib folder in the JDE install directory. The next big thing to understand is the command line options.

Use the import= to specify the location of the net_rim_api.jar. This flag takes the relative or absolute path of not only the rim APIs but any other jar or cod dependencies your application has. (I haven’t tried cod dependencies but the docs say you can list cod files.) Be careful because this is only for resolving dependencies. Anything you list here will not be included in the final output cod. What this means is any jar files listed here must either be compiled as separate cod libraries or their source files somehow inlined in the final compiled. Dependencies listed here are scope as provided. In other words it is assumed they will be made available at runtime by either the RIM platform or by some pre-installed package.

(codname for 4.3.0+)
This parameter threw me for a minute when I attempted to roll back to an earlier version of the RIM compile toolchain. I believe (and I need to double check and update this fact later) that rapc version 4.3.0 and higher expect a codname flag while versions prior to 4.3.0 expect a codename flag. The value you specify here will be the value used in the output codfile name. Paths can be relative to the working directory (the folder rapc is called from) or possibly absolute paths. For instance if you give build/MyApp as the value then rapc will spit all of its output into the build folder relative to the working directory giving you a final MyApp.cod file there. (you’ll also have MyApp.jar, MyApp.csi, MyApp.debug and other files in this folder.)

This is the application type. Use this flag if you plan to develop a J2ME spec compliant midlet. Blackberry support another application type, cldc, which provides RIM specific features (more on that in possibly another post) and is specified by the mutually exclusive -cldc flag. A You have to -midlet or -cldc to specify which kind of application you are building.

Use this flag if your are building a CLDC application that adheres to RIM’s proprietary launching interfaces. CLDC apps offer abilities to run in the background and be started at device boot time. There are other hidden gems here that I’d have to get into with another post.

Use this parameter if you have a JAD file that includes your application’s meta data. This parameter is optional as you may substitue a .rapc file to include the meta data. Application meta data includes things like the icon to display on the desktop, the midlets contained in your aplication bundle, runtime properties, etc. Referr to the J2ME midlet spec for more detail.

rapc file
This would be the path to a “.rapc” file that includes meta data about your application. I’ll probably follow up with another post to explain the details behind this proprietary RIM spec. While a JAD file also includes meta data there are certain properties that can be included in this file that cannot be included in the JAD file and vice-versa.

class or java files
The last parameter should specify either a jar file containing the compiled classes or be a list of the .java files you intend to compile. Here’s where rapc gets really interesting. If you precompile your “.java” files you had better be certain you run the RIM preverify tool against these class files prior to invoking the compiler. Also I will repeat that you must use the same (or earlier) version preverify as your target platform. For eg., if you intend to target Blackberry devices running a 4.2.2 OS you should use both RIM’s rapc and preverify tools from the JDE version 4.2.2 or earlier. If you don’t pay attention to this step your product will work just fine until it’s time to ship… at which time you’ll experience weird, impossible to track/reproduce on a developer workstation, errors. (I said this before didn’t I?) Rapc is very picky here and the behavior varies between versions. It first calls out to javac assuming that the Java compiler is in the PATH. It then passes the output from javac to preverify. Note that in JDE 4.3.0 both “.java” source compiling and preverification is done from within the rapc.jar while earlier versions of rapc call out to the command line using the command “preverify” (not “preverify.exe”… this is an important distinction) and passing standard preverify parameters. Finally it creates the final “.cod” that will run on the device. What is included in the cod file depends on what is passed on the command line and what’s available from the working directory. For instance, your application icon (and I found this through trial and error) must be locatable from the working directory unless you specify an absolute path in the jad file. If you compile a jar file (not recommended for rapc prior to 4.3.0) then everything in that jar file (images, multimedia files, application resources, etc.) will be included in the final cod file. If you compile “.java” files directly from rapc then no application resources will make it into the final output cod file, only the compiled .class files will be there (along with the application icon specified in the jad if it can be found from the working directory). It took a lot of banging my head to figure this out. Finally instead of a jar file or space delimited list of “.java” files, you can specify a text file that lists (new line delimited style) all of the “.java” files you wish to pass to the compiler. This is a potential work-around to command line size restrictions that may be present on some versions of Windows. This file should be given with an @ prefix. For eg. if your have a MyApp.txt file then the last parameter for rapc will be @MyApp.txt. This is the preferred method of giving files to rapc because as your application grows so will the list of files.

One last got’cha I found was with trying to use rapc.jar directly instead of rapc.exe. If you are running a JDE earlier than 4.3.0 then you’ll get into trouble here because there is a secret parameter passed from rapc.exe that RIM won’t tell me about. Because the command line syntax for the jar is identical to the command line syntax for the “.exe” you can almost get away with it but because of a secret exchange happening between the Jar and exe you’ll get undefined behavior and weird errors in certain circumstances. This last gotcha is what makes it impossible to compile on an OS other than Windows while targeting a RIM OS earlier than 4.3.0.

In the end I’ve determined that it is not possible to accurately build a blackberry application on an operating system other than Windows. Also while there may be some truth to the myth of bugs in rapc I believe it all boils down to a bunch of nasty assumptions the RIM developers made during the design. (they assumed everyone would be so thrilled with their Swing based IDE that they would throw away Eclipse/Idea/Netbeans and burn their Macintosh laptops in celbration of the obviously superior RIM tool chain.) I do believe that if you pay close attention to the details outlined above you can diagnose most preverification and random erros you may be suffering from. I plan to revist this topic in the future as I’ve taken a break from mobile and begun work on some other things. However I do not plan to let rapc defeat me entirely. Keep an eye out because I’ll likely be updating this very same post with more details.

Groovy Ant Secrets

I was pondering about this earlier today. Wondering how I would invoke an Ant task that used nested text data using Groovy Ant. Why? Because I looked at the Ant script task while browsing Ant docs and said, “Hmm. that’d be neat to embed a script in a GroovyAnt build… but how do I pass text?” Well it just dawned on me that the random errors I’d been getting with Groovy’s AntBuilder exposed the answer. Occasionally I’ll forget to prefix a task attribute with an attribute name like:
ant.available(property:'menu.icon.found', '/menu.png', classpath:'target/MapQuestCoolStuff.jar')
And I’ll get an error saying:
ERR: SCRIPT FAILED: The type doesn't support nested text data ("/menu.png").
ERR: The type doesn't support nested text data ("/menu.png").

It’s telling me that the second parameter is being passed to AntBuilder as text instead of a task attribute.

GMaven for Blackberry development on a Mac?

Ok, it’s been a while but I finally posted the code for GMaven-Blackberry in another blog post.

I’ve gone clear outta my mind. I’m thinking let’s build a blackberry native app on the Mac. Sure! Why not? I mean there’s only the long-standing platform incompatibilities between J2ME and OS X, the windows-oriented nature of Blackberry tools, the mismatch of typical J2ME project structure and Maven project structure and looming deadlines to worry about. And while I’m at it, throw Groovy into the mix, because everybody knows that Groovy has everything to do with writing embedded apps right?

This week I’ve bounced between a Gant build, a legacy Ant build, our multi-module maven JEE project, and brand new GMaven plugin projects. Let’s just say I’m becoming very familiar with the build process in general. Project structure is important. The build system is critical. It is at the root of every successful and failed project in history. Many problems, and resolutions to problems stem from the build/deployment system in place on a given project. It makes sense that as a developer you pay close attention to what’s involved in the building and deployment of your software. Yet soooo many people fail to understand the variables involved in the build tool chain. Many a problem can be tracked to system incompatibilities resulting from version mis-match and/or conflicts. Something as simple as a micro point release difference in your web container or JDK can cause an app to behave wildly different. (Today I tracked a bug down to Tomcat 5.0.28 on a dev box where version 5.5.25 is running in production.) The version numbers of your build tool chain is often the last place developers look when researching a problem. It is the first thing I look for when trying to isolate an issue. I digress.

With all the build tools ranging from Ant to GAnt to Maven to Gradle… (BuildR anybody?) it’s easy to get overwhelmed. I had a buddy remind me how important it is to refrain from going overboard in leading edge-multi-faceted-combined technologies. Stick with proven products yet always look for the next thing. That’s how I feel about Maven. There are alternatives some have better features. However Maven is mature and ubiquitous. I don’t have to pull in yet another runtime to build my project. I don’t need to learn yet another platform. It does its job and does it well. So well that I try to use it for all of my Java work, bringing me to my point. J2ME Blackberry development can be done with Maven. It can be done on a Mac. And at the end of my ordeal it should be painless to do on a Mac.

So why Maven? Why not Ant? The app I’m working on includes several JEE complimentary pieces and I’m sorry but I would rather not touch Ant on a web app project. No way, no how. Furthermore with JEE making up over half of the codebase Maven is the perfect solution for managing the project. We currently have it in use and we have a wrappering pom around the Ant build we use for our J2ME client. That wrapper has been a source of utter pain. So I’ve been looking to figure this out for the longest.

What does Groovy have to do with it? Well I have the Pyx4ME J2ME plugin for Maven working pretty well but there are some obvious holes. First off is a decent deployment mechanism. I need to tie deployment into the modified Antenna OTA servlet we have. Secondly there is no support for Blackberry in the plugin so somebody needs to add it. I started to write a plugin and I just knew I wouldn’t want to be bothered writing it in Java. A couple of minutes after I finished bouncing between 20 different reference articles, APIs, and PDFs and I had the beginnings of the RIM plugin for Maven complete with OS X compatibility. It’s not totally finished yet. I still have issues running the jar signing tool on the Mac. (It should run fine on Windows, and possibly Linux.) But it runs the compiler. I’ve generated a .cod file in the target folder of a prototype project. Interesting stuff indeed.

If that’s not enough, I’ve also managed to launch GAnt from within Maven! Why would anybody want to do this? Maven is become widespread like Ant. If you or your team already use Maven but you want to toy with Something else like Gant you would have immediate access. You don’t need to open your browser, don’t launch Safari, just run Gant:

mvn gant:run

That’s it! And that’s perfect for my current situation where I’m trying to spread Gant across my team. The same applies for Groovy. Don’t go to the codehaus site, just run it!

mvn groovy:console

There’s something to be said about the convenience of a one stop shop. That’s my thing for now. Stop back because I’m gonna continue to post updates on not only Groovy, but Blackberry, GAnt, Maven, maybe even Gradle if I get that far. Holla!

Using Groovy with Maven

Quick update. If you use Maven and got used to Groovy a while ago (in the 1.0 and 1.1 days) like I did you might be looking for Groovy-1.5 in the Maven repos. Don’t look here. Instead look here. The groovy stuff is in a new location under org/codehaus/groovy.

Pyx4ME Hack

It’s late tonight but I gotta get this posted. I’m still trying to get a good dev cycle with the Maven Pyx4ME maven plug in and IntelliJ Idea. I’m running on Parallels on my Mac. Here’s an extra tag I added to the project generated from the pyx4ME archetype:

                  <property environment="env"/>
                  <echo message="Running WTK out of ${env.WTK_HOME}"/>
                  <taskdef resource="antenna.properties"/>
                  <wtkrun jadfile="target/${project.artifactId}-${project.version}-me.jad"/>
                <copy overwrite="true" file="${project.build.directory}/${project.build.finalName}-me.jar" tofile="${project.build.directory}/site/mymidlet.jar" />
                <copy overwrite="true" file="${project.build.directory}/${project.build.finalName}-me.jad" tofile="${project.build.directory}/site/mymidlet.jad" />
                <replace value="mymidlet.jar" token="${project.build.finalName}-me.jar" dir="${project.build.directory}/site">
                  <include name="mymidlet.jad"></include>

This lets me run my project using WTK assuming WTK_HOME is been set as a system env var. More stuff later.

IIOException can’t create cache file

Tomcat + ImageIO

What do you get? Trouble! Who would guess that doing PNG conversion with JDK APIs requires you to have a temp folder off of $CATALINA_BASE? After being labeled the build master unofficially, I took the initiative (if you can call totally screwing up a quasi-production deployment initiative) to prune what looked like cruft from a template tarball. Among this cruft was our precious temp folder. Let me explain better.

We use a deployment template for production builds. You know how this stuff goes when one developer gets something working and management says something like, “can you make yours work like his?” Thus begins the binary hand-off. A binary archive of the stuff he did bequeathed unto you to bring identical magic to your project. You then hand it down to the next developer that needs to do something similar. And so on and so forth. The entire idea is crazy but it’s the stuff major companies run on. by the time it gets handed to the 5th or 12th person down the line nobody remembers or understands anything from the original project only that this thing is what’s supposed to make everything else work just like it.

So I get this tarball, and I’ve got plenty others that look sorta like it that I tried creating from scratch. Some of mine half work while others just flop. I loose track because all correspondences with the deployment team and myself happens in a loosely coupled chain of random emails and secure copies to locked down servers that I have no business touching. I finally get something working that I can actually automate but I feel dirty because this binary template is checked into our version control system. So I begin cleaning the cruft, extracting the true thing that our project needs… structure not binary data. Everything begins to work. Everything except for a few execution paths that I have not had time to QA due to my tight schedule. I ask for QA from the QA team but I’ve been so tied up in performance tests that I can’t let the team anywhere near the deployment. I find other issues due to my not following certain design patterns. I fix issues. But the entire thing takes sooo much effort because I work in a highly secured environment. I work with one guy that’s been a blessing giving me higher than usual access rights as if I truly knew what I was doing on a Linux box. (Wasn’t I losing my Linux touch just days ago?)

The end of my plight is today. After hitting the “IIOException can’t create cache file” thing I Googled and found some info hinting to some more detailed discussion that only hints that you need a temp folder configured for Tomcat. I examined the startup script and plugged in the missing temp folder and voila! Everything worked as it should.

Moral to story
If you get: IIOException: can’t create cache file!
And you’re running in Tomcat
Then you need to make sure $CATALINA_BASE/temp actually exists and is writable.

By default Tomcat’s startup scripts set java.io.tmpdir to $CATALINA_BASE/temp but this could be overriden by a number of different ways. It is the java.io.tmpdir that is used by ImageIO and possibly numerous of other 3rd party APIs and open source components for cache files and other temporary stuff.