My goal during the last weeks was to enable our company's application to run via the Web Start client.

Further installation should be as uncomplicated as possible. This task implied that except Web Start installation itself, the application should just require one click to download and install and nothing more.

As our applications uses some larger packages like Java3D and drivers for an object-oriented database, packages that both include native code, additional problems had to be solved that were only poorly if at all documented in the official Sun docs.

I will post some of the lessons learned in this thread.

Regards,
Marc

 

 

Fun with native libs (1): Java3D

---

Motivation

An important part in the field of scientific visualization is the display of 3 dimensional geometric data. One option for Java developers to get this task done is to use the Java3D API (another one would be to OpenGL for Java bindings, like done in the JCanyon demo for the JDK 1.4 beta).

At the time of this writing (JDK 1.4 is beta), the Java3D API is not part of the standard Java JRE.
Usually, people who want to run a Java3D enabled Java program have to download and install the seperatly available Java3D installation files for their specific plattform themselves.
This would obviously ruin the goal of one click installation for someone who is interested to run the app but had not installed Java3D before.

Ideally for Web Start deployment purposes, Sun would have packed and signed Java3D in some nice jars. But it hasn't yet. (But they promised)

So one needs to do it oneself.

A first solution

The first working solution to this problem I noticed was by Dr. Masa Takatsuka in this thread

http://forums.java.sun.com/thread.jsp?forum=38&thread=71532

He works on an interesting application called JBeanStudio that features 3D data viewers

http://www.geovistastudio.psu.edu/JBeanStudio

and managed to deploy the necessary Java3D libs together with the application via Web Start.

Looking at the jnlp file he published, his approach was first to seperate all Java3D stuff into its own sub jnlp file:

<extension name="Java3D" href="lib/core/Java3D/Java3D.jnlp"/>

In that sub jnlp file he used this division of resources for the different platforms:

  <resources>
    <jar href="j3daudio.jar"/>
    <jar href="j3dcore.jar"/>
    <jar href="j3dutils.jar"/>
    <jar href="vecmath.jar"/>
  </resources>
  <resources os="Windows">
    <nativelib href="windows/j3dDLL.jar"/>
  </resources>
  <resources os="linux" arch="i386">
    <nativelib href="linux/i386/j3d.so.jar"/>
  </resources>
  <resources os="linux" arch="ppc">    
    <nativelib href="linux/ppc/j3d.so.jar"/> 
  </resources>

Thus the first four jar file were intended for all platform, with a specific jar for each architecture.

It tried this out with an easy example and it turned out that this code only worked for Windows, but failed under Linux.

The experimenting exposed several problems.


Different Vendors, different arch property

I tried it on a Linux box with an IBM 1.3 JRE. The interesting bit was, that the native lib never got loaded.
The reason being that the IBM JRE doesn't define arch as 'i386', like the SUN JRE does, but uses 'x86'.
No wonder the resource had not been loaded by Web Start.

However this lead to a rather ugly duplication of otherwise identic XML code:

  <!-- Linux IBM J2RE 1.3.0 -->
   <resources os="Linux" arch="x86">
     <nativelib href="jars/j3d/linux/i386/j3d.so.jar"/>
  </resources>
  <!-- Linux SUN JRE1.3.1 -->
  <resources os="Linux" arch="i386">
    <nativelib href="jars/j3d/linux/i386/j3d.so.jar"/>
  </resources>

Pure java, but a call to a Win32 method

Another problem that showed up, was that at least one of the four to be thought platform neutral jars contained a method call like Win32Foo, a method that exists only on Win32 platform implementations, but not under Linux.

Thus this, plus possible problems with versioning (the Linux Java3D release was slighty behind the Win32 release) lead me to organize the resources into this scheme:

  <resources os="Windows">
    <jar href="jars/j3d/windows/j3daudio.jar"/>
    <jar href="jars/j3d/windows/j3dcore.jar"/>
    <jar href="jars/j3d/windows/j3dutils.jar"/>
    <jar href="jars/j3d/windows/vecmath.jar"/>
    <nativelib href="jars/j3d/windows/j3dDLL.jar"/>
  </resources>
  <!-- Linux IBM J2RE 1.3.0 -->
  <resources os="Linux" arch="x86">
    <jar href="jars/j3d/linux/i386/j3daudio.jar"/>
    <jar href="jars/j3d/linux/i386/j3dcore.jar"/> 
    <jar href="jars/j3d/linux/i386/j3dutils.jar"/>
    <jar href="jars/j3d/linux/i386/vecmath.jar"/>
    <nativelib href="jars/j3d/linux/i386/j3d.so.jar"/> 
  </resources>
  <!-- Linux SUN JRE1.3.1 --> 
  <resources os="Linux" arch="i386"> 
    <jar href="jars/j3d/linux/i386/j3daudio.jar"/>
    <jar href="jars/j3d/linux/i386/j3dcore.jar"/>
    <jar href="jars/j3d/linux/i386/j3dutils.jar"/>
    <jar href="jars/j3d/linux/i386/vecmath.jar"/>
    <nativelib href="jars/j3d/linux/i386/j3d.so.jar"/>
  </resources>
  <resources os="Solaris" arch="sparc"> 
    <jar href="jars/j3d/solaris/j3daudio.jar"/> 
    <jar href="jars/j3d/solaris/j3dcore.jar"/>
    <jar href="jars/j3d/solaris/j3dutils.jar"/>
    <jar href="jars/j3d/solaris/vecmath.jar"/> 
    <nativelib href="jars/j3d/solaris/j3d.so.jar"/> 
  </resources>

This time I made sure, that I offer only jars from a specific platform release for each platform.

Regards,
Marc

 

 

Would you be will to post the whole .jnlp for your process?
Thanks

 

 

one program, three behaviours
---------------------------------

While trying to make our application work with Web Start, some of the problems turned out to be specific to how the application was ran.

There are basically three situations:

- running from a collection of class files ("java com.foo.bar")
- running from a jar ("java -jar app.jar")
- running from Web Start ("<click>")

Aside from the security issues, the problems were mostly about the loading of resources and the binding against native code.


running from a jar is already different

In one situation, some icons were missing in the application, when running via Web Start. I tried running the application directly from the jar and again the icons were still missing, thus Web Start was not the culprit.
However running from the unpacked class files it worked perfectly. Strange.

The solution was that a getResource("Foo.gif") had to match a "foo.gif" entry in the jar.

This problem shows not up when working from the unpacked class files under Win32, as its file system is case insensitive. But obviously the class loader that handles the loading from jar files insists on matching case.


Web Start and class loader selection

When running from Web Start, just another different class loader gets used, one that is able to access some jar from the Web Start cache and to trigger a load from the network, if necessary.
In contrast to the situation when running from a jar alone, this time it might happen that the Java JRE selects the wrong class loader during execution. So it was necessarry to change a few of the

getResource("Foo.gif");

Statements into

ClassLoader cl = this.getClass().getClassLoader();
cl.getResource("Foo.gif");

Using a different look and feel

When adding a different look and feel, the Kunsstoff look and feel, suddenly several of the menus didn't show up anymore under Web Start.

The solution was to kick the UIManager to use the correct Web Start class loader.

try
{
    com.incors.plaf.kunststoff.KunststoffLookAndFeel kunststoffLF = new com.incors.plaf.kunststoff.KunststoffLookAndFeel();
    kunststoffLF.setCurrentTheme( new com.incors.plaf.kunststoff.KunststoffTheme() );
    UIManager.setLookAndFeel( kunststoffLF );
}
catch ( javax.swing.UnsupportedLookAndFeelException x )
{
 
}
 
// make Web Start happy
// http://developer.java.sun.com/developer/bugParade/bugs/4155617.html
UIManager.getLookAndFeelDefaults().put( "ClassLoader", getClass().getClassLoader() );

 

 

Fun with native libs (2): Java3D
-----------------------------------

rolling a jar with native libs

Another insight gained from JBeanStudio was how the "directory" layout for a jar with native libs had to look.

If my package is organized into com.foo., the class files will reside in a com/foo/ directory hierarchy on the disk and result in a corresponding structure within the zip archive that the jar file really is.

So where to put the native libraries that are needed?

The answer is simple: just put it in the topmost directory ("/"), the jar is flat, except for the manifest sub directory that carries all that administrative stuff like entry method and cryptographic hashes.

loading the native libs

At first I had luck. My easy test sample and the Win32 platform I tried it out, were probably one of the easier cases.
I just supplied the native DLLs in the jar file and still it worked under Web Start:
I got fast Java3D, without any Java3D installation on that box!

However this approach never worked under Linux.

The next example I tried was more complicated in that it required not only one but several native libs loaded.
No luck under Win32 this time.

So I searched the Web Start forum for other examples that involved the loading of several libraries.

When I found one and studied it's jnlp file and main class, suddenly a part from the jnlp spec made more sense:

It is up to the launched application to actually cause the
loading of the library (i.e., by calling System.loadLibrary).
Each entry must contain a platform-dependent shared library with the correct naming convention, e.g., .dll on Windows, or lib.so on Solaris.

It turned out that I had to System.loadLibrary() calls into my main method.

public static void main(String[] args) throws Exception {
        //
        // Web Start needs explicit native lib loading!
        //
 
        String os = System.getProperty( "os.name" );
        System.out.println("loading " + os + " native libraries ..");
        if ( os.startsWith( "Windows" ) )
        {
            // order matters here!
            // load those libs that are required by other libs first!
            System.out.print( "j3daudio.dll .. " );
            // drop ".dll" suffix here
            System.loadLibrary( "j3daudio" );
            System.out.println( "OK" );
 
            System.out.print( "J3D.dll .. " );
            System.loadLibrary( "J3D" );
            System.out.println( "OK" );
        } else if ( os.equals( "Linux" ) )
        {
            System.out.print( "libj3daudio.so .. " );
            // drop "lib" prefix and ".so" suffix
            System.loadLibrary( "j3daudio" );
            System.out.println( "OK" );
 
            System.out.print( "libJ3D.so .. " );
            System.loadLibrary( "J3D" );
            System.out.println( "OK" );
        } else
            throw new Exception( "OS '" + os + "' not yet supported." );
 
        new MainFrame(new Java3dTest(args), 700, 700);
}

Please note two things:

1. Note how prefixes and suffixes of the name of the native lib have to be removed for the actual System.loadLibrary call

2. Note that order matters. First pull in the lib with the least dependencies.

This was good enought to make the Java3D example run under Win32 and Linux, except for the JRE issue (see below).

Note however that it might not always be possible to sort the libs in that linear form:
lib a might need something from lib b, while lib b needs something from lib a.
The problems I got with the database sample under Linux, seem to indicate that Web Start needs a modified loading approach here.

the right JRE version

Experience was, that everything is incredibly dependent on the version of the JRE!

The Java3D Test sample didn't work with the 1.3.0_02 JRE from Sun under Win32.
No luck at all with the IBM JRE under Linux (the Java3D libs want to link against a non existing libjawt.so).

For the database sample, under Linux, for a run from the console, everything worked with Blackdown's 1.3.1 JRE, but not with Sun's 1.3.1 JRE.
From Web Start both failed, but Blackdown came a little bit further than Sun.

I was surprised to learn such difference in behaviour between minor releases. We're not talking 1.2 -> 1.3 updates here.

 

 

Thanks Marc for your detailed report on Java3D and JWS.

I'll try your suggestions.


masa

 

 

What can one do when the native libraries needed have circular dependencies? I am running into that problem now. All libraries get loaded successfully up to the 2 that are dependent on each other. Then I get an unsatisfied link error. I am using jdk1.3.1.

 

 

I believe this was the problem I had with the database access libs under Linux.

Note that the jdk released by Sun that I tried at that time behaved a bit different when loading native libs than the Blackdown jdk release of the same release number. Blackdown managed to pull in one more native lib (but failed at the last one :).

So it might be worth to experiment with the latest 1.3.x releases or even the 1.4 beta from both JDK sources.

Back to the question of circular references.

I don't know yet how to solve this. But I would like to. :-)

It would be interesting to look, how the operating system copes with such an issue.

What operating system do you use?

Some things I want to try:

1. Looking closer what is going on.
One means I haven't tried yet is setting LD_TRACE_LOADED_OBJECTS, as described in

http://www.linuxdoc.org/HOWTO/GCC-HOWTO/x796.html

2. Avoiding the problem in the first place by repackaging
all the libs that are involved in the circular dependency into one giant lib.
Question is if there are tools that allow this repackaging.

More in the next days.

Regards,
Marc

 

 

I've noticed that an exception will be thrown if this procedure is followed and the user has already installed Java3D. In that case, the native libs are loaded twice and this causes an "Unsatisfied Link Error" when Java tries to access the library. To avoid this problem, I test for an already installed version of Java3D before loading the native libs in my java web start program. By the way, I am still searching for the proper method to test whether JAI is already loaded...

~Todd Jenkins

private void loadJava3D()
  {
    // test to see if native libs for java3d already loaded
    try
    {
      // this method causes the native libraries to be invoked
      GraphicsConfigTemplate3D gct3D = new GraphicsConfigTemplate3D();
      m_logger.info("java3d previously installed on this system");
      return;
    }
    catch(Error e) {}
    catch(Exception e) {}
    
    Throwable thrown;
    
    try
    {
      if(OS.getType() == OperatingSystem.TYPE_LINUX)
      {
        System.loadLibrary("j3daudio");
        System.loadLibrary("J3DUtils");
        System.loadLibrary("J3D");
        m_logger.info("loaded java3d for linux");
        return;
      }
      
      else if(OS.getType() == OperatingSystem.TYPE_MAC_OSX)
      {
        System.loadLibrary("J3DAudio");
        System.loadLibrary("J3DUtils");
        System.loadLibrary("J3D");
        m_logger.info("loaded java3d for osx");
        return;
      }
      
      else if(OS.getType() == OperatingSystem.TYPE_WINDOWS)
      {
        System.loadLibrary("j3daudio");
        System.loadLibrary("J3DUtils");
        System.loadLibrary("J3D");
        m_logger.info("loaded java3d for windows");
        return;
      }
      
      else
      { assert OS.getType() == OperatingSystem.TYPE_OTHER;
        m_logger.info("Java3D not automatically provided for this OS.");
        return;
      }
    }
    catch(Exception e) { thrown = e; }
    catch(UnsatisfiedLinkError e) { thrown = e; }
    
    m_logger.throwing(this.getClass().getName(), "loadJava3D", thrown);
    String str = "Unable to load native libraries for Java3D:\n" +
        thrown.getLocalizedMessage();
    this.abort(str);
  }

 

 

Here's the .jnlp file I use to define Java3D as an extension.
~Todd
[code]
<?xml version="1.0" encoding="utf-8"?>

<!-- JNLP File for Java3D -->
<jnlp codebase="http://yourdomain.com/aps/extensions/Java3D/"
href="Java3D.jnlp">

<information>
<title>Java3D</title>
<vendor>Sun Microsystems, Inc.</vendor>
</information>

<security>
<all-permissions/>
</security>

<!-- Windows -->
<resources os="Windows">
<jar href="windows/j3daudio.jar"/>
<jar href="windows/j3dcore.jar"/>
<jar href="windows/j3dutils.jar"/>
<jar href="windows/vecmath.jar"/>
<nativelib href="windows/j3dDLL.jar"/>
</resources>

<!-- OS X -->
<resources os="Mac OS X" arch="ppc">
<jar href="osx/j3daudio.jar"/>
<jar href="osx/j3dcore.jar"/>
<jar href="osx/j3dutils.jar"/>
<jar href="osx/vecmath.jar"/>
<jar href="osx/j3dsupport.jar"/>
<nativelib href="osx/j3d.jnilib.jar"/>
</resources>

<!-- Linux IBM J2RE -->
<resources os="Linux" arch="x86">
<jar href="linux/i386/j3daudio.jar"/>
<jar href="linux/i386/j3dcore.jar"/>
<jar href="linux/i386/j3dutils.jar"/>
<jar href="linux/i386/vecmath.jar"/>
<nativelib href="linux/i386/j3d.so.jar"/>
</resources>

<!-- Linux SUN J2RE-->
<resources os="Linux" arch="i386">
<jar href="linux/i386/j3daudio.jar"/>
<jar href="linux/i386/j3dcore.jar"/>
<jar href="linux/i386/j3dutils.jar"/>
<jar href="linux/i386/vecmath.jar"/>
<nativelib href="linux/i386/j3d.so.jar"/>
</resources>

<component-desc/>

</jnlp>
[\code]