Hey all,

I'm working on setting up JMX monitoring on the JVMs running in our environment. Most of those JVMs are running third party applications supplied by our customers and are heavily protected by firewalls. So i needed to use the JavaAgent strategy to be able to set a fixed port for the JMX agent. (see: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html)

This all works very nice except that the JVM will never exit. The RMI Reaper is a non-daemon thread and will have to be stopped manually before the JVM will shutdown. The application has no knowledge of the RMI Reaper (as it is started from the premain method). A ShutdownHook doesn't work because those are called after the JVM decided to shutdown or when an external event is triggered (Ctrl-C) for example.

For now i have this horrible workaround. I create a thread that checks if there is still a main thread in the JVM. If there isn't it shuts down the JMX server and that allows the JVM to close down. Is there a better way of doing this? below is the code:

public class JMXAgent {
    
    private static JMXConnectorServer cs;
    private static Logger logger;
    
    public static void premain(String agentArgs, Instrumentation inst) {
           /** snip */
           JMXServiceURL url =
                    new JMXServiceURL("service:jmx:rmi://0.0.0.0:" + RMIConnectorPort + "/jndi/rmi://:" + RMIRegistryPort + "/jmxrmi");
            cs = JMXConnectorServerFactory.newJMXConnectorServer(url, env, mbs);
            
        // Setup the failsafe
        Thread failsafeThread = new MainCheckerThread();
        failsafeThread.start();
        logger.fine("MainChecker Thread created.");
        
        // Setup the shutdown hook
        Thread shutdownThread = new ShutdownThread();
        Runtime.getRuntime().addShutdownHook(shutdownThread);
        logger.fine("SchutdownHook implemented.");
    }
    
    public static void stop() {
        try {
            logger.info("Stop called");
            cs.stop();
        } catch (IOException ex) {
            ex.printStackTrace();
        }
    }
    
    /**
     * Shutdown thread that allows the tracer to write out any final
     * information.
     */
    private static class ShutdownThread extends Thread {
        @Override
        public void run() {
            JMXAgent.stop();
            Runtime.getRuntime().halt(0);
        }
    }
    
    private static class MainCheckerThread extends Thread {
        @Override
        public void run() {
            boolean found = true;
            
            while (found) {
                Logger.getLogger(this.getClass().getName()).finest("Ping!");
                ThreadGroup root = Thread.currentThread().getThreadGroup();
                while (root.getParent() != null) {
                    root = root.getParent();
                }
                Thread[] list = new Thread[400];
                int threads = root.enumerate(list, true);
                found = false;
                for (int i=0; i<threads; i++) {
                    if (!list[i].isDaemon()) {
                        if (list[i].getName().equals("main")) {
                            found = true;
                        }
                    }
                }
                if (!found) {
                    /** No main found, stop the RMI server */
                    JMXAgent.stop();
                } else {
                    try {
                        Thread.sleep(10000);
                    } catch (InterruptedException ex) {
                        JMXAgent.stop();
                        found=false;
                    }
                }
            }
        }
    }
 

 

 

Hi Hugo,

This is a very interesting question.

What happens here is that the RMI object exported by the JMX connector server prevents
the RMIReaper thread from terminating: as long as the JMX connector server is active, its
exported RMIServer object will remain active, and the RMI Reaper thread will stay there
waiting for the RMIServer object to be unexported.

This is in fact a feature: JMX connector servers where designed to be controlled by
the application itself. In the usual way of using JMX Connector Server, you would start your
server in main() (or some application equivalent of main) and stop it later on before exiting
your application.

Things got a little more complex when we introduced the 'out-of-the-box management'.
In the 'out-of-the-box management' the default connector is not controlled by the application
but by the VM - or more exactly by an agent loaded in the VM. As you rightly noticed, the
agent has no idea as to when the application (the main) will exit.
The default 'out-of-the-box management' agent is thus conspiring with RMI (using sun
private APIs) to export its RMIServer object, so that its exported RMIServer doesn't
prevent the RMI Reaper thread to terminate. These APIs are unfortunately private and
non portables (and low-level).

Maybe we should think of supporting some kind of 'daemon connector servers' for the future.
The ability of starting JMX Connectors servers in premain agents obviously makes this desirable.

Concerning your 'hack' that detects the end of the main thread I will make two observations:

1) such a hack might work if indeed the application is expected to exit when the main thread
terminates. This is not always the case - for instance consider this:

    public void main(String[] args) {
           final Thread myMainThread = new Thread("My real main thread") {
                 public void run() {
                        try {
                              System.out.println("Strike <return> to exit: ");
                              System.in.read();
                        } catch(Exception x) {
                              x.printStackTrace();
                        }
                 }
             };
             myMainThread.start();
      }

Such an application will wait for the user to strike enter to exit, although the main thread is
already terminated and gone. So the termination of main thread doesn't necessarily means
that the application is finished.

2) The concept of detecting a main thread is tenuous. What happens if an application does
new Thread("main") ?

This being said - detecting the absence of "main" thread might be sufficient to your needs in
most of the case. If you simply call JMXConnectorServer.stop() when you detect that the
main thread is no longer there - the only risk you will be taking is stopping the
JMXConnectorServer ahead of time, before the application actually exits. If you want
to implement such a hack - you might wish to use the following logic:

    // Attempts to find a "main" thread
    //
    public static Thread findmain() {
        // When the premain agent is loaded from the command line the
        // current thread is the main thread....
        //
        final Thread current = Thread.currentThread();
        if (current.getName().equals("main")) return current;
        
        // If the premain agent was loaded from the attach API, or by some
        // other means, the current thread might not be the main thread.
        // 
        // Try to find a main thread using Thread.enumerate()
        // The main thread should be already there, allocate some
        // more space than needed just to be sure we do get it, even if
        // new threads are created in between.
        //
        final Thread[] all = new Thread[Thread.activeCount()+100];
        final int count = Thread.enumerate(all);
        
        // Try to find the main thread
        //
        for (int i=0;i<count;i++) {
            if (all[i].getName().equals("main")) return all[i];
        }
        return null;
    }
 
     public static void premain(String agentArgs) 
        throws IOException {
        ....
        final JMXConnectorServer cs = .....
        cs.start();
        final Thread main = findmain();
        
        final Thread clean = new Thread("JMX Agent Cleaner") {
            public void run() {
                try {
                    if (main != null) main.join();
                    System.out.println("main thread exited...");
                } catch (Exception x) {
                    x.printStackTrace();
                } finally {
                    try {
                        System.out.println("cleaning up JMX agent...");
                        cs.stop();
                    } catch (Exception x) {
                        x.printStackTrace();
                    }
                }
            }
        };
        clean.setDaemon(true);
        clean.start();
    }

If you really want your JMXConnectorServer to stop() at the latest possible time, you could
for instance modify this logic to find the first non daemon thread (excluding RMI Reaper and
DestroyJavaVM threads), join it, and loop over this until the only non daemon thread that
remains is/are the RMI Reaper threads, in which case you would close the
JMXConnectorServer.
This would not be completely foolproof either because the normal behavior of the application
could precisely be to be held alive by an exported object, and waiting on some
external communication to stop. So even counting the non-daemon thread until only
the RMI Reaper threads remain doesn't necessarily means that the application is
finished...
Here is an example of CleanThread class that implements the logic above:

public class JMXAgent { 
    ....
    public static class CleanThread extends Thread {
        private final JMXConnectorServer cs;
        public CleanThread(JMXConnectorServer cs) {
            super("JMX Agent Cleaner");
            this.cs = cs;
            setDaemon(true);
        }
        public void run() {
            boolean loop = true;
            try {
                while (loop) {
                    final Thread[] all = new Thread[Thread.activeCount()+100];
                    final int count = Thread.enumerate(all);
                    loop = false;
                    for (int i=0;i<count;i++) {
                        final Thread t = all[i];
                        // daemon: skip it.
                        if (t.isDaemon()) continue;
                        // RMI Reaper: skip it.
                        if (t.getName().startsWith("RMI Reaper")) continue;
                        if (t.getName().startsWith("DestroyJavaVM")) continue;
                        // Non daemon, non RMI Reaper: join it, break the for 
                        // loop, continue in the while loop (loop=true)
                        loop = true;
                        try {
                            System.out.println("Waiting on "+t.getName()+
                                    " [id="+t.getId()+"]");
                            t.join();
                        } catch (Exception ex) {
                            ex.printStackTrace();
                        }
                        break;
                    }
                }
                // We went through a whole for-loop without finding any thread
                // to join. We can close cs.
            } catch (Exception ex) {
                ex.printStackTrace();
            } finally {
                try {
                    // if we reach here it means the only non-daemon threads
                    // that remain are reaper threads - or that we got an 
                    // unexpected exception/error.
                    //
                    cs.stop();
                } catch (Exception ex) {
                    ex.printStackTrace();
                }
            }
        }
    }
 
    public static void premain(String agentArgs) 
        throws IOException {
        ....
        final JMXConnectorServer cs = .....
        cs.start();
        final Thread clean = new CleanThread(cs);
        clean.start();
    }

Thanks for bringing this to my attention! I'm probably going to have to update my entry
on this subject at
http://blogs.sun.com/jmxetc/entry/connecting_through_firewall_using_jmx

Hope this helps,

-- daniel
http://blogs.sun.com/jmxetc
--

 

 

Hey Daniel,

Thanks a lot for the answer. It is indeed a difficult, but interesting problem. Fortunately for me the applications that need to be monitored are not supposed to go down, so hopefully i would not need the 'hack' that often.

I really like your third solution, it's the most elegant solution. I'm going to implement it right away.


Regards,

Hugo