JMX connectivity through the firewall

Recently I’ve been asked to help out a customer who was having issues with JMX connectivity to Spring Source dmServer through the firewall. However, one thing I want to point out right up front is that the issue is rather generic and has nothing to do with dmServer. It is really about understanding JMX, RMI and proper configuration. But I will use dmServer and its configuration as an example.
Here is the sample JMX configuration options provided in the dmServer startup script:
-Dcom.sun.management.jmxremote.port=${jmxPort} \
-Dcom.sun.management.jmxremote.authenticate \
-Dcom.sun.management.jmxremote.password.file=${jmxUsersPath} \
-Djavax.net.ssl.keyStore=${keystorePath} \
-Djavax.net.ssl.keyStorePassword=${keystorePassword} \
-Dcom.sun.management.jmxremote.ssl=true \
-Dcom.sun.management.jmxremote.ssl.need.client.auth=false”
This will enable JMX agent (MBean Server) when you start dmServer. Once started you can now monitor your process via JMX-compliant tool such as jconsole. Connectivity could be local or remote.
The above configuration seem to provide everything we need to access this process through the firewall, since com.sun.management.jmxremote.port is obviously the port that we need to open in the firewall. However there is a caveat.
Once connected to JMXRegistry running on the port specified by com.sun.management.jmxremote.port property, the actual objects are served by RMIServer which is running on different port. Unfortunately this port is chosen randomly by default instance of JMX Agent and there is no –D option to specify it. Obviously going through the firewall would require opening up two ports and with random port it presents a delicate issue.
Fortunately it is easily solvable by writing a custom Java Agent where you can programmatically specify each port and externalize it through custom properties (I am attaching sample code).
More info here: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
In the nutshell, the custom agent will take the port value provided by the com.sun.management.jmxremote.port property and will create a second port (RMIServer port) by incrementing it by 1. (in our case the port specified is 44444 which makes RMIServer port 44445)
Once such agent is in place (JAR) and the appropriate ports are open in the firewall all you need is modify the startup script to include –javaagent option providing the JAR.
. . . . .
$JAVA_HOME/bin/java \
-javaagent:/Users/olegzhurakousky/../../../dmserver.jmx-0.0.1-SNAPSHOT.jar
$JAVA_OPTS \
. . . . .
Well, that really only solved one half of the problem, since by default RMI stubs sent to the client contain the server’s private address instead of the public

Just look at this tcpdump fragment while monitoring the client’s access (jconsole running on the local network):
. . . . . . .
09:41:23.778663 IP 72.234.14.89.44444 > 192.168.1.156.52926: . ack 71 win 65535 <nop,nop,timestamp 919359579 313492>
09:41:23.779958 IP 192.168.1.152.44444 > 72.234.14.89.52926: P 20:251(231) ack 71 win 65535 <nop,nop,timestamp 919359579 313492>
09:41:23.780456 IP 72.234.14.89.44444 > 192.168.1.156.52926: P 20:251(231) ack 71 win 65535 <nop,nop,timestamp 919359579 313492>
09:41:23.796075 IP 192.168.1.156.37861 > 192.168.1.152.44445: S 1334070579:1334070579(0) win 5840 <mss 1460,sackOK,timestamp 313496 0,nop,wscale 6>
09:41:23.796328 IP 192.168.1.152.44445 > 192.168.1.156.37861: S 1760846938:1760846938(0) ack 1334070580 win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 919359579 313496,sackOK,eol>
. . . . . . .
You can clearly see that 192.168.1.156 (client i.e., jconsole) is attempting to connect directly to 192.168.1.152 (server) instead of 72.234.14.89 which is a public IP, although the JMX URL is:
service:jmx:rmi://72.234.14.89:44445/jndi/rmi://72.234.14.89:44444/jmxrmi

If I was behind the firewall I would obviously had problems connecting to 192.168.1.152
Fortunately, this one is easy to fix. All you need is to provide additional option on the server side (java.rmi.server.hostname) and add it to the script This option represents the host name string that should be associated with remote stubs for locally created remote objects, in order to allow clients to invoke methods on the remote object:
. . . . . . .
JMX_OPTS=” \
$JMX_OPTS \
-Dcom.sun.management.jmxremote.port=${jmxPort} \
-Djava.rmi.server.hostname=72.234.14.89 \
. . . . . . .
That is all .
Start jconsole: ./jconsole.sh service:jmx:rmi://<pub-ip>:<rmi-port>/jndi/rmi://<pub-ip>:<registry-port>/jmxrmi
Once you modify the script and start the dmServer you should see output similar to this:
. . . . . .
oleg-2:bin olegzhurakousky$ ./startup.sh
com.springsource.rmiregistry.port:44444
com.springsource.rmiserver.port:44445
Getting the platform’s MBean Server
Local Connection URL: service:jmx:rmi://oleg-2.local:44445/jndi/rmi://oleg-2.local:44444/jmxrmi
Public Connection URL: service:jmx:rmi://72.234.14.89:44445/jndi/rmi://72.234.14.89:44444/jmxrmi
Creating RMI connector server
[2009-02-26 18:53:34.031] main                     <SPKB0001I> Server starting.
[2009-02-26 18:53:35.943] main                     <SPOF0001I> OSGi telnet console available on port 2401.
[2009-02-26 18:53:41.558] main                     <SPKE0000I> Boot subsystems installed.
. . . . .

The sample code could be downloaded from here: JMXAgent.java

OSGi and “uses” directive

In current Java model, uniqueness of the class is defined by fully qualified name of the class plus the defining class loader. This mechanism actually creates a problem, where it is quite easy to have multiple versions of the same class loaded in a single JVM space. This happens quite a lot, giving Java’s ability to load classes dynamically. Modern framework such as Spring, Hibernate and others utilize this feature quite a bit. However if not used properly, having multiple instances of class loaded often results in memory leaks and various class type exceptions such as CCE, NCDF, Linkage etc. . .

OSGi approaches this problem rather differently and it does so by bringing the concept of class/packages version into the runtime environment, thus allowing you to still have multiple versions of the same class loaded without creating conflicts.  But it comes with the price. OSGi defines a very strict visibility and class sharing rules that must be explicitly declared, while at the same time moving us a way from the legacy mentality of the linear class path we are all so used to.

The following article article concentrates on one area of OSGi dependency resolution mechanism and that is “uses” directive. . .

To Spring or not to Spring. . .

Just another rant on the whole JEE vs. Spring and specifications in general . . .
Here is how I see it:
JEE(J2EE) is a specification which addresses some areas of concerns derived from the concepts of Enterprise Java.
Spring is an open source framework which also addresses some areas of concerns derived from the concepts of Enterprise Java.
One is managed by a large group of people and vendors, each with their own egos and potentially contradicting agendas, while the other one is managed by a smaller like-minded group of people with two common interests and that is to “make it better then other guy” and “make it quicker then other guy”.

So the real sticking point here is the speed and quality of innovation. In the privately managed framework such as Spring a good idea could be turned into concrete implementation in days (some times hours), while it takes years to accomplish anything in any managed specification (not only JEE). Take Hibernate’s capabilities of today vs. its derived specification such as JPA for example. Or take capabilities of Spring vs. various JEE implementations? You can clearly see how far behind JEE really is.
One might argue that specification gives you portability across platforms. . . well not true!
JEE, although meant to be portable, never has been. JEE application on JBoss is not portable to another platform such as WAS unless you spend some time making it portable. I spent two years of my career migrating apps from WAS and WebLogic to JBoss and it was anything but trivial due to all kinds of proprietary extensions and APIs on either platforms which resulted in a typical vendor lock-in.
Spring on the other hand, is giving you the same functionality as JEE plus much more, but in the detached non-JEE container called Application Context, while providing basic integration of Application Context with standard JEE services (e.g., JTA, DataSources, Messaging etc…), thus making your Spring based application completely portable across all JEE containers (You don’t have WAS version of Spring vs. the same of BEA). Yes that raises another question; If Spring can do all of what JEE does, why do I need JEE in the first place? Well, you don’t. . ., but that is a different discussion 🙂

Going back to the question of portability and innovation; If I can’t achieve portability and innovation, what is the value in following any specification? The way I see it I will always be behind the “other guy”.
Spring and other privately managed frameworks (open source and not) are evolving into conventions that developers understand and feel comfortable with.  And good conventions are just as good if not better then any specification. . .

OSGi and Class.forName(..)

While I was presenting “Intro to OSGi” at the recent Philadelphia JUG, I’ve been asked several questions about java class loader dynamics in OSGI environment.  The following post discusses the implication of using Class.forName(..) in OSGi environment.

Let’s say we have the following code:

Class.forName(”org.bar.Foo”);

Usually the same class loader that loaded the class which contains this code will be used. This class loader will follow standard java class loading delegation mechanism to discover and load the class.

However. . .

In the case of the OSGi, each bundle will have its own class loader and outside of java.* packages, this class loader doesn’t follow standard java class loading delegation mechanism and strictly relies on visibility rules and constraint resolution mechanism defined by OSGi – such as providing Import/Export-Package, Required-Bundle and other manifest headers.

This class loader has two responsibilities

1. Load classes available to the Bundle Space, where Bundle Space = Bundle JAR itself + attached Fragment Bundles
2. Collaborate with class loaders of other bundles to reach all classes that are visible to a bundle based on OSGi visibility rules and contsraint resolution mechanism. In other words it must colaborate with Bundle’s Class Space, where Bundle’s Class Space = Bundle-Space + Import-Pckage + Import-Bundle etc…

The issue with using Class.forName(..) in the OSGi environment starts when an attempt is made to resolve such Bundle’s Class Space. To do that we have to make sure that the target class is in the Bundle’s Class Space, which means that advertising bundle with proper Export-Package: org.bar.Foo declaration must be available and appropriate dependency strategy is provided in the bundle that relies on such dependency (e.g., Import-Package: org.bar.foo or Require-Bundle: Foo).

The work of discovering dependencies and providing appropriate manifest headers is not a difficult task (after all in the conventional application model all dependent JARs are in the class path) and could be performed manually or using automated tools such as maven-bundle-plugin, by recursively evaluating dependencies of your JAR

However, giving dynamic nature of Class.forName(..) and lack of compile time validation, the target class might not be available or even known at the time of manifest creation, thus making such determination impossible. The result are missing Import/Export-Package, Require-Bundle etc…, headers which creates a nasty problem, when target Class is not available to the Bundle’s Class Space resulting in ClassNotFoundException being thrown.

Trip to Japan

Next week (March 13th) I’ll be flying to Japan, to help yet another very large client to migrate from BEA to JBoss and being strong supporter of Open Source I love every chance I get to do something like that. But besides the fact that I’ve never been to Japan and feel rather excited to actually go there, I am starting to think more and more about the fact that what I am actually going to be doing is migrating from proprietary implementation of legacy technology (J2EE) to its open source counterpart. Yes, in the past few years J2EE has become legacy technology especially with products and frameworks led by Spring (full stack), Terracotta, ASF and technologies such as OSGi as well as many others. J2EE (with all its variations) has become outdated and can no longer support demands of the current enterprise. . . And of course I’ll be pitching these thoughts and all my other IT passions while I am in Tokyo, but for now I am simply looking forward to lots of Sushi and whatever else that is cool to do or see while I am there. . .