Oleg’s Weblog

My Tech Rant

Archive for the ‘JMX’ Category

JMX connectivity through the firewall

Posted by Oleg Zhurakousky on March 23, 2009

Recently I’ve been asked to help out a customer who was having issues with JMX connectivity to Spring Source dmServer through the firewall. However, one thing I want to point out right up front is that the issue is rather generic and has nothing to do with dmServer. It is really about understanding JMX, RMI and proper configuration. But I will use dmServer and its configuration as an example.
Here is the sample JMX configuration options provided in the dmServer startup script:
-Dcom.sun.management.jmxremote.port=${jmxPort} \
-Dcom.sun.management.jmxremote.authenticate \
-Dcom.sun.management.jmxremote.password.file=${jmxUsersPath} \
-Djavax.net.ssl.keyStore=${keystorePath} \
-Djavax.net.ssl.keyStorePassword=${keystorePassword} \
-Dcom.sun.management.jmxremote.ssl=true \
-Dcom.sun.management.jmxremote.ssl.need.client.auth=false”

This will enable JMX agent (MBean Server) when you start dmServer. Once started you can now monitor your process via JMX-compliant tool such as jconsole. Connectivity could be local or remote.
The above configuration seem to provide everything we need to access this process through the firewall, since com.sun.management.jmxremote.port is obviously the port that we need to open in the firewall. However there is a caveat.
Once connected to JMXRegistry running on the port specified by com.sun.management.jmxremote.port property, the actual objects are served by RMIServer which is running on different port. Unfortunately this port is chosen randomly by default instance of JMX Agent and there is no –D option to specify it. Obviously going through the firewall would require opening up two ports and with random port it presents a delicate issue.
Fortunately it is easily solvable by writing a custom Java Agent where you can programmatically specify each port and externalize it through custom properties (I am attaching sample code).
More info here: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
In the nutshell, the custom agent will take the port value provided by the com.sun.management.jmxremote.port property and will create a second port (RMIServer port) by incrementing it by 1. (in our case the port specified is 44444 which makes RMIServer port 44445)
Once such agent is in place (JAR) and the appropriate ports are open in the firewall all you need is modify the startup script to include –javaagent option providing the JAR.
. . . . .
$JAVA_HOME/bin/java \
-javaagent:/Users/olegzhurakousky/../../../dmserver.jmx-0.0.1-SNAPSHOT.jar
$JAVA_OPTS \
. . . . .

Well, that really only solved one half of the problem, since by default RMI stubs sent to the client contain the server’s private address instead of the public

Just look at this tcpdump fragment while monitoring the client’s access (jconsole running on the local network):
. . . . . . .
09:41:23.778663 IP 72.234.14.89.44444 > 192.168.1.156.52926: . ack 71 win 65535 <nop,nop,timestamp 919359579 313492>
09:41:23.779958 IP 192.168.1.152.44444 > 72.234.14.89.52926: P 20:251(231) ack 71 win 65535 <nop,nop,timestamp 919359579 313492>
09:41:23.780456 IP 72.234.14.89.44444 > 192.168.1.156.52926: P 20:251(231) ack 71 win 65535 <nop,nop,timestamp 919359579 313492>
09:41:23.796075 IP 192.168.1.156.37861 > 192.168.1.152.44445: S 1334070579:1334070579(0) win 5840 <mss 1460,sackOK,timestamp 313496 0,nop,wscale 6>
09:41:23.796328 IP 192.168.1.152.44445 > 192.168.1.156.37861: S 1760846938:1760846938(0) ack 1334070580 win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 919359579 313496,sackOK,eol>

. . . . . . .

You can clearly see that 192.168.1.156 (client i.e., jconsole) is attempting to connect directly to 192.168.1.152 (server) instead of 72.234.14.89 which is a public IP, although the JMX URL is:
service:jmx:rmi://72.234.14.89:44445/jndi/rmi://72.234.14.89:44444/jmxrmi

If I was behind the firewall I would obviously had problems connecting to 192.168.1.152
Fortunately, this one is easy to fix. All you need is to provide additional option on the server side (java.rmi.server.hostname) and add it to the script This option represents the host name string that should be associated with remote stubs for locally created remote objects, in order to allow clients to invoke methods on the remote object:
. . . . . . .
JMX_OPTS=” \
$JMX_OPTS \
-Dcom.sun.management.jmxremote.port=${jmxPort} \
-Djava.rmi.server.hostname=72.234.14.89 \
. . . . . . .

That is all .
Start jconsole: ./jconsole.sh service:jmx:rmi://<pub-ip>:<rmi-port>/jndi/rmi://<pub-ip>:<registry-port>/jmxrmi
Once you modify the script and start the dmServer you should see output similar to this:
. . . . . .
oleg-2:bin olegzhurakousky$ ./startup.sh
com.springsource.rmiregistry.port:44444
com.springsource.rmiserver.port:44445
Getting the platform’s MBean Server
Local Connection URL: service:jmx:rmi://oleg-2.local:44445/jndi/rmi://oleg-2.local:44444/jmxrmi
Public Connection URL: service:jmx:rmi://72.234.14.89:44445/jndi/rmi://72.234.14.89:44444/jmxrmi
Creating RMI connector server
[2009-02-26 18:53:34.031] main                     <SPKB0001I> Server starting.
[2009-02-26 18:53:35.943] main                     <SPOF0001I> OSGi telnet console available on port 2401.
[2009-02-26 18:53:41.558] main                     <SPKE0000I> Boot subsystems installed.
. . . . .

The sample code could be downloaded from here: JMXAgent.java

Advertisements

Posted in JMX, osgi, Spring | Tagged: , | 3 Comments »