Yesterday I had a heck of a time trying to figure out my my JSPs weren’t passing validation since I upgraded Ganymede from 3.4.0 to 3.4.1. There was a consistent issue where the JSP Validation did not like my scriptlet code, even when I dumbed it down to try even the simplest scriptlet. The validation errors looked like:

Syntax error on token "}", delete this token
Syntax error on token "catch", Identifier expected
Syntax error, insert "Finally" to complete TryStatement

I actually found out what my problem was through the Eclipse Web Tools usergroup. The issue for me was the use of the Spring form custom tag library. If you self-close the tag…

<form:errors path="*" />

…then you get the goofy JSP validation error. If you close the tag as if there was body content…

<form:errors path="*"></form:errors>

Then the error goes away. I need to follow-up with a bug for the Ganymede team.

Sometimes an oversight can lead to interesting results. We were testing a servlet-based application and in our test scripts, we had a hitch where if the script encountered an error, the re-try cycle didn’t have think time included. Which means anytime the script encountered an application error, it would then send re-try requests at a sub-second (inhuman) rate.

This increased our load from 10 pages per second to over 100 … our first indicator that something was wrong. We saw a corresponding spike in CPU from 8% to over 90% as the single JVM system tried to handle this new load. The application seized and thread dumps uncovered an interesting side effect. (more…)

Some development teams I work with have been reporting an issue where the JVM runs out of memory during deployment of an application using our standard heap size settings (initial 2 GB and maximum 2 GB). When they changed the initial heap size from 2 GB to 512 MB, the no longer receive the OutOfMemory error. To understand what’s happening, continue on. (more…)

A colleague and I were having trouble tracking down the root cause of a production issue recently and we learned an interesting trick involving the JIT compiler. The issue was that after a certain period of time the JVM would go from 60 second GC intervals to less than 2 second intervals, coupled with a spike in CPU usage from 3% to 30% (light load). There was no correlation found in the application logs, but we could show that the reduction in interval was occurring at the same time as the increase in CPU usage.
(more…)

The Server Side reports that Ounce Labs discovered two vulnerabilities in the Spring Framework’s MVC component.

SpringSource, the company behind Spring, issued a security advisory as a result. The two issues are entitled Data Submission to Non-Editable Fields and ModelView Injection.

Pass along the word to your friends and co-workers.

I saw this in a friend’s batch of interview questions and thought it was amusing.

public class DividingThatIsNotDividing {
    public static void main(String[] args) {
        int dividend = args[0];
        int divisor = args[1];
        divide(dividend, divisor);
    }
    /**
     * Subtract the divisor from the dividend and keep track of the number of times
     * it happens to find the quotient.
     * @param dividend
     * @param divisor
     */
    public static void divide(int dividend, int divisor) {
        int quotient = 0;
        int remainder = dividend;
        while(remainder >= divisor) {
            quotient++;
            remainder -= divisor;
        }
        System.out.println(dividend + " divided by " + divisor + " equals " +
            quotient + " and " + remainder + "/" + divisor);
        }
}

One of the most common operations in the JSTL framework is the <c:out> tag:

<c:out value="${name}" />

In applications that use this tag heavily, there is a high memory cost when you do not specifying the default attribute. The cost occurs when a NullAttributeException is thrown by the JSTL framework when an attribute is missing while evaluating the tag attributes. The creation and try/catch of the NullAttributeException uses 1,250 bytes of memory per tag instance, all of which can be avoided by simply providing an empty default:

<c:out value="${name}" default="" />

Although 1,250 bytes may seem fairly innocuous, imagine the costs magnified in the example of a multi-row, multi-cell table with potentially hundreds of renderings. Then you’re talking about serious memory churn for your transaction. So if you are using <c:out> tags in a exponential capacity (i.e. nested in a <c:foreach> tag), then be sure to implement a default, even if it is an empty space.

You might ask “But then why don’t I have to implement the escapeXml attribute?”

The answer is because the escapeXml attribute requires a boolean value. The evaluateExpressions() method of the OutTag code does not have to check for null because the when the value is evaluated it is converted to a Boolean object. The Boolean is then evaluated and cannot result in a NullAttributeException.

A colleague of mine recently asked me about a past performance issue I worked on revolving around the use of “body content” in custom tag libraries. Body content is best described in by the following examples:

Tag with Body Content:

<myTag:foo name="name1">Some body content...<myTag:foo/>

Tag without Body Content:

<myTag:foo name="name1" />

When you design a tag to manipulate that body content, you end up internally using the class org.apache.jasper.runtime.BodyContentImpl. This is the source of the performance concern because this class initializes a char array to a length of 512 each time it is instantiated [source: jasper-runtime.jar in Tomcat 5.5]. Then it contains a growth strategy for when your body content expands outside of that container.

The problem comes when you inadvertently support body content when your usage doesn’t need it to. In one application that makes heavy use of custom tags, I saw a small percentage of instances where the tag was used with body content. On some pages, there were 30-40 tag instances, all of which were not using any body content. Which meant most of the time we were burning that 512 character array for no good reason.

Now 40 instances of wasting 1,024 bytes (512 x 2 bytes per char) on a page might not be a big deal, but in older application servers (WebSphere 5.0, for example), this character array buffer size was set to 8,192, which meant over 16kb per tag instance was being wasted. In an application that heavily uses tags, this was a huge waste of memory in the JVM. Not to mention if these tags aren’t pooled, then you’re creating these wasted blocks of memory every time the page is executed.

The solution in general is to evaluate the usage of your tag. If your tag needs body content support, but most of the instances do not actually supply that content, consider splitting the code into two tags: one that supports body content and one that does not. Another approach was to attempt to cache some of the tags using application server settings, but I have some details on that I will share some other time.

My colleague Phil gave me a simple but effective breakdown of the difference between these two settings in WebSphere Application Server. Here he describes the shared setting:

Shareable tells the WAS container, when a servlet transport thread closes a connection, don’t actually put that connection back in the pool. Instead, keep it in some local context so that the thread can get access to it when it asks again during the processing of this request. Then finally, when the request is completed, then put the connection back in the pool.

Shareable is an optimization. You would use this for connections which you know will be used many times during processing of a single request. WPS database connections are a good example. We always recommend the number of WPS portal database connections equal the number of web container thread pool size to avoid queuing for portal database connections. I’ve never seen an application database connection that would need to gain from using shareable connections.

The downside of shareable is that if you use it when you don’t need it, you need a larger pool of connections than you would with unshareable connections in order to avoid a connection bottleneck since each thread is holding a connection for a longer period of time.

He goes on to describe the unshared setting:

Unshareable has the opposite behavior. When a thread closes a connection, the thread puts the connection back in the pool.

As an aside if you don’t have a resource-ref defined with shareable/unshareable property and/or don’t use this ref name in the code - you will get shareable by default - with a warning in wps.err/out. In this circumstance your pool size may be causing queuing if the size of the connection pool is less than the size of the thread pool. We suggest to use a resource-ref and set this to unshareable for applications.

While profiling a transaction in an application at work, I found this trace to be interesting:

Trace 6 | total: 275,780 | total: 276,320
identical: 45 | average: 6,128 | average: 6,140
percent: 0.32% | percent: 1.41% | percent: 1.41%
--------------------+--------------------------+--------------------------
com/ibm/jvm/ExtendedSystem,resizeArray
java/lang/StringBuffer,expandCapacity
java/lang/StringBuffer,append
com/ibm/ws/classloader/CompoundClassLoader,toString
java/lang/String,valueOf
java/lang/StringBuffer,append
org/apache/xerces/util/ObjectFactory,newInstance


Some code in the Xerces jar looks to be leveraging the toString() method on the ClassLoader implementation, which in WAS 5.1 is creating a StringBuffer full of class names. Looking at the newInstance() method in the Xerces ObjectFactory class reveals the code that invokes the CompoundClassLoader.toString():

debugPrintln("created new instance of " + class1 + " using ClassLoader: " + classloader);

The debug doesn’t check for any debug flag before doing the expensive String concatenation that results in our expensive CompoundClassLoader.toString() call. So every time we use this factory to get a new instance of a class, we’re generating a new StringBuffer of all the classes loaded by the application.

Further problems arise when one goes to actually look at the debugPrintln() method’s implementation.

/** Set to true for debugging */
private static final boolean DEBUG = false;
 
/** Prints a message to standard error if debugging is enabled. */
private static void debugPrintln(String msg) {
    if (DEBUG) {
        System.err.println("JAXP: " + msg);
    }
} // debugPrintln(String)

There’s no way to ever set DEBUG to true at runtime, so this is purely for the Xerces developers. This class concatenates Strings all over the ObjectFactory class to send them to a method that depends on a hard-coded class variables to operate. This is not a good practice.

For the one transaction in this application, there are 13 stack traces that report this issue (@ a 100 byte threshold) for a total of 829,920 bytes. A significant waste of memory in my estimation. Note that your mileage may very as this number is directly related to the number of classes loaded and the frequency of newInstance() calls in your application.

I checked the following version of Xerces and they all have the same unchecked debug code:

• Xerxes-2.4.0.jar
• Xerxes-2.5.0.jar
• Xerxes-2.6.0.jar