Syntax error on token "}", delete this token
Syntax error on token "catch", Identifier expected
Syntax error, insert "Finally" to complete TryStatement

I actually found out what my problem was through the Eclipse Web Tools usergroup. The issue for me was the use of the Spring form custom tag library. If you self-close the tag…

<form:errors path="*" />

…then you get the goofy JSP validation error. If you close the tag as if there was body content…

<form:errors path="*"></form:errors>

Then the error goes away. I need to follow-up with a bug for the Ganymede team.

/**
 * @version svn:$Id$
 */

After the commit, the file did not update as expected. I did some further digging and found a forum post that discussed another option where you set the property on the file. So I experimented with that using the Team > Set Property … command in eclipse on a single file. Setting the property name to svn:keywords and value to Id. I made a simple change to my file and committed it again. Voila! It worked.

The last step was to apply this keyword to all of my files…and preferably not manually. The answer was to go to the root of my project and set the property there, noting to check the Set property recursively check box. If I want to add more keywords later on, I can by going to the root, modifying the svn:keywords property by adding other keywords (refer to the manual again) separated by a single space.

- Seattle’s players hate Ichiro.
- Seattle won’t want to lose their marquee player without getting a marquee-level player in return.
- Seattle is in flux and would be interested in making a deal.
- Their contracts are similar: Ichiro’s is 09:$17M, 10:$17M, 11:$17M, 12:$17M and Soriano’s is 09:$16M, 10-14:$18M. This is where the Cubs might have to add value to a deal since Ichiro is cheaper, albeit older by 2 years, than Soriano. Throw in Guzman or Samardzija. Neither projects to more than relief or 4th/5th starter roles.
- Cubs would get their lead-off hitter (career .377 OBP)
- Cubs would break-up the Soriano-Ramirez-Lee righty tri-fecta which has two years of post-season stank on them.
- Hoffpauir could play LF and platoon with DeRosa & Fontenot (cycling DeRosa back and forth from 2B to LF).
- Fukudome would have a countryman to play with in the OF. He seemed overwhelmed and lost in the second half and getting a fellow Japanese player could do wonders for his play in ‘09.

Or…trade Lee instead of Soriano…you still get all the same benefits and Seattle might even like this more as Lee is more of a team guy and costs less than Soriano. Talk this up…I think this is a realistic move that could actually give Cub fans hope in 2009.

Cotts - contract up - re-sign
Fox - contract up - let go
Gaudin - contract up - re-sign
Guzman - contract up - re-sign
Howry - contract up - let go
Lieber - contract up - let go
Marmol - contract up - re-sign
Marshall - no data available - re-sign … take Harden’s spot in rotation
Samardzija - 5y/10m til 2011
Wood - contract up - re-sign
Dempster - contract up - re-sign
Harden - contract up - 2009 option @ 7m - decline
Lilly - 4y/40m til 2010
Marquis - 3y/21m through 2009
Weurtz - contract up
Zambrano - 5y/91.5m til 2012

Blanco - 2009 option @ 3m - accept
K.Hill - contract up - let go
Soto - contract up - re-sign

Cedeno - contract up - re-sign
DeRosa - 3y/13m through 2009
Fontenot - contract up - re-sign
Hoffpauir - contract up - re-sign
Lee - 5y/65m til 2010
Ramirez - 5y/75m til 2011
Theriot - contract up - re-sign

Edmonds - contract up … replace with FA or Pie … keep if can’t find good lefty replacement
Fukudome - 4yr/48m til 2011
Johnson - contract up - re-sign
Pie - contract up - re-sign
Soriano - 8y/136m til 2014
Ward - contract is up … replace with Hoffpauir

Contract data courtesy of Cot’s Contracts.

This increased our load from 10 pages per second to over 100 … our first indicator that something was wrong. We saw a corresponding spike in CPU from 8% to over 90% as the single JVM system tried to handle this new load. The application seized and thread dumps uncovered an interesting side effect. Forty-eight of the 65 allocated web container threads were waiting to lock on the synchronized method getEvaluatorByName() in ExpressionEvaluatorManager, a class provided by the Java Standard Tag Library. Here’s a sample from the thread dump:

"WebContainer : 49" daemon prio=5 tid=0x04a86830 nid=0xa1 waiting for monitor entry [0x59a7d000..0x59a7fc28]
    at org.apache.taglibs.standard.lang.support.ExpressionEvaluatorManager.getEvaluatorByName(Unknown Source)
    - waiting to lock <0xeb30d958> (a java.lang.Class)
    at org.apache.taglibs.standard.lang.support.ExpressionEvaluatorManager.evaluate(Unknown Source)

Looking at the source code, you can see the Evaluator retrieval and lazy instantiation of the Evaluator cache is done within this synchronized method. The intent was to prevent duplicate Evaluators from entering the cache, but the side effect of this approach is that every “get” is synchronized.

Synchronization Blocking – standard-1.0.3.jar

public static synchronized ExpressionEvaluator getEvaluatorByName(String name) throws JspException {
		try {
			Object oEvaluator = nameMap.get(name);
			if (oEvaluator == null) {
				ExpressionEvaluator e = (ExpressionEvaluator)
				Class.forName(name).newInstance();
				nameMap.put(name, e);
				return (e);
			} else
				return ((ExpressionEvaluator) oEvaluator);
		} catch (ClassCastException ex) {
			// just to display a better error message
			throw new JspException("invalid ExpressionEvaluator: " +
			ex.getMessage(), ex);
		} catch (ClassNotFoundException ex) {
			throw new JspException("couldn't find ExpressionEvaluator: " +
			ex.getMessage(), ex);
		} catch (IllegalAccessException ex) {
			throw new JspException("couldn't access ExpressionEvaluator: " +
			ex.getMessage(), ex);
		} catch (InstantiationException ex) {
			throw new JspException("couldn't instantiate ExpressionEvaluator: " + ex.getMessage(), ex);
		}
	}

A better approach would be to move the synchronization “lower” in the chain by removing the lock at the method level and creating a static block around the lazy instantiation of the cache. The project owners recognized this as a bottleneck in 1.0.6 and released the following revised version of the same method.

Fixed – standard-1.0.6.jar

public static ExpressionEvaluator getEvaluatorByName(String name) throws JspException {
	Object oEvaluator = nameMap.get(name);
	if (oEvaluator != null) {
		return ((ExpressionEvaluator) oEvaluator);
	}
	try {
		synchronized (nameMap) {
			oEvaluator = nameMap.get(name);
			if (oEvaluator != null) {
				return ((ExpressionEvaluator) oEvaluator);
			}
			ExpressionEvaluator e = (ExpressionEvaluator)
			Class.forName(name).newInstance();
			nameMap.put(name, e);
			return (e);
		}
	} catch (ClassCastException ex) {
		// just to display a better error message
		throw new JspException("invalid ExpressionEvaluator: " + ex.getMessage(), ex);
	} catch (ClassNotFoundException ex) {
		throw new JspException("couldn't find ExpressionEvaluator: " + ex.getMessage(), ex);
	} catch (IllegalAccessException ex) {
		throw new JspException("couldn't access ExpressionEvaluator: " + ex.getMessage(), ex);
	} catch (InstantiationException ex) {
		throw new JspException("couldn't instantiate ExpressionEvaluator: " + ex.getMessage(), ex);
	}
}

This code solution reduces contention to almost nothing, as there is only one Evaluator defined (hard-coded, acutally) in the out-of-the-box version of the standard tag libraries. In conclusion, it’s always a good idea to absorb updates in the thirty party libraries you are using! Most Java applications won’t see this issue unless they have extremely heavy traffic or are using automated load or procedures to churn through their site.

I also enjoyed John Resig’s take on where this takes browsers and the web in general.

Alas, my first task was to find XCode, the free IDE from Apple for developing anything Mac-related.  My Mac-savy uncle told me that XCode comes with OS X so there’s nothing to buy.  True words, though, after combing through the Finder list, I was unsuccessful in finding the darn thing.  Then it dawned on me that XCode may not be a default installation option and after a quick trip to the Apple Developer Center (ADC) I found out why.  XCode is a weighty 1+ GB depending on the version.

Version.  Hmm … the download page for XCode 3.1 mentioned that its built for “Intel or PowerPC Mac running Leopard.”  Well nuts.  Even being a Mac newbie, I remember that my Mom told me that her MacBook is on Tiger and not running Leopard yet because she’s in the middle of an iMovie/iPhoto project and heard there were changes moving from 10.4 to 10.5 in that software.  Thwarted in my first attempt, I would not be denied.  I quickly googled for “XCode on 10.4″ and learned that the ideal version of XCode for Tiger is 2.4.  Googling for XCode 2.4 came up with a couple links to XCode (I ignored these, the URIs seemed generic and they just pointed to the latest XCode downloads) and a legacy link to the XCode 2.4 download.  Huzzah!

I downloaded the disk image (entertained by the DMG extension; pronouncing it as “damage”) and when its done I’ll install and finally get to work!  I only have an hour this morning, so I’ll be installing and starting my development poking and prodding later today.

Issue Details
The garbage collection logs tell the story.

<AF[32]: Allocation Failure. need 67108880 bytes, 297 ms since last AF>
<AF[32]: managing allocation failure, action=2 (1727452832/2147482112)>
19:41:58.409 0x80e573c 0046F0   -   GC(VFY-SUM): pinned=33731(classes=32384/freeclasses=0) dosed=5180 movable=6205831 free=3549
19:41:58.409 0x80e573c 0046F1   -   GC(VFY-SUM): freeblocks: max=34664504 ave=491688 (1745002528/3549)
19:42:00.241 0x80e573c 0046F7   -   GC(VFYAC-SUM): freeblocks: max=53984992 ave=1500426 (1744996176/1163)
<GC(36): mark stack overflow[72]>
<GC(36): GC cycle started Wed Jun 25 19:42:00 2008
<GC(36): freed 17541296 bytes, 81% free (1744994128/2147482112), in 2620 ms>
<GC(36): mark: 399 ms, sweep: 25 ms, compact: 2196 ms>
<GC(36): refs: soft 309 (age >= 32), weak 0, final 9, phantom 0>
<GC(36): moved 1180447 objects, 67685368 bytes, reason=1>
<AF[32]: managing allocation failure, action=3 (1744994128/2147482112)>
<AF[32]: managing allocation failure, action=4 (1744994128/2147482112)>
<AF[32]: clearing all remaining soft refs>

The JVM requests a block of 67,108,880 bytes and the heap reports that it’s largest contiguous block of free memory is 53,984,992 bytes. This allocation failure forces the JVM to report that the heap is out of memory, even though the total space indicates 81% free.

Generally the scenario of the deployment is that the JVM has been up for a period of time (days, weeks) and during the time, data is written all over the 2 GB allocated. After this period of time, logic tells us that the heap’s largest free, contiguous block of memory is at it’s smallest size. Knowing this, it makes it a less than desirable time to try to write large blocks of memory to the heap. Chances are the blocks aren’t large enough after being chopped and used over and over.

Solution

The change in heap settings is a valid work-around for this issue.

By reducing the initial heap size, we are keeping a chunk of memory pristine and unused. This JVM has an action it can take when it encounters the allocation failure scenario we described in figure 2. Rather than declare the heap out of memory, the JVM can try to expand the heap in order to write the requested block of 67 MB.

The total amount expanded depends on an algorithm, so the example 100 MB here is for illustration purposes only. The allocation failure may even trigger multiple expansions, should the algorithm produce an expansion number less than required by the size of the requested block. Once the expansion can handle the requested size, the allocation can occur.

Summary

The deployment team has two options to potentially avoid this issue:

1. Change the initial heap size from 2 GB to 512 MB or 1 GB and restart the JVM.
2. Restart the JVM prior to deploying, in order to clear the heap (not guaranteed since the JVM could use a funky allocation strategy for the objects generated during server start-up and application allocation).

The application team should analyze the contents of their deployment files to ensure only the bare minimum amount of files are deployed in order to keep the deployment file sizes as low as possible. Common file clutter includes source control files (.pj files from MKS), source code (.java files) in .jar files and project documentation.