Posts Tagged maven

Web’s fear and maven

All is well and good on your jetty or tomcat servers, then one of your client want to deploy your application in websphere application server, and trouble begins

 - JNDI lookup for datasources
 - Classloading mess
    - Verbose classloading and parent last
    - Jboss tattletale
    - Cleanup undesired dependencies
        - Maven exclusions
        - Correct scope
        - Maven war plugin : packagingExcludes
        - Patched jar
    - Keep it clean
         - maven-enforcer-plugin and friends
         - Combining Groovy-Sonar-Jenkins

Jndi lookup

If your client plan to use websphere, may be he wants to use the built-in websphere datasource, an implementation collecting various statistics about connection, prepared statement,…
You probably want to keep your jetty/tomcat compliance and if you are in webphere switch to the specific implementation (jndi datasource, jta transaction manager,…)
You can use the spring profiles to lookup your datasource via jndi instead of using dbcp or another datasource implementation.

	<bean id="dataSource" class="org.springframework.jndi.JndiObjectFactoryBean" abstract="false"
		scope="singleton">
		<property name="lookupOnStartup" value="false" />
		<property name="cache" value="true" />
		<property name="proxyInterface" value="javax.sql.DataSource" />
		<property name="expectedType" value="javax.sql.DataSource" />
		<property name="jndiName" value="java:comp/env/jdbc/MyDataSource" />
	</bean>
 

If you plan to use jta transactions, and multiple datasources/queues, don’t forget to use XA transactions or tweak your transactions to avoid mixing access in a single transaction (and design it for possible data loss).
Also reduce the isolation level through the datasource property webSphereDefaultIsolationLevel (the default one is repeatable read).
If you have long running transaction like quartz job, test them extensively.

Classloading mess

We are in 2012… osgi is there since a long time and I’m still struggling with websphere and its bundled xerces.

Verbose classloading and parent last

To diagnose classloading issues like NoClassDefFound, Violation Constraint,… you can enable the verbose classloading.
To minimize the side effect of the bundled jars in websphere you setup the classloader policy of your application and module to parent last.

Jboss tattletale

I know that it’s ironic but this tool developed by JBoss will save you hours of trial and errors.
To audit your web-inf/lib, jboss tattletale is THE tool to identify :
– undesired dependencies like the one bundling javax.** classes
– duplicate jars (often due to maven relocation)
– duplicate classes

	 		<plugin>
				<groupId>org.jboss.tattletale</groupId>
				<artifactId>tattletale-maven</artifactId>
				<version>1.1.2.Final</version>
				<executions>
					<execution>
						<goals>
							<goal>report</goal>
						</goals>
					</execution>
				</executions>
				<configuration>
					<source>./target/mywebapp-${project.version}/WEB-INF/lib</source>
					<destination>./target/reports</destination>
				</configuration>
			</plugin> 
 

Launch mvn clean package
Then take a look at the report, you will perhaps discover duplicates classes like the one from commons-logging and use jcl-over-slf4j

or duplicate quartz jar :

  <groupId>opensymphony</groupId>
  <artifactId>quartz-all</artifactId>
vs
  <groupId>opensymphony</groupId>
  <artifactId>quartz</artifactId>

and many other undesired dependencies.

Cleanup Undesired dependencies

Maven exclusions

Since maven 2.x resolves dependencies transitively, it is possible for unwanted dependencies to be included in your project’s classpath. Projects that you depend on may not have declared their set of dependencies correctly, for example. In order to address this special situation, maven 2.x has incorporated the notion of explicit dependency exclusion. Exclusions are set on a specific dependency in your POM, and are targeted at a specific groupId and artifactId. When you build your project, that artifact will not be added to your project’s classpath by way of the dependency in which the exclusion was declared.

<exclusion>
    <groupId>xstream</groupId>
    <artifactId>xstream</artifactId>
</exclusion>
<exclusion>
    <groupId>com.thoughtworks.xstream</groupId>
    <artifactId>xstream</artifactId>
</exclusion>
 

Correct scope

For example excluding test artifact by specifying the correct scope.

	<dependency>
		<groupId>junit</groupId>
		<artifactId>junit</artifactId> 
                <scope>test</scope>
	</dependency>
 

Exclude jdbc drivers by defining them as provided (idem for your datasource implementation)


				<dependency>
					<groupId>com.ibm.data.db2</groupId>
					<artifactId>db2jcc</artifactId>
                                        <scope>provided</scope>
				</dependency>

 

packagingExcludes

In extreme case… putting exclusions is just too long and boring. Configuring the maven war plugin to exclude the jar can be a faster way but remember that if this dependency breaks something in your application, it’s still there in your unit tests.

			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-war-plugin</artifactId>
				<version>2.1.1</version>
				<configuration>
					<packagingExcludes>WEB-INF/lib/commons-logging-*.jar</packagingExcludes>									
					<warSourceDirectory>WebContent</warSourceDirectory>
				</configuration>
			</plugin>
	
 

Patched jars

Some open source jars bundles multiple times the same classes, for example org.w3c.dom.UserDataHandler is bundled in xom, jaxen and many more.
This interface was also bundled in websphere and two jars in the web-inf/lib, one of them was sealed leading to java.lang.LinkageError: loading constraint violation: loader.
So I removed them from the jar and upload a xom-1.1.patched.jar to the corporate maven repository. It’s really ugly but it’s working.

Keep it clean

maven-enforcer-plugin and friends

Maven provide a default rules to enforce some rules, on[e] of them is for banneddependencies.

But there is another set of rules provided by the pedantic pom enforcer

Have you ever experienced symptoms like headaches, unfocused anger or a feeling of total resignation when looking at a Maven project where everybody adds and changes stuff just as they need? Do people call you a “POM-Nazi” if you show them how to setup proper and well organized projects?

If so, the Pedantic POM Enforcers are absolutely the thing you need!

An you have also an extra rule set @codehaus

Combining Groovy-Sonar-Jenkins

It’s quite easy to create a small groovy script that
– will check the jars in web-inf/lib against a baseline list
– failed the build or if you are less paranoid…
– send a mail to your team,
– or just contribute to a sonar manual measure

Let’s define our baseline, for some jar you want to get noticed if a different version is bundled, for your module you accept any version.
And use this baseline as a whitelist if it’s a different version or if there’s no match then it’s a new dependency -> requires to test a websphere deployment.

xom-1.0.jar
mymodule-.*.jar
...
 

Then create the manual measure in sonar

You can define a manual measure

And now the groovy script to analyze the latest war file and post the manual measure to sonar and send you a mail ūüėČ :


import java.util.zip.ZipFile;

//authenticated post
def postSonarMeasure = { resource,metric,val, sonarhost,token ->
	def script = "resource=${resource}&metric=${metric}&val=${val}&text=fromgroovy&description=fromgroovy";
	println script
	URL url = new URL("${sonarhost}/api/manual_measures?"+script);
	URLConnection conn = url.openConnection();
	conn.setRequestMethod("POST");
	conn.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
	conn.setRequestProperty ("Authorization", "Basic ${token}")
	conn.setDoOutput(true);
	OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
	wr.write(script);
	wr.flush();
	result=  conn.getInputStream().getText()
	println 'metrics created '+result;
	return result
}

def sonar = 'https://continuousbuild.com/sonar'
def mavencoordinate='com.company:mywebapp'
def token = 'myuser:mypassword'.bytes.encodeBase64().toString()

//http://docs.codehaus.org/display/SONAR/Web+Service+API
//curl http://continuousbuild.com/sonar/api/manual_measures?resource=com.company:mywebapp&metric=unverifiedwebinfjars
//http://jira.codehaus.org/browse/SONAR-2966 <not_supported/>  
// 1. define the metrics
// 2. add a measure manually
// 3. launch an analysis
// 4. check the data through api/manual_measures
def postSonarUnverifiedWebInfJars = { value ->
	postSonarMeasure(mavencoordinate,'unverifiedwebinfjars',value,sonar,token)
}

def getActualContentOfWebInfLibFromLastestWar ={
	// find latest war file in target directory
	fileWar = new File("./target").listFiles().findAll(){ it.getName().endsWith('.war')}.sort() { a,b ->
		a.lastModified().compareTo b.lastModified()
	}.getAt(0);
	println "Checking WEB-INF/lib from "+fileWar.canonicalPath;
	//and create actuals with content
	ZipFile file = new ZipFile(fileWar)
	actuals = file.entries().collect { entry -> if (entry.getName().startsWith('WEB-INF/lib/')) return entry.getName().substring('WEB-INF/lib/'.length()) }
	actuals = actuals.findAll {it!=null && !it.equals('')}
	return actuals
}

def getBaseLine = {
	allowed=[]
	new File("./baseline.txt").eachLine { if (!it.trim().isEmpty())allowed.add(it) }
	return 	allowed
}
	actuals =getActualContentOfWebInfLibFromLastestWar();
	allowed =getBaseLine();

	println "************************************ "
	println "actuals "+actuals.size()
	println "allowed "+allowed.size()
	println "************************************ "

	unallowed = [];
	unmatched = [];

	allowedNonMatching = [];
	allowedNonMatching.addAll(allowed);

	actuals.each { actual ->
		ok = allowed.find() { allow ->
			boolean match= (actual =~ '^'+allow)
			if (match) {
				allowedNonMatching.remove(allow)
				println "matching " +actual +" "+ allow
			}
			return match;
		}
		if (ok==null) {
			unallowed.add("unmatched dependencies ! '${actual}' ")
			println "unmatched dependencies ! '${actual}' "
			unmatched.add(actual)
		}
	}
	if (!unallowed.isEmpty() || !allowedNonMatching.isEmpty) {
		def msg =  "The ${project} problem dependencies : \n"+unallowed.join('\n')+" \n add exclusions or adapt baseline.txt check if websphere deployment is ok.\nplease.\n"+actuals.join('\n');
		 ant = new AntBuilder()
		 ant.mail(mailhost:'mysmtp.server.com', subject:"${project} : undesired dependencies detected !" ,tolist:'myaddress@mestachs.com'){
		         from(address:'jenkins@mestachs.com')
		         replyto(address:'myaddress@mestachs.com')
		         message(msg.toString())
		 }

		println msg.toString()
	}
	println "************************************ unused constraint from baseline.txt"
	allowedNonMatching.each {println it}
	println "*************************"
	println "************************************ append content to baseline.txt"
	unmatched.each {println it}
	println "*************************"
	postSonarUnverifiedWebInfJars(unallowed.size())
 

Enable the run of this scripts via maven plugin in a dedicated profile

	
   	            <plugin>
				<groupId>org.codehaus.groovy.maven</groupId>
				<artifactId>gmaven-plugin</artifactId>
				<version>1.0</version>
				<executions>
					<execution>
						<phase>verify</phase>
						<goals>
							<goal>execute</goal>
						</goals>
						<configuration>
							<source>./comparewebinf.groovy</source>
						</configuration>
					</execution>
				</executions>
			</plugin> 	
 

or via jenkins groovy post scripts.

,

1 Comment

Maven Best Practices

TL;DR

Love or hate it… he will stay for a moment.
So let’s apply the best practices to our poms and maven builds.

Make the build reproducible
  - Always specify a version for Maven2 plugins
  - Minimize number of SNASPHOT dependencies
  - Use dependency management section
  - Beware of relocation in maven repo
  - After a dependency modification, double check the produced artifacts
Use and abuse of modules
  - more ‚Äútechnical/layered‚ÄĚ
  - more business oriented
Make the build maintainable
  - Prefer default directory layout
  - Avoid duplication by moving common tags to parent pom
  - Always specify a version of dependencies in a parent pom
  - Use Properties Liberally
  - Minimize the number of Profiles
Make the build portable
  - Don’t commit eclipse and maven artifacts
  - Don't modify pom/artifactsin your "enterprise" repository

Make the build reproducible

Always specify a version for Maven2 plugins

Wrong way

<plugin>
 <groupid>org.apache.maven.plugins</groupid>
 <artifactid>maven-surefire-plugin</artifactid>
</plugin>

Correct way

<plugin>
 <groupid>org.apache.maven.plugins</groupid>
 <artifactid>maven-surefire-plugin</artifactid>
 <version>2.3</version>
 </plugin>

This matters because if you leave out the version, maven2 defaults to LATEST  available.

So this can mean that

  • the result of the plugin can become unpredictable if the implementation of the plugin has changed (new/removed feature, …)
  • your project unknowingly became automatic beta testers for third party plugins if the latest version is a SNAPSHOT

Newer version fo m2clipse will this show as warnings in the console. See also the maven enforcer plugin

Minimize number of SNASPHOT dependencies

It’s strictly unrecommended to use SNAPSHOT¬†version in your project dependencies since it’s never guaranteed that a SNAPSHOT¬†version is available in any repository. This can lead to well-know build errors due to missing dependencies . So for projects that don’t belong to ‘you’… it’s preferable to use a real version.

how to detect if you have a snapshot version

Use dependency management section

Transitive dependencies is great feature… but in the end you want to be sure of the version you are using and shipping to production

Dependency Management allows to consolidate and centralize the management of dependency versions without adding dependencies which are inherited by all children. This is especially useful when you have a set of projects (i.e. more than one) that inherits a common parent.

Another extremely important use case of dependencyManagement is the control of versions of artifacts used in transitive dependencies. This is hard to explain without an example. Luckily, this is illustrated in the documentation.

    
    <dependencyManagement>
        <dependencies>
                 ...

Beware of relocation in maven repo

Relocating an artifact is changing the “maven id” (groupId:artifactId) of a project.

xstream:xstream
com.thoughtworks.xstream:xstream

One common pitfall in maven relocation is having double jars even with correct usages of dependencyManagement.
Use the m2clipse and his dependency hierarchy views, to detect and exclude the undesired artifacts.

After a dependency modification, double check the produced artifacts

A good habit is to double check your war/ear produced after the addition of a new dependency or the upgrade of an existing one.
2 greats to tools to help you in this

Use and abuse of modules

the modules can be more “technical”

<modules>
  <module>mymodule_api</module> <!-- service interface, value objects, exception -->
  <module>mymodule_impl</module> <!-- service & dao implementation -->
  <module>mymodule_web</module> <!-- web components, controllers, templates.. -->
</modules>

or

more business oriented

<modules>
   <module>contract_modules</module>
   <module>finance_modules</module>
   <module>time_modules</module>
   <module>agenda_modules</module>
</modules>
 

Make the build maintainable

 

Prefer default directory layout

this will make the plugin configuration more easy :

src/main/java         Application/Library sources
src/main/resources    Application/Library resources
src/main/webapp       Web application sources (for war packaging)
src/test/java         Test sources
src/test/resources    Test resources

Avoid duplication by moving common tags to parent pom

Do you really want to say that you compile for 1.5 jdk in all your projects ?

    
    <!-- Use Java 1.5 -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-compiler-plugin</artifactId>
       <version>2.3.1</version>
        <configuration>
            <fork>${javacFork}</fork>
            <executable>${javacExecutable}</executable>
            <verbose>${javacVerbose}</verbose>
            <compilerVersion>${jdk.version}</compilerVersion>
             <source>${jdk.version}</source>
            <target>${jdk.version}</target>
        </configuration>
    </plugin>

move this in a parent pom !

for example, in

  corporate_base_pom  ->  app_basecom  ->       app_module1         -> app_sub_modules

    jdk 1.5                    technical                application                     
 enterprise repo             dependencies               dependencies
                             (spring,...)               (app_module2,app_module3,..)

Always specify a version of dependencies in a parent pom

prefer a central place for version definition.
Don’t specify version in a specific sub-modules

Use Properties Liberally

Grouping Dependencies with properties… to avoid copy/pasting the version everywhere.
and help upgrading easily to 3.0.5.RELEASE ūüėČ
And move this to a parent pom : cfr Avoid duplication move to parent pom

<properties>
    <spring.version>3.0.0.RELEASE</spring.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework</groupId>
         <artifactId>spring-webmvc</artifactId>
        <version>${spring.version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
         <artifactId>spring-tx</artifactId>
        <version>${spring.version}</version>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
         <artifactId>spring-jdbc</artifactId>
        <version>${spring.version}</version>
    </dependency>
</dependencies>

Minimize the number of Profiles

Profiles can help a lot… but will inevitability complexify the process.
So don’t use profiles if for example adding a new project with and assembly will do the trick. best practices

  • The build must pass when no profile has been activated
  • Use profiles to adapt to build-time context, not run-time context, and not (with rare exceptions) to produce alternative versions of your artifact

Remark: The support for profiles outside of the POM or the settings.xml has been removed in Maven 3.x.

Make the build portable

don’t commit eclipse and maven artifacts

to make the checkout easier… avoid commiting the following files and directory

.project
.classpath
.settings
.wtpmodules
target

these files are often :

  • referencing local settings like JRE name/path/…
  • specific to a version of plugins (wtp,…)

so let m2clipse handle this and maintain/generate the .project, .classpath,…

see svnignore/cvs ignore

Don’t modify pom/artifacts in your “enterprise” repository

It’s always tempting to fix a pom or a jar in the central repository… don’t do this.
Identify it a special version in your repo ‘artefact-1.0.5.corporatepath’ or manage the correct exclusions.
see log4j example

12 Comments

Take care of your Jenkins

TL;DR

Secure your jenkins

  • ldap
  • shell scripts output
  • user/pwd

Manage diskspace requirement

  • Install nagios checks for diskspace monitoring
  • Install diskusage plugin
  • Discard Old Builds
  • Disable maven artefact archiving

Improve supportability

  • maven -e option
  • install job config history plugin
  • use template project plugin
  • groovy system scripts

TL

Secure your jenkins

configure jenkins to use your ldap or active directory. by default jenkins is really open… even “manage jenkins” is available to anonymous user.

you can disable logging of shell commands via

set +o interactive-comments
set +o xtrace

there’s a plugin where you can centralize your user/password and that will mask them in the console logs

Manage diskspace requirement

Installl nagios checks for diskspace monitoring

Install diskusage plugin to gain visibility over the big consumer

Discard Old Buils

enable one of the 2 options :

Days to keep builds
Max # of builds to keep

Disable maven artefact archiving

This option will tell jenkins to collect pom,jars,wars,ears as they are produced by maven. This is rarely usefull when you use an enterprise repository. This option is enabled by default… so you aren’t using it… disable it !

Build > Advanced > Disable automatic artifact archiving

Improve supportability
specify maven -e option
get detailed error from maven

install job config history plugin
knowing that something has changed in the project configuration is always good when something goes bad

use template project plugin
you can reuse builder, publisher from other projects.

get to know system groovy scripts. With a simple script you detect/fix the various highlighted issues in this post.

,

1 Comment