-
Notifications
You must be signed in to change notification settings - Fork 3
User Guide
- Running Tests
- Writing Tests
- Resources
The harness can be used to test different projects that use the JAX-WS RI. Normally, the harness will be able to automatically guess which project you are working on, and set up classpath automatically. However, you can always use one of the following options to manually specify it to override this default behavior.
The following option tells the harness to pick up the JAX-WS RI classes from the JAX-WS workspace.
Specifically, with this option, the harness will look into various build/classes and library jars.
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-cp:jaxws,path/to/jaxws-ri-workspace,test-dir1,test-dir2,...The following option tells the harness to pick up the RI classes from a JAX-WS binary dist image. Using this option,
the harness will only look at the RI library jars found under the lib directory of the top-level dist image directory
that you specify.
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-cp:jaxws-image,path/to/jaxws-ri-image,test-dir1,test-dir2,...The following option tells the harness to pick up the WSIT classes from a WSIT workspace. Specifically, with this option,
the harness will look into various build/classes and library jars.
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-cp:wsit,path/to/wsit-workspace,test-dir1,test-dir2,...The following option tells the harness to pick up the WSIT classes from a binary dist image. Using this option, the harness
will only look at the WSIT library jars (webservices.jar & webservices-tools.jar) found under the lib directory
of the dist image directory that you specify.
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-cp:wsit-image,path/to/wsit-image,test-dir1,test-dir2,...Each test is packaged in a directory. The following command runs a single test stored in testdata/abc/def:
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=testdata/abc/defIf you specify multiple directories, all the tests in the specified directories are run.
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=testdata/abc/*The -r option tells the harness to recursively search for all tests inside the given directory:
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-r,testdata/addressing|
Tip
|
Consider defining an alias to |
The following command executes a test with debugger support enabled. Run this command from a console, then set up a remote debugging with your IDE to port 8000. This allows you to set breakpoints anywhere both on the client and the server.
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dws.jvmOpts='-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000' -Dargs=testdata/abc/def|
Tip
|
You’ll use this command very often, so consider defining an alias to
|
If your test data resides in the Tango workspace or the JAX-WS workspace, the harness recognizes their directory layouts
and pick up classes from respective build/class directories and libraries. So if you make a change to the source code,
all you need to do is to compile them to class files.
|
Tip
|
Do not run any targets that generate jar files, as those unnecessary work will slow you down. |
If your change to the code is to the runtime and does not affect the code generation, then use the -skip option to skip
the code generation. This allows you to quickly rerun the runtime portion of the tests without wasting time running
all the tools just to produce the same results.
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-skip,testdata/abc/defThe test harness leaves all the artifacts generated during a test execution inside a working directory called work
in each test directory. Sometimes inspecting what’s generated becomes important, so this is the directory
structure of the work directory:
work/ (root test directory)
+- client-source/ (client Java artifacts generated by wsimport from service WSDL)
+- client-classes/ (compiled client Java artifacts. Generated from client-source by Javac)
+- services
+- <serviceName> (this intermediate directory is only if you have multiple services)
+- war (root of the exploded war file image that's deployed to the server)
+- gen-src (generated server Java artifacts by wsimport)Unless the -skip option is used, the work directory is "rm -rf"'ed before a test execution.
There’s no need to manually remove them.
The harness can run tests by using a variety of transports and containers.
By default, the test harness uses the in-vm transport. There’s no additional option required for this mode. This is the preferred mode of testing because of its quick turn-around time. This mode also allows you to use a single debug session to debug both the client and the server side.
Some delivery vehicles (notably the distribution images) do not have the code needed to use the in-vm transport.
To test those, find the matching jaxws-local-transport.jar and use the -transport option to specify it.
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-cp:wsit-image,path/to/wsit,-transport,path/to/jaxws-local-transport.jar,...This test harness supports Tomcat 5.0.x and newer. There are a few different ways of using Tomcat.
Applies to: Tomcat 5.0.x and newer
The following command causes the harness to run Tomcat within the same VM. This mode is bit like the local transport, in the sense that both the server and the client runs inside the same VM, but this mode run tests by using HTTP and full servlet container.
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-tomcat-embedded,path/to/tomcat,test-dir1,test-dir2,...path/to/tomcat should point to the directory where you have Tomcat installed. The harness will load Tomcat jar files
from there.
This mode is preferable mode of testing for Tomcat, as this provides the quickest turn-around time, and worry-free. This mode is the only mode in Tomcat where you can use a single debug session to debug both the client and the server.
Applies to: Tomcat 5.0.x and newer
The following command causes tests to run by using Tomcat launched externally. The parameter to the -tomcat-remote option
is adminUserName:adminPassword@hostname:port.
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-tomcat-remote,admin:admin@localhost:8080,test-dir1,test-dir2,...It is your responsibility to launch Tomcat. In this mode, the harness will only deploy and undeploy services from
this running instance. The tomcat used for this purpose should NOT have JAX-WS/Metro binaries installed in common/lib
or anywhere else. The harness will include everything necessary into a war file, and this is the only way to make sure
that you are testing the bits that you are supposed to be testing.
|
Tip
|
To find out the admin username and password, look at <role rolename="manager"/>
<user username="admin" password="admin" roles="manager"/> |
|
Tip
|
To debug the server side in this set up, you need to launch Tomcat with the debugger support enabled.
Run |
|
Tip
|
(Tomcat 5.0.x only) By default, Tomcat is configured to output logs to multiple places, including files
in the <Logger className="org.apache.catalina.logger.SystemOutLogger" /> |
|
Tip
|
When tests are aborted via Ctrl+C, web applications may be left running on this remote machine. When the test is run next time, they will be removed (or you can remove them manually.) |
Applies to: Tomcat 5.0.x and newer
Run the following command to use Tomcat in the local mode. In this mode, the haress launches Tomcat as a child process, run tests, and terminates Tomcat when everything is done.
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-tomcat-local,path/to/tomcat,test-dir1,test-dir2,...See the tips for the remote tomcat support for how to configure Tomcat to report logs to one place.
|
Tip
|
On Windows systems, aborting the harness via Ctrl+C will leave the child Tomcat process running (does anyone know how to fix that?) Those orphan processes can be killed from task manager (be careful not to kill other Java processes), or better yet, process explorer. |
This test harness supports Jetty 6.x.
The following command causes the harness to run Jetty within the same VM. See the embedded tomcat description for more about what it means to embed a container.
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-jetty-embedded,path/to/jetty-6.1.0,test-dir1,test-dir2,...This test harness supports GlassFish 1.0.
To test with existing instance of GlassFish, use the following option. This mode assumes that you have already started a domain, and the harness will simply deploy/undeploy applications to this running service:
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-glassfish-remote,localhost,test-dir1,test-dir2,...The "-glassfish-remote" takes the connection string as the parameter. The full syntax of the paramter is
[USER:PASS@]HOST[:PORT][-HTTPURL]. The first four parts up to "-" specifies the GlassFish admin port.
USER and PASS is used for the administrator login, and PORT is the admin TCP port.
The last HTTPURL specifies the URL with which client test programs talk to the service.
All components but HOST are defaulted, so if your glassfish installation is out of the box, then all you need to
specify is the host name where GlassFish runs.
|
Tip
|
GlassFish has its copy of JAX-WS, so testing with GlassFish requires us to replace the runtime of GlassFish. At this point the harness is not capable of doing that. |
The test harness is capable of starting and stopping GlassFish on its own before/after running tests. This mode assumes that you have GlassFish installed locally. Use the following command line option to run in this mode:
$ mvn -U org.glassfish.metro:harness-maven-plugin:ws-test -Dargs=-glassfish-local,path/to/glassfish,test-dir1,test-dir2,...The harness will create a domain configuration in a temporary directory and run tests from there, so this running mode
will not touch the files in path/to/glassfish --- it is simply used to load jar files. All the TCP ports
are chosen randomly from 20000-50000 ranges, to avoid collisions with other tests that happen to be running on the same machine.
See the console output for the HTTP port and admin port, in case you need to manually access them for debugging.
The admin user name and password will be set to "admin" and "adminadmin" respectively.
This mode is convenient for fully automated GlassFish testing, because you do not need a dedicated GlassFish installation per each test set up, and the harness chooses the right TCP ports to avoid conflicts. However, starting and stopping GlassFish takes significant amount of time, so it’s almost impossible to use for interactive debugging.
There are a few command-line options that often go with automated batch test for a large number of tests. First,
use -p N to run tests by using N number of threads. This should be used on a multi-processor system to reduce
the turn-around time of the test, and therefore ideal for continuous testing.
The -report DIR option causes the harness to generate JUnit test report files into the specified directory.
This allows the reports to be combined with other JUnit or TestNG-based tests (such as per-class unit test.)
These XML reports can be then formatted into HTML (via <junitreport> ant task) or picked up by Hudson for further processing.
The test harness can generate EMMA coverage report of the tests by specifying
the -emma OUTPUTFILE option. The coverage will include both the runtime as well as tools.
See Emma User’s Guide for how to generate various reports from the coverage data file.
Following system properties can be passed to the harness:
- harness.deploy.timeout
-
deployment timeout in milliseconds, defaults to 30000
A single end-to-end test consists of a client talking to services. Each test is organized into one directory. A test directory could look like the following:
testdir/
+- test-descriptor.xml (metadata file that describes a test)
+- custom-client.xml (optional client customization file)
+- *.bsh (client test scripts)
+- common/ (optional common source files)
| +- *.java
+- client/ (optional client source files)
| +- *.java
+- resources/ (optional resource files if needed by test)
| +- mascot.jpeg
+- AddNumbers.wsdl (optional WSDL that represents service)
+- **/*.java (Java code that implements service)
+- web.xml (optional web.xml file, if specified used for packaging the war)The file test-descriptor.xml is the heart of the test data, and it contains the required configuration information for each test.
A typical "fromwsdl" test descriptor could look like the following:
<descriptor>
<description>this is a very useful test that tests stuff</description>
<client href="*.bsh"/>
<service>
<wsdl href="AddNumbers.wsdl"/>
</service>
</descriptor>The structure of the test descriptor is defined in a schema, and the harness performs an XML validation to make sure the data is in the expected format.
It is recommended that you link test cases to relevant bugs, so that in the future we can take advantage of information programmatically. To do this, use the bugs attribute on the descriptor element. It takes whitespace-separated list of bugs, where each of them are either 7-digit bugster ID or a GitHub project name followed by an issue number.
<descriptor bugs="6420352 wsit-123 jax-ws-234">
...
</descriptor>If your test relies on a certain capability from the container, specify such requirements in uses attribute.
Harness checks the capabilities of the container to continue with running the test or skipping it.
Supported values are servlet30, servlet, ri, multi-endpoint, skip-lwhs.
Please use this option with caution, so that tests can retain its generality.
<descriptor uses="servlet30">
<description>this test runs only on servlet 3.0 based container</description>
....
</descriptor>If your test requires a certain wsimport command line options, you can specify verbatim by
using the nested wsimport-client and wsimport-server elements.
These are passed as-is to when the harness invokes wsimport for generating client artifacts
and generating server artifacts, respectively. Options can be separated by tab, space, CR, and NL.
Please use this option with caution, so that tests can retain its generality.
<descriptor>
<description>this is a very useful test that tests stuff</description>
<wsimport-client>-abc -def -ghi</wsimport-client>
</descriptor>If your test special configuration from the default harness behavior, test-options is the place to specify those.
Supported Options are:
- -noPackage
-
to not use the default harness behavior of specifying
-poption to wsimport.
<descriptor>
<description>this is a very useful test that tests stuff</description>
<test-options>-noPackage</test-options>
</descriptor>If your test sources require special options to be compiled, javac-options is the place to specify those.
Supported are any javac options.
<descriptor>
<description>this is a very useful test that tests stuff</description>
<javac-options>-XDignore.symbol.file=true</javac-options>
</descriptor>Client test scripts are the programs that are executed after all the services are set up. These scripts are expected to make some service invocations and perform assertions.
Test scripts can be written as:
-
files. They can be anywhere and have any name, since the harness locates them through
<client href="…">inside the descriptor. However, we recommend keeping them at the top directory and use.bshsuffix. -
inline inside the test descriptor, as illustrated below. This is particularly convenient for small test clients. The
nameattribute indicates human readable short summary of a test:<descriptor> <description>this is a very useful test that tests stuff</description> <client name="echo test"> assertEquals("hello",fooPort.echo("hello")); </client> ... </descriptor>
Scripts are written in the BeanShell scripting language. BeanShell is mostly compatible with Java, so you can write any valid Java programs, such as:
import java.math.*;
BigInteger bi = new BigInteger("35.12");As a scripting language, BeanShell allows you to save typing. Specifically,
-
You need not perform any explicit casting, since variables are not statically typed.
-
No explicit variable declaration is required (although allowed)
-
Auto-boxing/unboxing ala Tiger
-
Type parameters are not needed (in fact they are not supported)
So you can write a program like this:
import java.math.*;
bi = new Holder(new BigInteger("35.12")); // create JAX-WS holder object with BigIntegerSee beanshell syntax documentation for more about BeanShell syntax.
To isolate the tests from the environment, and to make tests as reusable as possible, the harness creates services and port objects on behalf of the test programs. This allows the harness to configure ports correctly, so that they can talk to the right service.
For this reason, it’s important for the client program not to create port objects on its own. See the following illustration:
// :-< BAD EXAMPLE
port = new HelloService().getHelloPort();
port.sayHelloTo("duke");
// :-) GOOD EXAMPLE
helloPort.sayHelloTo("duke");To find out what port variables are imported under what names, check the output from the test harness. Right before the client script executes, you should see a line like this:
injected services: helloService
injected ports: addNumbersPort|
Caution
|
Not fully cooked. Feedback welcomed. The only exception to this "don’t create your ports" rule is when you need |
Junit based tests can use System property to access the endpoint address, for ex: System.getProperty("helloPortAddress")
// create dispatch
helloService.addPort(new QName("foo","bar"), javax.xml.ws.soap.SOAPBinding.SOAP11HTTP_BINDING, endpointAddress);For productivity, scripts are run in a configured environment. Client scripts are expected to take advantage of those.
-
Script can invoke any of the
junit.framework.Assertmethods without any qualifier. So you can just writeassertEquals(5, 3 + 2). This is the prefered way of testing assertions. -
The following Java packages are implicitly imported.
java.lang java.util java.util.cocurrent javax.xml.soap javax.xml.ws javax.xml.ws.soap javax.xml.bind javax.xml.namespace javax.xml.transform javax.xml.transform.sax javax.xml.transform.dom javax.xml.transform.stream (client artifact packages)
So you could just say
JAXBContext.newInstance(…)instead ofjavax.xml.bind.JAXBContext.newInstance(…). Client artifact packages are all imported, too, so for example if you need to create a bean for service invocation, you can do as follows:bean = new FooBarBean(); // instead of new test.fromwsdl.helloLit.client.FooBarBean(); bean.setName("Duke"); bean.setId(53);
-
Helper methods. The harness defines a few helper methods that are available to the script as if they are global functions. See the source code of ScriptBaseClass.java and the source code of util.bsh for the complete list and their semantics.
Often it’s useful to factor out the common variable/function/etc for multiple <client> scripts in a single test suite.
This can be done by defining <pre-client> element in the test descriptor. If specified,
the pre-script will be executed before each client test script is executed.
Many tests are written in such a way that they do not have any side-effect on both server or client
(when I say side-effect, I mean things like updating global counters or touching database or anything of that sort).
Test descriptor can mark those tests as side-effect free by adding the attribute as follows:
<descriptor>
...
<client href="abc.bsh" sideEffectFree="true" />
</descriptor>With the -concurrent-side-effect-free option, tests marked in this way will be run in concurrent environment.
That is to say, the same test will be run repeatedly in multiple threads. This allows the runtime to be exercised in a new way.
If your client test scripts need some Java helper code (such as handlers), or if you have some JUnit test classes
written as Java classes, you can put them under the testdir/client directory. All classes that ends with Test
will be considered JUnit tests and executed like client test scripts.
Also, all the Java source files are compiled and made available to client test scripts (for example so that you can access
those classes.) The client package itself will be also imported into the client test scripts, so classes
in the client package can be accessed by their short name.
Code that constitutes a service needs to be written by using Java. By default, those files should be placed into the same test directory. If you are converting existing test services to this harness, most likely the only thing that needs to change is the declared package name.
In a rare circumstance, where you need multiple independently deployed services, you can use <service basedir="…"/> value
to move service source files into subdirectories to isolate them.
|
Tip
|
To set a binding in fromjava scenarios, use the @WebService
@BindingType(SOAPBinding.SOAP11HTTP_BINDING)
class FooBarSgervice {
...
} |
For scenarios which requiring building the service from a WSDL, there is a sub-element of service named wsdl
which has an attribute named href. href should name the WSDL file, relative to the service’s basedir.
When present, WSDL is first complied and then the artifacts are compiled together with the
rest of the Java source files. The harness uses the -p option of wsimport to place
the generated artifacts into the "right" package (infered from the directory name of the service.)
To further customize the compilation process, place a JAX-WS customization file named custom-server.xml to the service
top directory (to refer to WSDL file and schema files from this customization file, just use the relative path
from the customization file to WSDL/schemas like "./foo.xsd".) You can also use custom-schema-server.xml
to specify additional JAXB customization file.
A test can declare that it requires a particular kind of a transport by declaring the transport attribute.
The possible values are the values you specify in the scheme portion of the endpoint address URI.
(The major values include "http" and "local", but open-ended as new transports are added.)
<descriptor transport="http">
...A test can also declare that it’s applicable only to a certain verion(s) of JAX-WS/WSIT, by using the version attributes
@since, @until, and @excludeFrom.
-
The
@sinceattribute indicates that a test will require the given version of the test subject or later. For example,since="2.0"indicates that it will run on 2.0 FCS, 2.0 EAs, etc, but not on 1.1. -
The
@untilattribute indicates that a test can only be run against the given version of the test subject or earlier. For exampleuntil="1.*"indicates that the test will run on all versions of 1.x but not on 2.0. -
The
@excludeFromtakes a list of whitespace-separated version numbers, and it indicates that tests be excluded from these specific versions. This is used to mark known regressions and such things.
Those three attributes can be combined, and they can be specified either on <descriptor> element (in which case
the entire test is subject to the constraints), or on individual <client> element (in which case only that client test
is subject to the constraints.)
See this document for more formal definition of what can be a valid version number, and how they are ordered.
Similarly, Junit based tests can specify version requirements by using annotation @com.sun.xml.ws.test.VersionRequirement(since="2.1.*", until="2.2",excludeFrom="") on the TestClass.
If test programs need to use image files, binary files, or other data files, they should be put in resources directory
at test’s top-level. This is optional. Those resources can be loaded by using the helper methods as follows:
File f = resource("foo.jpg"); // load resources/foo.jpgFor resources that are XML, you can define them by using <xml-resource> elements.
<!-- define it inline -->
<xml-resource name="foo">
<echoRequest>123</echoRequest>
</xml-resource>
<!-- define it externally -->
<xml-resource name="bar" href="my.xml" />Such XML resources are injected into clients as variables of the type
XmlResource,
so you can use them like this:
echoPort.echoString(foo.asString());If test programs need to share some Java source files between the server and the client, those files can be put
into the common directory. They will be compiled and made available to both client and server.
|
Caution
|
Not fully cooked. Feedback welcomed. We are hoping to define more convenience methods and define how to access resources from server. |
See the LICENSE file for details.
We use the GitHub metro-ws-test-harness project for developing the harness. We use the metro-dev@eclipse.org mailing list for communication.
Our goal is to improve your developer productivity. Please use the issue tracker for keeping track of RFEs and problems.