Thursday, December 08, 2011

OSGi testing with Gradle and Pax Exam

I've recently been working on a Spring issue which involved writing a Gradle-built test to ensure that the Spring framework 3.1.x bundles successfully resolve in an OSGi container.

The Gradle side of this was a very simple and pleasant experience, as the following extract of the file build.gradle shows:

apply plugin: 'java'
apply plugin: 'eclipse'

repositories {
    mavenCentral()
}

dependencies {
    ...                
}

There were minor usability issues1 with the Gradle Eclipse plugin, but I gather these will soon be resolved.

I decided to use Pax Exam to launch the testcase inside Equinox, and this require specifying some dependencies in build.gradle:

dependencies {
    paxExamVersion = '2.3.0.M1'
    equinoxVersion = '3.6.0.v20100517'
    
    testCompile "junit:junit:4.+",
                "org.ops4j.pax.exam:pax-exam-junit4:$paxExamVersion",
                "org.eclipse.osgi:org.eclipse.osgi:$equinoxVersion",
                "javax.inject:javax.inject:1"
                
    testRuntime "org.ops4j.pax.exam:pax-exam-container-native:$paxExamVersion",
                "org.ops4j.pax.exam:pax-exam-link-mvn:$paxExamVersion",
                "org.ops4j.pax.url:pax-url-aether:1.3.5"
}

Let's go through the dependencies in turn:
  • JUnit 4 is the test framework.
  • @RunWith(JUnit4TestRunner.class) marks the test for launching under Pax Exam.
  • Equinox provides OSGi types at compile time and an OSGi container at runtime.
  • A bundle context is injected into the test using @javax.inject.Inject.
  • The test uses the native Pax Exam container - all the tests run in a single JVM.
  • The test uses Pax Exam to install Spring bundles directly from Maven repositories.2
  • The test caches bundles from the SpringSource Enterprise Bundle Repository using the Pax URL Mvn protocol.
The Pax Exam developers kindly helped me sort a basic setup issue which wasn't too obvious from the documentation. Thanks Harald and Toni! I needed to explicitly install a JUnit bundle, by calling junitBundles from the configuration method:

    @Configuration
    public static Option[] configuration() throws Exception {
        return options(//
            provisionSpringBundle("spring-core"), //
            provisionSpringBundle("spring-beans"), //
            // etc.
            junitBundles());
    }

The full source of the project is available on github (under the Apache 2.0 license) for anyone who wants to give it a spin or use it for their own purposes. If you'd prefer to write your tests in Groovy, take a look at Hamlet D'Arcy's blog.

Footnote:
1: The generated .classpath contains absolute paths rather than paths relative to the project and so is not suitable for committing and sharing with others. Also re-running gradle eclipse without first deleting .classpath can result in it containing duplicate paths. Luke Daley tells me that both these problems will soon be fixed, the latter fix appearing in gradle 1.0 milestone 7.
2: If the testcase was intended a a stand-alone test of published versions of Spring, it would cache Spring bundles in the local Maven repository for efficiency. But we plan to integrate the testcase into the Spring build in due course and so we'll want to grab the latest built version each time to cope with re-spins, so cacheing isn't helpful.

Tuesday, November 29, 2011

Virgo 3.0.2 released

Virgo 3.0.2 is available for download. Please see the release notes for details of this bug fix release.

Friday, November 18, 2011

Opening SourceTree from the command line

I've recently started using SourceTree to visualise and manage my git repos. It's very impressive.

The one thing I missed from gitx was the ability to launch SourceTree from the command line, in any directory of the repo. One solution is to set a git alias:

git config --global --add alias.sourcetree '!open -a SourceTree .'

and add an alias in the shell profile:

alias st='git sourcetree'

Now I can issue "st" from the command line and the relevant SourceTree window opens up.

(Thanks to Chris Frost for providing part of this solution.)

Monday, November 14, 2011

What did we learn in 100 sprints?

This week marks the 100th Sprint on Eclipse Virgo! That includes the precursor project, SpringSource dm Server. What have we learned about scrum? Well, obviously, it's pretty good otherwise we would have ditched it long ago (and believe me I've been tempted a couple of times). But specifically...


Firstly, that one week sprints are by far the easiest to manage, both in terms of planning effort, but also psychologically. Even in a two week sprint, it's easy to put off unpleasant tasks until it's too late.


Secondly, that the main value of scrum is to keep the team focussed on specific tasks during a sprint and enable progress to be shared easily at daily stand-up meetings.


Thirdly, that story points are more effort than they are worth, so it's more effective to go with gut feel.


Fourthly, that you have to take a relaxed attitude towards planning design work as it's almost impossible to predict the effort up front. Spikes are ok for small items, but sometimes you just need plenty of time to get your head round a large area.

Thursday, October 27, 2011

Enterprise Bundle Repository Samples

After opening up the Enterprise Bundle Repository build system, I've added more sample bundlor templates along with Ant/Ivy build files to github.

So while we are trying to come up with a community-based successor to the EBR, if you need to get a new version of one of the bundles in the EBR, you may be able to find a bundlor template to use as a starting point.

Wednesday, October 12, 2011

Latest draft subsystems spec

The OSGi Alliance has made an early draft of some RFCs available. This includes the subsystems RFC 152 I blogged about earlier: see pages 72-144 of the PDF.

Wednesday, October 05, 2011

Eclipse Dance Steps


Sometime Eclipse fails to work as expected. The following 'dances', in increasing order of desperation, may help. Remember to maintain a fixed grin throughout and try not to tread on your own toes.

Refresh and clean all projects ("Quickstep")

Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Hit F5 to refresh. Select Project->Clean..., select "Clean all projects", and click Ok.

Open and close all projects ("Foxtrot")

Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Right click the projects and select "Close Project". Right click the projects again and select "Open Project".

Open and close all projects restarting Eclipse in between ("Tango")

Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Right click the projects and select "Close Project". Restart Eclipse. Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Right click the projects again and select "Open Project".

Delete and re-import all projects ("Paso Doble")

Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Press the delete key, select "Do not delete contents", and click Ok. Right click in the Package Explorer and select Import..., then choose General and "Existing Projects into Workspace" and click Next. Browse to select the platform checkout directory and click Choose. Select all the projects for import. Click Finish.

Rebuild the search indices ("Slow Waltz")

Exit the workbench and delete the Java search index:
  • go into \plugins\.metadata\.plugins\org.eclipse.jdt.core
  • delete 'savedIndexNames.txt'
  • delete all *.index
    Restart

If all else fails

If you're still stuck, try the "Hokey Cokey" by repeating all of the above until you've had enough.

Monday, October 03, 2011

How Infor ION Uses Virgo

Infor have kindly made available a presentation on how their ION integration product is based on the Virgo kernel. They migrated from Equinox 3.5 to Virgo 2.1, added a lightweight web server (in preference to using the pre-built Virgo Tomcat or Jetty servers) and a service wrapper, added Spring integration, upgraded Spring framework, added a watched repository, and then added various enterprise service bundles. This is a perfect example of a Virgo kernel application, a so-called OSGi stackless stack.

The Infor team also helpfully pointed out some enhancements they'd like to see, some of which are already supported or planned for Virgo 3.5.

According to their website Infor is the world's third largest provider of enterprise applications and services, so it's nice to see another large company adopt Virgo.

Thanks to Ian Skerrett for bringing this user to my attention. See his blog for more.

Friday, September 30, 2011

Extenders: Pattern or Anti-Pattern?

The extender pattern has become popular in recent years and has even been utilised in OSGi standards such as the Blueprint service and the Web Applications specification. In Virgo, we've been working with extenders from the start, but in spite of their advantages, they have some significant downsides. Since the OSGi Alliance is considering using extenders in other specifications, I agreed to document some of the issues.

The first difficulty is knowing when an extender has finished processing a bundle. For example, a bundle containing a blueprint XML file will transition to ACTIVE state as soon as any bundle activator has been driven. But that's not the whole story. Administrators are interested in when the bundle is ready for use and so the management code in Virgo tracks the progress of the extender and presents an amalgamated state for the install artefact representing the bundle. The install artefact stays in STARTING state until the application context has been published, at which point it transitions to ACTIVE. Without such additional infrastructure, administrator cannot tell when a bundle processed by an extender really is ready for business.

That's the successful case, but there are complications in error cases too. The first complication is that since an extender runs in a separate thread to that which installed the bundle, if the extender throws an exception, this is not propagated to the code which installed the bundle. So the installer needs somehow to check for errors. Therefore Virgo has infrastructure to detect such errors and propagate them back to the thread which initiated deployment of the bundle: the deployment operation fails with a stack trace indicating what went wrong.

The other error complication is where there is a (possibly indefinite) delay in an extender processing a bundle. For this kind of error Virgo tracks the progress of extender processing and issues warnings to the event log (intended for the administrator's eyes) saying which pieces of processing have been delayed and in some common situations, for example when a blueprint is waiting for a dependency, what is causing the delay.

Extenders suffer from needing to be able to see bundle lifecycle events and so for systems that partition the framework, it is necessary to install each extender into multiple partitions. On the flip side it is crucial to prevent multiple instances of an extender from ever seeing the same bundle event otherwise they will both attempt to extend the bundle.

Another issue with extenders is the need to keep them running and healthy as there is little indication that an extender is down or sickly other than bundles not being processed by the extender. Virgo takes care to ensure its extenders are correctly started and its infrastructure for detecting delays helps to diagnose extender crashes or sickness (both of which are extremely rare situations).

There is also an issue in passing parameters to an extender to affect its behaviour. This is typically done by embedding extender configuration in the bundles being processed or by attaching a fragment containing configuration to the extender bundle. But since the extender is not driven by an API, the normal approach of passing parameters on a call is not available. Essentially, an extender model implies that the programming model for deployment is restricted to BundleContext.installBundle.

With considerable investment in additional infrastructure, Virgo has managed to support the Blueprint and Spring DM extenders reasonably well. But in the case of the Web Applications extender, Virgo couldn't make this sufficiently robust and so it drives the underlying web componentry directly from the Virgo deployment pipeline to avoid the above issues.

I understand at least one other server runtime project has encountered similar issues with extenders, so Virgo is not alone. There is a trade-off between loosely coupling the installer from the resource-specific processing, the main strength of the extender pattern (but far from unique to that pattern), and providing a robust programming model and usable management view -- crucial features of a server runtime -- which is far more straightforward without extenders.

Thursday, September 22, 2011

OSGi Subsystems in Virgo

A public draft(*) of the OSGi subsystems RFC (152) should soon emerge from the OSGi Alliance. A subsystem is a multi-bundle application, not dissimilar to a PAR or plan in Virgo. IBM is leading the spec work and a number of other vendors, including SpringSource/VMware, are contributing. Quite a few projects have multi-bundle application constructs, so it makes sense to agree a standard form.

After going through the RFC again with my implementer's hat on, I listed the features necessary to support subsystems in Virgo. Most of the changes are in the Virgo kernel, although I hope to structure the support into non-subsystem specific generalisations of the kernel plus subsystem-specific code running in the user region.

Currently, it is not possible to deploy a plan which contain artefacts that are already deployed. This will need generalising so that we can support subsystems using a common data structure of deployed artefacts: a directed acyclic graph (DAG) rather than the current collection of trees.

The switch to a DAG has interesting implications for lifecycle management of shared subgraphs. With today's tree, when a node is stopped, started, or uninstalled, any subtrees are also stopped, started, or uninstalled, respectively. With a DAG, shared subgraphs need to be sensitive to all their parents. This boils down to keeping each shared subgraph at the maximum state required by any of its parents. States are ordered: ACTIVE > RESOLVED > UNINSTALLED.

Lifecycle management will also get interesting when a shared subgraph belongs to one or more atomic subgraphs as then lifecycle changes in the common subgraph will propagate to all the containing atomic subgraphs. I think that will just "work", but users might need to be careful in their use of atomic plans if they want to avoid management operations on one application affecting other applications.

I'm also considering using garbage collection as a means of uninstalling artefacts which are no longer needed. Given the number of types of dependency that are possible, this is likely to be more reliable than the alternative of maintaining reference counts.

I'm a little concerned about a possible race between garbage collection detecting an artefact as dead and a new dependency being created on the artefact just before garbage collection goes ahead and uninstalls the artefact, but there would be a similar concern for reference counting. The basic issue in this race is that a dead artefact may be found by a live bundle and a new dependency created before the dead artefact can be uninstalled. For instance, a dead bundle may be found by using the OSGi API to list all bundles. It may be possible to use some technique such as a special region in the region digraph to isolate dead bundles, although this issue is probably something to discuss among those working on the RFC.

Anyway, there's plenty of work to be getting on with. I haven't done detailed estimates of the features identified so far, but I guess there's a person year or so of effort needed, so I'm initially targeting Virgo 4.0. If you feel like lending a hand, please get in touch on virgo-dev.

* - RFC 152 has changed quite a bit since the version in the Enterprise spec early access draft dated 16 May 2011. A new draft is being prepared.

Thursday, August 25, 2011

New Releases: Virgo 3.0 and Gemini Web 2.0

Virgo 3.0 and Gemini Web 2.0, collectively known as the Maya release, are available for download and download, respectively.

The team of committers has expanded considerably and, although the team is now distributed geographically, we have a great working relationship. I'm delighted that these projects are now community collaborations rather than the work of a single vendor.

The theme of Virgo 3.0 is better integration with EclipseRT technologies. To that end, we have created a Jetty variant of the Virgo web server and have switched from Felix to Equinox implementations of some OSGi services. Integration with p2 is proving a tough nut to crack and is deferred to the Bondi (3.5) release, although we expect to issue a milestone soon to get some feedback from the user community.

Gemini Web and the Tomcat variant of the Virgo web server have been upgraded to Tomcat 7 and Servlet 3.0.

Gemini Web 2.0 passes the OSGi Web Applications compliance tests and will replace the SpringSource OSGi Web Container as the OSGi Web Applications reference implementation.

The snaps framework for modular web applications is also released as part of Maya. This started life as the SpringSource slices prototype.

Virgo kernel 3.0 has a new shell based on Apache Felix Gogo for both the kernel and user region and accessible via telnet or ssh.

The 3.0 kernel also uses a new implementation of regions, the region digraph, based on OSGi standard framework hooks. The digraph should position Virgo well for implementing the OSGi Subsystems specification in a future release. (The specification is still under development, but we expect a public draft from the OSGi Alliance before long.) The digraph has moved from Virgo to Equinox to enable it to be used by other projects, especially those that are aiming to implement the Subsystems spec.

Highlights:

  • New Virgo Jetty Server
  • Virgo Tomcat Server and Gemini Web upgraded to Tomcat 7 and Servlet 3.0
  • New Snaps framework for modular web applications with new guide
  • New Gogo shell
  • New user guide for Gemini Web
  • Updated GreenPages sample with new guide
See the release notes for details.

Wednesday, August 03, 2011

OSGi Running in the Cloud: Virgo in CloudFoundry

I'm just been tryout out Hristo Iliev's support for Virgo in CloudFoundry, described on his blog. The VM took a few hours to download and a few minutes to unzip, but then starting CloudFoundry and deploying an application to a Virgo instance worked first time.

CloudFoundry takes a little while to get started:






























But then it's trivial to deploy an application to a Virgo instance:



after which the application, in this case the Virgo web admin console, runs as expected at the application URL http://virgo.vcap.me:






















and jconsole shows where the Virgo instance has been created:















The cloud nature of this setup becomes more apparent when another application is deployed. Since I'm lazy, I deployed the same application under a different name:




















The application is then available at http://virgo2.vcap.me and jconsole shows there are now two Virgo instances running:





















I'm looking forward to Hristo's pull requests become merged into the CloudFoundry base and maybe eventually Virgo showing up in some clouds near you.

Tuesday, July 26, 2011

Virgo 3.0 (Maya) Release Candidate 1

The first Release Candidate of Virgo 3.0 (Maya) is available for download. See the release notes for details.

RC1 finally brings the snaps framework for modular web applications out of prototype status. The separate snaps download includes a completely new Virgo Snaps User Guide and a comprehensive sample application. The User Guide describes how to install snaps on top of Virgo Tomcat Server and uses the sample application to verify the installation and step through all the major features of snaps.

RC1 of Virgo Tomcat Server embeds Gemini Web 2.0 RC1 which now comes with new documentation in the form of a User Guide plus a minimal Programmer Guide.

We encourage users to take RC1 for a spin and provide feedback via bugzilla or the Virgo forum. Unless critical bugs are found, we'll be focussing on the documentation until 3.0 is released, probably towards the end of August.

Thursday, July 14, 2011

Exploiting p2 in EclipseRT projects

Virgo committer Borislav Kapukaranov has written an interesting blog entitled "RT meets p2". Here he gives some of the background behind the work to integrate p2 into Virgo (bug 343543) which is targetting the Azure (3.5) release of Virgo.

Tuesday, July 05, 2011

Virgo Maya (3.0) feature freeze at milestone 6

Milestone 6 of the Virgo Maya (3.0) release is available for download. This milestone marks the feature freeze of Maya, so we'll be working on bugs and documentation from now on. We plan to ship the release in August.

The milestone includes the new shell based on Apache Felix GoGo, accessible via telnet or ssh as described in the release notes. There are a few bugs in this support which need fixing, but apart from that the new function is in good shape for trying out.

This milestone also includes snaps for the first time. Snaps is a framework supporting modular OSGi web applications. We are bringing snaps out of prototype status in Maya, primarily by documenting its function and limitations in the next few weeks. Several groups are using snaps, so this should give them a more stable base. Including snaps in the release process will ensure it stays in step with Virgo Tomcat Server as the two evolve.

Finally, this milestone moves Virgo Jetty Server out of prototype status as the User and Programmer Guides now include appropriate documentation.

Monday, June 27, 2011

First milestone of Virgo tooling

The first milestone of the Virgo IDE tooling is available. See the wiki for instructions. It should provide approximately the same level of function as the Virgo tooling that was provided as part of SpringSource Tool Suite (STS). It works nicely on Eclipse Indigo and Helios SR2, for example.

The basic functions allow you to develop OSGi bundles, including Web Application Bundles, and PARs and deploy them to an instance of Virgo Tomcat Server (or Virgo Web Server if you prefer to run v2.1 rather than a 3.0 milestone) running under the control of the tooling. It is also possible to develop plans, but this needs documenting better as it's not immediately obvious how to do it. See an earlier blog post if you need to run the Virgo kernel instead of Virgo Tomcat Server.


Please direct discussion to the Virgo forum and raise bugs on bugzilla (thanks Daniël van 't Ooster for the bugs you raised while I was writing this blog!). The source is available in Eclipse git for anyone who wants to make changes and/or send in patches. We are keen to encourage new contributors to Virgo in general and the Virgo tooling in particular.

What next? Well, the most urgent feature, bug 341910 which is a pre-req to the first release, is to remove the dependency on Spring IDE.

Then, over time, we expect to align the Virgo tooling with the Eclipse Libra project. Libra provides tooling for OSGi standards, so we aim to move the OSGi standard features of the Virgo tooling to Libra and end up with the Virgo tooling being a Virgo-specific extension of Libra.

Congratulations to Martin Lippert who started with the STS code and single-handedly achieved this milestone! You can see him in last week's OSGi Development Tooling Panel.

Jigsaw déjà vu

I recently questioned one of the Jigsaw requirements on the jigsaw-dev mailing list. I agree with the general direction for interoperability with OSGi, but I wanted to suggest an improvement:

Hi Mark

I'm pleased to see the explicit acknowledgement of some basic OSGi interoperation requirements in the requirements document ([1]).

I agree with David Bosschaert ([2]), that it would make sense for OSGi to support the Java SE 8 module format and, for modules which can serve equally well as OSGi bundles, I'd like to avoid dual-maintenance of module metadata and OSGi manifest. I'd like to be able to "decorate" the standard metadata.

However, the requirement "The syntax must place all extended metadata after all standard metadata, with a clear delineation between them." precludes inline decorations. The result would be duplication and clunkiness.

I propose that this requirement be changed so that standard metadata could be decorated inline (the decorations would be ignored by the Java SE module system).

What do you think?

Regards,
Glyn
[1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12
[2] http://osgithoughts.blogspot.com/2011/05/java-se-8-modularity-requirements.html

The response: zilch!

After a follow-up, I got a single supportive response, but that was from OSGi aficionado Peter Kriens.

I would have preferred some defence of the current requirement rather than silence. This is awfully like the days of JSRs 277 and 294. Plus ça change, plus c'est la même chose...

Tuesday, June 14, 2011

Virgo 3.0 plan update

The feature to provision Virgo using p2 is taking a while to finalise. Also, it changes the directory layout of Virgo and so will need some changes to the tooling. So we've decided to defer the feature to a later release, possibly 3.5 to indicate the relatively big jump from 3.0.

This means the 3.0 release content is pretty much complete, so we just have to fix a few remaining bugs and polish the documentation before releasing, probably in July or August 2011.

See the plan on the Virgo wiki for details.

Wednesday, June 01, 2011

Eclipse Virgo 3.0.0 Milestone 5

Milestone 5 of Virgo 3.0.0 is available for download. This is an important milestone which adds significant functional enhancements, upgrades several dependencies to their latest levels including Spring 3.0.5, Tomcat 7.0.12, and Servlet 3.0, and fixes a number of bugs. Full details are available in the release notes.

Monday, May 09, 2011

Equinox digraph ready for use

Previously I described how Virgo used OSGi framework hooks to implement a directed graph, or digraph, of isolated groups of bundles known as regions. The region digraph has now moved to Equinox and I'm modifying Virgo to use the Equinox region digraph in place of Virgo's own (see bug 343364 for details).

I always prefer making such changes in an incremental way, so the first step was to add the Equinox region digraph bundle to the Virgo kernel without exploiting it. The Virgo kernel runs fine with the Equinox region digraph in situ. This is because the Equinox digraph adds all bundles to its kernel region and so provides no isolation between those bundles. Later, Virgo's region digraph partitions the bundles in Virgo kernel and Virgo user regions with appropriate isolation and things continue to work as normal.

This is interesting, but not particularly useful. The whole point of the region digraph abstraction was to make it easy for multiple parties wishing to provide isolation via the OSGi framework hooks to be able to collaborate together by sharing and manipulating a single region digraph. It was never intended that multiple digraphs should co-exist.

But there is a useful point here. If you have an Equinox-based system, of which there are many, you can introduce the Equinox digraph with zero functional impact. All your bundles, including those installed before and after the Equinox region bundle, go in the Equinox kernel region and there is no isolation between those bundles.

When you want to start exploiting the digraph, you can use the RegionDigraph service to create a new region, attach it with appropriate filters, if any, to and from the kernel region, and then install a bundle into the new region using the Region.installBundle operation. After starting this bundle, you will then have a bundle context that can be used to install further bundles into the new region using the OSGi standard BundleContext API.

I hope this will inspire you to have a go at using the Equinox region digraph if you are an Equinox user and need to isolate groups of bundles.

Meanwhile, the standardisation of multi-bundle applications, including isolation, continues in the OSGi Alliance and a public draft of this spec should be available in the not too distant future. I hope the Equinox region digraph will be used more widely and this will help to provide feedback on the spec work.

Opening up the Enterprise Bundle Repository

We'd like to enable the SpringSource Enterprise Bundle Repository to turn into a shared community resource.

As a first, small step in that direction, I've made available the EBR build system (a few Ant/Ivy scripts) and some sample bundle projects on github. If you want to try it out, clone the repository and refer to doc/README.txt.

The idea is that the community will be able to add bundles to this repository and then the content can be hosted in some cloud storage similar to the way the SpringSource EBR is hosted in Amazon S3.

Tuesday, May 03, 2011

Virgo survey results

Thanks to everyone who responded to the recent Eclipse Virgo and SpringSource dm Server survey. The results make interesting reading.

Close to one quarter of respondents report they are "in production".


The transfer to Eclipse.org is a success. It is 6 months since the first Virgo release and 84% of respondents are now using Virgo exclusively with only 13% using dm Server exclusively.

Virgo is popular with early adopters. Of the 84% using Virgo exclusively, 59% are using 2.1.x or 3.0 milestones and 25% are on 3.0 milestones only (some of these already in production!). Please note that Virgo has versions 2.1.0 and greater whereas dm Server covers versions 1.x and 2.0.x.

The Virgo kernel is seeing significant usage. The kernel underpins the Virgo Web Server and provides Virgo's advanced deployment, provisioning, isolation, scoping, and diagnostic capabilities. The kernel has no web-specific code and so can be used to build messaging and other non-HTTP servers and to act as a container for non-server applications. 16% of respondents are using the kernel exclusively; 5% are in production.

Documentation and blogs were the most useful information sources closely followed by the Virgo forum, Virgo wiki, and Virgo development mailing list (1= not at all useful, 5 = very useful).

The survey was advertised on Twitter, the Virgo forum, the Virgo development mailing list, at www.springsource.org, and in this blog. 166 people responded. If you are interested, the raw results of the survey are available online or a summary can be downloaded.

Finally, I'd like to thank Chris Frost for coming up with the idea for the survey and then delivering the goods.

Tuesday, April 05, 2011

OSGi comes of age?

Early in my career at SpringSource, my colleagues presented me a T-shirt with "CICS does that!" printed on the front. However, nobody anticipated what that statement might eventually include.

IBM has just announced that the latest version of the CICS transaction monitor includes support for applications packaged as OSGi bundles. From the announcement letter:

The JVM server infrastructure with integrated OSGi support and tooling provides a new industry standard Java runtime environment in CICS.
[...] 
The JVM server environment supports the development and deployment of Java applications through the use of industry standard OSGi bundles. This will require Java applications to be packaged into an OSGi bundle, and then deployed as a CICS BUNDLE resource with reference to the OSGi bundle and its version, removing the need to load Java applications from a statically defined class path. This function will significantly improve the system management of Java applications in CICS through the provision of application isolation and versioning, cross-package prerequisite checking, and simplified package redeployment.
This is heart-warming news to me, both as an OSGi proponent and as a former member of the CICS development team. I'm guessing CICS is using Equinox as its OSGi framework, but I'm looking forward to reading the documentation for the details.
 

Thursday, March 31, 2011

Java update breaks jps, jconsole, etc

The latest update of Java broke jps and jconsole when Java is launched specifying "-Djava.io.tmpdir" with a value of anything other than $TMPDIR. This affects Virgo and Tomcat.

The problem was introduced in Java 6 update 21 and is fixed by bug sunbug 7009828 in Java 6 update 25.

A workaround for Virgo is as follows. On Mac and Linux to delete the following line in dmk.sh:

-Djava.io.tmpdir=$TMP_DIR \
and on Windows to delete the following line in dmk.bat:
set KERNEL_JAVA_PARMS=%KERNEL_JAVA_PARMS% -Djava.io.tmpdir="%TMP_DIR%" 

Eclipse Virgo diagnostics for "uses" conflicts

Congratulations to Felix on producing some really nice diagnostics for uses conflicts, and thanks to Neil Bartlett for his blog on the subject.

I thought I'd try a similar example in Virgo for comparison. Here are Virgo's diagnostics:

Unable to satisfy dependencies of bundle 'org.example.foo.xydata' at version '0.0.0': Cannot resolve: org.example.foo.xydata
    Resolver report:
        Uses violation: ⟨Import-Package: org.example.foo.bar.datastore; version="0.0.0"⟩ in bundle ⟨org.example.foo.xydata_0.0.0[1301563814803]⟩
            Found conflicts:
                package        'org.example.foo.util.verify_1.0.0' in bundle 'org.example.foo.platform.util_0.0.0[1301563814806]' used by 'org.example.foo.bar.datastore_0.0.0' in bundle 'org.example.foo.bar.datastore_0.0.0[1301563814805]'
                conflicts with 'org.example.foo.util.verify_2.0.0' in bundle 'org.example.foo.framework_0.0.0[1301563814804]' imported by bundle 'org.example.foo.xydata_0.0.0[1301563814803]'

(The numbers in square brackets are bundle ids - they are enormous because Virgo is attempting resolution in a "side state". Bundles with enormous ids have either been deployed directly or auto-installed from the Virgo repository and are in the process of being resolved. They are committed to the OSGi framework only if the directly deployed bundle(s) can be resolved successfully.)

The Virgo formatting isn't quite as pretty as Felix's, but it looks reasonable on a wide terminal window. However, there are some subtle differences:
  • Felix dumps out the dependency chains whereas Virgo does not attempt that. We had in mind examples where the dependency chains are unmanageably long, i.e. those where good diagnostics really are essential. But having seen Felix's diagnostics, we would like to dump out the chains in simple cases. On the other hand, that area of code is pretty complex, so this is not something we'll be rushing to do.
  • Felix depicts wirings from import to export which could be especially useful for newcomers to OSGi.
  • The Felix diagnostics are built in to the framework whereas non-Virgo users of Equinox do not get the Virgo diagnostics.
  • The Virgo diagnostics highlight the fact that there is a uses violation whereas Felix leaves the user to infer that from the rest of the message. It would be nice if Felix was more explicit because there are other reasons a bundle cannot be resolved.
  • Felix highlights the duplicated package whereas Virgo highlights the import that encountered the uses constraint. Virgo is designed to cope well when there are multiple duplicated packages. It's not clear what the Felix diagnostics would look like in such cases.
Note: on a resolution failure Virgo dumps the "side state" to disk for offline analysis. With this example, the offline analysis produced poorer diagnostics than were printed to the console, so I raised bug 341460 to investigate. The beauty of offline analysis is it lets you explore the details of the bundles. For example, the following screenshots shows the details of the duplicated package, including the import version ranges which result in the attempt to use the distinct versions of the duplicated package:



Anyway, I'm really pleased that we are starting to see some stiff competition on diagnostics. This will help OSGi users enormously.

Postscript
Bug 341460 is fixed in the Virgo 3.0 line. The admin console now displays the same diagnostics that were printed to the console.

Wednesday, March 30, 2011

Enterprise OSGi Runtimes and Tooling

It was great to see several enterprise OSGi runtimes in action at EclipseCon.

Readers of this blog will be familiar with Eclipse Virgo, so suffice to say that tutorials covered developing  applications using Eclipse Gemini components (reference implementations of several OSGi Enterprise standards) in Virgo and configuring and managing web applications in Virgo.

Several of the tutorials developed Web Application Bundles, or WABs, which are part of the OSGi Enterprise standard.  A Web Application Bundle is like a WAR file which specifies the web context path in its manifest and which obtains its dependencies using OSGi instead of having to include them all in WEB-INF/lib. The tutorials also implemented persistence using the OSGi Enterprise standard mapping of Java Persistence API (JPA).

Apache Aries provides a collection of components including implementations of several Enterprise OSGi standards and an application model similar to Virgo's. The EclipseCon tutorial had us creating a blogging application using the free IBM® Rational® tooling. The application consisted of an Aries application containing a plain OSGi bundle, a bundle utilising JPA persistence, and a WAB. The JPA bundle used Aries extensions to the standard blueprint service. The WAB used JNDI to look up the service published by the persistence bundle - the same approach that was taken in the Gemini and Virgo tutorial. We then deployed the application to a target platform which included Aries bundles.

Apache Karaf is similar in concept to the Virgo kernel. Karaf has several functions that Virgo lacks, some of which are good candidates to add to Virgo. Apache Geronimo is based on Karaf.

GlassFish now has good support for OSGi. The EclipseCon tutorial started with us developing a basic client bundle calling a service published by a back-end bundle. We then replaced the client bundle with a WAB and the back-end bundle by a JPA persistence bundle which published a service in the form of an EJB. The WAB used custom annotations to inject the service into the WAB's servlet. The EJB was used to demarcate transactions and security. The M2Eclipse "core" and "extras" plugins provided a basic Eclipse development environment, although the bundles were deployed via the command line. Kudos to the GlassFish team, including Arun Gupta and Sanjeeb Sahoo who led the tutorial, for possibly the most polished hands-on tutorial I've experienced.

I was also very pleased to see the Eclipse Libra project making progress. The version destined for inclusion in this year's Eclipse release train can be used to develop WABs. Runtime launching support is in the pipeline. We also hope to migrate standards based features from the Virgo tooling into Libra.

git tip: listing or editing the commits on a branch

Sometimes I need to list all the commits on a branch. I found the following command does the trick, even after I have merged from master into the branch one or more times:

git rev-list my-branch ^master

This lists all the commits reachable from my-branch but not reachable from master.

You can also change the committer or author info of commits on a branch by specifying  my-branch ^master as the on git filter-branch (see git help filter-branch). 

Tuesday, March 29, 2011

Quick survey of Virgo and SpringSource dm Server usage

I would be grateful if users of Virgo and SpringSource dm Server would take a short, anonymous survey to help us in our planning. Should take no more than 2 minutes to complete.

Correction: Oracle do not contribute to Virgo

A recent Oracle blog stated:

Also presented were Eclipse projects, that Oracle engineers are contributing to: Gemini, Virgo, Sapphire and Web Tool Platform.
I submitted a comment on the blog pointing out that Oracle engineers contribute to Gemini but not Virgo. Unfortunately, the comment was not published and the blog was not corrected so perhaps this blog will help to set the record straight.

That said, Virgo would welcome contributions from Oracle (and others). We've enjoyed collaborating with Oracle in the Gemini project and we really appreciate having Virgo committers from three different companies and a few more would be even better.

Monday, March 21, 2011

Incremental OSGi with Virgo, Gemini, and Libra

As I anticipate the start of EclipseCon 2011 later today, I'm reflecting on the current state of the Eclipse Virgo, Gemini, and Libra projects.

As I'll be discussing on Tuesday, Virgo evolved out of a relatively mature project whose aim was to make it easy for enterprise applications and enterprise application developers to adopt OSGi. The goal was to make the transition to OSGi incremental. Essentially you can start with a standard WAR file and deploy it to Virgo as-is. Then you can incrementally refactor the WAR file into an application comprising multiple OSGi bundles while continuing to use familiar technologies such as Spring, Hibernate, JPA, etc. We based Virgo on Tomcat suitably embedded into OSGi, again to provide familiarity for enterprise developers as well as systems administrators.

In parallel with the evolution of Virgo, the OSGi Alliance Enterprise Expert Group, being discussed on Tuesday, produced a series of enterprise specifications again with the goal of enabling enterprise applications to be migrate straightforwardly to OSGi. The reference implementations of these specifications became today's Eclipse Gemini project:

  • Gemini Blueprint - an OSGi standard dependency injection container which also supports the popular Spring and Spring DM namespaces
  • Gemini Web - support for servlets in the form of OSGi standard Web Application Bundles (WABs) or standard WAR files
  • Gemini JPA - support for JPA persistence in OSGi
  • Gemini DBAccess - modularised JDBC drivers for use in OSGi
  • Gemini Management - JMX management for OSGi
  • Gemini Naming - support for JNDI naming in OSGi
Until recently, integrating the Gemini projects into Virgo has been a fairly specialised skill. As we'll see in a workshop on Thursday, things are now really starting to come together and Gemini Blueprint, Gemini Web, and various other Gemini components can now be deployed and used in Virgo. We'll also be touching on how to manage and configure WABs in Virgo as part of a workshop later this morning.

So much for the past. What about the future? The primary goal of Virgo 3.0 is better integration with other EclipseRT technologies such as Equinox, Jetty, and p2. This will provide richer function and greater choice for enterprise applications as they migrate to OSGi. We'll be hearing about how Virgo and EclipseRT technologies work together on Wednesday. The recently released third milestone of this release shows the progress towards the goal of better integration with EclipseRT:
  • Virgo now uses Equinox Configuration Admin and Event Admin services instead of their Apache equivalents.
  • Virgo has moved to the new framework hooks model for isolation in Equinox which is part of the OSGi 4.3 standard soon to be issued for public review.
  • Virgo now provides two web servers: Virgo Tomcat Server, which continues to be based on Gemini Web, and Virgo Jetty Server. Some exciting technology for modularising the view portion of web applications is also emerging into the Virgo mainstream as we'll hear on Wednesday.
  • p2 support in Virgo is also making good progress although this is currently in prototype form.
Development tooling is also entering an exciting new phase which will greatly help applications migrate to enterprise OSGi:
  • The Libra project, which will be presented on Wednesday, is providing development tooling for building WABs and will ship in the Eclipse Indigo release train later this year.
  • The Virgo development tooling has entered the Eclipse IP process and will soon emerge in the Virgo project. Over time, the standards-based features of the Virgo tooling will be factored out and migrated into Libra.
  • OSGi runtime launchers are also being contributed to Libra. I would hope to see a Virgo launcher for Libra emerge so that Libra can start to take over from the Virgo tooling for many aspects of enterprise OSGi application development.
If you're attending EclipesCon, come and hear the talks, join in the workshops and BoFs and help to shape incremental OSGi in the Virgo, Gemini, and Libra projects.

Friday, March 18, 2011

Importing packages into a 3rd party bundle

I regularly come across the situation where someone needs to import additional packages into a 3rd party OSGi bundle. Sometimes this is because the bundle manifest is incorrect, but more often it's because the bundle uses Class.forName or similar to load classes that are somehow configured at runtime.


Unless the bundle is signed, it's possible to edit the manifest to add the import, but this is a really bad idea. If the original bundle needs upgrading, you have to remember to re-edit the manifest. Also, it's easy to mess something up in the process. Finally, it's bad practice to change the contents of a bundle without modifying the bundle version.

Thankfully, there is a simple way to add imports to a bundle without resorting to such invasive surgery: use a fragment bundle. Fragment bundles can extend the manifest of their host in various ways. In particular, they can add package imports. The fragment bundle need only contain a manifest, for example:


Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-SymbolicName: my.fragment
Bundle-Version: 1
Bundle-Name: My Fragment
Fragment-Host: host.symbolic.name;bundle-version="[2.4,2.5)"
Import-Package: extra.package;version="[1.1,1.2)"

More than one fragment bundle can attach to a given host bundle, so this technique can be used when unrelated groups of packages need to be imported: a separate fragment can be used for each group.

Each fragment bundle needs to be installed before its host is resolved at which point the fragment's manifest is merged into the host's and the resultant Bundle instance imports the packages added by the fragment. On Virgo, this can be achieved by placing the fragment bundles in repository/usr. When the host is deployed, the fragments are automatically installed along with the host.

Note: make sure you use the "bundle-version" attribute of Fragment-Host. "version" is not a standard attribute of Fragment-Host and if you use it accidentally in place of "bundle-version", chances are your fragment will not attach to its host.

Thursday, March 17, 2011

Virgo 3.0.0.M03 and EclipseCon

Milestone 3 of Virgo 3.0 is available for download. Apart from several bug fixes, this milestone includes a contribution from Tom Watson who is collaborating with the Virgo team to prepare the region digraph for its move to Equinox so that it is available to all Equinox users rather than just Virgo users.

We will be using milestone3 in tutorials at next week's EclipseCon. If you're attending, watch out for Virgo committers delivering the following sessions:

Hope to see you there!

Monday, February 28, 2011

Thread Context Class Loading in Virgo

A number of popular open source libraries, particularly persistence providers, use thread context class loading to load application types. This is an issue in OSGi once an application is divided into more than one bundle. Each bundle has its own class loader and no particular one of these bundle class loaders is necessarily suitable for loading all the application classes that libraries need to load.

Virgo overcomes this problem with its notion of a scoped application. A scoped application consists of either a PAR archive file containing the application artefacts (bundles, property files, plans, etc.) or a scoped plan referring to the application  artefacts. The scope limits the visibility of application packages and services so that scoped applications do not interfere with each another. But Virgo also creates an additional synthetic context bundle for each scope and it's this bundle's class loader which is used for thread context class loading when the application calls out to libraries.

The synthetic context bundle is very simple. It's a bundle which uses Virgo's import-bundle manifest header to import each of the bundles in the scope. Thus all the packages exported by the application are available for thread context class loading by libraries. There is a minor restriction implied by this approach: no two bundles in a scoped application may export the same package, which would be pretty unusual.

The following example should help make things clear:

An Example Scoped Application
Here a scoped application contains two bundles A and B. A calls B which calls out to a library bundle L. If L uses the thread context class loader to load application types, it will have access to all the packages exported by bundles A and B. The synthetic context bundle, which imports bundles A and B, provides the thread context class loader for the scoped application.

This simple solution enables a variety of existing open source libraries, once converted to OSGi bundles, to function correctly in their use of thread context class loading.

Thursday, February 10, 2011

Stumbling towards a better design

Some programs have a clear design and coding new features is quick and easy. Other programs are a patchwork quilt of barely comprehensible fragments, bug fixes, and glue. If you have to code new features for such programs, you're often better off rewriting them.

However there is a middle ground that I suspect is pretty common in these days of clean code and automated test suites: you have a good program with a clear design, but as you start to implement a new feature, you realise it's a force fit and you're not sure why. What to do?

I've recently being implementing a feature in Eclipse Virgo which raised this question and I hope sheds some light on how to proceed. Let's take a look.


The New Feature
I've been changing the way the Virgo kernel isolates itself from applications. Previously, Equinox supported nested OSGi frameworks and it was easy to isolate the kernel from applications by putting the applications in a nested framework (a "user region") and sharing selected packages and services with the kernel. However, the nested framework support is being withdrawn in favour of an OSGi standard set of framework hooks. These hooks let you control, or at least limit, the visibility of bundles, packages, and services -- all in a single framework.

So I set about re-basing Virgo on the framework hooks. The future looked good: eventually the hooks could be used to implement multiple user regions and even to rework the way application scoping is implemented in Virgo.


An Initial Implementation
One bundle in the Virgo kernel is responsible for region support, so I set about reworking it to use the framework hooks. After a couple of weeks the kernel and all its tests were running ok. However, the vision of using the hooks to implement multiple user regions and redo application scoping had receded into the distance given the rather basic way I had written the framework hooks. I had the option of ignoring this and calling "YAGNI!" (You Ain't Gonna Need It!). But I was certain that once I merged my branch into master, the necessary generalisation would drop down the list of priorities. Also, if I ever did prioritise the generalisation work, I would have forgotten much of what was then buzzing around my head.


Stepping Back
So the first step was to come up with a suitable abstract model. I had some ideas when we were discussing nested frameworks in the OSGi Alliance a couple of years ago: to partition the framework into groups of bundles and then to connect these groups together with one-way connections which would allow certain packages and services to be visible from one group to another.

Using Virgo terminology, I set about defining how I could partition the framework into regions and then connect the regions together using package, service, and bundle filters. At first it was tempting to avoid cycles in the graph, but it soon became clear that cycles are harmless and indeed necessary for modelling Virgo's existing kernel and user region, which need to be connected to each other with appropriate filters.


A Clean Abstraction
Soon I had a reasonably good idea of the kind of graph with filters that was necessary, so it was tempting to get coding and then refactor the thing into a reasonable shape. But I had very little idea of how the filtering of bundles and packages would interact. In the past I've found that refactoring from such a starting point can waste a lot of time, especially when tests have been written and need reworking. Code has inertia to being changed, so its often better to defer coding until I get a better understanding.

To get a clean abstraction and a clear understanding, while avoiding "analysis paralysis", I wrote a formal specification of these connected regions. This is essentially a mathematical model of the state of the graph and the operations on it. This kind of model enables properties of the system to be discovered before it is implemented in code. My friend and colleague, Steve Powell, was keen to review the spec and suggest several simplifications and before long, we had a formal spec with some rather nice algebraic properties for filtering and combining regions.

To give you a feel for how these properties look, take this example which says that "combining" two regions (used when describing the combined appearance of two regions) and then filtering is equivalent to filtering the two regions first and then combining the result:


Being a visual thinker and to make the formal spec more useful to non-mathematicians, I also drew plenty of pictures along the way. Here's an example graph of regions:


A New Implementation
I defined a RegionDigraph ("digraph" is short for "directed graph") interface, implemented it, and defined a suit of unit tests to give good code coverage. I then implemented a fresh collection of framework hooks in terms of the region digraph and then ripped out the old framework hooks and code supporting what in retrospect was a poorly formed notion of region membership and replaced this with the new framework hooks underpinned by the region digraph.


I Really Did Need It (IRDNI?)
It took a while to get all the kernel integration tests running again, mainly because the user region needs to be configured so that packages from the system bundle (which belongs in the kernel region) are imported along with some new services such as the region digraph service.

As problems occurred, I could step back and think in terms of the underlying graph. By writing appropriate toString methods on Region and RegionDigraph implementation classes, the model became easier to visualise in the debugger. This gives me hope that if and when other issues arise, I will have a better chance of debugging them because I can understand the underlying model.

A couple of significant issues turned up along the way, both related to the use of "side states" when Virgo deploys applications.

The first is the need to temporarily add bundle descriptions to the user region.

The second is the need to respect the region digraph when diagnosing resolver errors. This is relatively straightforward when deploying and diagnosing failures. It is less straightforward when dumping resolution failure states for offline analysis: the region digraph also needs to be dumped so it can also be used in the offline analysis.

These issues would have been much harder to address in the initial framework hooks implementation. The first issue would have involved some fairly arbitrary code to record and remove bundle descriptions from the user region. The second would have been much trickier as there was a poorly defined and overly static notion of region membership which wouldn't have lent itself to including in a state dump without looking like a gross hack. But with the region digraph it was easy to create a temporary "coregion" to contain the temporary bundle descriptions and it should be straightforward to capture the digraph alongside the state dump.

Ok, so I'm convinced that the region digraph is pulling its weight and isn't a bunch of YAGNI. But someone challenged me the other day by asking "Why do the framework hooks have to be so complex?".

Unnecessary Complexity?
Well, firstly the region digraph ensures consistent behaviour across the five framework hooks (bundle find, bundle event, service find, service event, and resolver hooks), especially regarding filtering behaviour, treatment of the system bundle, and transitive dependencies (i.e. across more than one region connection). This consistency should lead to fewer bugs, more consistent documentation, and ease of understanding for users.

Secondly, the region digraph is much more flexible than hooks based on a static notion of region membership: bundles may be added to the kernel after the user region has been created, application scoping should be relatively straightforward to rework in terms of regions thus giving scoping and regions consistent semantics (fewer bugs, better documentation etc), and multiple user regions should be  relatively tractable to implement.

Thirdly, the region digraph should be an excellent basis for implementing the notion of a multi-bundle application. In the OSGi Alliance, we are currently discussing how to standardise the multi-bundle application constructs in Virgo, Apache Aries, the Paremus Service Fabric, and elsewhere. Indeed I regard it as a proof of concept that the framework hooks can be used to implement certain basic kinds of multi-bundle application. As a nice spin-off, the development of the region digraph has resulted in several Equinox bugs being fixed and some clarifications being made to the framework hooks specification.


Next Steps
I am writing this while the region digraph is "rippling" through the Virgo repositories on its way into the 3.0 line. But this item is starting to have a broader impact. Last week I gave a presentation on the region digraph to the OSGi Alliance's Enterprise Expert Group. There was a lot of interest and subsequently there has even been discussion of whether the feature should be implemented in Equinox so that it can be reused by other projects outside Virgo.

Postscript (30 March 2010)
The region digraph is working out well in Virgo. We had to rework the function underlying the admin console because there is no longer a "surrogate" bundle representing the kernel packages and services in the user region. To better represent the connections from the user region to the kernel, the runtime artefact model inside Virgo needs to be upgraded to understand regions directly. This is work in progress in the 3.0 line.

Meanwhile, Tom Watson, an Equinox committer, is working with me to move the region digraph code to Equinox. The rationale is to ensure that multiple users of the framework hooks can co-exist (by using the region digraph API instead of directly using the framework hooks).

Tom contributed several significant changes to the digraph code in Virgo including persistence support. When Virgo  dumps a resolution failure state, it also dumps the region digraph. The dumped digraph is read back in later and used to provide a resolution hook for analysing the dumped state, which ensures consistency between the live resolver and the dumped state analysis.

Projects

OSGi (130) Virgo (59) Eclipse (10) Equinox (9) dm Server (8) Felix (4) WebSphere (3) Aries (2) GlassFish (2) JBoss (1) Newton (1) WebLogic (1)

Blog Archive