Dynamically loading Guice modules as external plugins

Hi,

today I will present some experiments I am currently working on with Guice.

Nowadays, it is more and more common to see applications that users can customize simply by dropping "plugins" in some defined directory. A well-known example is Eclipse, but there are many such examples.

One way to realize this is to use an "OSGi container" and write plugins that obey OSGi specifications. But OSGi is quite a complex stuff (in particular if you intend to write an OSGi container).

While working with Guice, I felt the need to have the possibility to plugin new functionality to a Guice-based application by dropping jars containing classes implementing the Guice Module interface. But I did not want to depend on OSGi for this -quite simple- stuff.

So I have started some work on my own. The main problem for this kind of system is to discover all Modules that can be used to create a Guice Injector.

First of all, in order to permit several implementations, I have defined an interface for Module discovery:

public interface ModulesDiscoveryManager
{
    public Iterable<Module> getModules();
}

In this post, I will show two implementations of this interface.

The first implementation, ClassPathModulesDiscoveryManager, is quite straight-forward but does not fulfill completely the initial requirements (but we will reuse this class for the other implementation class).

public class ClassPathModulesDiscoveryManager implements ModulesDiscoveryManager
{
    static private final String MANIFEST_PATH = "META-INF/MANIFEST.MF";
    static private final String MANIFEST_GUICE_NAME = "Guice-Modules";

    public ClassPathModulesDiscoveryManager()
    {
        this(Thread.currentThread().getContextClassLoader());
    }

    public ClassPathModulesDiscoveryManager(ClassLoader loader)
    {
        _logger.log(Level.INFO, "Loading all dynamic Guice Module");
        try
        {
            // Get all MANIFEST files in the classpath
            Enumeration<URL> manifests = loader.getResources(MANIFEST_PATH);
            while (manifests.hasMoreElements())
            {
                addModule(loader, manifests.nextElement());
            }
        }
        catch (IOException e)
        {
            // Should not happen
            _logger.log(Level.SEVERE, "Could not add all dynamic Modules", e);
        }
    }

    private void addModule(ClassLoader loader, URL manifestUrl)
    {
        InputStream input = null;
        try
        {
            input = manifestUrl.openStream();
            Manifest manifest = new Manifest(input);
            String modules = manifest.getMainAttributes().getValue(MANIFEST_GUICE_NAME);
            if (modules != null)
            {
                for (String module: modules.split("[ \t]+"))
                {
                    addModule(loader, module);
                }
            }
        }
        catch (IOException e)
        {
            _logger.log(Level.SEVERE, "addModule problem with URL: " + manifestUrl, e);
        }
        finally
        {
            if (input != null)
            {
                close(input);
            }
        }
    }

    private void addModule(ClassLoader loader, String module)
    {
        try
        {
            Class<?> clazz = Class.forName(module, true, loader);
            _modules.add((Module) clazz.newInstance());
            _logger.log(Level.INFO, "Guice Module %s dynamically added.", module);
        }
        catch (Exception e)
        {
            _logger.log(Level.SEVERE, "addModule problem with: " + module, e);
        }
    }

    public Iterable<Module> getModules()
    {
        return _modules;
    }

    static final private Logger _logger =
        Logger.getLogger(ClassPathModulesDiscoveryManager.class.getName());
    private final List<Module> _modules = new ArrayList<Module>();
}

Note that the snippet above does not include the obvious close() method.
The main idea for discovering classes implementing Module is to require some little help from the plugins developers: add a special attribute in the "META-INF/MANIFEST.MF" file of every jar to define the Module class names (full name, including package) included in that jar:

Guice-Modules: package1.MyModule1 package2.MyModule2

ClassPathModulesDiscoveryManager implementation is quite straightforward, given a ClassLoader, the constructor searches for all META-INF/MANIFEST.MF resources, then looks for a "Guice-Modules" attribute, extracts each Module class (separated by spaces) and finally loads each class with the provided ClassLoader and instantiates it (it must have a public no-arg constructor).

Now comes the second implementation of the ClassPathModulesDiscoveryManager interface, DirectoryModulesDiscoveryManager; this one is a subclass of ClassPathModulesDiscoveryManager.

public class DirectoryModulesDiscoveryManager extends ClassPathModulesDiscoveryManager
{
    public DirectoryModulesDiscoveryManager(
        boolean includeSubDirs, File... directories)
    {
        this(Thread.currentThread().getContextClassLoader(), includeSubDirs,
            directories);
    }

    public DirectoryModulesDiscoveryManager(
        ClassLoader parent, boolean includeSubDirs, File... directories)
    {
        super(ClassLoaderHelper.buildClassLoader(
            Arrays.asList(directories), includeSubDirs, parent));
    }
}

This class is quite simple, but most of its work is performed by another helper class. That helper is actually in charge of creating a new ClassLoader that will add all jars found in a given set of directories to the classpath:

final class ClassLoaderHelper
{
    static ClassLoader buildClassLoader(
        List<File> directories, boolean includeSubDirs)
    {
        return buildClassLoader(
            directories, includeSubDirs, Thread.currentThread().getContextClassLoader());
    }

    static ClassLoader buildClassLoader(
        List<File> directories, boolean includeSubDirs, ClassLoader parent)
    {
        List<URL> allJars = new ArrayList<URL>();
        // Find all Jars in each directory
        for (File dir: directories)
        {
            fillJarsList(allJars, dir, includeSubDirs);
        }
        return new URLClassLoader(allJars.toArray(new URL[allJars.size()]), parent);
    }

    static private void fillJarsList(List<URL> jars, File dir, boolean includeSubDirs)
    {
        try
        {
            for (File jar: dir.listFiles(_jarsFilter))
            {
                jars.add(jar.toURI().toURL());
            }
     
            if (includeSubDirs)
            {
                for (File subdir: dir.listFiles(_dirsFilter))
                {
                    fillJarsList(jars, subdir, true);
                }
            }
        }
        catch (Exception e)
        {
            // Should not happen
        }
    }

    static final private FileFilter _jarsFilter = new FileFilter() {...};
    static final private FileFilter _dirsFilter = new FileFilter() {...};
}

The snippet above doesn't show the obvious FileFilters used to traverse directories and search for jar files.

The main bits are in:

• fillJarsList(): add all jar files to a List (after converting these files into URLs)
• buildClassLoader(): new URLClassLoader(allJars.toArray(new URL[allJars.size()]), parent);

That's all there is to it!

Now you can build Guice-based applications with Guice-friendly plugins! Just do that in your main entry point method:

ModulesDiscoveryManager manager = new DirectoryModulesDiscoveryManager(
    true, new File("SomePluginDirectory"));
Guice.createInjector(manager.getModules());

You can find a complete maven project with tests in this zip. The tests need 4 simple plugins that are built by 4 sub-projects.
To use the source code, just unzip the file somewhere on your disk, it will create a "plugins" directory that contains a maven pom.xml. Type mvn install there and it will create all plugins used by the tests, then the main "manager" project that contains the source code presented in this post.

Eventually, this code may be part of my future Guice-GUI framework which shall be released by the end of August.

Enjoy!

New OSS project responsibility!

About one month ago, I have been assigned owner of the DesignGridLayout project.
I was very happy for that because:

1. I like this project a lot (since the first time I knew it, in November 2007)
2. But it seemed to have been dormant for more than one year

Most of the few issues open on the project were posted by me, hence I had to fix them by myself in the past.

Recently, I thought about completely refactoring DesignGridLayout source code, which I did, in order to improve its API furthermore and make it easier to enhance later on.

Since I have several improvement ideas, I have then contacted the curren project owner to ask him if he was interested in my work. His answer was that unfortunately he is much too busy to keep the project living and thus he suggested that I could become owner of the project and keep it alive! That's how it all happened!

From then on, I have started to take hold of the project and suggest important changes:

• more open license (GPL currently)
• upload to maven2 repository
• switching from CVS to Subversion
• API improvements

I truly believe that DesignGridLayout is a revolutionary LayoutManager for Swing applications but unfortunately, there was not much advertising about it so far.

If you are fed up of standard Swing layouts (GridBagLayout in particular;-)) and you find 3rd party OSS layouts a bit steep to learn, then you should definately take a look at DesignGridLayout, it is worth your (little) time!

I'll blog more about DesignGridLayout soon after I have released a new version including the whole refactored source code.

How to intercept Guice instance creation?

In this first post, I'll talk about some of my experiments with Google Guice Dependency Injection library.

I am using Guice 1.0 although 2.0 is planned for release this summer (but I prefer to wait and see...) Anyway, the problem I'm going to talk about seems not to be addressed in the future Guice 2.0 either (although the solution might be simpler later in 2.0, we'll see).

Guice is a very good library for Dependency Injection, it is heavily depending on Java 5 Generics thus ensuring very high confidence in your code robustness at compile-time. It is also using annotations a lot.

Guice API is quite simple for the end user, however it may be considered too simple, considering a few missing features that look fundamental.

As an example, I have a use case where I need to process annotations (not Guice annotations, but other unrelated annotations) to any class instance created and injected by Guice. How can I do that? Guice 1.0 (and 2.0 as far as I understand) provides no "hook" system that would enable me to be notified whenever a new instance has been created and injected by Guice Injector. But that's exactly what one needs to be able to add the necessary annotation processing code for all new instances created by Guice!

So how can we do that?

After several experiments, I think I have found one way, not perfect (I'll explain why later), but still acceptable, that works fine with Guice 1.0 without any hack.

I call this way "Guice creation hooks through Scope interception". Wait a minute! Guice does not provide any way to intercept Scope methods! Correct, in Guice, Scopes are not first-class citizen and cannot be injected, intercepted... That seems quite natural however (just imagine a Scope that can be scoped!).

So how do we intercept Scope methods then? Well, there are not too many possibilities: you have to create your own Scopes, and have them delegate to real Scopes (like Scopes.SINGLETON which is one of the few standard Scopes that Guice brings out of the box).
Creating a new Scope means creating a class implementing the Scope interface, and defining an Annotation for it. Since we are creating Scopes that wrap other Scopes, we'll have to define one new Annotation for each Scope we want to wrap (that's the most boring part).

In my example, I'll define only 2 annotations used to define 2 scopes that respectively wrap Singleton and Guice default scope (which has no specific name):

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@ScopeAnnotation
public @interface HookedSingleton
{
}

And exactly the same for the "no scope" annotation:

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@ScopeAnnotation
public @interface HookedDefaultScope
{
}

Unfortunately I couldn't find better names for these annotations:-(

Now let's see the implementation of the class implementing Scope:

public class HookedScope implements Scope
{
    private final Scope _wrappedScope;
    private final List<CreationHook> _hooks = new ArrayList<CreationHook>();

    public HookedScope(Scope scope)
    {
        _wrappedScope = scope;
    }

    public void addHook(CreationHook hook)
    {
        _hooks.add(hook);
    }

    public <T> Provider<T> scope(Key<T> key, final Provider<T> creator)
    {
        Provider<T> hookedCreator = new Provider<T>()
        {
            public T get()
            {
                // Actually create a new T instance
                T instance = creator.get();
                // Add processing to newly created instance
                for (CreationHook hook: _hooks)
                {
                    hook.process(instance);
                }
                return instance;
            }
        };
        // Make sure that we are called back whenever an instance is actually
        // created by unscoped
        if (_wrappedScope != null)
        {
            return _wrappedScope.scope(key, hookedCreator);
        }
        else
        {
            return hookedCreator;
        }
    }
}

This class has to be instantiated once for every scope used in the application (you want to hook ALL objects instantiation, don't you?)

This class works as follows:
First it stores the actual Scope that it is wrapping.
Then, when its scope() method is called, it is passed (creator, 2nd argument) a special Provider that is the one Provider that unconditionnally instantiates a T. Thus if we just decorate creator, and pass it along to the real Scope that we wrap, the decorated Provider will be called every time a new creation is about to happen. Our decorated Provider (named hookedCreator) first simply calls the real Guice creator, then it passes the newly created instance to any hook that is registered to it (through its addHook method).

Hooks just have to implement the CreationHook interface below:

public interface CreationHook
{
    public <T> void process(T instance);
}

Now the next step is to actually instantiate our HookedScopes and bind them to their
respective annotations, this is done in a Guice Module:

public class ScopeModule extends AbstractModule
{
    static HookedScope SINGLETON = new HookedScope(Scopes.SINGLETON);
    static HookedScope NO_SCOPE = new HookedScope(null);

    @Override protected void configure()
    {
        bindScope(HookedSingleton.class, SINGLETON);
        bindScope(HookedDefaultScope.class, NO_SCOPE);
    }
}

And you pass this module when you create the Guice Injector:

    Injector injector = Guice.createInjector(new ScopeModule(), ...);

Note the static SINGLETON and NO_SCOPE defined above, they will be needed so that you can
add your hooks when you need:

    CreationHook hook = ...;
    ScopeModule.SINGLETON.addHook(hook);
    ScopeModule.NO_SCOPE.addHook(hook);

As you can see, this leads to a new problem: you will have to add your hooks to each individual
HookedScope that you have created; in the example above that's only 2 scopes, but you may have more in your application!

Thus, a better way is to implement the whole logic of creating the HookedScopes and adding hooks in one single helper class:

final public class GuiceHookHelper
{
    static private final Map<Scope, Scope> _allScopes =
        new HashMap<Scope, Scope>();
    static private final List<CreationHook> _hooks =
        new ArrayList<CreationHook>();

    // Use the following improved scopes in bind()...in(scope) instead
    // of Scoped.SINGLETON or no scope at all.
    static final public Scope SINGLETON = hook(Scopes.SINGLETON);
    static final public Scope NO_SCOPE = hook(null);

    static public Scope hook(Scope wrappedScope)
    {
        Scope result = _allScopes.get(wrappedScope);
        if (result == null)
        {
            result = new HookedScope(wrappedScope);
            _allScopes.put(wrappedScope, result);
        }
        return result;
    }

    static public void addHook(CreationHook hook)
    {
        _hooks.add(hook);
    }

    static public void removeHook(CreationHook hook)
    {
        _hooks.remove(hook);
    }

    static private class HookedScope implements Scope
    {
        private final Scope _wrappedScope;

        private HookedScope(Scope scope)
        {
            _wrappedScope = scope;
        }

        public <T> Provider<T> scope(Key<T> key, final Provider<T> creator)
        {
            Provider<T> hookedCreator = new Provider<T>()
            {
                public T get()
                {
                    // Actually create a new T instance
                    T instance = creator.get();
                    // Add processing to newly created instance
                    for (CreationHook hook: _hooks)
                    {
                        hook.process(instance);
                    }
                    return instance;
                }
            };
            // Make sure that we are called back whenever an instance is actually
            // created by unscoped
            if (_wrappedScope != null)
            {
                return _wrappedScope.scope(key, hookedCreator);
            }
            else
            {
                return hookedCreator;
            }
        }
    }
}

Now this class centralizes all HookedScopes and all hooks, hiding all internals from the end user.
With this class, it is now easy to wrap other scopes and add hooks.

The main drawback of this approach is that you cannot use standard Guice scope annotations (@Singleton or no annotation at all) for scoping your classes. You have to use new annotations, even for the default scope! Compared with what it brings, I think this drawback is not that big.

These classes will be part of my future Guice-GUI framework (to be released this summer on SourceForge) where adding annotation processing to Guice instantiated classes makes sense when you want to use EventBus with its annotations for instance (common in GUI development).

For the time being, you can find the complete source code for the experiments presented in this post, a maven pom.xml and eclipse project settings here. Just unzip it and either import it into eclipse or install it with maven. The project includes a couple of unit tests (with TestNG).

Enjoy!