Dynamically loading Guice modules as external plugins

Hi,

today I will present some experiments I am currently working on with Guice.

Nowadays, it is more and more common to see applications that users can customize simply by dropping "plugins" in some defined directory. A well-known example is Eclipse, but there are many such examples.

One way to realize this is to use an "OSGi container" and write plugins that obey OSGi specifications. But OSGi is quite a complex stuff (in particular if you intend to write an OSGi container).

While working with Guice, I felt the need to have the possibility to plugin new functionality to a Guice-based application by dropping jars containing classes implementing the Guice Module interface. But I did not want to depend on OSGi for this -quite simple- stuff.

So I have started some work on my own. The main problem for this kind of system is to discover all Modules that can be used to create a Guice Injector.

First of all, in order to permit several implementations, I have defined an interface for Module discovery:

public interface ModulesDiscoveryManager
{
    public Iterable<Module> getModules();
}

In this post, I will show two implementations of this interface.

The first implementation, ClassPathModulesDiscoveryManager, is quite straight-forward but does not fulfill completely the initial requirements (but we will reuse this class for the other implementation class).

public class ClassPathModulesDiscoveryManager implements ModulesDiscoveryManager
{
    static private final String MANIFEST_PATH = "META-INF/MANIFEST.MF";
    static private final String MANIFEST_GUICE_NAME = "Guice-Modules";

    public ClassPathModulesDiscoveryManager()
    {
        this(Thread.currentThread().getContextClassLoader());
    }

    public ClassPathModulesDiscoveryManager(ClassLoader loader)
    {
        _logger.log(Level.INFO, "Loading all dynamic Guice Module");
        try
        {
            // Get all MANIFEST files in the classpath
            Enumeration<URL> manifests = loader.getResources(MANIFEST_PATH);
            while (manifests.hasMoreElements())
            {
                addModule(loader, manifests.nextElement());
            }
        }
        catch (IOException e)
        {
            // Should not happen
            _logger.log(Level.SEVERE, "Could not add all dynamic Modules", e);
        }
    }

    private void addModule(ClassLoader loader, URL manifestUrl)
    {
        InputStream input = null;
        try
        {
            input = manifestUrl.openStream();
            Manifest manifest = new Manifest(input);
            String modules = manifest.getMainAttributes().getValue(MANIFEST_GUICE_NAME);
            if (modules != null)
            {
                for (String module: modules.split("[ \t]+"))
                {
                    addModule(loader, module);
                }
            }
        }
        catch (IOException e)
        {
            _logger.log(Level.SEVERE, "addModule problem with URL: " + manifestUrl, e);
        }
        finally
        {
            if (input != null)
            {
                close(input);
            }
        }
    }

    private void addModule(ClassLoader loader, String module)
    {
        try
        {
            Class<?> clazz = Class.forName(module, true, loader);
            _modules.add((Module) clazz.newInstance());
            _logger.log(Level.INFO, "Guice Module %s dynamically added.", module);
        }
        catch (Exception e)
        {
            _logger.log(Level.SEVERE, "addModule problem with: " + module, e);
        }
    }

    public Iterable<Module> getModules()
    {
        return _modules;
    }

    static final private Logger _logger =
        Logger.getLogger(ClassPathModulesDiscoveryManager.class.getName());
    private final List<Module> _modules = new ArrayList<Module>();
}

Note that the snippet above does not include the obvious close() method.
The main idea for discovering classes implementing Module is to require some little help from the plugins developers: add a special attribute in the "META-INF/MANIFEST.MF" file of every jar to define the Module class names (full name, including package) included in that jar:

Guice-Modules: package1.MyModule1 package2.MyModule2

ClassPathModulesDiscoveryManager implementation is quite straightforward, given a ClassLoader, the constructor searches for all META-INF/MANIFEST.MF resources, then looks for a "Guice-Modules" attribute, extracts each Module class (separated by spaces) and finally loads each class with the provided ClassLoader and instantiates it (it must have a public no-arg constructor).

Now comes the second implementation of the ClassPathModulesDiscoveryManager interface, DirectoryModulesDiscoveryManager; this one is a subclass of ClassPathModulesDiscoveryManager.

public class DirectoryModulesDiscoveryManager extends ClassPathModulesDiscoveryManager
{
    public DirectoryModulesDiscoveryManager(
        boolean includeSubDirs, File... directories)
    {
        this(Thread.currentThread().getContextClassLoader(), includeSubDirs,
            directories);
    }

    public DirectoryModulesDiscoveryManager(
        ClassLoader parent, boolean includeSubDirs, File... directories)
    {
        super(ClassLoaderHelper.buildClassLoader(
            Arrays.asList(directories), includeSubDirs, parent));
    }
}

This class is quite simple, but most of its work is performed by another helper class. That helper is actually in charge of creating a new ClassLoader that will add all jars found in a given set of directories to the classpath:

final class ClassLoaderHelper
{
    static ClassLoader buildClassLoader(
        List<File> directories, boolean includeSubDirs)
    {
        return buildClassLoader(
            directories, includeSubDirs, Thread.currentThread().getContextClassLoader());
    }

    static ClassLoader buildClassLoader(
        List<File> directories, boolean includeSubDirs, ClassLoader parent)
    {
        List<URL> allJars = new ArrayList<URL>();
        // Find all Jars in each directory
        for (File dir: directories)
        {
            fillJarsList(allJars, dir, includeSubDirs);
        }
        return new URLClassLoader(allJars.toArray(new URL[allJars.size()]), parent);
    }

    static private void fillJarsList(List<URL> jars, File dir, boolean includeSubDirs)
    {
        try
        {
            for (File jar: dir.listFiles(_jarsFilter))
            {
                jars.add(jar.toURI().toURL());
            }
     
            if (includeSubDirs)
            {
                for (File subdir: dir.listFiles(_dirsFilter))
                {
                    fillJarsList(jars, subdir, true);
                }
            }
        }
        catch (Exception e)
        {
            // Should not happen
        }
    }

    static final private FileFilter _jarsFilter = new FileFilter() {...};
    static final private FileFilter _dirsFilter = new FileFilter() {...};
}

The snippet above doesn't show the obvious FileFilters used to traverse directories and search for jar files.

The main bits are in:

• fillJarsList(): add all jar files to a List (after converting these files into URLs)
• buildClassLoader(): new URLClassLoader(allJars.toArray(new URL[allJars.size()]), parent);

That's all there is to it!

Now you can build Guice-based applications with Guice-friendly plugins! Just do that in your main entry point method:

ModulesDiscoveryManager manager = new DirectoryModulesDiscoveryManager(
    true, new File("SomePluginDirectory"));
Guice.createInjector(manager.getModules());

You can find a complete maven project with tests in this zip. The tests need 4 simple plugins that are built by 4 sub-projects.
To use the source code, just unzip the file somewhere on your disk, it will create a "plugins" directory that contains a maven pom.xml. Type mvn install there and it will create all plugins used by the tests, then the main "manager" project that contains the source code presented in this post.

Eventually, this code may be part of my future Guice-GUI framework which shall be released by the end of August.

Enjoy!