TestOSGi task fails when including native Netty bundle

chrisr3 · March 22, 2021, 12:19pm

Hi,

My TestOSGi task is failing because this unwanted transitive dependency is being waved in my general direction:

io.netty:netty-transport-native-kqueue:4.1.60.Final:osx-x86_64

The stack trace is:

! Failed to start bundle io.netty.transport-native-kqueue-4.1.60.Final, exception Unable to resolve io.netty.transport-native-kqueue [6](R 6.0): missing requirement [io.netty.transport-native-kqueue [6](R 6.0)] osgi.native; (&(osgi.native.osname~=macosx)(osgi.native.processor~=x86_64)) Unresolved requirements: [[io.netty.transport-native-kqueue [6](R 6.0)] osgi.native; (&(osgi.native.osname~=macosx)(osgi.native.processor~=x86_64))]
org.osgi.framework.BundleException: Unable to resolve io.netty.transport-native-kqueue [6](R 6.0): missing requirement [io.netty.transport-native-kqueue [6](R 6.0)] osgi.native; (&(osgi.native.osname~=macosx)(osgi.native.processor~=x86_64)) Unresolved requirements: [[io.netty.transport-native-kqueue [6](R 6.0)] osgi.native; (&(osgi.native.osname~=macosx)(osgi.native.processor~=x86_64))]
        at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:4368)
        at org.apache.felix.framework.Felix.startBundle(Felix.java:2281)
        at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:998)
        at aQute.launcher.Launcher.start(Launcher.java:688)
        at aQute.launcher.Launcher.startBundles(Launcher.java:668)
        at aQute.launcher.Launcher.activate(Launcher.java:574)
        at aQute.launcher.Launcher.launch(Launcher.java:404)
        at aQute.launcher.Launcher.run(Launcher.java:185)
        at aQute.launcher.Launcher.main(Launcher.java:161)
        at aQute.launcher.pre.EmbeddedLauncher.executeWithRunPath(EmbeddedLauncher.java:170)
        at aQute.launcher.pre.EmbeddedLauncher.findAndExecute(EmbeddedLauncher.java:135)
        at aQute.launcher.pre.EmbeddedLauncher.main(EmbeddedLauncher.java:52)

I am running on Linux rather than OSX, so I understand why the framework cannot install this native bundle. But what I don’t understand is why Bnd is trying to install it the first place. I would have expected it to be filtered out in favour of an alternative which can satisfy my platform requirements.

I have the simplest of .bndrun files:

-tester: biz.aQute.tester.junit-platform

-runee: JavaSE-11
-runsystemcapabilities: ${native_capability}
-runfw: org.apache.felix.framework
-resolve.effective: active
-runproperties: \
    org.osgi.framework.bootdelegation=sun.reflect

-runrequires: \
    bnd.identity;id='${project.archivesBaseName}-tests',\
    bnd.identity;id='io.netty.transport-native-kqueue',\
    bnd.identity;id='junit-jupiter-engine',\
    bnd.identity;id='junit-platform-launcher'

-runstartlevel: \
    order=sortbynameversion,\
    begin=-1

-runbundles:

My Gradle dependencies are:

dependencies {
    testImplementation "org.junit.jupiter:junit-jupiter-api:5.7.1"
    testRuntimeOnly "org.junit.jupiter:junit-jupiter-engine:5.7.1"
    testRuntimeOnly "org.junit.platform:junit-platform-launcher:1.7.1"
    testRuntimeOnly "org.apache.felix:org.apache.felix.framework:6.0.4"
    testRuntimeOnly "io.netty:netty-transport-native-kqueue:4.1.60.Final"
    testRuntimeOnly "org.apache.activemq:artemis-server-osgi:2.15.0"
}

I know a Gradle incantation which can exclude the bundle with the osx-x86_64 classifier, but suspect that MacOSX users might object to me casting it .

Is this just anothe case of b0rked OSGI metadata, please? Am I missing some important piece of configuration? Or is there an issue with bundle resolution?

Thanks for any assistance here,
Cheers,
Chris

bjhargrave · March 22, 2021, 12:38pm

testOSGi does not know about dependencies. It runs the bundles listed in -runbundles which is either manually edited by the developer or generated by the a resolve operation.

You display an empty -runbundles which is odd since then nothing will happen.

bjhargrave · March 22, 2021, 12:43pm

You may want to look at https://github.com/bndtools/bnd/blob/master/bndtools.core/bndtools.gtk.linux.x86_64.bndrun which show how we define a processor specific bndrun file which includes processor specific things and excludes other things. It uses an included shared bndrun file for that which is common across all processors.

chrisr3 · March 22, 2021, 12:43pm

Hi, thanks for replying. I removed the runbundles for brevity, since I generate them using the Resolve task anyway. They actually look like this:

-runbundles: \
        bnd-artemis-tests;version='[1.0.0,1.0.1)',\
        io.netty.buffer;version='[4.1.60,4.1.61)',\
        io.netty.common;version='[4.1.60,4.1.61)',\
        io.netty.resolver;version='[4.1.60,4.1.61)',\
        io.netty.transport;version='[4.1.60,4.1.61)',\
        io.netty.transport-native-kqueue;version='[4.1.60,4.1.61)',\
        io.netty.transport-native-unix-common;version='[4.1.60,4.1.61)',\
        junit-jupiter-api;version='[5.7.1,5.7.2)',\
        junit-jupiter-engine;version='[5.7.1,5.7.2)',\
        junit-platform-commons;version='[1.7.1,1.7.2)',\
        junit-platform-engine;version='[1.7.1,1.7.2)',\
        junit-platform-launcher;version='[1.7.1,1.7.2)',\
        org.opentest4j;version='[1.2.0,1.2.1)'

The bnd-artemis bundle contains a single “stub” test, and only exists to prevent Bnd from hanging. (Because it does that when there are no tests to run.)

Cheers,
Chris

bjhargrave · March 22, 2021, 12:56pm

When I look at https://search.maven.org/remotecontent?filepath=io/netty/netty-transport-native-kqueue/4.1.60.Final/netty-transport-native-kqueue-4.1.60.Final.jar, I don’t even see a Bundle-NativeCode header in the manifest. However, this file https://search.maven.org/remotecontent?filepath=io/netty/netty-transport-native-kqueue/4.1.60.Final/netty-transport-native-kqueue-4.1.60.Final-osx-x86_64.jar does have a Bundle-NativeCode header in the manifest for macOS. Both have the same Bundle-SymbolicName (sigh).

So, your gradle build is somehow putting the osx-x86_64 classifier jar in the “source” list of bundles for the resolver. If you run the resolve task with --debug you should see all the data for the resolve operation. I expect you will see both classifier versions of the jar. The resolver seems to be picking the one you don’t want. So you may need to figure out how to convince gradle to exclude the osx-x86_64 classifier jar when not running on macOS.

chrisr3 · March 22, 2021, 2:21pm

So, your gradle build is somehow putting the osx-x86_64 classifier jar in the “source” list of bundles for the resolver. If you run the resolve task with --debug you should see all the data for the resolve operation. I expect you will see both classifier versions of the jar. The resolver seems to be picking the one you don’t want. So you may need to figure out how to convince gradle to exclude the osx-x86_64 classifier jar when not running on macOS.

That can be achieved (for a sufficiently recent Gradle version) using:

configurations {
    testRuntimeClasspath {
        resolutionStrategy.dependencySubstitution {
            substitute module('io.netty:netty-transport-native-kqueue') using module("io.netty:netty-transport-native-kqueue:$netty_version") withoutClassifier()
        }
    }
}

However, this doesn’t sound like it would work on MacOSX. But to be clear: is the underlying problem here that both kqueue bundles have the same BSN, which means that Bnd’s file repository can only store one of them? (I.e. the “wrong” one, in my case.)

Cheers,
Chris

bjhargrave · March 22, 2021, 3:01pm

Well the first question is whether Gradle is exposing you to both jars. If so, then they are both indexed by the FileSetRepository, and both resources are providers of capabilities to the resolve operation. Although I am not sure why your -runsystemcapabilities which, since you are running on linux, should have prevented the resolver from resolving the macos dependency in the one jar, thus excluding it.

Does running the resolve task with --debug shed any light?

chrisr3 · March 22, 2021, 3:31pm

Gradle is indeed exporting both jars from the testRuntimeClasspath configuration into Bnd:

2021-03-22T14:12:12.641+0000 [DEBUG] [aQute.bnd.repository.fileset.FileSetRepository] resolve: parsing /home/chris/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.60.Final/f4a482cc9d2c6027c4856c922c580869b2e7f01/netty-transport-native-kqueue-4.1.60.Final.jar
2021-03-22T14:12:12.643+0000 [DEBUG] [aQute.bnd.repository.fileset.FileSetRepository] resolve: parsing /home/chris/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.60.Final/b1f14b5d5b0e52e4385f1cd1a6c4934521a8ed3c/netty-transport-native-kqueue-4.1.60.Final-osx-x86_64.jar

followed by

2021-03-22T14:12:12.798+0000 [DEBUG] [aQute.bnd.repository.fileset.FileSetRepository] resolve: adding resource io.netty.transport-native-kqueue version=4.1.60.Final
2021-03-22T14:12:12.798+0000 [DEBUG] [aQute.bnd.repository.fileset.FileSetRepository] resolve: adding resource io.netty.transport-native-kqueue version=4.1.60.Final