Specific release url for each release (humble request)

GENERAL
1

Hi all,

is there a latest release url for bndtools besides https://bndtools.jfrog.io/bndtools/update-latest/?

This url is pointing to a composite p2 repo, which means that all contained repos are contacted and p2 metadata is downloaded. This is an increasing amount of network connections, which are really slowing down the installation process of bndtools on slow networks, like inside companies :wink:
see snippet below for the detailed info.

I was also using the direct jar url jar:https://bndtools.jfrog.io/bndtools/libs-release-local/org/bndtools/org.bndtools.p2/6.4.0/org.bndtools.p2-6.4.0.jar!/, but this is also taking long, as it requires to download the whole release jar, before accessing the p2 metadata inside.

So I wanted to ask if it would be feasible to provide also release specific p2 repository url.

This would spped up the general installation process significantly, and allows for more convenient consumption.

kind regards,
Peter

Fetching compositeArtifacts.xml from https://bndtools.jfrog.io/bndtools/update-latest/ (4,01kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/5.0.0/org.bndtools.p2-5.0.0.jar!/ (1,57kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/4.3.1/org.bndtools.p2-4.3.1.jar!/ (1,57kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/4.3.0/org.bndtools.p2-4.3.0.jar!/ (1,58kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/4.2.0/org.bndtools.p2-4.2.0.jar!/ (1,58kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/5.0.1/org.bndtools.p2-5.0.1.jar!/ (1,57kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/5.1.0/org.bndtools.p2-5.1.0.jar!/ (1,56kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/5.1.1/org.bndtools.p2-5.1.1.jar!/ (1,58kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/5.1.2/org.bndtools.p2-5.1.2.jar!/ (1,59kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/5.2.0/org.bndtools.p2-5.2.0.jar!/ (1,62kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/5.3.0/org.bndtools.p2-5.3.0.jar!/ (1,62kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/bnd-build/bndtools/bndtools-3.0.0.REL.jar!/ (1,45kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/bnd-build/bndtools/bndtools-3.1.1.REL.jar!/ (1,52kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/bnd-build/bndtools/bndtools-3.2.0.REL.jar!/ (1,59kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/bnd-build/bndtools/bndtools-3.3.0.REL.jar!/ (1,6kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/bnd-build/bndtools/bndtools-3.4.0.REL.jar!/ (1,63kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/bnd-build/bndtools/bndtools-3.5.0.REL.jar!/ (1,62kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/bnd-build/bndtools/bndtools-4.0.0.REL.jar!/ (1,65kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/bnd-build/bndtools/bndtools-4.1.0.REL.jar!/ (1,54kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/bnd-build/bndtools/bndtools-4.2.1.REL.jar!/ (1,57kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/6.0.0/org.bndtools.p2-6.0.0.jar!/ (1,7kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/6.1.0/org.bndtools.p2-6.1.0.jar!/ (1,95kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/update-4.2.2/org/bndtools/org.bndtools.p2/4.2.2/org.bndtools.p2-4.2.2.jar!/ (1,58kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/6.4.0/org.bndtools.p2-6.4.0.jar!/ (1,95kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/6.2.0/org.bndtools.p2-6.2.0.jar!/ (1,94kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/6.3.0/org.bndtools.p2-6.3.0.jar!/ (1,93kB)
Fetching artifacts.jar from https://bndtools.jfrog.io/artifactory/update-latest/libs-release-local/org/bndtools/org.bndtools.p2/6.3.1/org.bndtools.p2-6.3.1.jar!/ (1,95kB)
2

I think the problem is more that this is not really a p2 repository, but a jar that contains a p2 repository.

Due to the usage of jar:https:// .jar!/, syntax this is ultra slow, because p2 need to download the whole jar for each and every file inside (not only once as one might think)!

You can install m2e and then use the mvn:org.bndtools:org.bndtools.p2:6.4.0:jar scheme instead what is exactly designed for the use case of a maven deployed (packed) updatesite.

Or you can install m2e-pde and use maven artifacts directly in your target.

3

We are using bndtools inside oomph eclipse installations. Those are executed multiple times per day
(for each development task - providing a new, well-defined IDE setup and workspace).

Oomph is using internally p2 installation via p2 director.
So the official p2 behaviour is afaik …
1st network request - check for p2.index file ( defines in which format metadata is stored xz|jar|xml )
2nd network request - download content.(xz|jar|xml)
3rd network request - download compositeContent(jar|xml)
and then for each repo inside composite it is again starting from 1st request
2 and 3 are also happening for artifacts metadata

This is very inefficient implementation of p2. The developer installation experience is getting worse with each bndtools release, even though bndtools is not much increasing in bytecode size.

Inside artifactory a “generic” type deployment of p2 repos is feasible and could be used for providing a latest release url to one p2 repo.

4

This is very inefficient implementation of p2.

It is not P2 to blame for how bnd deploys its update-site…

I can only say that the used syntax is very inefficient, because this says:
download the jar and give me a pointer to one file, so your sequence is even:

1st network request - download the jar and check for p2.index
2nd network request - download the jar again and check for files provided in p2.index(xz|jar|xml)
… and so on … so each file request is a full download of the jar (32mb!) even though p2 caches some files after first retrival.

So first rule would be: never using jar:https://...!/ syntax in p2, this is simply a bad idea unless it is used with a local file like jar:file://...!/, and the cumulative nature of the bnd site makes this even worse.

Unless this is fixed by BND (you should better open an issue here: Issues · bndtools/bnd · GitHub ) I would recommend to donwload the jar, extract it and deploy it to some of your company web-servers and use that URL instead.

5

p2 has always been a giant pita because it was so badly documented. Originally Neil set it up to generate with the help of p2 eclipse code. (I think we used ant then …) I’ve been frustrated with p2 all my life (ok, they picked the basic ideas of OBR but missed the key idea completely. However, the inefficiency that you now run into was designed in, it is very indirect, using lots of referrals as well as having lots of redundant metadata.

However, we now use a virtual p2 repo from jfrogs. As I understand, they generate the metadata for p2 and seem to be the cause of your issue?

The problem I always ran into with p2 (it might have improved, one can always dream) is that it never documented its XML file formats. I recall that the idea was that the Java library creating the artifacts would be the API, not the file formats. (See the pattern with the .classpath files?) This was probably one of the worst ideas in the Eclipse eco system. File formats are much better than Java APIs for this stuff.

I think we also have a problem that modern Java’s removed MD5 and our old Eclipse code still uses it.

In the org.bndtools.p2 project we generate a p2 jar: generated/org.bndtools.p2.jar. I think this is the file you want? Could you verify that this JAR, if placed somewhere to download, would solve your problem? You can download the workspace and run cd org.bndtools.p2;../gradlew p2 to generate it.

I think we have two actions here. The quick fix for you is when we upload the p2 jar to Github packages. Would this work for you? Would you require snapshots as well or are releases good enough? We can add this to our Github Actions workflow.

However, in the long run we need to generate the p2 jar in another way as well because with the asinine changes, and the speed with which they make them, in Java it will run out one day. I think it should be straightforward to make an Export plugin for p2 but that requires:

  • valid p2 file format documentation
  • 5 days funding if nobody else wants to provide a PR for this (contact me)

And as side note, I don’t want to use Tycho for this … I’ll accept a PR for the build and release process that uses this but I’ve lots of bad memories that I do not want to trigger. :frowning:

6

@pkriens This has actually nothing to do with file format or documentation or whatever complained, its a plain deployment issue.

Deploying a P2 repo as one fat jar is and was never part of P2 is just a misuse of the jar protocol here. so the fix is rather simple, do not deploy a fat jar but an exploded version of the jar (I don’t think that github packages makes anything better here but you probably want to use github-pages).

Second issue is, that it seems here the “latest” is not really pointing to the latest release but to all current and previous releases, and therefore require additional downloads, so again nothing specific to P2 nor something that requires specific documentations.

Nerveless if you think you are missing an information regarding P2 for any of these aspects just let me know.

7

Actually its the choice of the BNDTools project if one wants to re-implement everything, but just in case you can save you 4.5 of the 5 days :sweat_smile: by either using:

Tycho p2 Extras Plugin – tycho-p2-extras:publish-features-and-bundles (this accepts just a plain folder of jars and generates a site from it)

or you might want to use:

https://tycho.eclipseprojects.io/doc/latest/tycho-p2/tycho-p2-repository-plugin/assemble-maven-repository-mojo.html

that assembles a site using pure maven build (jetty uses that) with all artifacts stored in a maven repository (e.g. maven central)

8

No, but there are other problems.

9

So if we just explode the org.bndtools.jar somewhere then we’re done?

Could you make a PR for the release process and the snapshot process? It should be clear p2 and I are not friends and you seem to be knowledgeable.

And nothing ever is 1/2 day with Tycho :frowning:

… And having a bnd bndrun p2 exporter would be cool!

10

So if we just explode the org.bndtools.jar somewhere then we’re done?

Yes, and latest should then always point to the single latest update site (not a composite)

Could you make a PR for the release process and the snapshot process?

I have no deeper knowledge about the release and deployment procedure so I can only give some hints, e.g. this uses a github action do deply an update-site to github-pages:

So given you already have a build that produces that jar, it kind of

  • unzip org.bndtools.p2-6.4.0.jar
  • … copy directory structure somwhere … (e.g. github pages)

And nothing ever is 1/2 day with Tycho :frowning:

I actually was a bit cautious here so maybe an hour or so :slight_smile: is more realistic :wink:

… And having a bnd bndrun p2 exporter would be cool!

Well actually everything is at your hands:

That’s what actually Tycho p2 Extras Plugin – tycho-p2-extras:publish-features-and-bundles is doing in a more convenient way from maven, and you can just use a single module pom.xml so its quite easy, so I’ll push a demo for this.

11

The demo is now here:

12

What purpose is this? We already have the p2 jar?

What is needed to explode it on a public place? Am I missing something?

13

What purpose is this?

You complained that its hard to generate a p2 site, the example shows how one can produce one with just having a jar (build by whatever tool) without any knowledge of P2 internals.

We already have the p2 jar?

Seems like that …

What is needed to explode it on a public place?

An unzip tool and a web-server accessible with http

14

Thanks for all the input and discussion. I did not want to spawn a p2 vs obr discussion.
I’m investigating into the build and deploy of bnd to see if I can create and provide a PR for storing an exploded version of the release jar in addition to the existing version.
Thank you all for your time, contributions and hints.