The PBI Format for 9.0 and Beyond
This document is intended to serve as a place for brainstorming ideas to improve and enhance the PBI (Push Button Installer) format for PC-BSD® 9.0 and beyond.
What is the PBI format?
The idea behind creating the PBI format is that one of the things holding back mainstream adoption of open-source desktops is the package management format. With almost all open-source desktops, software is simply treated as part of your system itself. Thus when performing an update of some seemingly trivial application, you run the risk of potentially needing to upgrade other packages which in turn could break other critical parts of your desktop.
- User wants to upgrade to latest version of FireFox. However, this in turn requires new version of libFoo*,
- which is also used by Xorg and many other libs/apps. In order to perform update of libFoo, Xorg and any other
- dependent libraries must be updated as well. Newer Xorg may in turn require different version of libOtherFoo and
- so-on and so forth. All it takes is one failure at any point of the process, and the user may end up with a broken
- system, no GUI and a huge loss of productive time.
While package management systems have gotten better at resolving these issues, trying to fix conflicts and prevent breakage before it occurs, it still does not fix the underlying problem, that every software package is simply a part of the system, and pulling one any one thread has the potential of causing a break somewhere farther down the line.
The PBI format tries to correct the underlying flaw in this system for desktop computing. Rather than making every application a part of your base system, PBIs are self-contained, including their own dependent library tree and related data. As a result, when you install a PBI there are no dependency issues to resolve, and applications can be added / removed freely, without fear of causing breakage to the desktop or any other installed software.
Why update the format?
The current PBI format, while still very useful is beginning to show its age. The format when started was a new concept for the open-source world and now, after many years of real-world usage, it is time to take what we've learned, refine and implement it in a cleaner way, for the benefit of both PC-BSD® and FreeBSD users.
The current PBI format
A .pbi file today is made up of several different components and organized as indicated below.
- At the front of a PBI file is a small C binary, which makes the application executable, and starts the installation process.
- Next is a small tarball which contains configuration meta-data, such as icon information, mime-data and install / uninstall scripts - all used by the GUI portion of the installer.
- After the tarball, there is a location for an optional "icon" which can be used by thumbnailer utilities, to display the applications icon in file-managers.
- Lastly is the program archive, which is made up of a LZMA compressed tarball for both maximum compression and fast extraction.
By using these components, the QT4 PBI installer is able to run, confirm some settings from the user, such as destination directory, and then extract the program archive to disk. The installer reads much of the included meta-data and uses it in the creation of XDG formatted desktop / menu icon creation, as well as registering mime types, etc.
While the current format is useful and working in many ways, it is lacking some features and using some "hackish" methods to force programs (which often are not designed to be run self-contained) to work for end-users. Below are some of the ideas we want to implement in the new format to correct these issues, and make the format work on traditional FreeBSD.
One of the drawbacks of the current implementation is its handling of libraries and resource files. With each program including its own complete set of libraries, there is often much duplication between applications, causing a waste of hard-disk space, as well as memory usage during runtime, since libraries are shared via inode lookups.
To improve upon this situation, a managed directory can be used for "sharing" the common files between applications, allowing less disk-space to be used and memory to be saved during runtime. This can be accomplished by creating hash-lists of files within a package and installing resources to a shared directory, creating hard-links back to the original file-names within the installed PBIs tree.
Pitfall: Different applications may include the same library with the same version (and sub-version and sub-sub...), but with different compile-time options or generated by different compilers. Thus either the PBI installer must compare all candidates byte-wise (making nonsense of the choice of LZMA for fast decompression) or (better?) a strong hash is needed for an abstract of the library, which must of course include all compile-time options. For C/C++, tools to extract or generate these informations exist, but care has to be taken for other major languages.
Create new PBI management system for add / remove of software
With the PBI format now, installation is performed by each application itself, using the included QT4-based binary installer. Other utilities for PBI management are also written in QT4, such as the creator / removal tool.
This should be changed, with a new suite of tools written in shell / C, that perform the actual installation, creation and removal portions. This would allow a variety of front-ends to be developed, and help break the PBI format from being more QT/KDE specific, allowing FreeBSD users to install support for the PBI format via ports. This would also removed the need for a binary loader at the front of .pbi files, instead making the file-format look like this:
[meta-data]|[optional icon]|[program archive]
pbi_add / pbi_delete / pbi_create / pbi_info
Not very original, but easy to remember for those coming from a BSD background.
Explore the usage of $ORGIN as a replacement for LD_LIBRARY_PATH
One of the ways the current version of PBI handles program execution using self-contained libraries is by creating wrapper scripts for executable, and setting some variables such as LD_LIBRARY_PATH. While this works, it has a few problems in that it is A: Hack-ish and B: Causes breakage with setuid.
One possible solution would be with the usage of $ORGIN for the compiling of PBI applications. This feature could eliminate the idea of needing to use LD_LIBRARY_PATH and fix issues with using some setuid programs, such as VirtualBox in user-mode.
There is another way: Use same approach as used this util: http://www.freshports.org/devel/chrpath/ I tested such situation: 1. Build PBI to directory /usr/Programs/VERY-long-name-here-very-very-long-and-unique-and-fixed-length 2. For all executable files, change DT_RPATH with chrpath; 2. For all other binary files; find all byte positions of this PATH with strings, change for example PATH/etc to NEW_PATH/etc, and write back; dd if=/file-with-zero of=$FILE offset=$offset-from-strings-output bs=1 count=strlength(PATH/ETC) dd if=/file-with-new-path of=$FILE offset=$offset-from-strings-output bs=1 count=strlength(NEW_PATH/ETC) 3. Find all text files (.sh, .xml, etc.), change there PATHes too.
With such tweak scripts, I was able to build PBI to one PREFIX, and dynamically install it to other PREFIX. Program worked just fine. However, need more testing with some complex Programs, OpenOffice probably, etc.
Integrate PBI building closer with FreeBSD Ports
The most common way to build a PBI currently, is by using the PBI Builder Software. This software acts as a front-end to the FreeBSD ports tree, by doing port builds with a custom PREFIX for a PBI, and using "modules" which contain additional meta-data used in the creation of the resulting PBI file.
One possible solution to improve upon this method, is by creating a new Mk/ entry, that will allow the usage of "make pbi" as a build option to ports. This make would then need to perform some steps to build a workable PBI file.
- Run a complete 'make' on the port with PREFIX set to a target location for this PBI. (Build port + dependencies)
- Include any optional meta-data for desktop icons / mime registration
- Run 'pbi_create' on the resulting PREFIX that the port + dependencies were installed to, resulting in a finished PBI file.
While this portion of the build is fairly straight forward and should be easy, the difficult part will be in the inclusion of additional meta-data to the FreeBSD ports tree. Meta-data will be necessary for most applications, since the PBI manager will need to know which binaries need to be placed in the systems $PATH for execution, what mime-entries should be created, optional plugins / ports to include, rc scripts, etc.