Difference between revisions of "PBI9 Format"

From PC-BSD Wiki
Jump to: navigation, search
(The PBI Format for 9.0 and Beyond)
m
(8 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 +
{{NavHome}}
 
== The PBI Format for 9.0 and Beyond ==
 
== The PBI Format for 9.0 and Beyond ==
  
This document is intended to serve as a place for brainstorming ideas to improve and enhance the PBI (Push Button Installer) format for PC-BSD 9.0 and beyond.  
+
This document is intended to serve as a place for brainstorming ideas to improve and enhance the PBI (Push Button Installer) format for PC-BSD® 9.0 and beyond.  
 +
 
 +
 
  
 
=== What is the PBI format? ===
 
=== What is the PBI format? ===
Line 7: Line 10:
 
The idea behind creating the PBI format is that one of the things holding back mainstream adoption of open-source desktops is the package management format. With almost all open-source desktops, software is simply treated as part of your system itself. Thus when performing an update of some seemingly trivial application, you run the risk of potentially needing to upgrade other packages which in turn could break other critical parts of your desktop.  
 
The idea behind creating the PBI format is that one of the things holding back mainstream adoption of open-source desktops is the package management format. With almost all open-source desktops, software is simply treated as part of your system itself. Thus when performing an update of some seemingly trivial application, you run the risk of potentially needing to upgrade other packages which in turn could break other critical parts of your desktop.  
  
  '''Example:'''
+
:'''Example:'''
  User wants to upgrade to latest version of FireFox. However, this in turn requires new version of libFoo*,  
+
::User wants to upgrade to latest version of FireFox. However, this in turn requires new version of libFoo*,  
  which is also used by Xorg and many other libs/apps. In order to perform update of libFoo, Xorg and any other  
+
::which is also used by Xorg and many other libs/apps. In order to perform update of libFoo, Xorg and any other  
  dependent libraries must be updated as well. Newer Xorg may in turn require different version of libOtherFoo and
+
::dependent libraries must be updated as well. Newer Xorg may in turn require different version of libOtherFoo and
  so-on and so forth. All it takes is one failure at any point of the process, and the user may end up with a broken
+
::so-on and so forth. All it takes is one failure at any point of the process, and the user may end up with a broken
  system, no GUI and a huge loss of productive time.  
+
::system, no GUI and a huge loss of productive time.  
  
While package management systems have gotten better at resolving these issues, trying to fix conflicts and prevent breakage before it occurs, it still doesn't fix the underlying problem, that every software package is simply a part of the system, and pulling one any one thread has the potential of causing a break somewhere farther down the line.
 
  
The PBI format tries to correct the underlying flaw in this system for desktop computing. Rather than making every application apart of your base system, PBI's are self-contained, including their own dependent library tree and related data. As a result, when you install a PBI there are no dependency issues to resolve, and applications can be added / removed freely, without fear of causing breakage to the desktop or any other installed software.  
+
While package management systems have gotten better at resolving these issues, trying to fix conflicts and prevent breakage before it occurs, it still does not fix the underlying problem, that every software package is simply a part of the system, and pulling one any one thread has the potential of causing a break somewhere farther down the line.
 +
 
 +
The PBI format tries to correct the underlying flaw in this system for desktop computing. Rather than making every application a part of your base system, PBI's are self-contained, including their own dependent library tree and related data. As a result, when you install a PBI there are no dependency issues to resolve, and applications can be added / removed freely, without fear of causing breakage to the desktop or any other installed software.
  
 
=== Why update the format? ===
 
=== Why update the format? ===
  
The current PBI format, while still very useful is beginning to show its age. The format when started was a new concept for the open-source world and now, after many years of real-world usage, it is time to take what we've learned, refine and implement it in a cleaner way, for the benefit of both PC-BSD and FreeBSD users.  
+
The current PBI format, while still very useful is beginning to show its age. The format when started was a new concept for the open-source world and now, after many years of real-world usage, it is time to take what we've learned, refine and implement it in a cleaner way, for the benefit of both PC-BSD® and FreeBSD users.  
 +
 
  
  
Line 37: Line 42:
  
 
By using these components, the QT4 PBI installer is able to run, confirm some settings from the user, such as destination directory, and then extract the program archive to disk. The installer reads much of the included meta-data and uses it in the creation of XDG formatted desktop / menu icon creation, as well as registering mime types, etc.
 
By using these components, the QT4 PBI installer is able to run, confirm some settings from the user, such as destination directory, and then extract the program archive to disk. The installer reads much of the included meta-data and uses it in the creation of XDG formatted desktop / menu icon creation, as well as registering mime types, etc.
 +
  
  
Line 42: Line 48:
  
 
While the current format is useful and working in many ways, it is lacking some features and using some "hackish" methods to force programs (which often are not designed to be run self-contained) to work for end-users. Below are some of the ideas we want to implement in the new format to correct these issues, and make the format work on traditional FreeBSD.
 
While the current format is useful and working in many ways, it is lacking some features and using some "hackish" methods to force programs (which often are not designed to be run self-contained) to work for end-users. Below are some of the ideas we want to implement in the new format to correct these issues, and make the format work on traditional FreeBSD.
 +
 +
  
 
==== Managed hard-links for resources ====
 
==== Managed hard-links for resources ====
Line 48: Line 56:
  
 
To improve upon this situation, a managed directory can be used for "sharing" the common files between applications, allowing less disk-space to be used and memory to be saved during runtime. This can be accomplished by creating hash-lists of files within a package and installing resources to a shared directory, creating hard-links back to the original file-names within the installed PBI's tree.
 
To improve upon this situation, a managed directory can be used for "sharing" the common files between applications, allowing less disk-space to be used and memory to be saved during runtime. This can be accomplished by creating hash-lists of files within a package and installing resources to a shared directory, creating hard-links back to the original file-names within the installed PBI's tree.
 +
 +
  
 
==== Create new PBI management system for add / remove of software ====
 
==== Create new PBI management system for add / remove of software ====
Line 53: Line 63:
 
With the PBI format now, installation is performed by each application itself, using the included QT4-based binary installer. Other utilities for PBI management are also written in QT4, such as the creator / removal tool.  
 
With the PBI format now, installation is performed by each application itself, using the included QT4-based binary installer. Other utilities for PBI management are also written in QT4, such as the creator / removal tool.  
  
This should be changed, with a new suite of tools written in shell / C, that perform the actual installation, creation and removal portions. This would allow a variety of front-ends to be developed, and help break the PBI format from being more QT/KDE specific, allowing FreeBSD users to install support for the PBI format via ports.  
+
This should be changed, with a new suite of tools written in shell / C, that perform the actual installation, creation and removal portions. This would allow a variety of front-ends to be developed, and help break the PBI format from being more QT/KDE specific, allowing FreeBSD users to install support for the PBI format via ports. This would also
 +
removed the need for a binary loader at the front of .pbi files, instead making the file-format look like this:
 +
 
 +
[meta-data]|[optional icon]|[program archive]
  
 
Possible names?:
 
Possible names?:
 +
 +
ports/sysutils/pbimanager?
  
 
pbi_add / pbi_delete / pbi_create / pbi_info
 
pbi_add / pbi_delete / pbi_create / pbi_info
  
 
Not very original, but easy to remember for those coming from a BSD background.
 
Not very original, but easy to remember for those coming from a BSD background.
 +
  
  
 
==== Explore the usage of $ORGIN as a replacement for LD_LIBRARY_PATH ====
 
==== Explore the usage of $ORGIN as a replacement for LD_LIBRARY_PATH ====
  
One of the ways
+
One of the ways the current version of PBI handles program execution using self-contained libraries is by creating wrapper scripts for executable, and setting some variables such as LD_LIBRARY_PATH. While this works, it has a few problems in that it is A: Hack-ish and B: Causes breakage with setuid.
 +
 
 +
One possible solution would be with the usage of [http://docs.sun.com/app/docs/doc/817-1984/appendixc-13?l=en&a=view $ORGIN] for the compiling of PBI applications. This feature could eliminate the idea of needing to use LD_LIBRARY_PATH and fix issues with using some setuid programs, such as VirtualBox in user-mode.
 +
 
 +
There is another way: Use same approach as used this util: http://www.freshports.org/devel/chrpath/
 +
I tested such situation:
 +
1. Build PBI to directory /usr/Programs/VERY-long-name-here-very-very-long-and-unique-and-fixed-length
 +
2. For all executable files, change DT_RPATH with chrpath;
 +
2. For all other binary files; find all byte positions of this PATH with strings, change for example PATH/etc to NEW_PATH/etc, and write back;
 +
dd if=/file-with-zero of=$FILE offset=$offset-from-strings-output bs=1 count=strlength(PATH/ETC)
 +
dd if=/file-with-new-path of=$FILE offset=$offset-from-strings-output bs=1 count=strlength(NEW_PATH/ETC)
 +
3. Find all text files (.sh, .xml, etc.), change there PATHes too.
 +
 
 +
With such tweak scripts, I was able to build PBI to one PREFIX, and dynamically install it to other PREFIX. Program worked just fine. However, need more testing with some complex Programs, OpenOffice probably, etc.
 +
 
 +
==== Integrate PBI building closer with FreeBSD Ports ====
 +
 
 +
The most common way to build a PBI currently, is by using the [http://wiki.pcbsd.org/index.php/Using_the_PBI_Builder PBI Builder Software]. This software acts as a front-end to the FreeBSD ports tree, by doing port builds with a custom PREFIX for a PBI, and using "[http://wiki.pcbsd.org/index.php/PBI_Module_Builder_Guide modules]" which contain additional meta-data used in the creation of the resulting PBI file.
 +
 
 +
One possible solution to improve upon this method, is by creating a new Mk/ entry, that will allow the usage of "make pbi" as a build option to ports. This make would then need to perform some steps to build a workable PBI file.
 +
 
 +
1. Run a complete 'make' on the port with PREFIX set to a target location for this PBI. (Build port + dependencies)
 +
 
 +
2. Include any optional meta-data for desktop icons / mime registration
 +
 
 +
3. Run 'pbi_create' on the resulting PREFIX that the port + dependencies were installed to, resulting in a finished PBI file.
 +
 
 +
While this portion of the build is fairly straight forward and should be easy, the difficult part will be in the inclusion of additional meta-data to the FreeBSD ports tree. Meta-data will be necessary for most applications, since the PBI manager will need to know which binaries need to be placed in the systems $PATH for execution, what mime-entries should be created, optional plugins / ports to include, rc scripts, etc.

Revision as of 10:28, 10 October 2012

Contents

The PBI Format for 9.0 and Beyond

This document is intended to serve as a place for brainstorming ideas to improve and enhance the PBI (Push Button Installer) format for PC-BSD® 9.0 and beyond.


What is the PBI format?

The idea behind creating the PBI format is that one of the things holding back mainstream adoption of open-source desktops is the package management format. With almost all open-source desktops, software is simply treated as part of your system itself. Thus when performing an update of some seemingly trivial application, you run the risk of potentially needing to upgrade other packages which in turn could break other critical parts of your desktop.

Example:
User wants to upgrade to latest version of FireFox. However, this in turn requires new version of libFoo*,
which is also used by Xorg and many other libs/apps. In order to perform update of libFoo, Xorg and any other
dependent libraries must be updated as well. Newer Xorg may in turn require different version of libOtherFoo and
so-on and so forth. All it takes is one failure at any point of the process, and the user may end up with a broken
system, no GUI and a huge loss of productive time.


While package management systems have gotten better at resolving these issues, trying to fix conflicts and prevent breakage before it occurs, it still does not fix the underlying problem, that every software package is simply a part of the system, and pulling one any one thread has the potential of causing a break somewhere farther down the line.

The PBI format tries to correct the underlying flaw in this system for desktop computing. Rather than making every application a part of your base system, PBI's are self-contained, including their own dependent library tree and related data. As a result, when you install a PBI there are no dependency issues to resolve, and applications can be added / removed freely, without fear of causing breakage to the desktop or any other installed software.

Why update the format?

The current PBI format, while still very useful is beginning to show its age. The format when started was a new concept for the open-source world and now, after many years of real-world usage, it is time to take what we've learned, refine and implement it in a cleaner way, for the benefit of both PC-BSD® and FreeBSD users.


The current PBI format

A .pbi file today is made up of several different components and organized as indicated below.

1. At the front of a PBI file is a small C binary, which makes the application executable, and starts the installation process.

2. Next is a small tarball which contains configuration meta-data, such as icon information, mime-data and install / uninstall scripts - all used by the GUI portion of the installer.

3. After the tarball, there is a location for an optional "icon" which can be used by thumbnailer utilities, to display the applications icon in file-managers.

4. Lastly is the program archive, which is made up of a LZMA compressed tarball for both maximum compression and fast extraction.


By using these components, the QT4 PBI installer is able to run, confirm some settings from the user, such as destination directory, and then extract the program archive to disk. The installer reads much of the included meta-data and uses it in the creation of XDG formatted desktop / menu icon creation, as well as registering mime types, etc.


New Goals

While the current format is useful and working in many ways, it is lacking some features and using some "hackish" methods to force programs (which often are not designed to be run self-contained) to work for end-users. Below are some of the ideas we want to implement in the new format to correct these issues, and make the format work on traditional FreeBSD.


Managed hard-links for resources

One of the drawbacks of the current implementation is its handling of libraries and resource files. With each program including its own complete set of libraries, there is often much duplication between applications, causing a waste of hard-disk space, as well as memory usage during runtime, since libraries are shared via inode lookups.

To improve upon this situation, a managed directory can be used for "sharing" the common files between applications, allowing less disk-space to be used and memory to be saved during runtime. This can be accomplished by creating hash-lists of files within a package and installing resources to a shared directory, creating hard-links back to the original file-names within the installed PBI's tree.


Create new PBI management system for add / remove of software

With the PBI format now, installation is performed by each application itself, using the included QT4-based binary installer. Other utilities for PBI management are also written in QT4, such as the creator / removal tool.

This should be changed, with a new suite of tools written in shell / C, that perform the actual installation, creation and removal portions. This would allow a variety of front-ends to be developed, and help break the PBI format from being more QT/KDE specific, allowing FreeBSD users to install support for the PBI format via ports. This would also removed the need for a binary loader at the front of .pbi files, instead making the file-format look like this:

[meta-data]|[optional icon]|[program archive]

Possible names?:

ports/sysutils/pbimanager?

pbi_add / pbi_delete / pbi_create / pbi_info

Not very original, but easy to remember for those coming from a BSD background.


Explore the usage of $ORGIN as a replacement for LD_LIBRARY_PATH

One of the ways the current version of PBI handles program execution using self-contained libraries is by creating wrapper scripts for executable, and setting some variables such as LD_LIBRARY_PATH. While this works, it has a few problems in that it is A: Hack-ish and B: Causes breakage with setuid.

One possible solution would be with the usage of $ORGIN for the compiling of PBI applications. This feature could eliminate the idea of needing to use LD_LIBRARY_PATH and fix issues with using some setuid programs, such as VirtualBox in user-mode.

There is another way: Use same approach as used this util: http://www.freshports.org/devel/chrpath/ I tested such situation: 1. Build PBI to directory /usr/Programs/VERY-long-name-here-very-very-long-and-unique-and-fixed-length 2. For all executable files, change DT_RPATH with chrpath; 2. For all other binary files; find all byte positions of this PATH with strings, change for example PATH/etc to NEW_PATH/etc, and write back; dd if=/file-with-zero of=$FILE offset=$offset-from-strings-output bs=1 count=strlength(PATH/ETC) dd if=/file-with-new-path of=$FILE offset=$offset-from-strings-output bs=1 count=strlength(NEW_PATH/ETC) 3. Find all text files (.sh, .xml, etc.), change there PATHes too.

With such tweak scripts, I was able to build PBI to one PREFIX, and dynamically install it to other PREFIX. Program worked just fine. However, need more testing with some complex Programs, OpenOffice probably, etc.

Integrate PBI building closer with FreeBSD Ports

The most common way to build a PBI currently, is by using the PBI Builder Software. This software acts as a front-end to the FreeBSD ports tree, by doing port builds with a custom PREFIX for a PBI, and using "modules" which contain additional meta-data used in the creation of the resulting PBI file.

One possible solution to improve upon this method, is by creating a new Mk/ entry, that will allow the usage of "make pbi" as a build option to ports. This make would then need to perform some steps to build a workable PBI file.

1. Run a complete 'make' on the port with PREFIX set to a target location for this PBI. (Build port + dependencies)

2. Include any optional meta-data for desktop icons / mime registration

3. Run 'pbi_create' on the resulting PREFIX that the port + dependencies were installed to, resulting in a finished PBI file.

While this portion of the build is fairly straight forward and should be easy, the difficult part will be in the inclusion of additional meta-data to the FreeBSD ports tree. Meta-data will be necessary for most applications, since the PBI manager will need to know which binaries need to be placed in the systems $PATH for execution, what mime-entries should be created, optional plugins / ports to include, rc scripts, etc.