diff --git a/en_US.ISO8859-1/articles/Makefile b/en_US.ISO8859-1/articles/Makefile index 95a40d704b..259deea664 100644 --- a/en_US.ISO8859-1/articles/Makefile +++ b/en_US.ISO8859-1/articles/Makefile @@ -1,46 +1,47 @@ # $FreeBSD$ SUBDIR = SUBDIR+= 5-roadmap SUBDIR+= checkpoint SUBDIR+= committers-guide SUBDIR+= console-server SUBDIR+= contributing SUBDIR+= contributors SUBDIR+= cvs-freebsd SUBDIR+= cvsup-advanced SUBDIR+= dialup-firewall SUBDIR+= diskless-x SUBDIR+= euro SUBDIR+= explaining-bsd SUBDIR+= fbsd-from-scratch SUBDIR+= filtering-bridges SUBDIR+= fonts SUBDIR+= formatting-media SUBDIR+= freebsd-questions SUBDIR+= hats SUBDIR+= hubs SUBDIR+= ipsec-must SUBDIR+= java-tomcat SUBDIR+= laptop SUBDIR+= mailing-list-faq SUBDIR+= mh SUBDIR+= multi-os SUBDIR+= new-users SUBDIR+= pam +SUBDIR+= portbuild SUBDIR+= pr-guidelines SUBDIR+= problem-reports SUBDIR+= pxe SUBDIR+= releng SUBDIR+= releng-packages SUBDIR+= serial-uart SUBDIR+= solid-state SUBDIR+= storage-devices SUBDIR+= vinum SUBDIR+= vm-design SUBDIR+= zip-drive # ROOT_SYMLINKS+= new-users DOC_PREFIX?= ${.CURDIR}/../.. .include "${DOC_PREFIX}/share/mk/doc.project.mk" diff --git a/en_US.ISO8859-1/articles/portbuild/Makefile b/en_US.ISO8859-1/articles/portbuild/Makefile new file mode 100644 index 0000000000..f61284a9be --- /dev/null +++ b/en_US.ISO8859-1/articles/portbuild/Makefile @@ -0,0 +1,18 @@ +# +# $FreeBSD$ +# +# Article: Portbuild Procedure +# + +DOC?= article + +FORMATS?= html +WITH_ARTICLE_TOC?= YES + +INSTALL_COMPRESSED?= gz +INSTALL_ONLY_COMPRESSED?= + +SRCS= article.sgml + +DOC_PREFIX?= ${.CURDIR}/../../.. +.include "${DOC_PREFIX}/share/mk/doc.project.mk" diff --git a/en_US.ISO8859-1/articles/portbuild/article.sgml b/en_US.ISO8859-1/articles/portbuild/article.sgml new file mode 100644 index 0000000000..5501afd063 --- /dev/null +++ b/en_US.ISO8859-1/articles/portbuild/article.sgml @@ -0,0 +1,714 @@ + +%man; + +%freebsd; + +%authors; + +%teams; + +%mailing-lists; + +%trademarks; +]> + +
+ + Package Building Procedures + + + The &os; Ports Management Team + + + $FreeBSD$ + + + 2003 + The &os; Ports + Management Team + + + + &tm-attrib.freebsd; + &tm-attrib.intel; + &tm-attrib.sparc; + &tm-attrib.general; + + + + + Introduction and Conventions + + In order to provide pre-compiled binaries of third-party + applications for &os;, the ports collection is regularly + built on one of the Package Building Clusters. + Currently, there are two such clusters: + pointyhat.FreeBSD.org and + dosirak.kr.FreeBSD.org. + + Most of the package building magic occurs under the + /var/portbuild directory. Unless + otherwise specified, all paths will be relative to + this location. ${arch} will + be used to specify one of the package architectures + (&i386;, alpha, &sparc64;, ia64, and amd64), and + ${branch} will be used + to specify the build branch (4, 5, 4-exp). + + + + + Build Client Management + + The &i386;, alpha, amd64, and two sparc64 clients currently + netboot from pointyhat; the other sparc64 client + and ia64 clients are self-hosted. In all cases they set themselves + up at boot-time to prepare to build packages. + + The + ports-${arch} + user can &man.ssh.1; as root onto + each of the ${arch} nodes. + + The scripts/allgohans script can + be used to run a command on all of the + ${arch} clients. + + The scripts/checkmachines script + is used to monitor the load on all the nodes of the + build cluster, and schedule which nodes build which ports. + This script is not very robust, and has a tendency to die. + It is best to start up this script on the build master + (either pointyhat or dosirak) + after boot time using a &man.while.1; loop. + + + + + Chroot Build Environment Setup + + Package builds are performed in a + chroot populated by the + portbuild script using the + ${arch}/${branch}/tarballs/bindist.tar + file. This tarball is created by the + mkbindist script which reads the + ${arch}/${branch}/mkbindist.conf + file to decide how to create the tarball. + + The script should be run as root + with the following command: + + /var/portbuild&prompt.root; scripts/mkbindist ${arch} ${branch} + + If ftp=1 in + mkbindist.conf then a prebuilt release + will be downloaded via FTP from the location specified by + ftp://${ftpserver}/${ftpurl}/${rel}. + If ftp=0 and + buildworld=1 then + mkbindist will call + makeworld to build a new world + [XXX This is currently broken]. + + If both ftp=0 and + buildworld=0 then + mkbindist will use the pre-existing + contents of ${worlddir} to + create bindist.tar. In practice + this means that you must have already installed a world + in ${worlddir}, which is typically installed with the + makeworld script: + + /var/portbuild&prompt.root; scripts/makeworld ${arch} ${branch} [-nocvs] + + This command builds a world from the + ${arch}/${branch}/src + tree and installs it into + ${worlddir}. The tree will + be updated first unless -nocvs is + specified. + + The bindist.tar file is extracted + onto each client at client boot time, and at the start of + each pass of the dopackages + script. + + + + + Starting the Build + + The scripts/dopackages* scripts + are used to perform the builds. Most useful are: + + + + dopackages.5 - Perform a + 5.X build + + + + + dopackages.4 - Perform a + 4.X build + + + + + dopackages.4-exp - Perform + a 4.X build with experimental patches + (4-exp branch) + + + + + These are wrappers around dopackages, + and are all symlinked to dopackages.wrapper. + New branch wrapper scripts can be created by symlinking + dopackages.${branch} to + dopackages.wrapper. These scripts + take a number of arguments. For example: + + dopackages.5 ${arch} [-options] + + [-options] may be zero or more of the + following: + + + + -nofinish - Do not perform + post-processing once the build is complete. Useful + if you expect that the build will need to be restarted + once it finishes. This option should always be used + for normal build operations. + + + + + -finish - Perform + post-processing only. + + + + + -restart - Restart an interrupted + (or non-finished) build from the + beginning. Ports that failed on the previous build will + be rebuilt. + + + + + -continue - Restart an interrupted + (or non-finished) build. Will not + rebuild ports that failed on the previous build. + + + + + -cdrom - This package build is + intended to end up on a CD-ROM, so + NO_CDROM packages and distfiles + should be deleted in post-processing. + + + + + -nobuild - Perform all + the preprocessing steps, but do not actually do + the package build. + + + + + -noindex - Do not rebuild + INDEX during preprocessing. + + + + + -noduds - Do not rebuild the + duds file (ports that are never + built, e.g. those marked IGNORE, + NO_PACKAGE, etc.) during + preprocessing. + + + + + -nocvs - Do not + cvs update the + src tree during preprocessing. + + + + + -noportscvs - Do not + cvs update the + ports tree during preprocessing. + + + + + -norestr - Do not attempt to build + RESTRICTED ports. + + + + + -plistcheck - Make it fatal for + ports to leave behind files after deinstallation. + + + + + -distfiles - Collect distfiles + that pass make checksum for later + uploading to ftp-master. Use this + sparingly because it takes up a lot of disk space. + You should remove the distfiles once they have been + transfered to ftp-master. + + + + + -fetch-original - Fetch the + distfile from the original MASTER_SITES + rather than ftp-master. + + + + + Make sure the ${arch} build + is run as the ports-${arch} user + or it will complain loudly. + + The actual package build itself occurs in two + identical phases. The reason for this is that sometimes + transient propblems (e.g. NFS failures, FTP sites being + unreachable, etc.) may halt the build. Doing things + in two phases is a workaround for these types of + problems. + + Be careful that ports/Makefile + does not specify any empty subdirectories. This is especially + important if you are doing a 4-exp build. If the build + process encounters an empty subdirectory, both package build + phases will stop short, and an error similar to the following + will be written to + ${arch}/${branch}/make.[0|1]: + + + don't know how to make dns-all(continuing) + + To correct this problem, simply comment out or remove + the SUBDIR entries that point to empty + subdirectories. After doing this, you can restart the build + by running the proper dopackages command + with the -restart option. + + + + + Anatomy of a Build + + A full build without any -no + options performs the following operations in the + specified order: + + + + A CVS checkout of the current ports + tree [*] + + + + + A CVS checkout of the running branch's + doc tree [*] + + + + + A CVS checkout of the running branch's + src tree [*] + + + + + Checks which ports do not have a + SUBDIR entry in their respective + category's Makefile [*] + + + + + Creates the duds file, which + is a list of ports not to build [*] [+] + + + + + Generates a fresh INDEX + file [*] [+] + + + + + Sets up the nodes that will be used in the + build [*] [+] + + + + + Builds a list of restricted ports [*] [+] + + + + Builds packages (phase 1) [++] + + + + Performs another node setup [+] + + + + Builds packages (phase 2) [++] + + + + [*] Status of these steps can be found in + ${arch}/${branch}/build.log + as well as on stderr of the tty running the + dopackages command. + + [+] If any of these steps fail, the build will stop + cold in its tracks. + + [++] Status of these steps can be found in + ${arch}/${branch}/make.[0|1], + where make.0 is the log file used by + phase 1 of the package build and make.1 + is the log file used by phase 2. Individual ports will write + their build logs to + ${arch}/${branch}/logs + and their error logs to + ${arch}/${branch}/errors. + + + + + Interrupting a Build + + Sending a HUP signal to the + dopackages* shell processes or to any + make process invoked by those scripts + is usually sufficient to interrupt the build. The + package builds dispatched by make to + the client machines will clean themselves up after a + few minutes (check with ps x until they + all go away). The following command usually does the trick: + + &prompt.user; killall -HUP sh ssh make + + Remove the + ${arch}/lock + file before trying to restart the build. + + + + + Monitoring the Build + + The + scripts/stats ${branch} + command counts the number of packages currently built. + + Running cat /var/portbuild/*/loads/* + shows the client loads and number of concurrent builds in + progress. + + Running tail -f ${arch}/${branch}/build.log + shows the overall build progress. + + If a build is failing, and it is not immediately obvious + from the port build log as to why, you can preserve the + WRKDIR for further analysis. To do this, + touch a file called .keep in the port's + directory. The next time the cluster tries to build this port, + it will tar, compress, and copy the WRKDIR + to + ${arch}/${branch}/wrkdirs. + + + Keep an eye on &man.df.1; output. If the + /var/portbuild file system becomes full + then Bad Things happen. + + + + + Release Builds + + When building packages for a release, it may be + necessary to manually update the ports + and src trees to the release tag and use + -nocvs and + -noportscvs. + + To build package sets intended for use on a CD-ROM, + use the -cdrom option to + dopackages. + + Assuming disk space is available on the cluster, use + -distfiles to collect distfiles. + + You must run the initial build with + -distfiles to collect + all the fetched distfiles. + + After the initial build completes, restart the build + with + -restart -distfiles -fetch-original + to collect updated distfiles as well. Then, once the + build is post-processed, take an inventory of the list + of files fetched: + + &prompt.user; cd ${arch}/${branch} +&prompt.user; find distfiles > distfiles-${release} + + This inventory file typically lives in + i386/${branch} + on the cluster master. + + This is useful to aid in periodically cleaning out + the distfiles from ftp-master. When space + gets tight, distfiles from recent releases can be kept while + others can be thrown away. + + Once the distfiles have been uploaded (see below), + the final release package set must be created. Just to be + on the safe side, run the + ${arch}/${branch}/cdrom.sh + script by hand to make sure all the CD-ROM restricted packages + and distfiles have been pruned. Then, copy the + ${arch}/${branch}/packages + directory to + ${arch}/${branch}/packages-${release}. + Once the packages are safely moved off, contact the &a.re; + and inform them of the release package location. + + Remember to coordinate with the &a.re; about the timing + and status of the release builds. + + + + + Uploading Packages + + Once a build has completed, packages and/or distfiles + can be transferred to ftp-master for + propagation to the FTP mirror network. If the build was + run with -nofinish, then make sure to + follow up with + dopackages -finish to post-process the + packages (removes RESTRICTED and + NO_CDROM packages where appropriate, + prunes packages not listed in INDEX, + removes from INDEX + references to packages not built, and generates a + CHECKSUM.MD5 + summary); and distfiles (moves them from the temporary + distfiles/.pbtmp directory into + distfiles/ and removes + RESTRICTED and NO_CDROM + distfiles). + + It is usually a good idea to run the + restricted.sh and/or + cdrom.sh scripts by hand after + dopackages finishes just to be safe. + Run the restricted.sh script before + uploading to ftp-master, then run + cdrom.sh before preparing + the final package set for a release. + + Packages can be copied to the staging area on + ftp-master with something like the following: + + &prompt.root; cd /var/portbuild/${arch}/${branch} +&prompt.root; tar cfv - packages/ | ssh portmgr@ftp-master tar xfC - w/ports/${arch}/tmp/${branch} + + Then log into ftp-master, verify that + the package set was transferred successfully, remove the + package set that the new package set is to replace (in + ~/w/ports/${arch}), + and move the new set into place. + + Some of the directories on + ftp-master are, in fact, symlinks. Be sure + you move the new packages directory over the + real destination directory, and not + one of the symlinks that points to it. + + Distfiles can be transferred via + rsync: + + &prompt.root; cd /var/portbuild/${arch}/${branch} +&prompt.root; rsync -r -v -l -p -c -n distfiles/ portmgr@ftp-master:w/ports/distfiles/ | tee log + + ALWAYS use -n + first with rsync and check the output + to make sure it is sane. If it looks good, re-run the + rsync without the -n + option. + + + + + Experimental Patches Builds + + Experimental patches builds are run from time to time to + new features or bugfixes to the ports infrastruction (i.e. + bsd.port.mk), or to test large sweeping + upgrades. The current experimental patches branch is + 4-exp on the i386 + architecture. + + In general, an experimental patches build is run the same + way as any other build. However, before running the + dopackages script, you must apply the required + patches to the ports tree. It is always a good idea to save + original copies of all changed files, as well as a list of what + you are changing. You can then look back on this list when doing + the final commit. + + In order to have a good control case with which to compare + failures, you should first do a package build of the branch on + which the experimental patches branch is based for the i386 + architecture (currently this is 4). Then, when + preparing for the experimental patches build, checkout a ports + tree and a src tree with the same date as was used for the control + build. This will ensure an apples-to-apples comparison + later. + + One build cluster can do the control build while the other + does the experimental patches build. This can be a great + time-saver. + + Once the build finishes, compare the control build failures + to those of the experimental patches build. Use the following + commands to facilitate this (this assumes the 4 + branch is the control branch, and the 4-exp + branch is the experimental patches branch): + + &prompt.user; cd /var/portbuild/i386/4-exp/errors +&prompt.user; find . -name \*.log | sort > /tmp/4-exp-errs +&prompt.user; cd /var/portbuild/i386/4/errors +&prompt.user; find . -name \*.log | sort > /tmp/4-errs + + If it has been a long time since one of the builds + finished, the logs may have been automatically compressed with + bzip2. In that case, you must use sort | sed + s,\.bz2,,g' instead. + + &prompt.user; comm -3 /tmp/4-errs /tmp/4-exp-errs | less + + This last command will produce a two-column report. The + first column is ports that failed on the control build but not in + the experimental patches build; the second column is vice versa. + Reasons that the port might be in the first column + include: + + + + Port was fixed since the control build was run, or was + upgraded to a newer version that is also broken (thus the + newer version should appear in the second column) + + + + + Port is fixed by the patches in the experimental patches + build + + + + + Port did not build under the experimental patches build + due to a dependency failure + + + + + Reasons for a port appearing in the second column + include: + + + + Port was broken by the experimental patches [1] + + + + Port was upgraded since the control build and has become + broken [2] + + + + + Port was broken due to a transient error (e.g. FTP site + down, package client error, etc.) + + + + + Both columns should be investigated and the reason for the + errors understood before committing the experimental patches set. + To differentiate between [1] and [2] above, you can do a rebuild + of the affected packages under the control branch: + + &prompt.user; cd /var/portbuild/i386/4/ports + + Be sure to cvs update this tree to the same date as + the experimental patches tree. + + The following command will set up the control branch for + the partial build: + + &prompt.user; /var/portbuild/scripts/dopackages.4 -noportscvs -nobuild -nocvs -nofinish + + The builds must be performed from the + packages/All directory. This directory should + initially be empty except for the Makefile symlink. If this + symlink does not exist, it must be created: + + &prompt.user; cd /var/portbuild/i386/4/packages/All +&prompt.user; ln -sf ../../Makefile . +&prompt.user; make -k -j<#> <list of packages to build> + + <#> is the concurrency of the build to + attempt. It is usually the sum of the weights listed in + /var/portbuild/i386/mlist unless you have a + reason to run a heavier or lighter build. + + The list of packages to build should be a list of package + names (including versions) as they appear in + INDEX. The PKGSUFFIX + (i.e. .tgz or .tbz) is optional. + + This will build only those packages listed as well as all + of their dependencies. You can check the progress of this + partial build the same way you would a regular build. Once all + the errors have been resolved, you can commit the package set. + After committing, it is customary to send a HEADS + UP email to ports@FreeBSD.org and + copy ports-developers@FreeBSD.org + informing people of the changes. A summary of all changes + should also be committed to + /usr/ports/CHANGES. + +