Pkgsrc bmake bootstrap, that is. There’s a new version of bmake, and it needs to be tested on every platform possible.
The pkgsrc ‘freeze’ in preparation for the pkgsrc-2011Q2 branch is coming up, starting this Sunday the 19th. This means the quarterly release will be tagged in about 2 weeks, and I’ll probably have binary packages built for DragonFly about a week or so after that.
There’s still a few packages in pkgsrc that don’t support DESTDIR (e.g. being built by someone other than root). If you want to help out, here’s a list of those 60 packages.
I moved to DragonFly 2.10 over the past few days, and I tried out deduplication, to see what kind of results I would get. The procedure is outlined below. I’m using /home here as an example, just to reduce the amount of text pasted in.
/pfs/@@-1:00004 966000640 566434576 399566064 59% /home
Move my various Hammer pseudo-file systems to version 5, which supports deduplication.
# hammer version-upgrade /home 5
Issue a deduplication simulate command, to see what it guesses will be the savings:
# hammer dedup-simulate /home
Dedup-simulate /home: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 4
Dedup-simulate /home succeeded
Simulated dedup ratio = 1.22
That ratio turned out to be pretty accurate for the actual deduplication. I didn’t time it, unfortunately. I don’t know if the time taken is proportional to the amount of deduplication or the total volume of data, though I suspect the latter.
# hammer dedup /home
Dedup /home: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 4
Dedup /home succeeded
Dedup ratio = 1.22
462 GB referenced
378 GB allocated
14 MB skipped
6869 CRC collisions
0 SHA collisions
0 bigblock underflows
The end result?
/pfs/@@-1:00004 966000640 505887504 460113136 52% /home
That data space is shared across all file systems, and it’s a 1TB disk, so it’s 7%, or 70GB. I was hoping for more, but I don’t have any obviously duplicated data (no local mail store, no on-disk backups), so perhaps this is normal. 70GB that I didn’t have before is no bad thing, though.
Incidentally, I was able to upgrade my installed software from pkgsrc-2009Q4 to pkgsrc-2011Q1 entirely using pkg_radd -u <pkgname>. Remarkably quick and painless, though pkgin may have been able to do it even faster since it would pull from the same place.
This new build is on x86_64, pkgsrc-2011Q1. It’s already uploaded, if you want to update. i386 coming soon. Several packages freeze up during build, so it’s been turning into a manual process.
One of the Google Summer of Code projects that will be valuable for DragonFly even though it isn’t a DragonFly project: “Add other package formats to pkgsrc”, where pkgsrc can interpret rpm, dpkg, and FreeBSD Ports files. Anyway, the project has a Sourceforge site.
This week: lots more reading!
- Michael Lucas describes an extra layer of protection for when you can’t force public key usage on every SSH user.
- Cool, but obscure Unix tools (via) The screenshots are all from a Mac… How many of the 24 tools listed are in pkgsrc/pkgsrc-wip? Almost all of them. (tpp sounds entertaining.)
- NYCBUG, in addition to having a really fun convention, has been regularly posting audio of the presentations they host. The most recent is “William Baxter’s NYCBUG presentation on The Unix Method of Development Management”. See the BSD Events tweet for the download.
- What Ubuntu means. (via)
- Here’s a nice explanation of Intel’s new Tri-Gate design and with it, an incidental explanation of the processor market.
- This ycombinator post about Hammer2 work has an in-depth comment from Venkatesh Srinivas about DragonFly’s network setup, memory allocator, and token use. (Ignore the trolling in other comments.)
- Michael Lucas’s next No Starch Press book is Absolute OpenBSD, second edition.
- Pictures and video are starting to show up from the just-passed BSDCan 2011. (via this and also thesjg on EFNet #dragonflybsd)
- My first experience of The Internet was very similar to this. It should be bizarrely unfamiliar to anyone under 20 or so. (via) Get this: I typed ‘exit’ instead of just closing the browser window when I was done messing with it, because some habits cannot be broken.
The usual way for building pkgsrc packages from source is ‘bmake install clean’, to build and install the package, and then clean the work files from building it. Since the recent change to DESTDIR, where a binary package is built before installation, you may want to add ‘package-clean’ to the list, so that the binary package is also removed after installation.
No, 2.10 is not out. I built packages for pkgsrc-2011Q1 on 2.9, and set it to think 2.10 so that the pkgsrc tools wouldn’t complain. We’re close enough to release that this shouldn’t be a problem. The packages are available for x86_64; i386 packages coming “soon”. See my note to email@example.com for details on accessing these packages.
Get out your wallet! I encourage purchasing here.
- You should buy a SSD. Not necessarily news to you, but that article does a good job of summarizing why.
- On the other hand, SSD prices are already on their way up/availability is way down. Japan’s disasters are having a ripple effect through the high-tech supply chain. Either buy immediately or get ready to wait for a while…
- Introduction to Architecting Systems for Scale – you either don’t care, or find scaling questions immediately engaging. I am one of the latter, so here’s the link.
- I’ve been watching firstname.lastname@example.org for a little while. One thing I’ve discovered: there’s a lot of updates going on! Another thing that’s nice to see: DragonFlyupdates, including ones that help with our move to gcc 4.4.
- Aw, no more Kermit. (via) Not that I have a use for it at this point, but still: aww. I bet in about 10 years I’ll say the same thing about… gopher? Remember that? It’s not even supported in Firefox 4 now, which kinda makes me feel sad. And old.
- Server plans: Facebook vs. Google. (warning: Facebook article is somewhat giddy.)
- The infinite hard drive. (via I lost it, sorry)
Here’s an extra little thing: next time you’re dealing with dusty computer equipment, remember this picture:
That is what happens to an exposed RJ45 port after a few years in a salt mine (my employer). This was inside an enclosed, mostly-sealed structure, too.
I already noted that the quarterly release is out, but the pkgsrc-2011Q1 release announcement is available now. There’s good reasons to link to it – the list of updated packages, new packages, and credits for the work people have been doing. Here’s the part I really want to pick out:
We’re aiming to make this the last branch to support non-DESTDIR packages. We have almost finished the transition to DESTDIR installation, where a staging directory is used to make a binary package, which is then managed by the pkg_install tools.
The reason I’m highlighting this is: it’s good news! One of the long-term complaints with pkgsrc is that the upgrade process is painful. If you try to build an upgrade and the build processfails after uninstalling the existing package, not only are you not getting the upgrade, but you’ve lost the existing package. Binary packages for download helps with this (and generally is faster), but only so many packages can be built separately and made available for download.
Building a package separately and then installing from there removes these issues. No binary redistribution issues, actual downtime is minimal, and the package is known to work when an upgrade happens. This removes most of the problems I’ve heard raised about pkgsrc over the years.
It’s tagged, though there’s no release announcement yet. Working on building binaries starting tonight…
There’s a number of pkgsrc packages that have a combination of security vulnerabilites and lack of updates for more than a year which is placing them on the chopping block. (Follow the discussion to see which ones make it off the list.) The removals will happen after the next branch, pkgsrc-2011Q1, which is itself due in two days.
As already mentioned on this Digest, the freeze for the next quarterly release of pkgsrc, pkgsrc-2011Q1, has started. I’ve also completed several bulk builds of pkgsrc-2010Q4 and pkgsrc-current using DragonFly system with GCC 4.4. Francois Tigeot has very kindly gone out of his way to get some of the (relatively few) broken packages listed in those builds to be fixed.
As noted in announcements, pkgsrc is entering a 10-day freeze period starting tomorrow. If everything goes to plan, the next quarterly release of pkgsrc, 2011Q1, will be released April 3rd.
My first bulk build of pkgsrc with gcc 4.4 has completed; the results are available. Notice that most of the errors are from checksum problems with downloads, not actual problems from the compiler change. I’m starting a new build to see if the checksum problems go away with fresh downloads.
The GIF format, or rather the LZW format it uses, is no longer patent-encumbered. (GIF patent worries led to the creation of the PNG format, if I’m not mistaken) Matthias Drochner has changed pkgsrc to use giflib instead of libungif.
According to Wikipedia, the patent expired more than 5 years ago, so this isn’t really news other than some packages need to be rebuilt. Still, memories of the general Internet Outrage from a decade ago are interesting compared to the events of today.
There’s two recent changes for pkgsrc and DragonFly:
- DragonFly-current (2.9) now pulls the most recent pkgsrc quarterly release (2010Q4) by default, instead of pkgsrc-current. This means more packages will be working with the default setup, plus pkg_radd and other tools will be pulling the same ‘generation’ of software.
- The DragonFly/git version of pkgsrc can now be created as a shallow clone. This means less file history, but also means a much faster download.
If you’re curious, I have a bulk build on DragonFly 2.9/x86_64/pkgsrc-current finished. Work on the programs that don’t build is always welcome. It’s pretty good for bleeding-edge, though!
… is to make its patches unnecessary, by getting the changes needed for any program to compile on DragonFly built right into the program. (Often called “pushing patches upstream”) That usually means creating a patch and then tracking down the program authors to get them to include those changes in the next release of a project. That tracking down can be a majority of the work. In that case: thanks, Rumko!
Update: Also, thanks, Matthias Rampke! He did the same thing for pcc.
Matthias Scheler is looking for Postfix testers. If you run it, he has a patch to version 2.8.1 he’d like you to try.
Also, the final list of GTK1-using packages that are not actively updated has been determined. These packages are leaving pkgsrc next week unless there’s any last-minute intercessions.
If you’re like me, you’ve been using XMMS for music playback since just about forever. It’s ancient, though. It uses GTK1, and since Thomas Klausner is trying to get GTK1 dependencies out of pkgsrc, he listed a roundup of alternatives on the pkgsrc-users mailing list, most/all of which are in pkgsrc. Pouya Tafti added some more.
Thomas Klausner is planning to get rid of the last bits of GTK1 in pkgsrc, which means some old/non-updated software has to go too. Speak up on the email@example.com mailing list if you need some of that listed software, or (better yet) provide patches to move it to GTK2.
So, I felt lucky recently. I updated shiningsilence.com from DragonFly 2.6 to DragonFly 2.8, and wanted to upgrade my pkgsrc packages from pkgsrc-2010Q3 to pkgsrc-2010Q4.
You can do this with pkg_rolling-replace, or various other tools, but I wanted to see if I could do it completely with binary packages. I used pkg_radd -uv <pkgname> for each of the major packages I had installed.
Surprisingly, it worked, for every package. I had to force-install some Perl modules because I was moving from 5.10 to 5.12, but I think I may have been able to use an additional -u switch to get by that problem. I did use pkg_leaves to identify packages I didn’t need, and removed them to reduce the number of items to upload.
It was exactly what I wanted. Previous pkgsrc upgrades had taken most of a day, as I had to build from source and figure out what went where. We’ve had a better success rate in bulk builds recently, and this paid off in an upgrade process that only took perhaps an hour.
If you’ve ever installed pkgsrc packages from source, you probably typed ‘bmake install’. There’s a ‘bin-install’ target that will use binary packages if they are available, but you have to set the appropriate environment variables to do so.
It’s now much easier, on DragonFly. If you have pkgsrc-current as of yesterday or later, or pkgsrc-2011Q1 when it arrives, you can type ‘bmake bin-install’ for a pkgsrc application and it will download the binary package automatically, if it’s available, and build from source if it’s not.
This is a setup I’ve wanted for a while – the speed of a binary install, plus a fallback if the binary isn’t available.
They’re finally uploaded! See my rather lengthy post about it on users@ for all the details.
The pkgsrc-2010Q4 branch is now available in DragonFly’s git repo, via ‘git checkout pkgsrc-2010Q4’ in /usr/pkgsrc. Enjoy!
Here’s where the binary build is: summarized in a post to users@. So far so good…
avalon.dragonflybsd.org, also known as mirror-master.dragonflybsd.org, is back up at a new location, with new disks and new connectivity. pkg_radd should work by default again, as should git.dragonflybsd.org.
Here’s the state of my build of pkgsrc-2010Q4 packages:
- DragonFly 2.8/i386 – in progress
- DragonFly 2.8/x86_64 – in progress
- DragonFly 2.9/i386 – just started (happens on Avalon)
- DragonFly 2.9/x86_64 – in progress
So it will be some days yet… building over 4000 packages total is never quick.
Normally I hold this for Sunday, but I’ve got a good batch of links already. Something here for everyone, this week.
- A git cheatsheet, and another git cheatsheet. I may have linked to the latter one before, as it looks vaguely familiar. Anyway, bookmark. (Thanks, luxh on EFNet #dragonflybsd)
- What should you do about bad blocks on a disk? Get a new disk.
- If you ever wanted to port software, there’s a pkgsrc developer’s guide (thanks Francois Tigeot) that shows you how.
- It’s NOT LINUX, for the billionth time. It’s BSD UNIX (certified, even) under there!
- “Children of the Cron“. An entertaining pun. (via)
- Nothing to do with BSD, or even computers, really: Gary Gorton, interviewed about the recent financial crisis, at a Fed bank website (!?). Interesting because I like economic matters, and because it’s the first web page where I’ve ever seen pop-up links added usefully, as a sort of footnote that you don’t have to scroll. (via)
- Michael Lucas recently had a machine broken into. Since everything on the machine is suspect, he’s using Netflow data to figure out when it happened, and how, which is not surprising given his most recent book. He has two posts describing how he backtracks his way to the probable source.
If you run pkgsrc-current, not the quarterly releases, the png library has just been updated to 1.5. This may break a few applications for now, and require a lot of rebuilding on your next update, since many packages depend on this.
If you are running a quarterly release of pkgsrc, you are unaffected.
Aleksey Cheusov is putting together a package manager for pkgsrc, called nih. (For “Not Invented Here”). It’s binary-only at this point, so you’d need to run distbb or pbulk to generate packages, or download from avalon.dragonflybsd.org.
The planned freeze is underway; so pkgsrc-2010Q4 should arrive soon. How soon? January 1st, if it’s by the traditional schedule.
I never really noticed this before, but it’s possible to include your own patchsets into pkgsrc and have them picked up as part of the build process, using $LOCALPATCHES.
Somehow I ended up with a zillion links for this week’s Lazy Reading. I hope you’ve got some spare time for this… Let’s get right into it:
- Michael Lucas, BSD book author (see links on site), has started Twittering. He’s also found the Wikileaks/NetBSD association that I didn’t know about, as Julian Assange even shows up in the NetBSD fortunes file. Also, while linking to his blog, I’ll point at his post on “Write what you don’t know“. Think of that article next time you feel you don’t know enough to contribute to something – especially open source.
- There’s a lengthy dialog on the firstname.lastname@example.org mailing list about pkgsrc, and “Making it easier to get and use pkgsrc“. You can follow the whole thread on the listing page. I am all for the idea. Everybody and their brother has an App Store these days. Ports/pkgsrc are perhaps the original app store ideas, and I’d like to see them brought to the same level as these commercial entitites. This is important: pkgsrc is perhaps the only app store equivalent in existence that is not tied to a platform; that exists only to get you software rather than to provide a way to tie a platform into its developers profits.
- Hey, a roguelike zombie apocalypse game! Aw, it’s Windows-only.
- Mikel King has an editorial that sums up the many places BSD serves as an underpinning to products – a good checklist, if you don’t know of them. He’s also written an instructional article on passwordless/SSH setup.
- Along the same lines, Promote Perl by Building Great Things. This applies to BSD products too; telling people it’s great doesn’t work as well as making something great and showing that a BSD system is part of what makes it so.
- Did you know there are even BSD Certification classes in Iran? I really need to do that… though probably not at that location.
- Yacc is not dead. (via) I link to this because I had a moment of nerd excitement realizing that blog’s title is intended to look like a bang path.
- Database design ideas. There’s been a good series of posts there lately, good for anyone wanting to move beyond the basic CRUD details.
The utility pkg_add has a -u option that tells it to upgrade any existing matched package with a given binary package. Since pkg_radd passes options on to the underlying use of pkg_add, after automatically setting a remote repository for binary files, pkg_radd -u <packagename> tells pkg_add to automatically find and upgrade a package.
I never thought this would work. However, I’m building a package on a system that has pkgsrc-2010Q1 packages installed, but a pkgsrc-2010Q3 /usr/pkgsrc. Every time I’ve encountered an error because installed software was too low a version, pkg_radd -uv <package_name> has resulted in a quick upgrade.
I’m not recommending this as a new upgrade method; I’m noting how unexpectedly well this experiment is going. It may be just blind luck, but this sure would be nice if it ‘just worked’.
A number of people have encountered this: while installing some larger pkgsrc package, the process stops on a strange DocBook error. Alex Hornung has a fix: symlink /usr/pkg/etc/xml/catalog to /usr/pkg/share/xml/catalog.
Alex Hornung has some patches that allow KDE4 to build on DragonFly. They aren’t in pkgsrc and not all in KDE yet, so try them out directly if you want KDE4, for now.
Update: based on something Alex said on IRC… they’re in KDE4 now.
Chris Turner wrote some notes about building pkgsrc packages in a chroot, including the handy tip of using
to run and display a GUI-using app under the chroot.
Our mirror of the never-quite-official git repository for pkgsrc is being rebuilt, so it will be temporarily inaccessible. Matthew Dillon is working on building a new one directly from pkgsrc CVS, which will have a different link.
Update: It’s finished. Matthew Dillon’s posted a summary of the changes and what you need to update in order to use it.
I’ve drastically revamped the pkgsrc howto on the dragonflybsd.org website. It’s also linked in that site’s menu, too. Comments please!
A bump in shared library version for libssl/libcrypto means that any programs dependent on it will require a rebuild – including any pkgsrc programs.
This only affects you if you are running 2.7, for now. It means that on upgrading from 2.6 to 2.8, any libssl-using programs will need to be updated. This may not be a big thing, since pkgsrc-2010Q3 will also be out and people will want to upgrade anyway.
A small crop for Lazy Reading this week – oh well.
- Bad things about SVN. I’ve always thought git is more interesting, but subversion is easier. Oh well. The article is also fun for the unfiltered Windows-hate.
- At least this hasn’t happened yet with pkgsrc.
- What happens if you put your hand in the way of the Large Hadron Collider beam? (via)
I’ve noticed that if you have older pkgsrc packages installed, and install binary packages for pkgsrc-2010Q2, those packages will refuse to install if pkg_install is an older version than what they were built with.
I ended up force-deleting pkg_install and bmake, and reinstalling by running pkgtools/bootstrap/bootstrap. There may be better solutions; I’m mentioning it now since it’s a known problem.
Update: “bmake replace USE_DESTDIR=yes” was suggested by Joerg Sonnenberger. “pkg_add -u /path/to/newer/pkg_install” should also work (untested).
This will probably apply to the upcoming pkgsrc-2010Q3, too. Building from source is a workaround for now.
Starting September 23rd, pkgsrc will freeze (i.e. bugfixes only) in preparation for the 2010Q3 release, which is planned for one week later, instead of the usual 2 weeks. This release will include some very new software like Postgres 9, too.
Some time ago, there was an application called pkgmanager, available in pkgsrc-wip. It worked by tracking ‘wanted’ packages in pkgsrc, and upgrading based on that list. It hasn’t been updated in some time, however, and may not even build.
‘Rumko’ has written a replacement, called rpkgmanager. The Gitorious page linked in the previous sentence includes the URL to download the code via Git, so it’s available to try now even though it’s not yet in pkgsrc.
I totally meant to post this yesterday. Oops!
- We’re using toeplitz. I just like the name; I don’t understand how it works.
- The idea of software forks has been around since, oh, BSD and System V Unix diverged, if not earlier. Here’s an article that talks about forking in general, rather breathlessly. After reading that, read this perhaps more accurate fork parody. (via)
- You know what we could use for pkgsrc, and all the other port/package collections? Explanation. They face the same problem phone application stores face: too many programs to easily select what you need. You could certainly build a whole site just around package reviews; it’s even possible to argue that Ubuntu or PC-BSD are built around just making some 3rd-party-app choices ahead of time on an existing operating system. Anyway, here’s an article talking about that idea specifically around the Apple App Store. Please won’t somebody who is not me do something like that for pkgsrc?
- This writeup of one man’s experience with Forth gives a good feel for the language, or at least as good a feel as I can understand. Posted in memoriam for our recently departed Forth bootloader. (via) There’s other enjoyable articles on that blog, too.
- This describes about two years of my life, except it was mostly Zangband.
As I found out directly, upgrading from pkgsrc version 2010Q1 to 2010Q2 has a minor quirk: binary packages for 2010Q2 will refuse to install with an older version of pkg_install. Rebuild pkgtools/pkg_install to the 2010Q2 version and the problem will go away.
- If you are running DragonFly 2.7, Matthew Dillon has made some kernel changes, so updating your 2.7 machine will require a full buildworld cycle, not quickworld.
- The binary packages for 2.6 and 2.7 have been updated to pkgsrc-2010Q2. This means that pkg_radd will automatically pull down newer packages, and you should make sure your /usr/pkgsrc is using the pkgsrc-2010Q2 release if you want to be sure there’s no version mismatches.
I recently sent out a description of what built for pkgsrc-2010Q2 , though the section on not changing the stable link is no longer true.