A few days ago, I posted about Python 2.4 and Python 2.5 leaving pkgsrc – it looks like it’ll be a little bit longer, at least for the 2.5 version. This means the Zope packages will be gone too, since they depend on Python 2.4. This won’t affect you if you aren’t using these packages, of course.
The default version of Python in pkgsrc is going to become 2.7. This will mean the 2012Q1 release will use that version by default. Older versions, meaning Python 2.4 and 2.5, may be going away. At least, that’s how the linked thread started but I’m not totally sure about it as I read farther through.
I was reading an article about how Tumblr scaled to handle the huge amount of data it’s regularly pushing out. Apparently, it started life as a traditional LAMP stack, but they’ve since moved on – to software packages I have not yet needed to ever use. Being open source software, it all has crazy names. Some of these packages are perfectly familiar to me now, but others are completely new.
Anyway, for fun, I decided to see how many of these sometimes new-to-me packages were present in pkgsrc. I’ll reproduce a paragraph from the story that lists the software they use, and link each one that I found in pkgsrc.
- PHP, Scala, Ruby
- Redis, HBase, MySQL
- Varnish, HA-Proxy, nginx,
- Memcache, Gearman, Kafka, Kestrel, Finagle
- Thrift, HTTP
- Git, Capistrano, Puppet, Jenkins
That’s actually more than I thought I’d find, though I can’t articulate why. Anyway, if any of the names are unfamiliar to you, now is the time to follow up. Redis, for example, looks more interesting to me at a casual glance than the normal NoSQL models I’ve heard about.
Here’s several things to look at:
Michael Lucas’s “BSD Needs Books” talk from NYCBSDCon 2010, on Youtube. I’ve talked about it before because I saw it in person; it’s a good talk. Ironically, he talks about getting a publisher interested in your book, and he just self-published.
Hubert Feyrer linked to the slides of two pkgsrc talks at FOSDEM; one about bringing pkgsrc to MirBSD, and one about pkgin, which is included in DragonFly.
There’s a NetBSD Hackathon going on February 10th through 12th, mostly online. I mention this because it may have some effect on pkgsrc, used by both NetBSD and DragonFly. Hackathons for pkgsrc usually happen separately, but no harm in keeping an eye out for any positive benefits.
Ulrich Habel wants to update some of the Perl 5 modules in pkgsrc. He published a request for comments, describing what he plans to do for changing some dependencies. He does note that Perl 5 in pkgsrc is at 5.14.2, which is very recent.
I was talking to a relative today who works at a large financial company, which is standardizing on Red Hat Enterprise. I find it strange that Red Hat, which has a lot of money behind it, still ships a years-old and arguably broken version of perl. By using pkgsrc, you’re getting more up-to-date software than people that actually shell out money for the privilege of compiling software.
The answer is “not very”. As I wrote in a post to kernel@, DragonFly 3.0 will be tagged soon, and released when there’s pkgsrc-2011Q4 packages to go with it. Probably a week if everything goes to plan.
John Marino has pointed out, with a number of examples, that gnat-aux is the best pkgsrc-based compiler for DragonFly right now, in terms of compatibility and support. It’s certainly good news if you are an Ada programmer. He lists some interesting numbers to demonstrate this superiority, though you can’t buildworld with it yet. (gcc 4.4, on DragonFly as part of the system, will do this normally.)
I’m linking to this small discussion about licensing and its documentation in pkgsrc, just because these paragraphs, out of context, are good for any pkgsrc user to know.
The links are sheer entertainment this week. No strong options or anything, not even about that U.S. legislative mess called SOPA.
- I greatly enjoyed this history of personal computer mishaps and blunders. Of course, nothing like any of that has ever happened to me. Ever. Ever ever ever.
- Nintendo Entertainment System stories. (a comic) Also familiar to anyone of a particular age.
- This is good advice about env(1) – use it.
- “Fork Yeah! The Rise and Development of illumos“, a video via matthiasr on EFNet #dragonflybsd.
- “What’s the strangest language feature you’ve ever encountered?” (via) Some of these are mind-boggling. I’ve also never seen APL before, yeesh.
- My Year in Roguelikes. I think a few of the games mentioned are in pkgsrc.
Your unrelated comic link of the week: Basic Instructions. Well, not totally unrelated, since BSD author Michael Lucas’s tweet about it reminded me. I’ve got the first book; I need to get the second and third.
The freeze for pkgsrc-2011Q4 has started. No updates to pkgsrc, other than for security, for the next two weeks.
The last quarterly release of pkgsrc for the year is scheduled for the end of this month. This means the freeze, where only bugfixes are applies, will be starting on the 17th.
Two tips for working with pkgsrc, derived in part from this mailing list post on users@ (follow the thread) and from my own experience. If you put
- You can clean up any leftover package building files by deleting the files in that directory and leave your pkgsrc files untouched.
- You can have a read-only /usr/pkgsrc, which means it can be shared over NFS (or SMB?) between multiple machines, DragonFly or otherwise.
Some time ago, Matthew Dillon worked on a bulk build system that built as much of pkgsrc in parallel as possible. It’s in the tree now as ‘fastbulk‘, for anyone wanting to try it out. I used it a bit; I didn’t measure the degree of speed increase, but was able to get about 70% of the packages built.
Almost all the packages in pkgsrc support non-root installation now… except these last 31. I recall something about their removal by the next quarterly release if they still don’t work, or maybe just after. Jump in if one of these packages is useful to you.
Here’s some recent x86_64 bulk builds: one on DragonFly 2.11, one on NetBSd 5.0.2, and one on Linux 188.8.131.52. Some data of note: DragonFly is within 8%-ish total packages built compared to NetBSD, which could be considered the baseline. Linux, the more common platform for most of the software built, is another step less. I don’t know if there’s any dramatic conclusion to get from this other than, “Hey, a lot of packages build on DragonFly!”
Not a lot of links this week, for some reason.
- The best obituary I’ve seen yet for Dennis Ritchie, where he’s contrasted with Steve Jobs.
- The best paper abstract ever.
- Michael Lucas documents his DragonFly update.
- Our tcplay TrueCrypt implementation is getting noticed.
- pkgin-0.51 is available in pkgsrc-current, though it’s not in the most recent quarterly release.
Your unrelated comics link for the week: Oglaf. This week’s OK, but it’s frequently NSFW, and frequently hilarious.
I have some pkgsrc-2011Q3 builds done, for x86_64 and i386. I performed them on DragonFly 2.11, but they should work fine for 2.12/2.13. They’re uploading to the pkgsrc-2011Q3 folder on mirror-master, so you’ll need to set PKG_PATH correctly to use them via pkg_radd.
The x86_64 package upload is done, and I anticipate the i386 one will be done within the next 24 hours.
The latest quarterly release of pkgsrc is out. You can download it via CVS, or update /usr/Makefile to pull down the correct branch. I’ll be building binaries as soon as I can. I like the release announcement style.
I finished a build of pkgsrc-current on x86_64, with a report on what built. I’ve kicked off a new build, and I expect at least 100 more packages to build thanks to John Marino’s work on pkgsrc and DragonFly.
The freeze for the next quarterly release of pkgsrc has been extended another week, to October 2nd. This will push the DragonFly 2.12 release out a ways, too.
As is common for the combination of new Postgres releases and new pkgsrc quarterly releases, Postgres 8.3 is going to be missing from pkgsrc-2011Q3. The default version of Postgres installed by pkgsrc will become 9.0 after that quarterly release. (9.1 is already present in pkgsrc.) This is all planned by Joerg Sonnenberger.
As predicted and now announced, pkgsrc is now frozen for the next week. If everything goes well, we’ll have pkgsrc-2011Q3 next week.
For reasons unknown to me, there’s enough functional change between PHP 5.2 and PHP 5.3 that it affected a lot of PHP-based programs. For that reason, PHP 5 in pkgsrc defaults to the 5.2 version. However, it’s going to be 5.3 for the next stable quarterly release of pkgsrc. In theory, all PHP5-dependent programs are ready to handle that now. Note that PHP 5.3 is already in pkgsrc; it just wasn’t the default. If you were using the php53 package, it may require some manual fiddling at your next upgrade of pkgsrc packages.
At some point, you may want to generate binary programs that are unstripped of debugging information. You may want to generate them with pkgsrc. Here’s a little note on what options will make that happen.
The ‘freeze’ period (when bugfixes are the only addition) for pkgsrc will start on September 18th, with the next quarterly release of pkgsrc, 2011Q3, scheduled for the 25th. Think of it as an early Christmas present.
John Marino, who already has commit access for DragonFly, now also has commit access for pkgsrc. What does this mean? It means if you have a pkgsrc problem, submit it through NetBSD’s Problem Report system as normal, and maybe let him know about it too. He’s already made some DragonFly-specific fixes.
The next release of pkgin, the binary package installer for pkgsrc, is imminent. I link to the note about this because the new features list sounds good, including a significant speedup.
This week has taught me one thing for sure: Always make sure your backup generator is working. And over-plan battery capacity. That’s actually two things, but what the heck. I’m tired, for reasons that can probably be inferred! I’m not the only one suffering these problems, it seems.
- There is a certain subset of readers here that will find this fascinating: a video of a game postmortem. Specifically, Elite. (via) Needs Flash.
- This is as good an article as any I’ve seen describing where the tablet computer market is going, at The Economist.
- Remember RetroBSD, mentioned here previously? Here’s some discussion of it.
- EuroBSDCon’s 2011 conference is open for registration, but the early bird discount only lasts until the end of August, so jump on it soon if you’re thinking of going. It’s the 10th anniversary of the event!
- PHP 5.3 is coming to pkgsrc as default, soon? The PHP 5.2 -> 5.3 transition seems to mess up a lot of code because of some changes in the way things are handled, or at least that’s my experience, so watch out.
- Make sure you aren’t running mod_deflate on your Apache 2.x server.
- Kristaps Dzonsons, the fellow behind mdocml (which is in DragonFly now and mentioned here before) is working on a mdoc manual. It’s an actual book, with examples. It’s titled “Practical UNIX Manuals: mdoc”, which sounds like part of a series, though I don’t know if there’s anything else. I’d sure like it if there was. (via Undeadly.) Look very closely at the mdoc web page and you will see the markup, too. Neat!
- Breakout treated as a musical instrument, in 1983. That’s too glib a summary of this explanation of an old book studying the game Breakout and playing it. Really, read the article, and remember that the book described would just be lost in a sea of
blog postsnoise today. (via)
There are only 45 packages out of over 10,000 in pkgsrc that do not support being installed by people who aren’t root, or in different locations. Thomas Klausner has that list of 45 packages. It’s very close to zero packages with this problem at this point, so if you want to make a big difference…
Anton Panev is working on a Google Summer of Code project for NetBSD, adding support in pkgsrc for RPM/Debian package formats. He posted a status report recently; will this come to DragonFly via pkgsrc? I don’t know!
I put together a post on users@ about updating to pkgsrc-2011Q2. I’ll just repeat it here after the break:
I recently saw some terse notes on firstname.lastname@example.org about compiling using clang for pkgsrc. I haven’t tried this on DragonFly…
Francois Tigeot has fixed wip/jdk16 to build on DragonFly. Note that this is in pkgsrc-wip, not ‘normal’ pkgsrc. The secret is to build lang/kaffe to bootstrap it, which requires CCVER=gcc41 to be set. Apparently kaffe will not build under gcc 4.4.
p.s. I hated “Stranger in a Strange Land”.
The builds of pkgsrc-2011Q2 are finishing up, so we have/will have binaries to download, for those who don’t want to build from source. The uploads should be complete by now for everything except maybe 2.11/x86_64. You’ll have to change $PKG_PATH to point at the new directories for now, though. There’s also some build reports to look at.
I spied a bulk build of pkgsrc using clang. It’s interesting to see the results… It’s on NetBSD, but it should be possible to try the same thing with CCVER on DragonFly. Any takers?
The latest quarterly release of pkgsrc, 2011Q2, has been branched. There’s no formal announcement yet to describe the highlights, but I’ll link it when it shows up. I’ve already started building binary packages for DragonFly 2.10 and 2.11.
Two completely separate and unrelated changes:
First, Alex Hornung has added a check to look for certain lines in a commit message, and add a MFC reminder note to the commit message if they are found. MFC, if you haven’t heard it, means ‘merge from current’, or moving a change from dragonfly-current to the last release version.
Second, with the next quarterly release of pkgsrc coming up, there’s some old packages that will get dropped. Speak up if you need them to stick around.
Pkgsrc bmake bootstrap, that is. There’s a new version of bmake, and it needs to be tested on every platform possible.
The pkgsrc ‘freeze’ in preparation for the pkgsrc-2011Q2 branch is coming up, starting this Sunday the 19th. This means the quarterly release will be tagged in about 2 weeks, and I’ll probably have binary packages built for DragonFly about a week or so after that.
There’s still a few packages in pkgsrc that don’t support DESTDIR (e.g. being built by someone other than root). If you want to help out, here’s a list of those 60 packages.
I moved to DragonFly 2.10 over the past few days, and I tried out deduplication, to see what kind of results I would get. The procedure is outlined below. I’m using /home here as an example, just to reduce the amount of text pasted in.
/pfs/@@-1:00004 966000640 566434576 399566064 59% /home
Move my various Hammer pseudo-file systems to version 5, which supports deduplication.
# hammer version-upgrade /home 5
Issue a deduplication simulate command, to see what it guesses will be the savings:
# hammer dedup-simulate /home
Dedup-simulate /home: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 4
Dedup-simulate /home succeeded
Simulated dedup ratio = 1.22
That ratio turned out to be pretty accurate for the actual deduplication. I didn’t time it, unfortunately. I don’t know if the time taken is proportional to the amount of deduplication or the total volume of data, though I suspect the latter.
# hammer dedup /home
Dedup /home: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 4
Dedup /home succeeded
Dedup ratio = 1.22
462 GB referenced
378 GB allocated
14 MB skipped
6869 CRC collisions
0 SHA collisions
0 bigblock underflows
The end result?
/pfs/@@-1:00004 966000640 505887504 460113136 52% /home
That data space is shared across all file systems, and it’s a 1TB disk, so it’s 7%, or 70GB. I was hoping for more, but I don’t have any obviously duplicated data (no local mail store, no on-disk backups), so perhaps this is normal. 70GB that I didn’t have before is no bad thing, though.
Incidentally, I was able to upgrade my installed software from pkgsrc-2009Q4 to pkgsrc-2011Q1 entirely using pkg_radd -u <pkgname>. Remarkably quick and painless, though pkgin may have been able to do it even faster since it would pull from the same place.
This new build is on x86_64, pkgsrc-2011Q1. It’s already uploaded, if you want to update. i386 coming soon. Several packages freeze up during build, so it’s been turning into a manual process.
One of the Google Summer of Code projects that will be valuable for DragonFly even though it isn’t a DragonFly project: “Add other package formats to pkgsrc”, where pkgsrc can interpret rpm, dpkg, and FreeBSD Ports files. Anyway, the project has a Sourceforge site.
This week: lots more reading!
- Michael Lucas describes an extra layer of protection for when you can’t force public key usage on every SSH user.
- Cool, but obscure Unix tools (via) The screenshots are all from a Mac… How many of the 24 tools listed are in pkgsrc/pkgsrc-wip? Almost all of them. (tpp sounds entertaining.)
- NYCBUG, in addition to having a really fun convention, has been regularly posting audio of the presentations they host. The most recent is “William Baxter’s NYCBUG presentation on The Unix Method of Development Management”. See the BSD Events tweet for the download.
- What Ubuntu means. (via)
- Here’s a nice explanation of Intel’s new Tri-Gate design and with it, an incidental explanation of the processor market.
- This ycombinator post about Hammer2 work has an in-depth comment from Venkatesh Srinivas about DragonFly’s network setup, memory allocator, and token use. (Ignore the trolling in other comments.)
- Michael Lucas’s next No Starch Press book is Absolute OpenBSD, second edition.
- Pictures and video are starting to show up from the just-passed BSDCan 2011. (via this and also thesjg on EFNet #dragonflybsd)
- My first experience of The Internet was very similar to this. It should be bizarrely unfamiliar to anyone under 20 or so. (via) Get this: I typed ‘exit’ instead of just closing the browser window when I was done messing with it, because some habits cannot be broken.
The usual way for building pkgsrc packages from source is ‘bmake install clean’, to build and install the package, and then clean the work files from building it. Since the recent change to DESTDIR, where a binary package is built before installation, you may want to add ‘package-clean’ to the list, so that the binary package is also removed after installation.
No, 2.10 is not out. I built packages for pkgsrc-2011Q1 on 2.9, and set it to think 2.10 so that the pkgsrc tools wouldn’t complain. We’re close enough to release that this shouldn’t be a problem. The packages are available for x86_64; i386 packages coming “soon”. See my note to email@example.com for details on accessing these packages.
Get out your wallet! I encourage purchasing here.
- You should buy a SSD. Not necessarily news to you, but that article does a good job of summarizing why.
- On the other hand, SSD prices are already on their way up/availability is way down. Japan’s disasters are having a ripple effect through the high-tech supply chain. Either buy immediately or get ready to wait for a while…
- Introduction to Architecting Systems for Scale – you either don’t care, or find scaling questions immediately engaging. I am one of the latter, so here’s the link.
- I’ve been watching firstname.lastname@example.org for a little while. One thing I’ve discovered: there’s a lot of updates going on! Another thing that’s nice to see: DragonFlyupdates, including ones that help with our move to gcc 4.4.
- Aw, no more Kermit. (via) Not that I have a use for it at this point, but still: aww. I bet in about 10 years I’ll say the same thing about… gopher? Remember that? It’s not even supported in Firefox 4 now, which kinda makes me feel sad. And old.
- Server plans: Facebook vs. Google. (warning: Facebook article is somewhat giddy.)
- The infinite hard drive. (via I lost it, sorry)
Here’s an extra little thing: next time you’re dealing with dusty computer equipment, remember this picture:
That is what happens to an exposed RJ45 port after a few years in a salt mine (my employer). This was inside an enclosed, mostly-sealed structure, too.
I already noted that the quarterly release is out, but the pkgsrc-2011Q1 release announcement is available now. There’s good reasons to link to it – the list of updated packages, new packages, and credits for the work people have been doing. Here’s the part I really want to pick out:
We’re aiming to make this the last branch to support non-DESTDIR packages. We have almost finished the transition to DESTDIR installation, where a staging directory is used to make a binary package, which is then managed by the pkg_install tools.
The reason I’m highlighting this is: it’s good news! One of the long-term complaints with pkgsrc is that the upgrade process is painful. If you try to build an upgrade and the build processfails after uninstalling the existing package, not only are you not getting the upgrade, but you’ve lost the existing package. Binary packages for download helps with this (and generally is faster), but only so many packages can be built separately and made available for download.
Building a package separately and then installing from there removes these issues. No binary redistribution issues, actual downtime is minimal, and the package is known to work when an upgrade happens. This removes most of the problems I’ve heard raised about pkgsrc over the years.
It’s tagged, though there’s no release announcement yet. Working on building binaries starting tonight…
There’s a number of pkgsrc packages that have a combination of security vulnerabilites and lack of updates for more than a year which is placing them on the chopping block. (Follow the discussion to see which ones make it off the list.) The removals will happen after the next branch, pkgsrc-2011Q1, which is itself due in two days.
As already mentioned on this Digest, the freeze for the next quarterly release of pkgsrc, pkgsrc-2011Q1, has started. I’ve also completed several bulk builds of pkgsrc-2010Q4 and pkgsrc-current using DragonFly system with GCC 4.4. Francois Tigeot has very kindly gone out of his way to get some of the (relatively few) broken packages listed in those builds to be fixed.
As noted in announcements, pkgsrc is entering a 10-day freeze period starting tomorrow. If everything goes to plan, the next quarterly release of pkgsrc, 2011Q1, will be released April 3rd.
My first bulk build of pkgsrc with gcc 4.4 has completed; the results are available. Notice that most of the errors are from checksum problems with downloads, not actual problems from the compiler change. I’m starting a new build to see if the checksum problems go away with fresh downloads.
The GIF format, or rather the LZW format it uses, is no longer patent-encumbered. (GIF patent worries led to the creation of the PNG format, if I’m not mistaken) Matthias Drochner has changed pkgsrc to use giflib instead of libungif.
According to Wikipedia, the patent expired more than 5 years ago, so this isn’t really news other than some packages need to be rebuilt. Still, memories of the general Internet Outrage from a decade ago are interesting compared to the events of today.
There’s two recent changes for pkgsrc and DragonFly:
- DragonFly-current (2.9) now pulls the most recent pkgsrc quarterly release (2010Q4) by default, instead of pkgsrc-current. This means more packages will be working with the default setup, plus pkg_radd and other tools will be pulling the same ‘generation’ of software.
- The DragonFly/git version of pkgsrc can now be created as a shallow clone. This means less file history, but also means a much faster download.
If you’re curious, I have a bulk build on DragonFly 2.9/x86_64/pkgsrc-current finished. Work on the programs that don’t build is always welcome. It’s pretty good for bleeding-edge, though!
… is to make its patches unnecessary, by getting the changes needed for any program to compile on DragonFly built right into the program. (Often called “pushing patches upstream”) That usually means creating a patch and then tracking down the program authors to get them to include those changes in the next release of a project. That tracking down can be a majority of the work. In that case: thanks, Rumko!
Update: Also, thanks, Matthias Rampke! He did the same thing for pcc.
Matthias Scheler is looking for Postfix testers. If you run it, he has a patch to version 2.8.1 he’d like you to try.
Also, the final list of GTK1-using packages that are not actively updated has been determined. These packages are leaving pkgsrc next week unless there’s any last-minute intercessions.
If you’re like me, you’ve been using XMMS for music playback since just about forever. It’s ancient, though. It uses GTK1, and since Thomas Klausner is trying to get GTK1 dependencies out of pkgsrc, he listed a roundup of alternatives on the pkgsrc-users mailing list, most/all of which are in pkgsrc. Pouya Tafti added some more.
Thomas Klausner is planning to get rid of the last bits of GTK1 in pkgsrc, which means some old/non-updated software has to go too. Speak up on the email@example.com mailing list if you need some of that listed software, or (better yet) provide patches to move it to GTK2.
So, I felt lucky recently. I updated shiningsilence.com from DragonFly 2.6 to DragonFly 2.8, and wanted to upgrade my pkgsrc packages from pkgsrc-2010Q3 to pkgsrc-2010Q4.
You can do this with pkg_rolling-replace, or various other tools, but I wanted to see if I could do it completely with binary packages. I used pkg_radd -uv <pkgname> for each of the major packages I had installed.
Surprisingly, it worked, for every package. I had to force-install some Perl modules because I was moving from 5.10 to 5.12, but I think I may have been able to use an additional -u switch to get by that problem. I did use pkg_leaves to identify packages I didn’t need, and removed them to reduce the number of items to upload.
It was exactly what I wanted. Previous pkgsrc upgrades had taken most of a day, as I had to build from source and figure out what went where. We’ve had a better success rate in bulk builds recently, and this paid off in an upgrade process that only took perhaps an hour.
If you’ve ever installed pkgsrc packages from source, you probably typed ‘bmake install’. There’s a ‘bin-install’ target that will use binary packages if they are available, but you have to set the appropriate environment variables to do so.
It’s now much easier, on DragonFly. If you have pkgsrc-current as of yesterday or later, or pkgsrc-2011Q1 when it arrives, you can type ‘bmake bin-install’ for a pkgsrc application and it will download the binary package automatically, if it’s available, and build from source if it’s not.
This is a setup I’ve wanted for a while – the speed of a binary install, plus a fallback if the binary isn’t available.