I meant to post this a while ago; it’s a few days old but still useful. John Marino gave some stats on DPorts progress, plus he and Francois Tigeot also had some tips on xorg setup. The successful build count is higher by now, and I think KDE3 is done, though I haven’t tried it.
It’s a very short week this week. I was on the road for work, so I didn’t see anywhere as much of the Internet as I may have liked. Count my dports writeup yesterday as part of this and it averages out to a good amount of reading.
- Favorite Linux Commands. Not all of them are Linux/bash specific. (via)
- Advanced Vim Registers. Or buffers, or clipboards, if you want to get messy with terms. (via)
- “I hate BSD so much!”, he yelled at his spittle-flecked monitor.
- TOME, a roguelike. Read through the comments for discussion of many other roguelike games.
John Marino’s DPorts project, mentioned here briefly before, is interesting. I had two separate people ask me how it works, so a better explanation is in order. I’ve tried it out on a test machine over the past few weeks.
Dports is an effort to use FreeBSD’s ports system as a base for DragonFly, and the pkg tool as a way to manage binary packages built from DPorts. This is complicated, so I’ll explain each part in order.
- FreeBSD ports are a FreeBSD-specific collection of software installation files that automate building 3rd-party software on FreeBSD. You’ve probably already heard of them. (Note there’s no mention of DragonFly.)
- DPorts is a collection of files that map to existing FreeBSD ports, and contain any changes necessary to make that port also build on DragonFly. Many of those programs build without changes on DragonFly. DPorts builds from source.
- pkg is used for package management, and is usable on FreeBSD and on DragonFly. The binary packages produced from building with DPorts can be installed from remote locations and managed separately using pkg, so that software upgrades and installation can be performed with binaries only. (It’s much faster that way.)
Every port seen in DPorts is known to build on DragonFly. John Marino adds a port only after it builds successfully, using poudriere as a bulk software tool. Ports are only updated to a newer version when that newer version builds, too, so once something arrives in DPorts, it should never break from being updated at some point in the future.
To use DPorts, you need two things:
- DragonFly 3.3 or later, though 3.3 is the most recent right now.
- You need to rename /usr/pkg so that your existing pkgsrc binary programs don’t get accidentally used while working with DPorts, causing confusion. If anything goes wrong with DPorts when you are installing it and you want to go back, remove all the DPorts packages and rename /usr/pkg back to normal.
(Don’t confuse pkg, the management tool, with /usr/pkg, the normal installation directory for pkgsrc. ) For the installation of the base port files:
cd /usr make dports-create-shallow
If you’ve already renamed your /usr/pkg directory, git won’t be in your path any more. You can instead download a tarball and unpack it, which also happens to be possible automatically via that same Makefile.
cd /usr make dports-download
Downloading via git is fastest, so if you do need to use the tarball via make dports-download, build devel/git, delete /usr/dports, and then pull it again with make dports-create-shallow. This all comes from John Marino’s Github site for DPorts.
DPorts doesn’t use pkg_info, pkg_add, and the other tools traditionally seen on DragonFly for pkgsrc. Instead, package management is done with pkg. Use pkg info, pkg install, pkg remove, and pkg update to list, install, delete, and upgrade various packages on your system. Packages built from source or downloaded as prebuilt binaries are managed the same way, using these tools.
Since DPorts doesn’t update a package until it gets a successful build, and installations are of successfully built binary packages, upgrades with prebuilt packages should always succeed. Since they’re binary, they should be fast. There’s a lot of ‘shoulds’ in this sentence, but these are reasonable suppositions.
What about pkgsrc?
Pkgsrc and DPorts shouldn’t be used at the same time, since one system’s packages may be at different versions but still get picked up during building for the other system. That’s about it for restrictions.
I intend to try building an experimental release of DragonFly with DPorts, to see if all the right packages can be added, but no guarantees. DPorts is brand new and does not yet have a repository for downloading packages, so the normal caveats apply; don’t install it on a mission-critical machine, and be ready to deal with any surprises from using it if you do try it out.
What packages are available?
Browsing the Github repo will show you all listed packages. More complex packages like xorg, openjdk7, and libreoffice install, as does xfce. Parts of KDE 3 and KDE 4 are in there. (I haven’t tried either.) I’m not sure about Gnome, but I don’t think anyone ever is. There’s no vim, but there is emacs.
That’s just what I see at this exact minute. It changes daily as more packages are built. Changes from DragonFly builds are sometimes relevant to the original FreeBSD port, so there’s benefits for everyone here.
Try it now if it has all the packages you need, or wait for a binary repository to be created to speed things up. Remember, this is a new project, so a willingness to deal with problems and contribute to fixes is necessary.
It’s actually been out since the start of January, but the release announcement is available now.
Stéphane Russell, on the users@ mailing list, pointed out an in-depth article about DragonFly’s 3.2 release, on linuxfr.org. It’s in French, which means I’m just going to have to trust his word about the contents.
The Open Graphics Project, which is building a completely open video card, needs a wiki maintainer. It’s a volunteer effort. If you were perhaps thinking you wanted to step up to a more complex project but didn’t want to just be writing code, here is a perfect opportunity.
(Not too different from maintaining a project work blog, after all, and I know that’s rewarding.)
If you were thinking you wanted to try gcc 4.7 with pkgsrc, John Marino’s described the option you need to set. It only works in pkgsrc-master right now (because of changes John made), and not every package in pkgsrc will build.
The advantage is that it’s also possible, with the same syntax, to set pkgsrc to build with gcc 4.4. This means the default compiler in DragonFly can be changed to gcc 4.7 and pkgsrc packages that aren’t compatible can still be built.
Update: Check this minor change: ‘?=’ instead of ‘=’.
Whomever submitted this story to Slashdot really doesn’t like FreeBSD; they’re describing FreeBSD’s annual end-of-year fund drive as failed. The month-long drive is only about a week old and has already picked up donations at a faster rate than any previous year’s donation drive, but apparently the poster – and Slashdot’s editors – can’t be bothered to do math. While we’re on the topic, donate to the FreeBSD Foundation; they do good things.
(There’s DragonFly too, though we’re not as ambitious or officially 501(c)(3) non-profit.)
Matthew Dillon has written up another update on his progress with HAMMER2. (I need to be consistent in how I write that.) He has disks being exported and mounted on other systems, and adds an explanation of some of the issues around creating reliable multi-master setups. Before you get too excited, no, multi-master isn’t working yet, and this is not production ready.
There’s more benchmarks for DragonFly vs. other systems on Phoronix. It has the same problem as previous benchmarks; some of the benchmarks may have no connection to reality (what does the “Himeno Poisson Pressure Solver” actually test?), and almost every system has a different version of the gcc compiler. So it’s meaningless in terms of comparative or absolute performance. On the other hand, DragonFly doesn’t do badly.
You can also look at the comments to see someone absolutely freak out over the very existence of things that aren’t Linux. I’m not sure if it’s actually trolling, since the comments are so exactly wrong.
Shopping! This is the big holiday shopping weekend in the US, and I usually put together something here.
- Buy an SSD for someone who doesn’t have one – including you if that’s the case. There’s better and worse SSDs out there, but you’ll get a speed benefit no matter what, and other bonuses are possible.
- The Tea Bag Buddy, which also comes in a color-changing version. Because tea.
- My perennial Science! suggestions: ThinkGeek, American Science and Surplus, Ward’s Scientific, Carolina, and United Nuclear, The Bone Room, and Skulls Unlimited.
- The Best of BSD 2011 and Last Year in BSD Security, from the BSD Magazine publisher.
- For more BSD, there’s always the orgs themselves. FreeBSD, NetBSD, and OpenBSD – no DragonFly, though there ought to be. Also, ISC.
- For lists of gifts, there’s the Verge Gift Guide, which has some interesting offshoots.
- Another long list: The Comics Reporter’s Shopping List.
If you have suggestions, please comment!
Because of the recent good results for pgbench on DragonFly 3.2, Phoronix has a new benchmark of DragonFly using other (possibly unrelated) tests. There’s not a lot of information to glean from them; they are testing operations different than what was optimized for pgbench in 3.2. I’d like to see DragonFly 3.0 tested the same way to see how much improvement there was between versions.
Today is the day that FreeBSD moves to using clang by default. This is not necessarily a surprise, but I like the finality of calling it “Clang-Day”. I think Clang will probably be the next compiler brought into DragonFly’s base system, instead of the next release of gcc. Don’t make any bets on my statement, though, cause I certainly won’t be the one doing it. (It’s hard.)
There was one more file to change for the bmake import, so if you are running DragonFly 3.3 and updated between the 28th and 30th of October, do a full rebuild.
I mentioned this before in the Lazy Reading from last Sunday, but it’s worth a second look: Apple’s new Fusion Drive product appears to be very much like DragonFly’s swapcache. DragonFly doesn’t have exclusive right to the idea of caching on a faster disk, clearly, so I’m not complaining that it’s “ours”. It’s frustrating to see product announcement/press releases stumbling all over this like it’s a new thing.
Then again, having new ideas about technology ideas and making sure they spread is one of the points of the BSD license, so perhaps there’s no good reason to complain at all.
(Before anyone reads too much into this: No, I don’t know of any direct relationship between swapcache and Fusion Drive; they may have no common background other than structure.)
A thread on pkgsrc-users@ reminds me: adding a specific line for bin-install will save time when rebuilding packages; pkgsrc will use existing binary packages instead of rebuilding from source when possible, when this is set. At least, I’m pretty sure that’s what it does.
Sandip Jadhav asked if anyone was working on an I/O scheduler. Chris Turner replied with a “no”, but also with a list of places to look for details on writing one, which I’m linking here for posterity.
John Marino is working on a very good idea: bringing bmake into DragonFly as a replacement for the current ‘make’. bmake is going through more active development and apparently also in use/will be used? on FreeBSD, so syncing up with the same make flavor as FreeBSD and NetBSD will help everyone. It’ll also remove the problem where you ‘make’ everything in DragonFly, except pkgsrc packages which you ‘bmake’. It’s not changed over yet.
(What does OpenBSD use for make?)
A conversation about compilers in the DragonFly base system led peeter (must) to describe his group’s use of OpenMPI on DragonFly for physics calculations. Apparently he’s had a significant performance improvement on DragonFly.
John Marino did a bulk build of pkgsrc using gcc 4.7.2, and posted the results. The result? About 1% of packages that built with gcc 4.4 did not build with 4.7.2. Whether that’s a problem with gcc or a problem with how each of those software packages were created by the original authors, I don’t know.
Google is hosting a ‘Doc Camp’, where people get together and write documentation for open source projects. There’s a page that talks about it. Last year’s inaugural event was apparently quite successful. I haven’t been to it, but I think a day just for documentation is a good idea.
I’m planning for DragonFly 3.2 to come with pkgsrc-2012Q3, the most recent release. I’m building binary packages to match, and the build should complete by the time we release on the 22nd…
Notice I said “should” – sometimes the universe conspires against bulk builds.
I branched 3.2 tonight. That means 2 weeks until release, so sharpen your bug-poking sticks!
(I’m very tired and unable to think of good analogies, sorry.)
As I typed elsewhere, my general plan is to branch DragonFly 3.2 on the 8th, and release on the 22nd. That should give the recent scheduler and gcc work a chance to settle, and perhaps get a new version of USB support in too. It will probably be using pkgsrc-2012Q3, also, though we may not have binary i386 packages. 3.2 is shaping up to be a much more significant release than I expected.
The machine that runs www.dragonflybsd.org and bugs.dragonflybsd.org is currently down. While it gets figured out, Alex Hornung has a static copy of the dragonflybsd.org main website available.
Sepherosa Ziehau has some suggestions for anyone looking for some kernel hacking. They’re mostly based around busdma(9).
dragonflybsd.org appears to be down right now so I’m linking to the MARC kernel@ post.
There’s a post on the mailing list firstname.lastname@example.org of currently broken packages for the next quarterly release. It’s not a lot of stuff, but if something you need is on there, don’t worry too much. If you follow the thread through its replies, there’s a lot of fixing going on.
MARC, which stands for Mailing list ARChives, has a lot of mailing lists. It now includes the DragonFly users@ list, along with the others. (It’s not linked in *BSD on the main MARC page yet, but it should be soon.) It’s worth digging through the massive, massive wall of text on that page to find a mailing list you didn’t know existed.
This latest commit for the new scheduler means that on your next update, you will want to build a new kernel, and probably a new world too. This only applies if you’re running DragonFly 3.1, of course.
The old mailing list software for @dragonflybsd.org mailing lists, bestserv, apparently allowed people not subscribed to a list to post to it, after answering a confirmation message for each message posted.
The closest way to duplicate that for Mailman is to sign up for the list you want, and then turn off mail delivery for your email address in the config page for that mailing list. This won’t affect a lot of people, since most people want list output in their mailbox, but there’s at least a few I’ve fixed that way.
Francois Tigeot benchmarked the recent Postgres 9.3 release. Postgres apparently switched to using mmap instead of SYSV shared memory, and Francois has done this to show the performance differences. (view the PDF in his post.) Of course, work has continued since this was posted, so there should be new numbers soon, and new changes I’ll document in a future post.
I haven’t found a reference to the exact decision Postgres made on how to handle memory; please post a link in comments if you know a good source.
See the note on pkgsrc-users@. The next quarterly release, pkgsrc-2012Q3, should be fully baked by the end of the month, if all goes well.
As seen in this pkgsrc-users@ post from Thomas Klausner, the freeze for pkgsrc-2012Q3 starts on Sunday and continues for (probably) two weeks before the release.
NYCBUG, the NY BSD user’s group, has an RSS feed for their speaker events, found via Dru Lavigne’s always useful BSD Events twitter. The next event at the start of October is a talk about SMPng in FreeBSD. Given that it was the project that in part led to the creation of DragonFly, I’d like to hear about it. (and even better, have someone more qualified than I compare and contrast that approach with what’s in DragonFly.)
If you look at new.pkgsrc.org, you will see what may become a new site. This is apparently a test, so don’t react as if this was the actual site.
If you ever wanted to read an extensive discussion about the scheduler, today’s your day. Mihai Carabas, who posted the details of a long discussion he had with Matthew Dillon about how the scheduler works. You may recall Mihai’s name from the very successful GSoC scheduler project that recently finished.
(look, a link to the new Mailman archive!)
If you’re on any of the dragonflybsd.org mailing lists, I’m converting them over from bestserv to Mailman. I’ve done bugs@, commits@, hammer@, and test@ so far, and I’ll move the old archives over to the same format as soon as I find an actual mbox file with the old messages in it. The remaining lists should be tomorrow.
(If you got a note tonight from a list you were sure you were unsubscribed from, that was my fault; sorry! I didn’t understand the format of the bestserv user lists.)
DragonFly user varialus has created a page on the DragonFly website (it’s a wiki, after all) with all the notes taken from trying installation, etc. There’s far more notes than I expected there, so it’s worth a read.
I’ve uploaded DragonFly 3.0.3 disk images, both ISO and IMG. They should start appearing on a mirror site near you in the next 24 hours. This took a while after the tagging, I know, but I wanted to make sure every one of them booted. I didn’t on a previous release, and regretted it.
Francois Tigeot benchmarked several different operating systems using Postgres 9.2b3, including DragonFly, and published the results. I have a local copy of the PDF since the attachment didn’t really survive the archiving. Follow the thread for discussion. The Linux results look abnormally high, so it is possible that something different is happening on that platform…
DragonFly had a successful Google Summer of Code even this year. It marks our 5th time participating, 7th if you count Google Code-In events.
Mihai Carabas worked on adding SMT/HT awareness to the DragonFly scheduler. This project was very successful. The original goal was just to take advantage of threading with the scheduler, but the benchmarks published by Mihai show in general a 5% speedup from these scheduler changes. His work has already been committed.
Vishesh Yadav implemented an inotify interface in DragonFly. inotify is an originally Linux-based system for monitoring files and directories for changes. A specific use for this is an inotify-aware locate program, so that a list of file locations can be kept ‘live’. His code for the inotify interface should be committed to DragonFly very soon.
(This was written in part for Google to use on their Open Source Blog.)
I’m working on building new images, but: DragonFly 3.0.3 has been tagged. If you’re running 3.0, you can update and get some of the recent bug fixes.
3.2 is the next major release of DragonFly, which will be relatively soon by the every-6-months release schedule. John Marino’s put together another catch-all bug report for that release.
Juraj Sipos wrote me to describe MaheshaDragonFlyBSD, a live DragonFly image that has additional software preinstalled, and can easily be set to understand Sanskrit. It’s available in DVD and USB versions.
If you want to put something towards DragonFly, and you don’t have time or hardware, cash is now an option. (It’s not tax-deductible.)
The release announcement for pkgsrc-2012Q2 is out. New in this quarterly release: statistics about clang and pkgsrc. A surprisingly large number of packages build just fine with clang instead of gcc.
Attention students and mentors: the Summer of Code midterms open up on July 9th. This means students fill out an evaluation, and mentors also fill out an evaluation. Don’t forget, because completed evals from mentor and student both are necessary for a project to continue being funded.
More benchmarks, in this case a comparison of Debian GNU/kFreeBSD and DragonFly. I’m not even sure how to derive meaning from it.
It’s almost an all-Vim week.
- Unix as IDE, a 6 part completed series. (via)
- VimOrganizer, Emac’s Org-mode for Vim. (via)
- The Vim Clutch. This is hilarious. (via) Also see “Chindogu“, though this might be too useful.
- Those who know UNIX are doomed to reinvent it. (via same place)
- Falsehoods programmers believe about time. Taken from the same thing about names. (via) Also, when someone says, “Oh, I can just look for email addresses using a regular expression”, you’re stepping over a similar cliff.
- Welcome image in your terminal. (via) Will this work in an xterm? rxvt? I don’t know…
- CanI make this Lazy Reading post more nerdy? Yes! Arbitrage and Equilibrium in the Team Fortress 2 Economy. I already enjoy economics writing, and this one is about virtual economies and games. The only thing I know of close to this is the economist that works on EVE Online.
If you have an Intel processor with multiple cores and hyperthreading support, you can compile a new kernel and try out Mihia Carabas’s GSoC work already; he’s created a test using the OpenSSL test case to time scheduling performance vs. number of threads.
I know I already posted that this was on the way, but this time, the quarterly pkgsrc freeze is starting with a detailed announcement. 2 weeks until the next release, if everything goes well.
The freeze for pkgsrc-2012Q2 starts on the 16th of June, as recently announced. Freezes are usually 2 weeks, so that means 2012Q2 should be tagged at the end of June.
Pkgsrc already runs on a large number of different platforms, but that’s not what I’m talking about. In this case, Joyent, which uses pkgsrc internally, has a suggested change that makes binaries usable on both 32 and 64 bit systems. I don’t know if this will go into pkgsrc proper, but it’s interesting to see.
There’s a number of packages out there that assume you are using the GNU versions of ls, wc, and so on. However, you aren’t when using a BSD system. Pkgsrc has historically dealt with this when GNU tools are needed for a package by prefixing them with a ‘g’. ‘ls’ becomes ‘gls’, and so on. Aleksey Cheusov proposed a fix to keep these utilities under their original names, which I think will go into the next quarterly pkgsrc release.
Pkgsrc packages that have source files that can’t be redistributed, and go missing for the length of an entire quarterly release, will get removed. They are effectively broken at that point anyway.
That policy is now formally in place; I don’t think there was a clear prescription before.