Fresh from BSDCan 2011, an interview with Ingo Schwartze and Kristaps Dzonsons, mostly about mdocml. (Which is already present in DragonFly.)
Month: May 2011
I moved to DragonFly 2.10 over the past few days, and I tried out deduplication, to see what kind of results I would get. The procedure is outlined below. I’m using /home here as an example, just to reduce the amount of text pasted in.
/pfs/@@-1:00004 966000640 566434576 399566064 59% /home
Move my various Hammer pseudo-file systems to version 5, which supports deduplication.
# hammer version-upgrade /home 5
Issue a deduplication simulate command, to see what it guesses will be the savings:
# hammer dedup-simulate /home
Dedup-simulate /home: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 4
Dedup-simulate /home succeeded
Simulated dedup ratio = 1.22
That ratio turned out to be pretty accurate for the actual deduplication. I didn’t time it, unfortunately. I don’t know if the time taken is proportional to the amount of deduplication or the total volume of data, though I suspect the latter.
# hammer dedup /home
Dedup /home: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 4
Dedup /home succeeded
Dedup ratio = 1.22
462 GB referenced
378 GB allocated
14 MB skipped
6869 CRC collisions
0 SHA collisions
0 bigblock underflows
The end result?
/pfs/@@-1:00004 966000640 505887504 460113136 52% /home
That data space is shared across all file systems, and it’s a 1TB disk, so it’s 7%, or 70GB. I was hoping for more, but I don’t have any obviously duplicated data (no local mail store, no on-disk backups), so perhaps this is normal. 70GB that I didn’t have before is no bad thing, though.
Incidentally, I was able to upgrade my installed software from pkgsrc-2009Q4 to pkgsrc-2011Q1 entirely using pkg_radd -u <pkgname>. Remarkably quick and painless, though pkgin may have been able to do it even faster since it would pull from the same place.
- Do you like the Opera browser? Apparently all it takes is a little misspelling to confuse it with a U.S. daytime talk show host. The “Best of Oprah emails to Opera“. (via) Mistaken identity on the Internet is always fun.
- Popular free software licenses, described. (via) One of the better, non-polemic descriptions I’ve seen.
- For the opposite effect, the Free Software Foundation’s license recommendations. Somehow, the BSD license isn’t even mentioned. (via) A commenter at the source link notes that the GNU Free Documentation License isn’t even considered ‘free’ by Debian. Along those lines, I’ve always thought that GPL licensing creates a perverse incentive to keep your software undocumented.
- The FreeBSD and NetBSD Foundations have acquired a license for libcxxrt from PathScale, which I assume is for C++ support in conjunction with clang. (or pcc?) This isn’t as much of an issue for DragonFly right now since we’re continuing down the GCC route.
- Temple of the Roguelike, a searchable database of roguelike games. It’s an idea that you would totally expect for this genre. (via trevorjk on EFNet #dragonflybsd) Also: a roguelikedev subreddit.
This new build is on x86_64, pkgsrc-2011Q1. It’s already uploaded, if you want to update. i386 coming soon. Several packages freeze up during build, so it’s been turning into a manual process.
One of the Google Summer of Code projects that will be valuable for DragonFly even though it isn’t a DragonFly project: “Add other package formats to pkgsrc”, where pkgsrc can interpret rpm, dpkg, and FreeBSD Ports files. Anyway, the project has a Sourceforge site.
I haven’t covered recent disk encryption work evenly, here, so I’ll point at a recent discussion instead. Alex Hornung mentioned a cryptsetup(8) man page that may help, as does any dm-crypt tutorial out there on the Internet. (DragonFly has the same userland tools.) The DragonFly installer will create encrypted disks at install time, too.
If you follow this thread, it has some discussion on how to handle a multi-disk setup and Hammer. If a disk is going bad, you can try mirroring, though you have to be careful how your pseudo-file systems are set up.
The SMP option is now in the GENERIC kernel config. This means you’ll have a SMP-capable kernel even on an uniprocessor machine, unless you configure a special kernel.
Apparently the mail(1) and battlestar(6) source code caused a problem when checked out onto a NTFS volume. Sascha Wildner fixed it, and Alexander Polakov found an explanation as to why. How long has this problem been around? Well, look at the email addresses at the end of the man page for battlestar(6), for instance, or guess how long mail(1) has been around…
The I/O APIC is now always on unless you say otherwise. This may not make a clear difference to you, but enabling that kernel option has always been a somewhat iffy thing; working for some configurations and not others. Now, it’s one less thing to worry about.
This week, the links are generally fun.
- Hey, can this work on BSD? Cause this. (yeah, I know, hype.)
- Michael Lucas has a summary of his experience at BSDCan 2011. His third point – anyone can experiment and publish results – is something I’d like to see. I love graphs, and I love being able to see quantifiable results.
- The many slides for ’10 years of pf’, also from BSDCan 2011, are online. (via) The background images are entertaining, though it’s using that font. Slide 78 mentions that other BSDs have much older versions of pf. I think DragonFly’s running the newest old version of any of them, actually, though the slide doesn’t mention it.
- There’s Gnome people saying “Ignore everything that isn’t Linux“. No, wait, it’s not that bad!
- The history of the computer mouse. Something I’d always heard about, but not with this detail.
- Git aliases. I don’t think it’ll save you hours of your life as the author claims, but it may be handy.
- How to build your own gaming PC. The author went for funny and true, rather than the multipage exposition of out-of-date numbers and graphs that usually make up these articles.
- Where the octothorpe came from. It has to do with Unix and telephones, which are close to the same thing if you go back far enough.
- Speaking of Unix-ish stuff, here’s an interview with Ken Thompson, who recently won the Japan Prize for the creation of Unix, along with Dennis Richie. (via) Yes, it’s their fault!
- “First was viruses, second was malware, third is facebook.“
Here’s some recent notes on running Java on DragonFly; I may have posted something similar before, but it doesn’t hurt to keep the information out there.
I posted something about this before, but now it’s definite: bleeding-edge users of DragonFly can boot a multiprocessor kernel on a single-processor machine.
If you’ve ever wanted to really make sure of all the network interfaces supported on your DragonFly system, you can create an exhaustive (and exhausting) list.
Samuel Greear has a totally untested update to the NVIDIA video driver available. It may not work, but it’s not like that’ll be any different than the current state of the driver.
This week: lots more reading!
- Michael Lucas describes an extra layer of protection for when you can’t force public key usage on every SSH user.
- Cool, but obscure Unix tools (via) The screenshots are all from a Mac… How many of the 24 tools listed are in pkgsrc/pkgsrc-wip? Almost all of them. (tpp sounds entertaining.)
- NYCBUG, in addition to having a really fun convention, has been regularly posting audio of the presentations they host. The most recent is “William Baxter’s NYCBUG presentation on The Unix Method of Development Management”. See the BSD Events tweet for the download.
- What Ubuntu means. (via)
- Here’s a nice explanation of Intel’s new Tri-Gate design and with it, an incidental explanation of the processor market.
- This ycombinator post about Hammer2 work has an in-depth comment from Venkatesh Srinivas about DragonFly’s network setup, memory allocator, and token use. (Ignore the trolling in other comments.)
- Michael Lucas’s next No Starch Press book is Absolute OpenBSD, second edition.
- Pictures and video are starting to show up from the just-passed BSDCan 2011. (via this and also thesjg on EFNet #dragonflybsd)
- My first experience of The Internet was very similar to this. It should be bizarrely unfamiliar to anyone under 20 or so. (via) Get this: I typed ‘exit’ instead of just closing the browser window when I was done messing with it, because some habits cannot be broken.
If you’ve used ‘free’ on Linux to find available memory, Steve O’Hara-Smith has described the way to do the same thing on DragonFly.
It looks like Sepherosa Ziehau is working on getting multiprocessor kernels able to boot on single-processor systems. This makes life a bit easier, since there’s only one kernel needed for any given processor. I don’t know if it’s in a finished state yet.
For anyone who missed/couldn’t join Summer of Code, there’s still lists of potentially interesting projects, as Alex Hornung points out.
I didn’t think of this, but I needed it: if you have an older Hammer system that now can perform deduplication because you upgraded to DragonFly 2.10, make sure to add it to the configuration for that file system, or else it won’t run.
If you have a really old DragonFly system, meaning you’ve been upgrading it since version… 1.8 (I think?), you may have libpthread linked to libc_r instead of libxu. This means that if you have a system that old, you will now need to set THREAD_LIB or just recompile your pkgsrc programs on your next upgrade to something after DragonFly 2.10. I don’t think this is going to apply to a lot of people.
(I hope I got the lib details right…)
Let’s see, what do I have now…
- Did you know we just released DragonFly 2.1? Neither did I.
- The AppleCrate II (][?), a set of parallel Apple //e systems. It makes me so happy. I love to see how
simpleuncomplex the old Apple systems were, almost at the level of programmable logic controllers today. I was struck by the fact that the Apple //e requires less than 5 volts, which means it could run off a USB port. (via lots of places)
- Removing the internet’s relics: a call to kill FTP now that it’s 40 years old. There’s no easy alternative, though…
- 20 years of Adobe Photoshop. (via) Obviously that’s not found on any BSD platform, but almost every raster-based image editor out there tries to emulate Photoshop in some way, on every platform. It casts a long shadow. Plus, I remember the Photoshop 2.0 loading screen, so now I feel old.
- Is tech blogging becoming worse? i.e not really tech any more? I’ve mumbled about this before, since this site is arguably a tech blog. Sites tend to diversify and lose focus to grow their audience. You can see the same pattern in the magazine market, back when there was a magazine market. You don’t have to worry about the Digest – I’m targeting BSD users, so I’m totally not growing my audience! (Joking, joking. Readership is staying even to slightly up, over the last while.)
On a separate note that has nothing to do with DragonFly: if you live outside the United States and have a postcard handy, can you send it to “St. John Neumann School, 31 Empire Blvd., Rochester, NY 14609 USA”? My daughters’ school is collecting international postcards this month as part of their geography lesson. It doesn’t have to have anything specific, other than be interesting to 8-year-olds.
ipfilter has now been removed from DragonFly, by Sascha Wildner. We now have “only” ipfw2 and pf for software firewalls.
John Marino has updated the GNU Debugger (GDB) from version 7.0 to version 7.2. The lengthy commit message describes how surprisingly complex the upgrade proved to be.
And here’s one last writeup/introduction for Google Summer of Code projects on DragonFly: kevent part 2. (Apparently school exams prevented this from being written sooner.)
The 10th EuroBSDCon is happening in Maarssen, The Netherlands, October 6th through the 9th. The call for proposals is up until the end of the month.
Matthew Dillon’s been thinking about Hammer, and how to implement clustering well enough to work as a sort of RAID replacement. He’s written up a document describing his plans. Some highlights:
- writable history snapshots
- quotas and accounting
- live rebuilds of data from mirrors
- and the same history, mirroring, and snapshots as before.
It’s going to be a while before this “Hammer 2” becomes a finished product, though, so don’t count on it for the next release.
The May issue of BSD Magazine is out. The cover says “Embedded FreeBSD” as a continuation of last month’s theme, but there’s really a wide mix in there. Of course, there’s a DragonFly news section from me, talking about 2.10. I’ll point at the zsh article that opens the magazine, since every zsh user I’ve met talks about zsh rapturously. Also, the iXSystems ad on page 2 has a rather fun illustration…
Tim Bisson has inital TRIM support working for UFS. His lengthy posting talks about how it’s done, and shows how much it speeds things up. He’s looking for testers, so please try it if you have a SSD. (The usual warnings apply about testing code that specifically deletes things.)
For those not familiar with TRIM in SSD context, here’s the least annoying page with an explanation that I could find in a few seconds of Googling.
The May 2011 issue of the Open Source Business Resource is “Technology Entrepreneurship“. You’ll want to read this because it’s all people who are their own boss, using software they can modify themselves. Seductively nerdy/utopian! They’re continuing the topic for next month’s issue, so if that describes you and you like writing, here’s your chance.
John Marino’s gone on a tear and updated GCC to version 4.4.6, diffutils from 2.87 to 3.0, and texinfo from 4.8 to 4.13. Each commit message that I linked to has plenty of notes on what’s different, so they’re worth following. This is the first update for texinfo in 6 years, so the quantity of updates is not surprising.
Kedar Soparkar applied for Google Summer of Code with DragonFly, but didn’t make the cut with the very few slots we have. So, he’s going to work on his i386 ABI implementation anyway. More student work is always wonderful news.
Then use the new LINT64 config added by Sascha Wildner. LINT kernels have every option turned on, so it’s pretty easy to have problems due to conflicts or untested parts and so on. You probably won’t get a kernel out of it, but now there’s a comprehensive list of your 64-bit kernel options for when you’re building a kernel that works.
There hasn’t been much to nab for Lazy Reading, lately. Oh well. The last few weeks were good so it has to even out sometime.
- Did you know GBC stands for Great Ball Contraption, a Lego device designed to move little plastic balls? Here’s 20 of them chained together. (via b3ta)
- The original University POSTGRES. (thanks, Jan) This is a source for PostgreSQL, as far as I can tell, which makes it in some ways contemporary to BSD’s origins. I am not surprised. PostgreSQL seems to be the thinking person’s alternative to MySQL like BSD is the thinking person’s alternative to Linux.
- Do you have a pf.conf? The people behind fwbuilder can use it for examples, so they can support pf in their config builder. (via)