Earlier this week, Daniel Flores posted the first-week report on his Google Summer of Code project, file compression in Hammer. He mentioned that the LZ4 algorithm he is using seems to have the best performance with repeating text data, as in logfiles. I asked for numbers, and he provided them. The important data in the results is the total compression column. It shows how many 64k blocks were able to be compressed, with that type of data.
If you have a mps(4) device (LSI Fusion-MPT 2 SAS disk controller), you may be interested in Matthew Dillon’s large commit of bugfixes from FreeBSD. Specifically, he notes that the drive gets ‘overtagged’, and performance can be greatly improved by reducing the number of tags used.
Thanks to Sepherosa Ziehau, Netperf will now calculate Jain’s fairness index. That is a formula that is both interesting and unfamiliar to me. Not that I understand it, of course – it’s just because it has a neat name.
It looks like OpenJDK7 works in pkgsrc for DragonFly, thanks to Ryo ONODERA, and I think it’s working in dports too.
Because Sepherosa Ziehau changed mbstat, anyone on bleeding-edge DragonFly will need to rebuild world, or else netstat will become confused.
All the Summer of Code students for DragonFly have posted their first week reports:
- Daniel Flores: HAMMER2 compression feature
- Larisa Grigore: System V IPC in userspace
- Pawel Dziepak: Make vkernels checkpointable
- Joris GIOVANNANGELI: Capsicum
- Mihai Carabas: hardware nested page table support for vkernels
If any of these projects are interesting to you, or if you have any tips for these students on work they are doing, please provide feedback.
There’s already been some previous conversation about how much longer to support the i386 platform for DragonFly. It looks like PC-BSD will be the first ‘flavor’ of BSD to make the jump. Support for PC-BSD on i386 will be dropped after release 9.2. That includes ‘TrueOS‘, the version of PC-BSD for servers, which I did not know about until just now.
There’s support for a new timer mechanism in DragonFly 3.5, for x86_64 users: TSC. Sepherosa Ziehau added support and has described how to disable it – it’s on by default. It speeds up some very basic (and frequently used) system calls.
It’s possible your Internet service provider uses a non-routeable IP range (like 10.*) and occasionally your border device picks that up via DHCP by accident instead of an Internet address. If that happens to you, and you’re using DragonFly as your border gateway, it’s possible to prevent it with PF dhclient.
If you get your Hammer drive really full, a normal cleanup won’t make enough space. When that happen, use ‘hammer reblock’ in increasing increments. That works because it does cleanup in much smaller steps.
If you’ve been reading the Digest for a while, you’ve seen me talk about the value of hosting or running your own services. It’s not too much of a surprise in my case; if you are working on an open-source operating system, you want to run it. It’s good to get the experience, and you can run programs the way you want, instead of picking from whatever vendors happen to sell you.
The PRISM disclosure, which I am going to assume everyone is familiar with at this point, is another facet. Every time you use another company for your email, your entertainment, your software, and so on, their information on you can be accessed. This isn’t a problem that can be fixed by going from one webmail provider to another. You can shop around, but notice that the author in that link effectively throws his or her hands in the air and says, “there’s no way out” by the end of the article. This is because corporations work as collecting agents for the government, even if they don’t plan to do so.
That sounds drastic, but there’s legal frameworks in every country for governments to require companies to give up data on any person, on request. It happens. I’ve seen it myself; I worked for Time Warner for several years, tracking down cable modem user information and handing it over as compelled by law. I know the lawyers at TW Corporate didn’t like doing it, but they didn’t have a choice. (I have some horrifying stories about what people would do to themselves and each other.)
Companies are increasingly working to create services to sell, not products to buy. A service never stops being consumed, so it forms an ongoing revenue stream. I’m not saying this is bad; I firmly believe that a financial incentive to be paid improves services. However, as only a consumer, you can end up not owning what you use. Other people have pointed this out, and I don’t want to sound like a frothing crazy person… but it is relevant, though not necessarily as catastrophic as some people pronounce.
What I’m working towards here is a reminder that you should run your own software, and running it on DragonFly is the best way. (Or some other operating system, I guess. If you have to.) Instead of trying to figure out what the least-bad commercial option can be, run it yourself. Good for privacy, good for learning. I know that’s not an option for everyone; fighting with Sendmail (for instance) is not an activity that many people pick voluntarily. But, if you’ve been thinking of setting up a replacement for Google Reader, or hosting your own mail, or own blog, etc… there’s never a better time than now.
(Follow all those links for some good information; consider it an early Lazy Reading post)
Sepherosa Ziehau has added a sort of queuing to altq, where TCP ACKs get higher priority. You may have seen this in any number of pf configurations, where returning data is given its own queue to keep high-volume transfers from slowing themselves down because the acknowledgements can’t get back to the sender. His commit has statistics on the performance improvement. He also added a ‘netrate‘ tool for calculating results from using netperf.
If you’re using DragonFly 3.5, your next update should be a full buildworld. That’s because John Marino is adding the framework for symbol versioning. This means that individual library (.so) files will internally keep track of newer and older symbols. The current behavior is to name the files differently, which can cause problems if an expected, linked file is missing – even if the needed symbols are present. The basic framework is being added now, and will be turned on all at once, to minimize the number of times that full buildworld is needed.
The ‘amd64’ specific parts of kernel architecture have been removed, since x86_64 covers all that. As a side effect of other changes, John Marino warns that upgrading DragonFly from a version older than 3.4, to a version newer than 3.4, will require an intermediate step of going to 3.4 first. e.g. If your machine is a DragonFly 3.0 system, you will need to upgrade to 3.4 before moving to, say, 3.6 once it is out. This won’t matter for some months, since the next release is months off.
Larisa Grigore posted an introduction of her Summer of Code project: Userland System V IPC in userland, and Daniel Flores wrote out his initial ideas for Hammer compression. That’s the remaining two projects introduced. If any of these interest you or you want to make suggestions, respond on the lists. Work starts on the 17th.
FreeBSDNews.net has a nice summary up of video from all (?) the presentations at BSDCan 2013. Of particular interest to DragonFly users: a video about pkg, the tool used for package maintenance in dports. In this presentation, it’s talking about use on FreeBSD, but the future stuff applies to DragonFly too.
Pawel Dziepak has posted details on his Summer of Code project for DragonFly. He will be making it possible to checkpoint vkernels, restoring network and console state. He even has a public repository for his work set up.
Not as wordy this week, but still wordy. And linky!
- Max Headroom and the Strange World of Pseudo-CGI. A discussion of how old fake CGI can look better than modern, real CGI. This is an opinion I’ve had for quite a while, and my children pretty much ignore it every time I bring it up. (via)
- The Colby Walkmac, which predates the Mac Luggable. Linked to because it includes good pictures of what the (external) hardware was like. I find all the old ports interesting, since it’s all USB and the occasional eSATA these days… not that I’m complaining! I’ve never had a good experience with a 9-pin serial port. (via)
- A brief education on escaping characters.
- I get worried when remotely rebooting a server in a different town or even state. In Praise of Celestial Mechanics covers much more stressful circumstances: interplanetary reboots. Does Voyager 1 or 2 have an ‘uptime’ function?
- The equivalent of what you are doing right now, 20 years ago. I personally never got to see this; my experience was MUDs. Speaking of which…
- The Birth of MMOs: World of Warcraft’s debt to MUD. MUD == MMO, Roguelike == Diablo/Torchlight, Doom == almost everything else. There’s a number of game archetypes that haven’t changed in some time. (via)
- Playing with powerlines. I used to work at a company that used these lines for data transfer. It was neat technology, but it sure wasn’t easy to set up. Imagine wiring a city but only being able to use Ethernet hubs. Not switches, hubs. That, combined with undersized ARP caches/MAC tables, made it really difficult.
- OpenVPN on FreeBSD, which will come in handy for at least several readers, I’m sure, as the directions should apply to any BSD.
- Is there anything DNS can’t be used for? Cause now it’s domain-based mail policy publishing. (via ferz on EFNet #dragonflybsd)
- “Have you tried DragonFly?” posts on various forums seem to pop up with some regularity.
- Uses of tmux, explained. A slide show talking about how tmux works. (via)
Unrelated link of the week: I’ve had several deadlines and a mail server with issues this week at work, so this is all I got.
Since dports uses FreeBSD ports as a base, adding something to FreeBSD ports means it will show in dports, too. However, it doesn’t have to go that way. It’s possible to have dports packages that exist only in dports. If you have changes to a port that make it compile on DragonFly, that can be added too. For all of that, go to the dports issues page on GitHub.
Another DragonFly/Google Summer of Code project introduction is up: Mihai Carabas wrote out his project on developing hardware nested page table support for vkernels. If Mihai’s name seems familiar, it’s because he was in Summer of Code for DragonFly last year, with a successful project.