Because of some structure changes made by Matthew Dillon while chasing a pf bug, you will need to do a full buildworld/buildkernel on your next update – if you are running DragonFly-master. 3.8 users are unaffected by the bug or the change.
Category: Committed Code
DragonFly’s dhclient will now retry failed interfaces and handle being re-run gracefully. This is a blessing for anyone who has had a flaky link. Matthew Dillon’s made two other improvements for booting that will also improve boot time when networks go missing.
Thanks to Zachary Crownover, rcreload is available in DragonFly. (It’s always good to see a new contributor name.)
Nuno Antunes brought in a significant number of fixes for libradius. He’s been doing other work recently on netgraph7 support, so I’m linking to this as a ‘signpost’ commit.
While Matthew Dillon was testing the new up-to-256-processor support for DragonFly, he added a few sysctls, one of which helps qemu performance when emulating a lot of processors. I note it here in case it’s helpful to someone else.
DRM (Direct Rendering, not Digital Rights) on DragonFly will normally eat all the memory it thinks it needs. However, vm.dma_reserved can now be set to a fixed limit in /boot/loader.conf. By default, vm.dma_reserved on DragonFly is set to 16M, and can be set higher. I think this is necessary when running higher-resolution screens… Don’t quote me on that, though.
Matthew Dillon changed the default keep-policy in DragonFly to:
set keep-policy keep state (pickups, sloppy)
This is to match other BSDs (which? I don’t know) and reduce overhead, according to the commit.
The obvious joke should be “how can you tell?” Anyway, the csprng in DragonFly has been updated and IBAA is being used more often, and there’s more updates on the way.
ATM support is gone in DragonFly, and frankly, I’m surprised it was still there.
DragonFly now has a ‘rescue’ system added in, which also functions as a way to mount encrypted filesystems. Does PAM work yet? I don’t know; I may be linking to this earlier than I need to.
A recent commit from Sepherosa Ziehau has a 5% improvement in the number of network connections per second a x86_64 machine can accept. He’s also reducing the number of IPIs during network activity. If this seems somewhat esoteric, it’s because network speeds are getting so fast that the benefits come from reducing the accompanying CPU load.
If you’re on DragonFly 3.7, you will need to build world before building the kernel again if you are updating to some point in the last 24 hours. Sascha Wildner points out the related commit.
Sascha Wildner brought in ACPICA 20140214, and his commit message has a list of the updates.
There seems to be a lot of ACPI-related updates lately: Sascha Wildner has updated ACPICA in DragonFly to what I think is the very latest version. See his commit for the differences.
John Marino updated daemon(8) on DragonFly. For some reason, I didn’t know it was a standalone program. I knew about the idea of daemons as helpers based inside the computer, which is why so many server programs end with a ‘d’ – sshd, ftpd, and so on. Inexplicably, I never actually saw the program itself.
Markus Pfieffer has committed Larisa Grigore’s Google Summer of Code work, “SysV IPC in userspace”. It’s been a bit since the event finished, but it’s in DragonFly now.
Matthew Dillon wrote a roundup post summarizing all the changes he’s made to DragonFly to improve SMP performance in the last few weeks. He’s removed almost all contention from DragonFly. This means better performance, scaling upward depending on the number of processors.
‘monster’, the system that builds all 20,000 items in dports, can complete the run in 15 hours. Compare this to the 2 weeks it used to take me to build the 12,000 packages in pkgsrc. This is admittedly on different hardware and different packaging systems, but it gives a sense of the scale of the improvement.
The ‘poweroff’ command, the equivalent of ‘halt -p’, has been added based on a suggestion from Robin Hahling.
Matthew Dillon was using poudriere, the dports build tool, on a 48-core system. Poudriere was building all 20,000+ dports, so the machine was quite busy. He decided to get rid of as much contention as possible, and he’s listed all the ways DragonFly’s been streamlined by these efforts. We need to revisit some of our previous benchmarks…
John Marino has accomplished the major task of updating gdb/kgdb, to version 7.6.1 for DragonFly.
John Marino has put in a large patch to DragonFly 3.5, updating all sorts of language-related items. As he warns, you will need a full buildworld/buildkernel in a specific order to update. On the plus side, you can now probably use your native language for nvi and for git.
Matthew Dillon’s committed the work by Daniel Flores on Hammer 2 compression and Mihai Carabas’s vkernel hardware support – both Summer of Code projects. There’s a good amount of detail in the commit messages describing the work and what it changed; I expect more Summer of Code work to be getting committed…
Note: you’ll want to do a full update.
Antonio Huete Jimenez has committed his work on “dirfs”, a filesystem that lets you mount directories from your host machine within the running vkernel environment. It’s a sort of shared folders for vkernels. See the commit message for usage details.
killall -T will now kill all processes associated with the current tty, except parents of the killall process itself. It’s a shortcut to “kill all these runaway items I started by accident”.
Sepherosa Ziehau has added a sort of queuing to altq, where TCP ACKs get higher priority. You may have seen this in any number of pf configurations, where returning data is given its own queue to keep high-volume transfers from slowing themselves down because the acknowledgements can’t get back to the sender. His commit has statistics on the performance improvement. He also added a ‘netrate‘ tool for calculating results from using netperf.
If you’re using DragonFly 3.5, your next update should be a full buildworld. That’s because John Marino is adding the framework for symbol versioning. This means that individual library (.so) files will internally keep track of newer and older symbols. The current behavior is to name the files differently, which can cause problems if an expected, linked file is missing – even if the needed symbols are present. The basic framework is being added now, and will be turned on all at once, to minimize the number of times that full buildworld is needed.
The ‘amd64’ specific parts of kernel architecture have been removed, since x86_64 covers all that. As a side effect of other changes, John Marino warns that upgrading DragonFly from a version older than 3.4, to a version newer than 3.4, will require an intermediate step of going to 3.4 first. e.g. If your machine is a DragonFly 3.0 system, you will need to upgrade to 3.4 before moving to, say, 3.6 once it is out. This won’t matter for some months, since the next release is months off.
OpenBSD has a new identd daemon. Is identd used for anything other than verification when connecting to an IRC network? I’ve never seen it in another context.
Peter Avalos has committed another batch of updates to sh(1), from FreeBSD. I was going to comment on how strange it was to see software getting updated so many years later; you’d think everything there was to update for /bin/sh had been done at this point. Digging casually, the oldest bit on sh that I can find is from 1991 – 22 years old. The man page mentions a rewrite in 1989 based on System V Release 4 UNIX, and there were versions of sh all the way back to version 1.
Here’s a trivia question – what’s the oldest Unix utility, and what’s the oldest code still in use? I don’t know the answer.
Here’s 3 recent and different commits to DragonFly that I’m commenting on all at once:
- Peter Avalos upgraded libarchive in DragonFly to 3.1.2, with a note of the changes. An ordinary and appreciated update.
- Sascha Wildner updated the ISO639 file to include the newest update: “Standard Moroccan Tamazight”. There’s no particular utility to that; I just like saying “Standard Moroccan Tamazight” out loud.
- Work on poudriere, the utility for bulk-building DPorts packages, has caused some nice speedups for DragonFly in extremely stressful situations. See one of Matthew Dillon’s recent commits.
I really wish the other BSD projects would include commit lines in the mail message subjects, so it was easier to catch things like these.
The new vm.read_shortcut option has been turned on by default by Matthew Dillon, which should lead to some performance improvements. That improvement has been measured for tmpfs, at least. There’s also some buffer cache improvments that help on x86_64 systems, too.
Update: As Venkatesh Srinivas pointed out, tmpfs also no longer uses the mplock, so it’ll take better advantage of multiple processors.
John Marino proposed a method for backing up world when upgrading, for those rare but catastrophic cases where the installed programs can’t run. After some discussion, he committed an automatic backup method, and there’s a ‘restoreworld’ target to take advantage of it.
The kernel already gets renamed to kernel.old as a backup, if I remember correctly.
As Sepherosa Ziehau mentions in his latest commit, DragonFly now collects IFNET/IFADDR statistics on a per-CPU basis. This makes it more accurate, but may mess with any third-party program that accessed it directly. I don’t know if there’s anything in pkgsrc that does that…
I know OpenSSL in DragonFly was just updated, but Peter Avalos has done it again, bringing it to version 1.01e. I assume this new version is to fix some recently-exposed problems. He also has updated libdialog, which was previously not located in contrib/, as sime third-party software needed a more modern version. As a side effect from that, tzsetup in DragonFly now matches the version in FreeBSD and NetBSD. And, Sascha Wildner has updated the locale files on DragonFly, also to match FreeBSD and NetBSD.
The emx(4) driver now has support for multiple TX queues, but it’s not on by default. There’s scenarios where multiple queues work out with that hardware, but you have to be sure you are actually in the right setup for that first. Check Sepherosa Ziehau’s commit message for the details.
John Marino has set gcc 4.7 as the default compiler in DragonFly. This replaces the previous default of gcc 4.4. The 4.4 version is still available, and while you can set NO_GCC44 to keep it from being built, John’s commit message notes that it’s still useful especially for some ports that don’t work with gcc 4.7.
Sepherosa Ziehau makes commits almost daily to DragonFly’s network infrastructure, but I have a hard time quantifying it into Digest posts in part because it’s often very technical. His most recent commits come with an explanation, however. He has done plenty of work to improve overall transmission speeds in DragonFly, and now he’s working on ‘fairness’. Fair, in this case, means ensuring that packet transmitting and receiving happen without either one monopolizing the connection. In real world terms, this translates to much more constant speeds. His recent commit details what he’s doing and some numbers to prove it.
Remember I said he’s improved speeds? Note that in his example, he’s reaching stable peaks of 981 Mbps. This is on a line that I assume theoretically maxes out at 1000.
Peter Avalos has updated m4 for DragonFly. This will bring us a little more in sync with the other BSDs. Also, John Marino has updated flex, which is apparently 17 years old? Meaning it hasn’t been updated in DragonFly ever, and then not in FreeBSD before that, for a long time. Looking at the timeline on the flex web page appears to match.
If you recall, Phoronix recently ran a bunch of benchmarks on DragonFly. One spot that didn’t look good was the “Himeno Poisson Pressure Solver”. I’m no closer to knowing what capability it actually tests other than itself, but Alex Hornung, Matt Dillon, and Venkatesh Srinivas figured out that cache coloring was the missing ingredient. DragonFly now scores the same as Linux.
Tangentially related, this cache coloring is happening in nmalloc, which is now used on 64-bit DragonFly systems. The previous one, dmalloc, had problems in long-running programs.
I’m not sure what IFQ stands for, but Sepherosa Ziehau’s added it. It appears to be based on an idea from Luigi Rizzo called ‘netmap‘. In this case, network packets are grouped together before being placed onto the network interface’s hardware queue. That means better packet per second performance without a corresponding increase in CPU usage, as Sepherosa Ziehau’s report lists, along with needed sysctls.
If you’re running DragonFly 3.3, make sure you perform a full buildworld and buildkernel when you next upgrade. Sascha Wildner is mentioning this as a cautionary note after experiencing issues when using quickkernel, after removing a number of syscalls. Once past that point, it should be safe to go back to quickworld/quickkernel.
I knew about files like /etc/services, for common IP port usages, and /usr/share/zoneinfo, for time zones, but I didn’t know that DragonFly (along with other systems) keeps a list of agreed names for various human languages defined by ISO639 in /share/misc/iso639, and it’s maintained at least in part by the Library of Congress. At least I didn’t know until Sascha Wildner updated it.
If you are one of the few people still wanting to read an OS/2 HPFS drive, support for it in DragonFly has been updated by Antonio Huete Jimenez. It’s read-only, but writing didn’t work well, and I’d be surprised if there’s any hpfs disks that aren’t archival, out there.
The initial download of pkgsrc via Git on DragonFly is a little bit faster now, with the ‘make pkgsrc-create-shallow’ option recently added by John Marino. Note that there’s a similar option for src. It skips downloading file history.
Matthew Dillon’s put more of his Hammer work into DragonFly, with notable parts being the creation of a ‘dmsg’ setup for advertising available block devices to share between machines using Hammer. To anticipate your next question: No, it’s not something you can run right now as a test; this is the underlying framework.