I usually link to new features, additions, and so on. That’s fine, but there’s often necessary work that goes on which doesn’t correlate to a new function – just better code. Rimvydas Jasinskas just did one of those cleanups, and I’m mentioning it to give credit where’s it’s due.
Category: Committed Code
If you happen to be testing kernel modules, DragonFly can now load them from a modules.local directory. This keeps modules that aren’t part of the base system, separate. This is probably of most use to developers. It’s controlled by local_modules being set in /boot/loader.conf, and defaults to on.
(Updated for correct file location – thanks, swildner)
If you’re using qemu and DragonFly, the latest update of ACPICA to version 20160422 may fix some issues introduced in a previous update. (I don’t have a specific bug report to point you at; sorry!)
Tomohiro Kusumi has been creating a near-constant stream of bugfixes and cleanups to Hammer for quite some time. I don’t often link to it, because they are incremental improvements and hard to linkblog, so to speak. In an effort to make up for this deficit, I do want to draw attention to his two recent commits: “Make hammer commands print root volume path“, and “Print volume list after volume-add|del“. Small changes, but this is what makes complex systems usable.
I keep posting about Sepherosa Ziehau’s work on sustaining extremely high traffic loads in DragonFly. Now I’m posting about a tool to create that load: kq_sendrecv. It creates tens of thousands of TCP connections, without creating a process for each, and uses kqueue, as you might guess from the name. This may be useful if you really want to tax another system.
unzip has been added to DragonFly, making it present in every BSD but I think OpenBSD.
Imre Vadasz has added the ability to create a UEFI bootloader in DragonFly. Can you use it? I don’t know; I haven’t tried it yet and I can’t tell from the commit.
John Marino has added the starting framework to use clang as the alternate base compiler in DragonFly. Note that it’s not hooked into the build yet. This is the first non-GCC compiler added into DragonFly, so there’s some work yet before you can have an all-clang system. This should replace GCC 4.7, which is the current alternate compiler. GCC 5.0 is the default, if you didn’t know.
Note that clang is present in dports, so it’s already been available for general use, for some time. This framework is for building DragonFly itself.
If you are running bleeding-edge DragonFly, Sepherosa Ziehau has made some networking changes that both reduce CPU usage in high-traffic situations and change some underlying network structures. This means a full buildworld is needed on your next update.
If you’re using DragonFly 4.4.x or older, you are unaffected.
Sepherosa Ziehau has continued his quest of making large-scale data transmission on DragonFly effortless; his latest change has cut the kqueue contention rate by two-thirds when dealing with a connection rate of nearly 400,000 connections per second. Note that’s number of connections, without even tracking the bandwidth used by each.
Bill Yuan has added ‘ipfwsync’ to ipfw3 in DragonFly. As you may expect from the name, it’s a way to sync ipfw3 configurations across multiple devices.
If you have a Broadwell system, the drm.i915.enable_execlists tunable added by Imre Vadász may keep your system stable. (thanks, zach on EFNet #dragonflybsd)
That’s a pretty cryptic headline, isn’t it? John Marino has ‘privatized’ several libraries in DragonFly, so that they can’t get included involuntarily as part of a port build. That may mean you will need to perform a full rebuild of your system if you are tracking DragonFly-current.
(This is the way to fix ‘system’ languages like Perl was in FreeBSD 4.x – keep them clearly separate from the port version. It’s about a decade too late for that idea to work out, though.)
A number of people have reported problems with qemu and DragonFly, both running locally and on a host. It turns out to be a problem with the getcontext(), setcontext(), and swapcontext() functions, but Matthew Dillon fixed it in a way that doesn’t affect performance very much.
That apparently wasn’t good enough, so he added _quick versions of those same functions, so it became not just a fix, but an improvement.
In related qemu news: qemu-devel can use vknetd similar to a vkernel, now.
Hammer now defaults to ‘noatime’, meaning the date and time of last access are not updated on every file action. Note that creation and modification date and time are still recorded. This will help with speed and disk activity.
This may cause a problem with any software expecting this to change – mutt, possibly? We will find out. This change was done after the 4.4 branch, so it’s not in the current release of DragonFly.
I am taking this moment away from my significant backlog of things to post to note that there have been a lot of games fixes in DPorts lately. Thanks to Rimvydas, many small bugs that kept games from compiling on DragonFly are now fixed. The easiest way to see is to look at the commits from December 8th and back, but the best way is to pick one and play.
I have a huge backlog of things to post, so this is originating from the 17th: Matthew Dillon has been working for some time on hardlinks and Hammer 2. Hardlinks are the same file, presented in multiple places. This can be a problem when your filesystem keeps infinite, writable snapshots. The solution he just commited is called ‘xlink’ and the commit message has details.
The default linker in DragonFly has been switched to gold, the newer version of ld. (get it, go-ld?) It’s faster, cleaner, going by the commit message. It’s possible to switch back to the old one if needed. This predates the recent branch for 4.4, so it will be default in the release, too.
Imre Vadász fixed top so that hitting ‘c’ filters displayed processes by command name. I am mentioning this not because it’s a huge change, but because I forget about all the interactive elements that are possible with top.
Does that count as alliteration? Anyway, Matthew Dillon has increased the size of the starting window in TCP. If you are on a higher-latency link and/or fetching lots of small files, you should notice better performance.
If you are using bleeding-edge DragonFly (4.3) on a machine with Intel video, the i915 module has been renamed. This means you will probably need to rebuild xf86-video-intel from source to have it match. There should be a matching binary package soon.
If you are on DragonFly 4.2, this does not affect you.
I think at this point, Sepherosa Ziehau is able to improve the DragonFly network stack by just standing near his computer and concentrating for a few minutes. For example, he’s unearthed another improvement to connect rate/reduction of CPU usage.
John Marino is working on versioning libc, and as part of that process, libc is no longer loaded into executable memory. Here is I think an explanation of lib versioning that may apply, and of course moving things that aren’t supposed to execute, out of executable memory areas, is good for security. There’s more on that topic, too – W^X may be a similar example.
This is a complicated topic that I’m not part of, so suggest better descriptions in the comments, please.
I don’t note it enough, but Tomohiro Kusumi has been making constant updates to HAMMER, the version we have now. Often they are the sort of update that makes the code more readable, or fixes possible problems, and so on. Very essential, but hard to post about it. In any case, I’m using his recent improvements to hammer volume-del to note his contributions, of which there are much more than the day’s worth I link here.
hostapd, for creating a wireless access point, has been included in DragonFly along with wpa_supplicant, for a long time. Like wpa_supplicant, there’s a version in dports that is the latest version and is easier to update (e.g. no system update required to get a newer version.) Unlike wpa_supplicant, there’s no chicken-and-egg installation problem if it’s not in the base system – so out it goes.
The more eagle-eyed may have noticed a branching for DragonFly 4.2, and for DragonFly 4.0.6. The 4.2 branch is currently only a release candidate, so don’t necessarily change over yet – it’s for testing, not release.
Note that packages for 4.2 are not yet built, so you’ll have to manually specify a package path to install with pkg on 4.2 – for now.. That won’t be the case for the actual release, of course. DragonFly 4.3 users will have to specify PKG_PATH manually to use 4.2 images until new ones are built. 4.2 release candidate users will be fine. (see comments for correction.)
The 4.0.6 release is mostly to get the recent OpenSSL update into a 4.0.x build.
I am working on image building for both.
Even sysctl accesses can be made to handle multiprocessor environments. This can actually make a difference when you’ve got a lot of processors building a lot of software, as in all of dports.
Those changes I mentioned yesterday for text console support? They’re in DragonFly-master now, along with a loader tunable to turn it on and off.
Hammer will perform daily housekeeping tasks each night. If you’re up late enough, it may kick off while you are working. If you want to stop the process after it’s already started (since it’s disk-intensive), John Marino has added the ‘abort-cleanup‘ command.
Sepherosa Ziehau has introduced a new sysctl:
Set this to zero and you won’t get endless ARP events from networks you aren’t on. For example, I’m hooked up to a cable modem. I only get a public routable IP address, but the network used for the cable modem network itself bleeds ARP packets out where my DragonFly machine can see it. Since it’s on a different network segment than the address I receive through DHCP, it always fails and the system logs it. For example:
May 11 05:20:52 www kernel: arplookup 100.68.112.145 failed: host is not on local network
I can’t do much about it since that layer 2 leakiness is going to happen, but I can shut it up with this sysctl – and thank goodness, cause I’ve been seeing these messages since first using a DOCSIS modem in… 2001 or so?
Tomohiro Kusumi has been quietly making a lot of commits to Hammer. I haven’t been linking them because they don’t necessarily equate to new features, but here’s an recent exception: the -A argument will make your Hammer command run on every PFS. It only affects reblocking/rebalancing – for now.
You can now export Hammer slave volumes as NFS mounts – but since slave volumes are updated from master, you’re mounting a snapshot of that point in time. That may actually be an advantage.
DragonFly builds two compilers by default. If you weren’t interesting in building both, there were switches to build only the default, like NO_GCC47. This changed with every compiler update.
With the switch to GCC 5, the new switch is “NO_ALTCOMPILER”. That will last through compiler changes. I’m mentioning this now because sooner or later, you’ll want to gain back some time on a buildworld.
I haven’t been drawing enough attention to it, but there’s been a bunch of HAMMER filesystem activity lately: First, Tomohiro Kusumi has been working on HAMMER – these posts are a small subset of his commits. Second, Matthew Dillon has been working full steam ahead on HAMMER2. The HAMMER2 design document has been updated (read this!), and he’s already accomplished master->slave disk syncing.
It’s not ready for production, of course, which you may already realize, so don’t install it unless you want to work on the code.
DragonFly 4.0.3 has been tagged; you can look at the tagging message for details, but the major reason for doing so is to include OpenSSL-1.0.1l. I will have images up soon.
The CAM layer in DragonFly has had its big lock removed/been marked MPSAFE, so you will notice a performance increase when using multiple disks. (assuming you aren’t throughput-limited, of course.)
Sepherosa Ziehau has posted a note that V4-mapped addressing is no longer supported in DragonFly. You will need to do a full buildworld/buildkernel if you are running master. Also, TCP MTU path discovery is on by default. Also also, he’s added a SOL_SOCKET/SO_CPUINT socket option for use to reduce load in heavy network activity. As usual, I don’t quite comprehend.
It’s possible, if you are several releases (years) behind, to end up with a DragonFly system that can’t compile and install the current release, due to incremental changes over time. It’s rare, but it could happen now between, say, version 3.4 and 4.0. The usual solution would be to incrementally upgrade in order, which is a lot of building and updating. The alternative is the new installworld-force option from Matthew Dillon that forces a new set of binaries into place. Use as a last resort.
In an effort to reduce my backlog of DragonFly things to post about, here’s quick notes:
- The path to xauth is now configurable, though correct by default. (that’s bit me in the past)
- There’s a new callout*() implementation.
- cpuctl(4) has been imported to allow CPU microcode updates.
- libm has been updated with math functions from FreeBSD and NetBSD, which because of library versioning support, won’t cause compatibility problems for older vs. newer DragonFly versions.
- C++11 support is also now available.
With a recent commit from Sascha Wildner, DragonFly now loads XHCI (meaning USB3) by default. If you had previously tried to install DragonFly via USB stick, and it inexplicably refused to mou t the installer drive… It may work much better now.
Chrome runs on DragonFly now, apparently possible now because of this ported fix from Joris Giovannangeli.
John Marino updated wpa_supplicant (in dports). He then suggested moving it out of base into dports, so that it could be updated independently of the base system. (this update, for instance, took years.) Since wpa_supplicant is necessary to get some systems online – and it can’t be installed if missing if you don’t have a network link – it may be too risky. I think other packages could be moved out, myself.
Markus Pfeiffer has imported FreeBSD’s if_lagg to DragonFly. It’s for talking LACP over multiple network ports, so that the traffic from those multiple ports can be aggregated – if what’s on the other end generally understands LACP. (Failover mode may not count.) Please test if you have that sort of surfeit of network ports.
This very long commit message from Sepherosa Ziehau details the UDP changes he’s made. It’s mostly technical details, but at the end he mentions this little tidbit:
“For ‘kq_connect_client -u’ test, this commit gives 400% performance improvement (31Kconns/s -> 160Kconns/s).”