Matthew Dillon has been doing a lot of Hammer 2 work lately. Well, he’s been doing it for quite some time, but the recent commits contain the sort of things that are easier to link to, like deletion speedups, freemap changes, and stats tracking/compression results.
Category: Committed Code
Hammer will perform daily housekeeping tasks each night. If you’re up late enough, it may kick off while you are working. If you want to stop the process after it’s already started (since it’s disk-intensive), John Marino has added the ‘abort-cleanup‘ command.
Sepherosa Ziehau has introduced a new sysctl:
Set this to zero and you won’t get endless ARP events from networks you aren’t on. For example, I’m hooked up to a cable modem. I only get a public routable IP address, but the network used for the cable modem network itself bleeds ARP packets out where my DragonFly machine can see it. Since it’s on a different network segment than the address I receive through DHCP, it always fails and the system logs it. For example:
May 11 05:20:52 www kernel: arplookup 100.68.112.145 failed: host is not on local network
I can’t do much about it since that layer 2 leakiness is going to happen, but I can shut it up with this sysctl – and thank goodness, cause I’ve been seeing these messages since first using a DOCSIS modem in… 2001 or so?
Tomohiro Kusumi has been quietly making a lot of commits to Hammer. I haven’t been linking them because they don’t necessarily equate to new features, but here’s an recent exception: the -A argument will make your Hammer command run on every PFS. It only affects reblocking/rebalancing – for now.
You can now export Hammer slave volumes as NFS mounts – but since slave volumes are updated from master, you’re mounting a snapshot of that point in time. That may actually be an advantage.
DragonFly builds two compilers by default. If you weren’t interesting in building both, there were switches to build only the default, like NO_GCC47. This changed with every compiler update.
With the switch to GCC 5, the new switch is “NO_ALTCOMPILER”. That will last through compiler changes. I’m mentioning this now because sooner or later, you’ll want to gain back some time on a buildworld.
I haven’t been drawing enough attention to it, but there’s been a bunch of HAMMER filesystem activity lately: First, Tomohiro Kusumi has been working on HAMMER – these posts are a small subset of his commits. Second, Matthew Dillon has been working full steam ahead on HAMMER2. The HAMMER2 design document has been updated (read this!), and he’s already accomplished master->slave disk syncing.
It’s not ready for production, of course, which you may already realize, so don’t install it unless you want to work on the code.
DragonFly 4.0.3 has been tagged; you can look at the tagging message for details, but the major reason for doing so is to include OpenSSL-1.0.1l. I will have images up soon.
The CAM layer in DragonFly has had its big lock removed/been marked MPSAFE, so you will notice a performance increase when using multiple disks. (assuming you aren’t throughput-limited, of course.)
Sepherosa Ziehau has posted a note that V4-mapped addressing is no longer supported in DragonFly. You will need to do a full buildworld/buildkernel if you are running master. Also, TCP MTU path discovery is on by default. Also also, he’s added a SOL_SOCKET/SO_CPUINT socket option for use to reduce load in heavy network activity. As usual, I don’t quite comprehend.
It’s possible, if you are several releases (years) behind, to end up with a DragonFly system that can’t compile and install the current release, due to incremental changes over time. It’s rare, but it could happen now between, say, version 3.4 and 4.0. The usual solution would be to incrementally upgrade in order, which is a lot of building and updating. The alternative is the new installworld-force option from Matthew Dillon that forces a new set of binaries into place. Use as a last resort.
In an effort to reduce my backlog of DragonFly things to post about, here’s quick notes:
- The path to xauth is now configurable, though correct by default. (that’s bit me in the past)
- There’s a new callout*() implementation.
- cpuctl(4) has been imported to allow CPU microcode updates.
- libm has been updated with math functions from FreeBSD and NetBSD, which because of library versioning support, won’t cause compatibility problems for older vs. newer DragonFly versions.
- C++11 support is also now available.
With a recent commit from Sascha Wildner, DragonFly now loads XHCI (meaning USB3) by default. If you had previously tried to install DragonFly via USB stick, and it inexplicably refused to mou t the installer drive… It may work much better now.
Chrome runs on DragonFly now, apparently possible now because of this ported fix from Joris Giovannangeli.
John Marino updated wpa_supplicant (in dports). He then suggested moving it out of base into dports, so that it could be updated independently of the base system. (this update, for instance, took years.) Since wpa_supplicant is necessary to get some systems online – and it can’t be installed if missing if you don’t have a network link – it may be too risky. I think other packages could be moved out, myself.
Markus Pfeiffer has imported FreeBSD’s if_lagg to DragonFly. It’s for talking LACP over multiple network ports, so that the traffic from those multiple ports can be aggregated – if what’s on the other end generally understands LACP. (Failover mode may not count.) Please test if you have that sort of surfeit of network ports.
This very long commit message from Sepherosa Ziehau details the UDP changes he’s made. It’s mostly technical details, but at the end he mentions this little tidbit:
“For ‘kq_connect_client -u’ test, this commit gives 400% performance improvement (31Kconns/s -> 160Kconns/s).”
You can now start moused with an argument, so it will look at the right device. In most cases, I imagine “
/etc/rc.d/moused start ums0” will be what anyone wants. Credit to Michael Neumann for the update. Perhaps
moused_flags="ums0" will do it too? I haven’t tried yet.
This will overwrite your /etc/devd.conf.
Because of some structure changes made by Matthew Dillon while chasing a pf bug, you will need to do a full buildworld/buildkernel on your next update – if you are running DragonFly-master. 3.8 users are unaffected by the bug or the change.
DragonFly’s dhclient will now retry failed interfaces and handle being re-run gracefully. This is a blessing for anyone who has had a flaky link. Matthew Dillon’s made two other improvements for booting that will also improve boot time when networks go missing.
Thanks to Zachary Crownover, rcreload is available in DragonFly. (It’s always good to see a new contributor name.)
Nuno Antunes brought in a significant number of fixes for libradius. He’s been doing other work recently on netgraph7 support, so I’m linking to this as a ‘signpost’ commit.
While Matthew Dillon was testing the new up-to-256-processor support for DragonFly, he added a few sysctls, one of which helps qemu performance when emulating a lot of processors. I note it here in case it’s helpful to someone else.
DRM (Direct Rendering, not Digital Rights) on DragonFly will normally eat all the memory it thinks it needs. However, vm.dma_reserved can now be set to a fixed limit in /boot/loader.conf. By default, vm.dma_reserved on DragonFly is set to 16M, and can be set higher. I think this is necessary when running higher-resolution screens… Don’t quote me on that, though.
Matthew Dillon changed the default keep-policy in DragonFly to:
set keep-policy keep state (pickups, sloppy)
This is to match other BSDs (which? I don’t know) and reduce overhead, according to the commit.
The obvious joke should be “how can you tell?” Anyway, the csprng in DragonFly has been updated and IBAA is being used more often, and there’s more updates on the way.
ATM support is gone in DragonFly, and frankly, I’m surprised it was still there.
DragonFly now has a ‘rescue’ system added in, which also functions as a way to mount encrypted filesystems. Does PAM work yet? I don’t know; I may be linking to this earlier than I need to.
A recent commit from Sepherosa Ziehau has a 5% improvement in the number of network connections per second a x86_64 machine can accept. He’s also reducing the number of IPIs during network activity. If this seems somewhat esoteric, it’s because network speeds are getting so fast that the benefits come from reducing the accompanying CPU load.
If you’re on DragonFly 3.7, you will need to build world before building the kernel again if you are updating to some point in the last 24 hours. Sascha Wildner points out the related commit.
Sascha Wildner brought in ACPICA 20140214, and his commit message has a list of the updates.
There seems to be a lot of ACPI-related updates lately: Sascha Wildner has updated ACPICA in DragonFly to what I think is the very latest version. See his commit for the differences.
John Marino updated daemon(8) on DragonFly. For some reason, I didn’t know it was a standalone program. I knew about the idea of daemons as helpers based inside the computer, which is why so many server programs end with a ‘d’ – sshd, ftpd, and so on. Inexplicably, I never actually saw the program itself.
Markus Pfieffer has committed Larisa Grigore’s Google Summer of Code work, “SysV IPC in userspace”. It’s been a bit since the event finished, but it’s in DragonFly now.
Matthew Dillon wrote a roundup post summarizing all the changes he’s made to DragonFly to improve SMP performance in the last few weeks. He’s removed almost all contention from DragonFly. This means better performance, scaling upward depending on the number of processors.
‘monster’, the system that builds all 20,000 items in dports, can complete the run in 15 hours. Compare this to the 2 weeks it used to take me to build the 12,000 packages in pkgsrc. This is admittedly on different hardware and different packaging systems, but it gives a sense of the scale of the improvement.
The ‘poweroff’ command, the equivalent of ‘halt -p’, has been added based on a suggestion from Robin Hahling.
Matthew Dillon was using poudriere, the dports build tool, on a 48-core system. Poudriere was building all 20,000+ dports, so the machine was quite busy. He decided to get rid of as much contention as possible, and he’s listed all the ways DragonFly’s been streamlined by these efforts. We need to revisit some of our previous benchmarks…
John Marino has accomplished the major task of updating gdb/kgdb, to version 7.6.1 for DragonFly.
John Marino has put in a large patch to DragonFly 3.5, updating all sorts of language-related items. As he warns, you will need a full buildworld/buildkernel in a specific order to update. On the plus side, you can now probably use your native language for nvi and for git.
Matthew Dillon’s committed the work by Daniel Flores on Hammer 2 compression and Mihai Carabas’s vkernel hardware support – both Summer of Code projects. There’s a good amount of detail in the commit messages describing the work and what it changed; I expect more Summer of Code work to be getting committed…
Note: you’ll want to do a full update.
Antonio Huete Jimenez has committed his work on “dirfs”, a filesystem that lets you mount directories from your host machine within the running vkernel environment. It’s a sort of shared folders for vkernels. See the commit message for usage details.
killall -T will now kill all processes associated with the current tty, except parents of the killall process itself. It’s a shortcut to “kill all these runaway items I started by accident”.