The more eagle-eyed may have noticed a branching for DragonFly 4.2, and for DragonFly 4.0.6. The 4.2 branch is currently only a release candidate, so don’t necessarily change over yet – it’s for testing, not release.
Note that packages for 4.2 are not yet built, so you’ll have to manually specify a package path to install with pkg on 4.2 – for now.. That won’t be the case for the actual release, of course. DragonFly 4.3 users will have to specify PKG_PATH manually to use 4.2 images until new ones are built. 4.2 release candidate users will be fine. (see comments for correction.)
The 4.0.6 release is mostly to get the recent OpenSSL update into a 4.0.x build.
I am working on image building for both.
Even sysctl accesses can be made to handle multiprocessor environments. This can actually make a difference when you’ve got a lot of processors building a lot of software, as in all of dports.
From IRC today:
“19:43 <@dillon> I’m bored at the moment. lemme try to speed up module builds“
Buildkernel, depending on your processor count, now may have tripled in speed.
Those changes I mentioned yesterday for text console support? They’re in DragonFly-master now, along with a loader tunable to turn it on and off.
If you’re using nginx on DragonFly, version 1.9.1 has specific DragonFly speedup options built in.
Hammer 2 now uses LZ4 compression by default. It also uses a new CRC algorithm that performs much better, and there’s numbers to prove it. It helps iSCSI too. When I say new, it appears to be from the 1980s? I may be looking at the wrong place.
Matthew Dillon has been doing a lot of Hammer 2 work lately. Well, he’s been doing it for quite some time, but the recent commits contain the sort of things that are easier to link to, like deletion speedups, freemap changes, and stats tracking/compression results.
Hammer will perform daily housekeeping tasks each night. If you’re up late enough, it may kick off while you are working. If you want to stop the process after it’s already started (since it’s disk-intensive), John Marino has added the ‘abort-cleanup‘ command.
Experimental automatic crypting of swap is now available in DragonFly-master. Recently added, though it may have been possible another way.
Sepherosa Ziehau has introduced a new sysctl:
Set this to zero and you won’t get endless ARP events from networks you aren’t on. For example, I’m hooked up to a cable modem. I only get a public routable IP address, but the network used for the cable modem network itself bleeds ARP packets out where my DragonFly machine can see it. Since it’s on a different network segment than the address I receive through DHCP, it always fails and the system logs it. For example:
May 11 05:20:52 www kernel: arplookup 100.68.112.145 failed: host is not on local network
I can’t do much about it since that layer 2 leakiness is going to happen, but I can shut it up with this sysctl – and thank goodness, cause I’ve been seeing these messages since first using a DOCSIS modem in… 2001 or so?
Tomohiro Kusumi has been quietly making a lot of commits to Hammer. I haven’t been linking them because they don’t necessarily equate to new features, but here’s an recent exception: the -A argument will make your Hammer command run on every PFS. It only affects reblocking/rebalancing – for now.
You can now export Hammer slave volumes as NFS mounts – but since slave volumes are updated from master, you’re mounting a snapshot of that point in time. That may actually be an advantage.
DragonFly builds two compilers by default. If you weren’t interesting in building both, there were switches to build only the default, like NO_GCC47. This changed with every compiler update.
With the switch to GCC 5, the new switch is “NO_ALTCOMPILER”. That will last through compiler changes. I’m mentioning this now because sooner or later, you’ll want to gain back some time on a buildworld.
The most recent (well, this is the mostest recent) update of ACPICA for DragonFly, by Sascha Wildner, is different from the usual import: it happens to include actual upstream support for DragonFly
I haven’t been drawing enough attention to it, but there’s been a bunch of HAMMER filesystem activity lately: First, Tomohiro Kusumi has been working on HAMMER – these posts are a small subset of his commits. Second, Matthew Dillon has been working full steam ahead on HAMMER2. The HAMMER2 design document has been updated (read this!), and he’s already accomplished master->slave disk syncing.
It’s not ready for production, of course, which you may already realize, so don’t install it unless you want to work on the code.
Do you use info pages? Yeah, me neither. John Marino’s removed or converted the various info pages already in DragonFly, and removed texinfo since it’s no longer required. (I’ve linked to a few examples, but there were a number more commits than this.)
The default .cshrc in DragonFly has had some changes, which shouldn’t be anything but handy… assuming you are using tcsh. Also: the loader menu defaults to a blue Fred, now.
In the process of committing binutils 2.25 to DragonFly, John Marino also broke its build into parallel parts and removed the build of the gold linker. Buildworlds should be noticeably faster now, though I don’t have a before/after.
Bill Yuan’s work on a new ipfw has been committed, and for clarity, called “ipfw3“.
Matthew Dillon pulled in a new USB update from FreeBSD to DragonFly. What does it change? I’m not completely sure, but he did it to get apcupsd working, so that may be a hint.
DragonFly 4.0.3 has been tagged; you can look at the tagging message for details, but the major reason for doing so is to include OpenSSL-1.0.1l. I will have images up soon.
DragonFly no longer has SCTP. Nobody minds, I think – I had to look up what it is.
The CAM layer in DragonFly has had its big lock removed/been marked MPSAFE, so you will notice a performance increase when using multiple disks. (assuming you aren’t throughput-limited, of course.)
Sepherosa Ziehau has posted a note that V4-mapped addressing is no longer supported in DragonFly. You will need to do a full buildworld/buildkernel if you are running master. Also, TCP MTU path discovery is on by default. Also also, he’s added a SOL_SOCKET/SO_CPUINT socket option for use to reduce load in heavy network activity. As usual, I don’t quite comprehend.
It’s possible, if you are several releases (years) behind, to end up with a DragonFly system that can’t compile and install the current release, due to incremental changes over time. It’s rare, but it could happen now between, say, version 3.4 and 4.0. The usual solution would be to incrementally upgrade in order, which is a lot of building and updating. The alternative is the new installworld-force option from Matthew Dillon that forces a new set of binaries into place. Use as a last resort.
The DragonFly boot menu has been cleaned up a bit, and Fred, the dragonfly drawn on DragonFly, is now in better color. In fact, there’s even an option to turn him blue.
Update: I wanted to see what this looked like, and I realized screenshots might help everyone else.
In an effort to reduce my backlog of DragonFly things to post about, here’s quick notes:
I’m running behind so this is a bit old, but: Matthew Dillon commited svc(8), a service manager program. Take a look at its man page to see the potential uses.
With a recent commit from Sascha Wildner, DragonFly now loads XHCI (meaning USB3) by default. If you had previously tried to install DragonFly via USB stick, and it inexplicably refused to mou t the installer drive… It may work much better now.
I’m working on the 4.0 release, but in the meantime, I wanted to point at a slew of updates from Sascha Wildner: ACPICA 20141107, pciconf(8), axge(4), and the kernel part of the netgraph7 Bluetooth stack.
Matthew Dillon’s added the reapctl() call, which gives a process control over all sub-processes, even when detached. This is initally useful for bulk builds of dports, but can probably be extended farther…
Chrome runs on DragonFly now, apparently possible now because of this ported fix from Joris Giovannangeli.
As Francois Tigeot has pointed out, recent Mesa upgrades have made very old graphics drivers using DRM1 no longer work. They’ve been removed. This won’t affect you unless your graphics card is 10+ years old.
Matthew Dillon has added /dev/upmap and /dev/kpmap to DragonFly in two commits. (er, three.) It’s an optimization of some sort, and it is unfortunately over my head. Please, someone comment in a way that fills it in. I’m tired.
John Marino updated wpa_supplicant (in dports). He then suggested moving it out of base into dports, so that it could be updated independently of the base system. (this update, for instance, took years.) Since wpa_supplicant is necessary to get some systems online – and it can’t be installed if missing if you don’t have a network link – it may be too risky. I think other packages could be moved out, myself.
Robin Hahling volunteered to update OpenSSH in DragonFly, which is good news. It’s a jump from version 6.1 to 6.7, so there’s some feature changes. tcpwrappers support is gone, for instance. If you have a reason to object to this change, speak up now.
John Marino has upgraded gcc, libedit, xz, and grep in DragonFly. Also, tzdata has jumped from version 2014e to 2014h, thanks to Sascha Wildner.
Sepherosa Ziehau’s recent changes to UDP in DragonFly mention some performance gains to sheer packet count.
Markus Pfeiffer has imported FreeBSD’s if_lagg to DragonFly. It’s for talking LACP over multiple network ports, so that the traffic from those multiple ports can be aggregated – if what’s on the other end generally understands LACP. (Failover mode may not count.) Please test if you have that sort of surfeit of network ports.
You can now see the packets, bytes, and drops in altq by using the -altq arg to systat, thanks to this recent commit from Matthew Dillon.
This very long commit message from Sepherosa Ziehau details the UDP changes he’s made. It’s mostly technical details, but at the end he mentions this little tidbit:
“For ‘kq_connect_client -u’ test, this commit gives 400% performance improvement (31Kconns/s -> 160Kconns/s).”
You can now start moused with an argument, so it will look at the right device. In most cases, I imagine “
/etc/rc.d/moused start ums0” will be what anyone wants. Credit to Michael Neumann for the update. Perhaps
moused_flags="ums0" will do it too? I haven’t tried yet.
This will overwrite your /etc/devd.conf.
Because of some structure changes made by Matthew Dillon while chasing a pf bug, you will need to do a full buildworld/buildkernel on your next update – if you are running DragonFly-master. 3.8 users are unaffected by the bug or the change.
DragonFly’s dhclient will now retry failed interfaces and handle being re-run gracefully. This is a blessing for anyone who has had a flaky link. Matthew Dillon’s made two other improvements for booting that will also improve boot time when networks go missing.
Thanks to Zachary Crownover, rcreload is available in DragonFly. (It’s always good to see a new contributor name.)
Nuno Antunes brought in a significant number of fixes for libradius. He’s been doing other work recently on netgraph7 support, so I’m linking to this as a ‘signpost’ commit.
While Matthew Dillon was testing the new up-to-256-processor support for DragonFly, he added a few sysctls, one of which helps qemu performance when emulating a lot of processors. I note it here in case it’s helpful to someone else.
DRM (Direct Rendering, not Digital Rights) on DragonFly will normally eat all the memory it thinks it needs. However, vm.dma_reserved can now be set to a fixed limit in /boot/loader.conf. By default, vm.dma_reserved on DragonFly is set to 16M, and can be set higher. I think this is necessary when running higher-resolution screens… Don’t quote me on that, though.
Alex Hornung has added a ChaCha algorithms and Fortuna-based CSPRNG to DragonFly’s random device. You can pick what runs with the sysctl kern.rand_mode, and some other changes.
Matthew Dillon changed the default keep-policy in DragonFly to:
set keep-policy keep state (pickups, sloppy)
This is to match other BSDs (which? I don’t know) and reduce overhead, according to the commit.
Predrag Punosevac noticed that turning on pf was slowing his machine down. Rearranging the rules fixed a lot of it for him. However, Matthew Dillon decided it was time to make pf work concurrently instead of in a single thread, and 24 hours later, it does. Quick, someone benchmark this!
Sepherosa Ziehau implemented a UDP echo response tool, which not surprisingly meant he also had some UDP performance improvements. As he points out in the commit, it makes lockless firewall state tables possible.
The obvious joke should be “how can you tell?” Anyway, the csprng in DragonFly has been updated and IBAA is being used more often, and there’s more updates on the way.
Sepherosa Ziehau has enabled GSI target CPU auto selection, by default, on x86_64. He says to let him know if there’s problems. I’m not sure what form the problems would take, cause I’m not sure what this does.
libpcap has been updated in DragonFly by Matthew Dillon, and file has been updated by Peter Avalos.
ATM support is gone in DragonFly, and frankly, I’m surprised it was still there.
Sascha Wildner’s updated ACPICA to version 20140424. Will that help you? Perhaps with newer motherboards; otherwise check the changelog.
Here’s the announcement from Francois Tigeot: DragonFly now uses dynamic binaries in the root filesystem. You will need to do a full buildworld/buildkernel if on 3.7 and upgrading.
DragonFly now has a ‘rescue’ system added in, which also functions as a way to mount encrypted filesystems. Does PAM work yet? I don’t know; I may be linking to this earlier than I need to.
Alex Hornung has updated tcplay in DragonFly to 2.0, and cryptdisks is updated to match. If you have a short memory, tcplay(8) is the tool on DragonFly to manage TrueCrypt volumes. Is DragonFly the only BSD to have this? I think so, based on very few seconds of googling.
A recent commit from Sepherosa Ziehau has a 5% improvement in the number of network connections per second a x86_64 machine can accept. He’s also reducing the number of IPIs during network activity. If this seems somewhat esoteric, it’s because network speeds are getting so fast that the benefits come from reducing the accompanying CPU load.
Sascha Wildner updated the time zone database on DragonFly to tzdata2014a. The odd thing isn’t that update – Sascha updates like clockwork, haha! – but the release notes. Apparently Even Microsoft is starting to support time zone names, sorta, finally.
If you’re on DragonFly 3.7, you will need to build world before building the kernel again if you are updating to some point in the last 24 hours. Sascha Wildner points out the related commit.
Sascha Wildner brought in ACPICA 20140214, and his commit message has a list of the updates.
See the announcement, and the commit. I’m not totally sure what this affects.
There seems to be a lot of ACPI-related updates lately: Sascha Wildner has updated ACPICA in DragonFly to what I think is the very latest version. See his commit for the differences.
John Marino updated daemon(8) on DragonFly. For some reason, I didn’t know it was a standalone program. I knew about the idea of daemons as helpers based inside the computer, which is why so many server programs end with a ‘d’ – sshd, ftpd, and so on. Inexplicably, I never actually saw the program itself.
Sepherosa Ziehau is continually trying to squeeze more network performance into DragonFly. I’m not always so good at pointing it out, but here’s several commits from him that improve performance on several chipsets.
Franco Fitchner is bringing in netmap to DragonFly. I don’t think it’s complete yet.
Markus Pfieffer has committed Larisa Grigore’s Google Summer of Code work, “SysV IPC in userspace”. It’s been a bit since the event finished, but it’s in DragonFly now.
Franco Fitchner has updated mdocml in DragonFly to 1.12.3. The changelog is right on the front page of the vendor site.
Update: Undeadly has a nice summary of the changes.
Matthew Dillon wrote a roundup post summarizing all the changes he’s made to DragonFly to improve SMP performance in the last few weeks. He’s removed almost all contention from DragonFly. This means better performance, scaling upward depending on the number of processors.
‘monster’, the system that builds all 20,000 items in dports, can complete the run in 15 hours. Compare this to the 2 weeks it used to take me to build the 12,000 packages in pkgsrc. This is admittedly on different hardware and different packaging systems, but it gives a sense of the scale of the improvement.
The ‘poweroff’ command, the equivalent of ‘halt -p’, has been added based on a suggestion from Robin Hahling.
Matthew Dillon was using poudriere, the dports build tool, on a 48-core system. Poudriere was building all 20,000+ dports, so the machine was quite busy. He decided to get rid of as much contention as possible, and he’s listed all the ways DragonFly’s been streamlined by these efforts. We need to revisit some of our previous benchmarks…
If you’re watching for a certain event, dmesg(8) on DragonFly now has a -f flag that will display new output as it’s logged, sort of like ‘tail -f’.
While looking for a different bug, Matthew Dillon made some changes in the way memory usage is totaled in DragonFly. You’ll see this most when using ‘systat -vm 1′ or ‘vmstat’, probably.
John Marino has accomplished the major task of updating gdb/kgdb, to version 7.6.1 for DragonFly.
John Marino has put in a large patch to DragonFly 3.5, updating all sorts of language-related items. As he warns, you will need a full buildworld/buildkernel in a specific order to update. On the plus side, you can now probably use your native language for nvi and for git.
Matthew Dillon’s committed the work by Daniel Flores on Hammer 2 compression and Mihai Carabas’s vkernel hardware support – both Summer of Code projects. There’s a good amount of detail in the commit messages describing the work and what it changed; I expect more Summer of Code work to be getting committed…
Note: you’ll want to do a full update.
It’s now possible to use systat(1) to see per-connection speeds and pftop status, thanks to Matthew Dillon.