Category: Hammer

More Hammer documentation


Thomas Nikolajsen has put together more information on Hammer, including formatting and the new deduplication features, conveniently located in the man pages and some other spots.

Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

Double buffering in Hammer usually useful


Enabling the vfs.hammer.double_buffer=1 sysctl will greatly improve Hammer performance when you’ve exceeded your memory cache (at a possible slight penalty when you have not) and also speed things up when using live deduplication.

Update: Venkatesh Srinivas says:

“double_buffer makes sense when: 1) you want all CRCs to be checked on reads. 2) you’re running live dedup and care about dedup performance rather than say read-heavy performance; 3) you have swapcache but are often running into the  vnode limit in what you can cache.”

So, not always useful.

Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

Hammer speed improvements


Yeah, so those Phoronix benchmarks are crap, but Matthew Dillon went and implemented some things that would speed up Hammer write performance in any case.  Read his summary for details.

Posted by     Categories: DragonFly, Hammer     1 Comment

New Hammer version


The default Hammer version in DragonFly is now version 5, which is the one that includes deduplication.  Enjoy, bleeding-edge users!  Otherwise, wait for the next release.

Version 6 is there, but don’t upgrade to it yet; there aren’t significant user-visible changes, and the usual disclaimers for new versions apply.

Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

Phoronix benchmarks for Hammer


A Phoronix test of DragonFly’s Hammer filesystem turned  up, via Siju George.  It’s not really a benchmark as much as it is a speed test, and it’s not a realistic comparison, but it’s interesting to see numbers.

They need a graph that shows how much historical data can be recovered by each file system, or how long fsck takes after a crash.

Update: Matthew Dillon points out the many ways these tests are wrong.

Posted by     Categories: Goings-on, Hammer     1 Comment

Deduplication arrives


Ilya Dryomov’s work on deduplication for Hammer has been committed to the tree in an early test form.  I guess I need to pay up as part of the code bounty.  If you’re wondering how much space it will save, but don’t want to try non-production code yet, there’s a ‘hammer dedup-simulate’ command that will estimate the saving ratio.

This is great news – deduplication is so valuable it adds an extra zero onto the price of any storage device that can do it.

Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

Encrypted HAMMER volumes possible


I haven’t covered this enough: thanks to Alex Hornung, it’s possible to create a HAMMER volume and have it be encrypted.  Matthias Schmidt has done just this, and has provided an rconfig(8) script to automate the process.  (Or to crib from if you prefer to do it by hand.)

Posted by     Categories: DragonFly, Goings-on, Hammer     2 Comments

Lazy reading: the return of ACID, SSI, weirdness


    A smaller set of links, but still the same volume of reading material.

    Posted by     Categories: Hammer, Lazy Reading     0 Comments

    Updates and improvements for HAMMER, crypto


    Matthew Dillon posted a summary of recent bugfixes in HAMMER and kqueue, which means if you are running a version of bleeding edge DragonFly build in the last few weeks, you should update.

    He also mentions a “significant improvement in performance” in disk encryption.  How significant?  Over three times as fast.

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    New HAMMER catastrophic recovery tool


    Matthew Dillon reports that DragonFly now has a catastrophic recovery tool for HAMMER filesystems, with pertinent details.

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    Summary of recent kernel work


    Matthew Dillon has provided some details about recent kernel work, along with a release forecast.

    Posted by     Categories: Committed Code, DragonFly, Goings-on, Hammer     0 Comments

    What of OpenSolaris?


    You have probably seen reports declaring the demise of OpenSolaris by now, many taking a less than conservative approach in reporting the news one way or the other. So what do you make of the news? By all accounts, the source code (including future changes) for things such as ZFS will continue to be published under the CDDL. Will Oracle closing up development make it impossible for operating systems like FreeBSD to maintain ZFS without forking it? What do you think the ramifications will be for DragonFly’s HAMMER and DragonFly in general?

    Posted by     Categories: DragonFly, Goings-on, Hammer, Off-Topic, UNIXish     5 Comments

    Dedup, please


    Matthew Dillon made a minor change to HAMMER that would help any future deduplication work.  There’s also a deduplication code bounty out on the recently-updated Code Bounties page…

    I’ve been NAS-shopping, and I’ve found that deduplication ability seems to add an extra zero on the end of a device’s price tag.  It would be very nice for HAMMER.

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    Messylaneous: reading, catchup


    I apologize; I’ve been missing.  Here’s some misc links while I get back in gear:

    • A very good reason to be interested in Hammer over ZFS: nobody will threaten lawsuits over Hammer.
    • 10 tricks for admins.  I’m posting it cause I can never remember that thing with tunneling ssh out.  (via)
    • This Gaming Life, as a free download.  An excellent book that is in physical form on my shelf right now.  Yes, unrelated.

    Rolling everything back


    If you have a Hammer filesystem, and you want to roll the entire thing back to a previous snapshot – all files, everywhere – it can be accomplished with one command.

    Posted by     Categories: DragonFly, Goings-on, Hammer     0 Comments

    Keeping a mirror-stream going


    A note, in part for my own benefit: the @reboot crontab entry is all you need to get a HAMMER mirror-stream going again after a reboot/shutdown.

    Posted by     Categories: DragonFly, Hammer     1 Comment

    More on Hammer and Samba


    Matthew Dillon went into detail on just how Hammer snapshots could be shared out via Samba.

    Posted by     Categories: DragonFly, Goings-on, Hammer     0 Comments

    Hammer via Windows


    Siju George is making a Hammer volume’s snapshots available through Samba, with the results that some Windows-using developers get historical snapshots for free.

    Posted by     Categories: DragonFly, Hammer     2 Comments

    2.6.2, 2.7.2 created; please update


    Matthew Dillon identified a possible data corruption bug in Hammer with a nearly-full filesystem.  It’s dramatic enough he’s tagged 2.6.2 and 2.7.2 so that people can update; his message about it describes how to check for corruption.

    Posted by     Categories: DragonFly, Hammer, Heads Up!     4 Comments

    Last-ditch ways to check your disk


    If you’re worried that your Hammer disk may be going bad – and I mean bad like physically bad – you can check it with dd, or see what the hammer tool lists as bad.

    Posted by     Categories: DragonFly, Hammer     0 Comments

    Even more PostgreSQL benchmarking


    Jan Lentfer’s done some new benchmarking of PostgreSQL on Hammer.  There’s further suggestions and a more complete benchmark is planned, taking advantage of the Hammer improvements in 2.6.  In the meantime, you can look at previous benchmarks.

    Posted by     Categories: DragonFly, Goings-on, Hammer     0 Comments

    Hammer and OS X


    Daniel Lorch has ported Hammer to Mac OS X, of all things.  It’s not complete, but he’s moving right along.

    Posted by     Categories: DragonFly, Hammer     2 Comments

    New location for Hammer on Linux


    Daniel Lorch’s work on porting Hammer to Linux (read-only, currently) has been moved to a new location.

    Posted by     Categories: DragonFly, Goings-on, Hammer     0 Comments

    REDOs done, and 64-bit vkernels too


    Matthew Dillon has implemented what he calls “REDO” records in Hammer, which reduce the amount of time taken flushing data to disk.  It’ll be in the 2.6 release, but it isn’t on by default.

    Jordan Gordeev’s work on 64-bit vkernels has also been brought in, so virtual systems are now available for x86_64 users.

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    32 to 64-bit Hammer mirroring fixed


    Michael Neumann has fixed the ability to stream Hammer data between 32 and 64 bit systems.  However, this is a change to 64-bit systems that requires them to match; make sure that you are not mixing 64-bit systems built before and after this commit on the 21st.

    I can’t find the commit message in the mail archive, so I’ll quote it here:

    More…

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    HAMMER config details


    Pulled from a larger conversation: a description of the settings for a HAMMER filesystem, and what they mean.  I can tell from experience that extremely active disks will need extra cleanup time…

    Posted by     Categories: DragonFly, Hammer     0 Comments

    A week’s worth of posts for you


    I can’t keep up with all the things to post.  I desperately want to clear my inbox, so here’s a week’s worth of posts all smushed together.  Enjoy!

    Phew.

    New HAMMER presentation


    Michael Neumann presented a talk on HAMMER at the Karlsruher Institut
    für Technologie
    on January 27th.  His slides (in English) are now available in PDF or ODP formats, and are listed on the dragonflybsd.org Presentations page.

    Posted by     Categories: DragonFly, Goings-on, Hammer     2 Comments

    Watch out on the bleeding edge with UFS


    If you’re running DragonFly 2.5 and updated in the past week or so, and have UFS disks, there’s some instability introduced by Matthew Dillon’s recent work.  It ought to be better by next week.

    Users of Hammer, or of UFS only as /boot, don’t have anything to worry about.

    Posted by     Categories: DragonFly, Hammer, Heads Up!     0 Comments

    More REDO


    That didn’t take long: Matthew Dillon has an update on his REDO work; he’s about halfway there.  His summary includes instructions on how to test this new work, including ways to change how Hammer syncs to disk.

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    Hammer downgrade and upgrade bug


    Thomas Nikolajsen experienced firsthand a bug where downgrading a Hammer PFS master to a slave and then later making it a master again lost all data.  Lucky him…   The problem’s now fixed.

    Posted by     Categories: DragonFly, Hammer, Heads Up!     0 Comments

    Hammer disk removal


    Thanks to Michael Neumann, it’s now possible to remove a drive from a Hammer volume.  It’s experimental, so all the standard warnings apply.

    This can’t be done on a root volume, for hopefully obvious reasons.

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    Hammer expanding


    Did you know you a Hammer volume can span multiple disks?  And that you can add extra disks later on?  There’s no RAID-like features – it’s just a straight multiple-disk volume, but it works.  The Hammer command to do it is now “hammer volume-add

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    Hammer saves my bacon


    Some of the ikiwiki configuration files on dragonflybsd.org were accidentally overwritten during a software upgrade.  Normally this would mean some work to locate and replace them from backups, but since it was a Hammer volume, a quick look in /var/hammer/usr/… found them for me.

    I want to point out what Hammer does, here.  Restoring from backup isn’t new – it is in fact probably one of the most basic and necessary of system administration duties.  However, Hammer makes it so easy that the incremental work of using it falls to almost nothing.  There’s no extra preparation or syntax to learn for retrieval, which is wonderful.  Hammer’s easy fix has helped me out several times now, saving me time that, while probably still successful with any other backup system, would have been taken up just restoring things back to normal.

    Posted by     Categories: Goings-on, Hammer     2 Comments

    Hammer version 4 now available


    Matthew Dillon has made version 4 of Hammer the default; the upgrade is a relatively painless ‘hammer upgrade’ command.   This new version cuts out a chunk of the disk syncs needed, speeding up Hammer disk operations.

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    More links again


    I like linkblogging, especially because there’s been a lot of good stuff floating about:

    Posted by     Categories: BSD, DragonFly, Hammer, UNIXish     2 Comments

    Accessing Hammer config via NFS


    Thomas Nikolajsen came up with some ideas for making the configuration files for a given Hammer volume accessible, even when that volume is being presented over NFS.  He’s looking for more ideas.

    Posted by     Categories: DragonFly, Hammer     0 Comments

    Small undo fix


    If anyone wants a project, there’s apparently a small undo bug that I’ve encountered.  It is a small fix in terms of changes, for any takers.

    Posted by     Categories: DragonFly, Hammer     0 Comments

    Hammer version 4 status


    There’s a status report from Matthew Dillon about his work on version 4 of Hammer, including the always enjoyable stories of tests that involve yanking the SATA cable from the drive.

    Posted by     Categories: DragonFly, Hammer     0 Comments

    New Hammer work


    Matthew Dillon’s working on a 4th iteration of Hammer, this time to address speed issues, among other things.   There’s a bunch of commits to look at so far.

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    Easier Hammer manipulation


    ‘mike’ made this interesting csh script that allows autocompletion of Hammer sub-commands.  e.g. type ‘hammer’ and then cycle through the available hammer commands as you would through file names.

    Posted by     Categories: DragonFly, Hammer     0 Comments

    More Postgres benchmarking


    Jan Lentfer repeated his Postgres tests on DragonFly with some system changes suggested by Matthew Dillon, and noticed a speed increase.  (See previous report.)

    Posted by     Categories: DragonFly, Goings-on, Hammer     1 Comment

    Bug report reading


    This description of a Hammer bug makes for interesting reading, since it delves into the sequence of events where data is actually laid down on disk.  Interesting reading for a geek, admittedly…

    Posted by     Categories: DragonFly, Hammer     0 Comments

    Hammer version 3 in testing


    Version 3 of Hammer is now available in bleeding-edge DragonFly, though it’s still experimental.  The biggest reason for this version bump is to move the /snapshots folder to /var for all Hammer filesystems.  This means an accidental <tt>rm -rf</tt> won’t destroy snapshots, as I’ve done.  The saved data is still on the original partition, as just the metadata is saved to /var.  More explication is available.

    Posted by     Categories: Committed Code, DragonFly, Hammer     1 Comment

    Postgres benchmarks


    Jan Lentfer performed some Postgres benchmarks on DragonFly.  It’s elaborate enough that it’s in the form of a PDF attached to the message I’ve linked. There’s some additional variations that haven’t been tried yet.

    Vigorous file system activity seemed to lower performance in the long term on Hammer, which is certainly something to investigate.  More testing please!

    Posted by     Categories: DragonFly, Hammer     4 Comments

    Saved by Hammer


    A script I was running on avalon.dragonflybsd.org yesterday afternoon removed the packages, iso images, and snapshots stored there.   (Sorry!)  Hammer saved my bacon, with a snapshot of the 182G of missing files immediately available.

    Posted by     Categories: DragonFly, Hammer     0 Comments

    Hammer mirror header output, just in case


    If you back up the pseudo-file-systems (PFS) on your Hammer volume to a non-Hammer disk, and then need to restore them to a new Hammer volume, and then realize you forgot to write down the shared-uuid, well, then, YONETANI Tomokazu has a patch for you.  I haven’t seen this committed yet, but it appears valuable.

    Posted by     Categories: DragonFly, Hammer     0 Comments

    Hammer speedups


    Matthew Dillon’s made some changes to Hammer that make performance during mixed operations (reading and writing requests at the same time) much faster.  This should work for everyone, though AHCI/SILI/SCSI users will notice it more.  The new writing system is called ‘BIOQ‘.

    Hammer improvements, tests


    Matthew Dillon’s made some improvements to Hammer’s read and write processes.  To quantize this, he’s tested Hammer and UFS with blogbench and written up the results.   The tl;dr summary: UFS performs well until the system cache runs out, and then it halts.  Hammer has some overhead from saving all history, but doesn’t stop working under a much heavier load.

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    Messylaneous: books, lawsuits, git, more


    Dear universe, including DragonFly people: stop doing so much stuff.  It’s hard to keep up.

    Posted by     Categories: BSD, DragonFly, FreeBSD, Hammer     0 Comments

    Spreading Hammer


    Want to bring Hammer to Slackware Linux?  People want it, and there’s some work already in progress.

    Posted by     Categories: DragonFly, Goings-on, Hammer     0 Comments

    PFS and NFS now play nicely together


    It is now possible to mount a Hammer PFS via NFS, though you’ll want to use NFSv3.

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    Hammer gets bigger


    Hey, look at what Michael Neumann’s doing: making Hammer expandable!  It will be possible to expand your Hammer volumes while online, even.

    (note: it’s experimental; don’t be surprised if it destroys data.)

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    New Hammer option


    The hammer command now has an ‘info’ option, which gives a great deal of information on your Hammer drives, thanks to Antonio Huete Jimenez.  (Committed)

    Posted by     Categories: Committed Code, DragonFly, Hammer     2 Comments

    New Hammer version


    Matthew Dillon has a new version of Hammer, which speeds up listings from programs like ‘ls -la’ and ‘find’.  This is only in 2.3.1.x code right now, so don’t force an upgrade via hammer version-upgrade if you’re still on DragonFly 2.2. His post includes some benchmarks.

    On a side note: sili(4) tests look good.

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    Other ways to back up Hammer


    As Jim Chapman found out, dump only works for UFS, and not for Hammer.  Matthew Dillon outlined the different mirroring and snapshot methods that Hammer makes available.

    Posted by     Categories: DragonFly, Hammer     0 Comments

    Mirroring with Hammer


    Siju George described his efforts to set up a continuous, automatic backup system using Hammer, with some interesting results.  Matthew Dillon chimed in with some suggestions.

    Posted by     Categories: DragonFly, Goings-on, Hammer     0 Comments

    Minor Hammer changes


    Matthew Dillon’s made some small changes to Hammer; it should result in a small speedup when copying data.

    Posted by     Categories: DragonFly, Hammer     0 Comments

    Better speed for cleanup


    A number of people have noticed that Hammer’s pruning (which by default runs once at midnight) makes systems temporarily unresponsive.   Matthew Dillon’s committed a fix for this, with warnings of more improvements to come.

    Posted by     Categories: Committed Code, DragonFly, Hammer     0 Comments

    Bulk build speed stats


    I recently did a bulk build of pkgsrc on two similar machines; the only significant difference being extra CPU work being done on one system, and Hammer snapshots on the other.  However, they’re diverging in speed over time, which is interesting but not yet conclusive.  Read my post about it for more details.

    A good benchmarking project would be testing Hammer with snapshots on and with snapshots off.

    Posted by     Categories: DragonFly, Hammer, pkgsrc     0 Comments

    Hammer presentation at BSDDay


    Sdävtaker is giving a “Hammer administration” presentation at BSDDay, May 29th and 30th in Argentina.  (His presentation is the second day.)

    Posted by     Categories: Conventions, DragonFly, Hammer     0 Comments

    Hammer porting notes


    Daniel Lorch, the student working on a port of Hammer to Linux, has a blog, with some notes on progress.  I found this April item entertaining.

    Posted by     Categories: DragonFly, Hammer     0 Comments

    lseek(2) extensions for Hammer?


    Pedro F. Giffuni suggested that the SEEK_HOLE and SEEK_DATA lseek extensions would be good additions to Hammer, and linked to a Sun paper that went into more detail.

    Posted by     Categories: DragonFly, Hammer     1 Comment

    Encryption ideas for backups


    Sdävtaker came up with a potential idea for encrypting backup files, and Matthew Dillon followed up with another way that uses Hammer.

    Posted by     Categories: DragonFly, Goings-on, Hammer     0 Comments

    Hammer for Linux, and others


    Daniel Lorch is working on a port of Hammer to Linux’s VFS, though since he’s using FUSE, it will be able to reach other systems, like NetBSD.  The code is accessible.

    Posted by     Categories: DragonFly, Hammer, NetBSD     0 Comments

    Lost any Hammer files?


    Matthew Dillon is trying to track down a Hammer bug where directory entries (files, usually) are missed, whether it’s with ls or find or similar.  Has this happened to you?  It’s apparently very hard to duplicate, so please speak up if it has.

    Posted by     Categories: DragonFly, Hammer     1 Comment

    DWIMmy undo


    Hammer’s ‘undo’ now has the ability to index and automatically diff historical versions of files for you, thanks to a patch from Joel K. Pettersson.  (He’s got more ideas, too.

    Posted by     Categories: Committed Code, DragonFly, Hammer     1 Comment

    Hammer for Linux?


    Simon ‘corecode’ Schubert has a student working on a Linux port of Hammer.  This will lead to a breakout of Hammer from the DragonFly tree, too.

    Posted by     Categories: DragonFly, Hammer     4 Comments

    New Hammer feature


    Matthew Dillon has added a “rebalance” feature to Hammer, which cleans up the underlying B-Tree structures in Hammer that might otherwise slow down searching. It’s considered experimental, so be careful with it for now.

    Posted by     Categories: Committed Code, DragonFly, Hammer     2 Comments

    Messylaneous, 03/12/2009


    A bunch of links, cause that’s the easiest way to get this all out:

    Posted by     Categories: DragonFly, Hammer     2 Comments

    Hammer this, Hammer that


    Hammer filesystem, meet Hammer OS.  Please don’t take this seriously.

    Posted by     Categories: DragonFly, Hammer     2 Comments