# Category: Hammer

## Checking HAMMER volumes for health

If you are looking to validate the data on your HAMMER volume, there’s several ways to do so, with one common-sense caveat.

## No atime in Hammer

Hammer now defaults to ‘noatime’, meaning the date and time of last access are not updated on every file action.  Note that creation and modification date and time are still recorded.  This will help with speed and disk activity.

This may cause a problem with any software expecting this to change – mutt, possibly?  We will find out.  This change was done after the 4.4 branch, so it’s not in the current release of DragonFly.

Posted by     Categories: Committed Code, DragonFly, Hammer

## Hammer2 mountall and info directives

Matthew Dillon has added two Hammer2 directives – ‘info’ and ‘mountall’.  See his commit message for a explanation of each.  This predates the 4.4 branch, so it’s available in the current release.  The usual caveat applies: Hammer2 is for development only; don’t use this to store data you want to use.

Posted by     Categories: DragonFly, Goings-on, Hammer

I have a huge backlog of things to post, so this is originating from the 17th: Matthew Dillon has been working for some time on hardlinks and Hammer 2.  Hardlinks are the same file, presented in multiple places.  This can be a problem when your filesystem keeps infinite, writable snapshots.  The solution he just commited is called ‘xlink’ and the commit message has details.

Posted by     Categories: Committed Code, DragonFly, Hammer

## No more dsched

The disk scheduler apparatus in DragonFly has been removed.  This may not affect you much, since alternate scheduling setups were never utilized much with it.  It may fix some rare Hammer cleanup issues, though, and you may need to adjust your custom kernel config, if you have one.

Posted by     Categories: Committed Code, DragonFly, Hammer

## HAMMER2 root mounts and live dedup too

HAMMER2 recently gained the ability to be used as the root mount for your DragonFly system.  Live deduplication of data is also now possible, which means fast copy operations, less space used, and no need to wait for an overnight batch process to do it.  If you want to try it, you need a bleeding edge DragonFly system and the WANT_HAMMER2 option.  It’s still not ready for production use, so don’t try it with any data you want to keep.

Posted by     Categories: DragonFly, Hammer

## Many Hammer 1 updates, and credit

I don’t note it enough, but Tomohiro Kusumi has been making constant updates to HAMMER, the version we have now.  Often they are the sort of update that makes the code more readable, or fixes possible problems, and so on.  Very essential, but hard to post about it.  In any case, I’m using his recent improvements to hammer volume-del to note his contributions, of which there are much more than the day’s worth I link here.

Posted by     Categories: Committed Code, DragonFly, Hammer

## More Hammer 2 improvements

Hammer 2 now uses LZ4 compression by default.  It also uses a new CRC algorithm that performs much better, and there’s numbers to prove it.  It helps iSCSI too.  When I say new, it appears to be from the 1980s?  I may be looking at the wrong place.

Posted by     Categories: Committed Code, DragonFly, Hammer

## Recent Hammer2 work

Matthew Dillon has been doing a lot of Hammer 2 work lately.  Well, he’s been doing it for quite some time, but the recent commits contain the sort of things that are easier to link to, like deletion speedups, freemap changes, and stats tracking/compression results.

Posted by     Categories: Committed Code, DragonFly, Hammer

Hammer will perform daily housekeeping tasks each night.  If you’re up late enough, it may kick off while you are working.  If you want to stop the process after it’s already started (since it’s disk-intensive), John Marino has added the ‘abort-cleanup‘ command.

Posted by     Categories: Committed Code, DragonFly, Hammer

DragonFly committer Joris Giovannangeli has a Google Summer of Code project.  He’s bringing Hammer2 to OpenBSD, in single-node form.  It’s a very difficult project, but Joris is a very talented worker.

Posted by     Categories: BSD, DragonFly, Google Summer of Code, Hammer, OpenBSD

## Reblock all the things!

Tomohiro Kusumi has been quietly making a lot of commits to Hammer.  I haven’t been linking them because they don’t necessarily equate to new features, but here’s an recent exception: the -A argument will make your Hammer command run on every PFS.  It only affects reblocking/rebalancing – for now.

Posted by     Categories: Committed Code, DragonFly, Hammer

## NFS and Hammer slaves

You can now export Hammer slave volumes as NFS mounts – but since slave volumes are updated from master, you’re mounting a snapshot of that point in time.  That may actually be an advantage.

Posted by     Categories: Committed Code, DragonFly, Hammer

## Hammer and Hammer2 details

I haven’t been drawing enough attention to it, but there’s been a bunch of HAMMER filesystem activity lately: First, Tomohiro Kusumi has been working on HAMMER – these posts are a small subset of his commits.  Second, Matthew Dillon has been working full steam ahead on HAMMER2.  The HAMMER2 design document has been updated (read this!), and he’s already accomplished master->slave disk syncing.

It’s not ready for production, of course, which you may already realize, so don’t install it unless you want to work on the code.

Posted by     Categories: Committed Code, DragonFly, Hammer

## BSDNow 083: woN DSB

This week’s BSDNow has the usual roundup of news, including some… suspicious items, plus an interview of Kamila Soucková about conferences and Google Summer of Code.  They note this Hammer2 proposal.

Posted by     Categories: BSD, Goings-on, Google Summer of Code, Hammer, Periodicals

## Clustering and copies in HAMMER2

Matthew Dillon answered some mailing list questions on how clustering and data copies will work in HAMMER2 – no due date, of course, because this is very complex.  If you’re really into it, there’s always watching the recent commits.

Posted by     Categories: DragonFly, Goings-on, Hammer

## Got Nagios? Got Hammer?

If you’re monitoring your DragonFly systems with Nagios, here’s a way to check the health of your Hammer mirror-streams.  Thanks, Mike!

Posted by     Categories: DragonFly, Hammer

## DragonFly catchup

Here’s a number of DragonFly links to clear out my backlog:

Posted by     Categories: DragonFly, Goings-on, Hammer

## ZFS and Hammer conversation

There’s a FreeBSD Forums thread about ZFS and Hammer, as several people have pointed out to me.  It’s interesting to see, but there isn’t a lot of quantitative discussion.  (It’s a forum post, not a white paper, though.)

Posted by     Categories: BSD, DragonFly, FreeBSD, Hammer

## Slider, for Hammer

John Marino has created something very useful: a graphical tool for Hammer file history.  It’s called ‘Slider’, and it uses curses to work in a terminal.  It shows historic versions of files and can restore those old versions as needed.  This was already possible in Hammer, of course, but it required a sequence of commands that were not straight-forward.  I’ve been slow enough posting it that version 2.0 is already out, offering a way to see files that no longer exist, but are still in history.  (i.e. deleted some time ago)  ‘Time Machine’ sounds like the best name, but that seems to be taken.

Posted by     Categories: DragonFly, Hammer, Someday you will need this

## A tip for Hammer disks and history

One way to keep file history on an very active Hammer disk from eating up all the space: more snapshots.  This may seem counterproductive, but disk pruning eliminates historical data between snapshots, so you can keep older data at the cost of some temporal accuracy.

Posted by     Categories: DragonFly, Hammer, Someday you will need this

## BSDNow 053: It’s HAMMER Time

Not the first time that pun has been deployed, of course, but: BSDNow 053 has a Hammer tutorial, along with an interview of Reyk Floeter about OpenBSD’s httpd, along with the usual recent news and some links to some things I haven’t seen, like pfSense tricks.

.

Posted by     Categories: BSD, Hammer, Periodicals

## Hammer and mirrored disks

A frequent question people ask when trying Hammer is “How can I do software RAID to cover a disk failure?”  Hammer provides for streaming one volume to another, so you can duplicate drives, but there isn’t an automatic failover mechanism as there is with a RAID setup.  The first answer is usually “get hardware RAID“; my preferred solution.   The remaining software solutions are vinum, ccd, and lvm for DragonFly.

Posted by     Categories: Device support, DragonFly, Hammer

## How to keep Hammer empty

A note for everyone: use Hammer default on a very busy filesystem, and you will eat a lot of disk space since all file changes are recorded.  (I’ve done this to myself a few times.)  Francois Tigeot has a list of tips on how to keep that from happening.

Posted by     Categories: DragonFly, Hammer, Someday you will need this

## Hammer and buffers

Are you running a Hammer filesystem on a low-memory system?  You may get some warnings.  It’s possible to tweak some settings to accommodate it, or just deal.

Posted by     Categories: DragonFly, Hammer, Someday you will need this

## Hammer in BSD Magazine

The May BSD Magazine is out, and Siju George has written an article about using Hammer on DragonFly.  It’s a free download to read.

Posted by     Categories: BSD, DragonFly, Hammer, Periodicals

## BSD Magazine for March

The March issue of BSD Magazine is out, and this month has an article written by Siju George about how his company is using DragonFly and Hammer for backups.

Posted by     Categories: BSD, DragonFly, Hammer, Periodicals

## Hammer 2 work and notes

Matthew Dillon committed the start of a Hammer 2 cluster API. I noticed, while looking at the commit, that there’s a design document, a freemap design document, a changes list, and – most important for anyone interested – a TODO list.

Posted by     Categories: DragonFly, Hammer

## Backing up Hammer to non-Hammer volumes

Hammer’s ability to stream to remote disks is great, but what if you have storage that uses some other file system?  Antonio Huete Jimenez put together a shell script that will dump out the contents of a Hammer PFS, for upload to whatever.  Read the README for the details.

Posted by     Categories: DragonFly, Hammer, Someday you will need this

## Remember: manually clean up offline Hammer volumes

If you have a Hammer volume that is offline, meaning that you don’t have the pseudo-file-systems null-mounted anywhere, it won’t get cleaned up in overnight processing.  You just have to manually specify it.

Posted by     Categories: DragonFly, Hammer, Someday you will need this

## Hammer2 status

This is a little old, but Matthew Dillon noted the status of his Hammer2 work a little while ago.  Some highlights: he’s intending Hammer2 to be usable on a single host by the time of the next DragonFly release (summer 2014), the Summer of Code project for compression has already been integrated, and he listed different parts of the work that may be interesting for anyone wanting to chip in.

Slightly related: Matt posted some Hammer2 comments on the DragonFly 3.6 release story on Slashdot that may be interesting.  Don’t bother reading the other comments; they’ll make your eyeballs bleed.

Posted by     Categories: DragonFly, Goings-on, Hammer

## Hammer2 boot support

If you want to boot from a Hammer 2 /boot volume, you now can.  Hammer 1 never worked well as /boot, though it was technically possible.  Hammer 2 will be just fine.

Note that you can’t turn on recently-added disk compression since the bootloader doesn’t understand it, and Hammer 2 is not ready for anything but being worked on.  Don’t try it unless you’re ready to be submitting code changes to fix Hammer2.

Posted by     Categories: DragonFly, Hammer

## Deduplication benefits, again

Remember my recent disk issues?  As a side effect of protecting myself, I have a good example of deduplication results.

I have a second disk in my server, with slave Hammer PFSs to match what’s on my main disk.  I hadn’t put them in fstab, so they weren’t getting mounted and updated.  I got them re-created, but they were nearly full.  Here’s an abbreviated df, from which you should be able to tell which drives I have :

Size   Used   Avail   Capacity
929G   729G   200G    78%    /slave/slavehome
929G   729G   200G    78%    /slave/slavevar
929G   729G   200G    78%    /slave/slaveusr
929G   729G   200G    78%    /slave/slaveslash

That 78% is how full the Hammer volume was.  I turned on Hammer deduplication, since it’s off by default.  The very next day:

Size   Used   Avail   Capacity
929G   612G   318G    66%    /slave/slavehome
929G   612G   318G    66%    /slave/slavevar
929G   612G   318G    66%    /slave/slaveusr
929G   612G   318G    66%    /slave/slaveslash

It’s a 1 terabyte disk, and I gained more than 10% back – That’s 100g of disk space that I gained overnight.  There might be more tomorrow, given that it was all of 5 minutes of dedup work.

This won’t surprise you if you’ve seen previous deduplication links here, like my previous results or some real-world tests.  It’s still great.  I’d suggest turning it on if you haven’t – hammer viconfig the appropriate PFS and uncomment the dedup line.

Posted by     Categories: DragonFly, Hammer

## Hammer and Go

Something new and odd: A port of the Hammer (1) filesystem into Go, for go-fuse.  As the author has said, it’s more for the practice of learning Go and Hammer than for producing anything useful.  Still, an interesting way to learn.

Posted by     Categories: DragonFly, Hammer

## Hammer 2 and encryption, briefly

Encryption seems to be the accidental theme tonight.  A question about Hammer 2 and encryption prompted this list of possible solutions from Matthew Dillon.  Hammer 2 is still months out, so these features both require time and someone interesting in doing them – though they sound quite possible.

Still not sure if I should be writing Hammer or HAMMER.

Posted by     Categories: DragonFly, Hammer

## Encrypting /home

If you were wanting to encrypt your /home directory, Pierre Abbat has written up the explicit steps he took to do that very thing.

Posted by     Categories: DragonFly, Hammer

## GSoC Hammer and compression performance numbers

Earlier this week, Daniel Flores posted the first-week report on his Google Summer of Code project, file compression in Hammer.  He mentioned that the LZ4 algorithm he is using seems to have the best performance with repeating text data, as in logfiles.  I asked for numbers, and he provided them.  The important data in the results is the total compression column.  It shows how many 64k blocks were able to be compressed, with that type of data.

Posted by     Categories: DragonFly, Google Summer of Code, Hammer

John Marino managed to update GCC from 4.7.2 to 4.7.3 (4.7 changelog), zlib from 1.2.7 to 1.2.8 (changelog), and awk from 20110810 to 20121220 (can’t find a changelog).

In other update news, Matt Dillon has been working on HAMMER2’s flush sequencing.

Update: tcsh too.

Posted by     Categories: DragonFly, Goings-on, Hammer

## HAMMER file system resizing

Posted by     Categories: DragonFly, Hammer, Someday you will need this

## Some Tux3 notes

If you’ve been feeling the need for reading about filesystems, Daniel Phillips has posted more notes about his Tux3 filesystem design, which can be contrasted with HAMMER.  (thanks, Venkatesh Srinivas)

Posted by     Categories: Goings-on, Hammer

## HAMMER2 update

Matthew Dillon has written up another update on his progress with HAMMER2.  (I need to be consistent in how I write that.)  He has disks being exported and mounted on other systems, and adds an explanation of some of the issues around creating reliable multi-master setups.  Before you get too excited, no, multi-master isn’t working yet, and this is not production ready.

Posted by     Categories: DragonFly, Goings-on, Hammer

## More Hammer2 work

Matthew Dillon’s committed a bunch more Hammer2 work.  No, it’s not usable yet.  Look at the commit messages for details on how he’s setting up multi-master volume information, though.

Posted by     Categories: Committed Code, DragonFly, Hammer

## More Hammer 2 in the tree

Matthew Dillon’s put more of his Hammer work into DragonFly, with notable parts being the creation of a ‘dmsg’ setup for advertising available block devices to share between machines using Hammer.  To anticipate your next question: No, it’s not something you can run right now as a test; this is the underlying framework.

Posted by     Categories: Committed Code, DragonFly, Hammer

The weather is finally turning cooler, which makes me happy.

• I don’t think I’ve seen this before: Very old UNIX releases, listed for running in emulation.  (via)
• Where the red-black tree name came from.  A red-black tree underpins Hammer 1’s data structures, though it does not in Hammer 2.  (also via)
• Someone with a HP passport login want to help this guy?  He just needs to reinstall Windows in IDE mode, or perhaps find the right sysctl to toggle.
• The acme editor, from Plan 9.  I didn’t realize it’s 20 years old.
• Speaking of editors, Replace in Multiple Files with Vim.  I haven’t seen the argdo command before, or the Vim Ninjas site.  Their color schemes article is useful just for the screenshots. (via)
• Adbuntu.  It’s not as bad or as inconsequential as most reactions would lead you to believe, but advertising within an OS seems heavy-handed.  The BSD model has been to use the operating system as a vehicle for selling hardware, and that’s been much more successful.  (see iOS, PC-BSD.)
• Where Did the Internet Come From?
• The map for Adventure.  (via)

Your unrelated link of the day: Victorian Sci-Fi.  It’s not just a reference list, it’s a link to a lot of the original material, since copyright no longer applies.

Posted by     Categories: BSD, Hammer, Lazy Reading, roguelike

## Do you have offline Hammer1 slave systems?

If you do, they don’t get cleaned up during the normal ‘hammer cleanup’ nightly routine.  Chris Turner has added a way to manually specify them as a cleanup target.

I’m pretty sure in this case ‘offline’ means ‘nothing streaming to it from a master disk’.  I think.

Posted by     Categories: Committed Code, DragonFly, Hammer

## Some new Hammer features: scoreboards, shells

These are small, but they make life easier: Hammer now has a scoreboard file, for viewing of mirror-streams running in the background.  There’s also a ssh-remote directive, so you can use ssh without enabling an interactive shell, and a HAMMER_RSH environment variable so different remote shells can be used.  These are all for Hammer 1.

Posted by     Categories: Committed Code, DragonFly, Hammer

There’s certainly no theme to this week’s links.  I even manage to avoid my usual git and vim links, strangely.

Your unrelated comic link of the week: Cul De Sac.   The strip is ending due to the creator’s health issues, but what he has done is marvelous.  This is one of the few newspaper strips that is both visually interesting and often abruptly laugh out loud funny, without being patronizing.

Posted by     Categories: Hammer, Lazy Reading, pkgsrc, roguelike, UNIXish

## Actually trying Hammer 2

If you wanted to try Hammer 2 and you have several DragonFly-current systems around (virtual or not), Matthew Dillon has the instructions.  Keep in mind that this is not something ready for use; it can’t actually free up space, for instance.  It’s neat that you can have multiple systems passing data back and forth already, though!

Posted by     Categories: DragonFly, Hammer

## Hammer 2 status report

Matthew Dillon recent posted a status report for Hammer 2.  Of interest is the spanning tree protocol being built to handle messages between Hammer volumes.  As he says in the message:

For example, we want to be able to have millions of diskless or cache-only clients be able to connect into a cluster and have it actually work…

(No, it doesn’t do this, yet.)

Posted by     Categories: Committed Code, DragonFly, Hammer

## Hammer 2 messages

Hammer 2 (or is it HAMMER2?) is nowhere near ready to test. But!  For laughs, I think it could be set up just so you can watch the messages go back and forth.  Someone want to set up a few DragonFly-current VMs and try?

Posted by     Categories: DragonFly, Hammer

## HAMMER2 and remote mounting

Reading this HAMMER2 commit carefully shows some future plans: remote cluster control, and the ability to mount nonlocal HAMMER2 volumes.  A reminder: those are future plans, not what you can do now.

Posted by     Categories: Committed Code, DragonFly, Hammer

## CCMS goes away, comes back

The cache coherency management code in DragonFly has been removed, but it’s coming back under HAMMER2, as part of how HAMMER2 maintains multiple master drives.

Posted by     Categories: DragonFly, Hammer

## More on HAMMER2 progress

As noted in a recent commit, it’s possible to set up a HAMMER2 /usr/obj and survive a buildworld.  That’s good progress.

Note that this is basic work, so features like multi-master and deduplication are not present yet, and it’s still work in progress, so don’t try HAMMER2 unless you like losing data.  Watch the branch for changes, though.

(I’m going with “HAMMER2” for the name.)

Posted by     Categories: DragonFly, Hammer

## Hammer 2 progress

Matthew Dillon’s recently added getaddr/setaddr support, dumping, and session encryption, among other things, to Hammer 2.  Or is it HAMMER2?  I’m not sure.

Posted by     Categories: DragonFly, Hammer

## Hammer2 messaging

If you’re curious about Hammer2 development, it’s been ongoing, but there haven’t been any more juicy commits to point at.  Here’s one – the start of the messaging system.

Posted by     Categories: DragonFly, Hammer

## Deleting too fast

Here’s an interesting side effect that came up in Hammer 2 development: deleting files can potentially require modification of only one parent element.  If I’m reading it right, that means deletion always takes about the same time, independent of the amount of data being deleted.  Your ‘rm -rf /largedrive’ could complete, removing multiple terabytes of data before you realize it.  I suppose it’s silly to complain about speedy results.  Of course, being Hammer, it would still be available in history.

Posted by     Categories: Committed Code, DragonFly, Hammer

## Hammer 2 branched

There’s a Hammer 2 branch in the DragonFly git repo now, for the next generation of DragonFly’s native file system.  Don’t get too excited; as Matthew Dillon explains, it won’t be operational for months, and features won’t get added until much later this year.  It’s neat to see the work happening, though, and there’s a new design document to show what’s coming.

Posted by     Categories: DragonFly, Goings-on, Hammer

## How low can you go? (with RAM and Hammer)

Matthew Dillon has a very detailed commit message with changes to make sure Hammer will run overnight cleanups in situations as low as 256M of RAM.  I think you can find that much RAM in breakfast cereal boxes these days.

Posted by     Categories: Committed Code, DragonFly, Hammer

## How much RAM is too little?

If you’re running DragonFly on a very low-end system, you may be wondering about memory requirements for Hammer.  Hammer is much less RAM-hungry than ZFS, so it looks like you can get away with 128M, as long as you don’t mind the occasional error message.  You can manually tweak settings for it if you like.  256M is plenty.

It still strikes me as odd to consider systems with less than 1G of RAM as “low-memory”.  What rich times we live in!

Posted by     Categories: DragonFly, Hammer, Someday you will need this

Antonio Huete Jimenez’s ‘libhammer‘, a library to make various Hammer functions available to userland programs, has been added.  It implements ‘hammer info’ only at this point, if I understand correctly.

Posted by     Categories: Committed Code, DragonFly, Hammer

## A new Hammer presentation

Francois Tigeot recently presented a set of slides about Hammer at a recent Irill conference.  PDFs of the slides are available at his site, in English and French.

Posted by     Categories: Conventions, DragonFly, Hammer

## Quotas almost sort of working

Francois Tigeot took an old Summer of Code proposal, VFS Quotas, and started running with it.  He’s made some progress, as he detailed in a recent post to kernel@ (with code!) , but the nullfs-mount nature of PFSs in Hammer are making it difficult.

Posted by     Categories: DragonFly, Goings-on, Hammer

## Deduplication now eats less RAM

Well, if you tell it to do so.  Matthew Dillon has added a user-settable limit to the amount of memory used during deduplication, so if your Hammer-using system is low on RAM, you can conserve.  This is probably most useful if you are running DragonFly in an extremely small VM, or if your name is Venkatesh.

(inside joke; Venkatesh has a crazy old desktop for DragonFly.)

Posted by     Categories: Committed Code, DragonFly, Hammer

## Pulse-width modulated time-domain multiplexer!

I really just like that phrase and the action movie feeling of using it, like “Watch out!  The pulse-width modulated time-domain multiplexer is targeting us!”  Sorta like a PU-36 space modulator.  It’s actually a recently-committed mechanism to improve write performance in Hammer, but my idea sounds more exciting.

Posted by     Categories: Committed Code, DragonFly, Hammer

## Deduplication real world results

I’ve posted about my own results with Hammer deduplication here before, but Siju George put together results from his workplace using actual files in production.  He recovered 138G from a 1T disk, and recovered 20% of space from another disk.  Not bad for something that’s nearly automatic, and completely free.

Posted by     Categories: DragonFly, Hammer

## Trying out deduplication

I moved to DragonFly 2.10 over the past few days, and I tried out deduplication, to see what kind of results I would get.  The procedure is outlined below.  I’m using /home here as an example, just to reduce the amount of text pasted in.

/pfs/@@-1:00004     966000640 566434576 399566064    59%    /home

Move my various Hammer pseudo-file systems to version 5, which supports deduplication.

# hammer version-upgrade /home 5

Issue a deduplication simulate command, to see what it guesses will be the savings:

# hammer dedup-simulate /home Dedup-simulate /home: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 4 Dedup-simulate /home succeeded Simulated dedup ratio = 1.22 

That ratio turned out to be pretty accurate for the actual deduplication.  I didn’t time it, unfortunately.  I don’t know if the time taken is proportional to the amount of deduplication or the total volume of data, though I suspect the latter.

# hammer dedup /home Dedup /home: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 4 Dedup /home succeeded Dedup ratio = 1.22 462 GB referenced 378 GB allocated 14 MB skipped 6869 CRC collisions 0 SHA collisions 0 bigblock underflows 

The end result?

/pfs/@@-1:00004     966000640 505887504 460113136    52%    /home

That data space is shared across all file systems, and it’s a 1TB disk, so it’s 7%, or 70GB. I was hoping for more, but I don’t have any obviously duplicated data (no local mail store, no on-disk backups), so perhaps this is normal. 70GB that I didn’t have before is no bad thing, though.

Incidentally, I was able to upgrade my installed software from pkgsrc-2009Q4 to pkgsrc-2011Q1 entirely using pkg_radd -u <pkgname>.  Remarkably quick and painless, though pkgin may have been able to do it even faster since it would pull from the same place.

Posted by     Categories: DragonFly, Google Code-In, Hammer, pkgsrc

## Hot-swap and a bad disk

If you follow this thread, it has some discussion on how to handle a multi-disk setup and Hammer.  If a disk is going bad, you can try mirroring, though you have to be careful how your pseudo-file systems are set up.

Posted by     Categories: DragonFly, Goings-on, Hammer

## More on Hammer design

I mentioned it before, but Matthew Dillon’s updated his Hammer document, and posted about it.  Read on, especially if you like extremely complex plans.

Edit: first link fixed, plus there’s a followup.

Posted by     Categories: DragonFly, Goings-on, Hammer

## Remember to enable deduplication

I didn’t think of this, but I needed it: if you have an older Hammer system that now can perform deduplication because you upgraded to DragonFly 2.10, make sure to add it to the configuration for that file system, or else it won’t run.

Posted by     Categories: DragonFly, Hammer, Someday you will need this

## SSDs and how to portion them

There’s been some discussion on buying a SSD and how to match it with a hard disk, and/or swapcache.  Follow the thread for more details.

Posted by     Categories: DragonFly, Goings-on, Hammer

## Hammer and the future

Matthew Dillon’s been thinking about Hammer, and how to implement clustering well enough to work as a sort of RAID replacement.  He’s written up a document describing his plans.  Some highlights:

• writable history snapshots
• quotas and accounting
• live rebuilds of data from mirrors
• and the same history, mirroring, and snapshots as before.

It’s going to be a while before this “Hammer 2” becomes a finished product, though, so don’t count on it for the next release.

Posted by     Categories: DragonFly, Google Code-In, Hammer

## RAM vs. deduplication

Tomas Bodzar asked about RAM usage with Hammer and deduplication, pointing at this example that shows ZFS requiring…  I’m not sure.  Lots?  Anyway, Matthew Dillon noted that offline deduplication in Hammer would use available RAM/swap for CRCs on all files, but only a limited subset for ‘live’ dedup.  For a real-world example, Venkatesh Srinivas described deduplicating about 600G down to 400G, with a machine having only 256M of RAM. Yes, only 256M.

Posted by     Categories: DragonFly, Goings-on, Hammer

## More Hammer documentation

Thomas Nikolajsen has put together more information on Hammer, including formatting and the new deduplication features, conveniently located in the man pages and some other spots.

Posted by     Categories: Committed Code, DragonFly, Hammer

## Double buffering in Hammer usually useful

Enabling the vfs.hammer.double_buffer=1 sysctl will greatly improve Hammer performance when you’ve exceeded your memory cache (at a possible slight penalty when you have not) and also speed things up when using live deduplication.

Update: Venkatesh Srinivas says:

“double_buffer makes sense when: 1) you want all CRCs to be checked on reads. 2) you’re running live dedup and care about dedup performance rather than say read-heavy performance; 3) you have swapcache but are often running into the  vnode limit in what you can cache.”

So, not always useful.

Posted by     Categories: Committed Code, DragonFly, Hammer

## Hammer speed improvements

Yeah, so those Phoronix benchmarks are crap, but Matthew Dillon went and implemented some things that would speed up Hammer write performance in any case.  Read his summary for details.

Posted by     Categories: DragonFly, Hammer

## New Hammer version

The default Hammer version in DragonFly is now version 5, which is the one that includes deduplication.  Enjoy, bleeding-edge users!  Otherwise, wait for the next release.

Version 6 is there, but don’t upgrade to it yet; there aren’t significant user-visible changes, and the usual disclaimers for new versions apply.

Posted by     Categories: Committed Code, DragonFly, Hammer

## Phoronix benchmarks for Hammer

A Phoronix test of DragonFly’s Hammer filesystem turned  up, via Siju George.  It’s not really a benchmark as much as it is a speed test, and it’s not a realistic comparison, but it’s interesting to see numbers.

They need a graph that shows how much historical data can be recovered by each file system, or how long fsck takes after a crash.

Update: Matthew Dillon points out the many ways these tests are wrong.

Posted by     Categories: Goings-on, Hammer

## Deduplication arrives

Ilya Dryomov’s work on deduplication for Hammer has been committed to the tree in an early test form.  I guess I need to pay up as part of the code bounty.  If you’re wondering how much space it will save, but don’t want to try non-production code yet, there’s a ‘hammer dedup-simulate’ command that will estimate the saving ratio.

This is great news – deduplication is so valuable it adds an extra zero onto the price of any storage device that can do it.

Posted by     Categories: Committed Code, DragonFly, Hammer

## Encrypted HAMMER volumes possible

I haven’t covered this enough: thanks to Alex Hornung, it’s possible to create a HAMMER volume and have it be encrypted.  Matthias Schmidt has done just this, and has provided an rconfig(8) script to automate the process.  (Or to crib from if you prefer to do it by hand.)

Posted by     Categories: DragonFly, Goings-on, Hammer

## Lazy reading: the return of ACID, SSI, weirdness

A smaller set of links, but still the same volume of reading material.

Posted by     Categories: Hammer, Lazy Reading