Joe “Floid” Kanowitz brought up the idea of RAM being used as disk cache, to which a number of replies were made. I mentioned softupdate snapshots as a possible mechanism, which Sander Vesik corrected to say that it would be more like a “write through” union mount of a MFS and real filesystem.
Kip Macy pointed out that ‘tmpfs‘, on Solaris and Linux does something similar. BSD also has mfs, which requires that you allocate memory ahead of time. (Most commonly used to speed up buildwords in /usr/obj/ .)
The concept of journaling file systems that save the state of a file at given chronological points was brought up, and several people noted ‘Elephant‘. (Link from Hiten Pandya.)
Matt Dillon also wrote up several paragraphs on using RAM for storage, which are sufficiently technical that I’ll paste them in the extended link for this entry, rather then sum up.
(Direct quote follows)
“Well, the existing buffer and path caches are not really well suited to
‘locking’ certain elements into ram. They really wants a ‘backing store’
to play with.But, that said, it will basically attempt to cache as much as it can in
memory. The two primary limitations which tends to cause cache flushes is
kern.maxvnodes, because in order to flush a vnode the underlying data
cached with that vnode will also be flushed even if memory is available
to hold it, and limitations with the buffer cache.I believe we could implement a general mechanism to turn off
delayed-writes, write-behind, and other elements on a per-filesystem
basis, and basically revert the associated data from dirty buffer cache
entries to dirty VM pages and then allow them to be paged out on demand
by the VM paging system. This removes the buffer cache limitations
and leaves us only with the vnode limitations.The MFS filesystem comes fairly close to doing this already since
its backing store is simply the memory associated with a process, but
it has the disadvantage of causing data to be stored in memory twice
(once in the VM / buffer cache, once in the MFS’s procs memory space).
Despite the duplication of data, the fact that MFS filesystems do not
automaticaly flush data to a physical storage device has proven to be
a huge performance advantage over the years.So, in short, we already have the pieces, we just have to get them to
fit together in a reasonable way.”
Anyone want a headache? I just found this while trying to refresh my memory on OpenBSD’s mfs:
http://www.blackant.net/other/docs/howto-full-system-mfs.php
—
I was thinking of the softupdate delayed-commit work* when I came up with my cockamamie idea; the moment of ‘insight’ was ‘What if the delay went on forever and the actual commit to disk was never made?’
To be honest, I have no idea how the snapshot ability could/would/wouldn’t relate here; refreshing my memory with the papers at the bottom of http://www.mckusick.com/softdep/ just proved interesting, but my feeble mind says “that’s just metadata?,” so if you’re making a brilliant point on the interaction between that and the buffer cache(?), it’s shot right over my head.** (Maybe you’re on to some other mode of filesystem-abuse for fun and profit?)
—
Did I miss something with this talk of ditching softupdates? (Forgive me if all the overloading of the word ‘journaling’ didn’t inspire confidence.) I gathered it would stop being the only way of life, might need reimplementation someplace other than it currently is, but I missed out or just plain forgot if the concept itself suddenly became old and busted. Is there a New Hotness? Am I reading too much into other people’s acknowledgement of simple work-in-progress breakages? Are other people reading too much into other people’s acknowledgement of simple work-in-progress breakages?
—
*vfs.sync_extdelay and the code behind it. In or planned for FreeBSD at least, can’t remember where or when I’ve otherwise seen it mentioned. (Deja vu that it was mentioned on dragonfly.kernel, but if I knew where, I think I’d have the answer to ‘Do we hate softupdates now?’ already.)
**Emphasis on ‘my.’ Let’s go shopping!
I was thinking a RAM-based snapshot would let you cache data in RAM, with the only speed hit when memory is initially accessed. Data would then be flushed to disk on an occasional basis.
You’d only get a speed benefit when accessing data that had been recently touched, though. Good for, say, a news server, and maybe a mail server, to some extent. It’s only a half-formed idea, in any case, and it can wait until the rest of the system is developed.
I have had a recurring geek fantasy of acquiring a motherboard with many gigabytes of RAM, and using that for /usr/obj/, /tmp, and perhaps /usr/ports/, just storing/refreshing it on shutdown/boot. It’s not terribly practical now, but maybe if RAM keeps doing the eternal price slide…
I’d like to be able to overlay a compact flash filesystem, over a hard disk filesystem, over a CDROM filesystem.
That way the system can be booted from CDROM, modified or speed critical data can be copied to HD, and if CF is available then the HD can be spundown until the CF gets full.
Backup just becomes a matter of preparing a new ISO, burning it, and swapping the new and active copies.
If CDROM is too restraining, then move to DVDROM.