Blog O’ Matty

Thomas Stromberg  on November 28th, 2007

I’m curious what bugs you are seeing. I haven’t had any corruption issues yet, but I’ve only been running it in earnest on FreeBSD. On FreeBSD, I’ve just had to deal with some deadlock issues under heavy I/O, but through tweaking the kernel memory allocation, I’ve managed to rid myself of those problems.

BTW, are you using 3par snapshots in addition to ZFS snapshots?

j  on November 29th, 2007

Any specifics on the bugs that caused the corruption? Would just like to know what to look out for. Thanks for any info.

Marcelo Leal  on November 29th, 2007

Hello Matty,
I’m making a lot of tests with ZFS, and thinking in try it in “development environment” in the first place. But my “first” concern is the one you have mentioned. How if the filesystem has bugs that make it “unmountable”? What if i have 1TB on a ZFS pool, and suddenly ZFS can not mount the pool anymore? There is tools to try to recover it? You have mentioned “3par snapshot”… nothing to do with ZFS, right? You had to rollback from backup, right?
Leal.

Makea  on November 29th, 2007

Oh crap, our Thumper just arrived yesterday!

This will be our first real use of zfs. Hope we can avoid these bugs.

matty  on November 29th, 2007

We hit bugs #6454482 and #6458218, and one bug that has yet to be root caused (the stack trace on this bug is different from the two above). Since we have had multiple pools self-destruct, we are using hardware snapshots to protect against data loss. If your pool gets into one of the states where you cannot mount it, you are at the mercy of Sun support (if you have a support contract), or the kernel engineers who reply to questions on zfs-discuss. Hopefully ZFS will become more mature over time, but currently it doesn’t give me a warm fuzzy feeling.

- Ryan

Marco  on November 29th, 2007

Good to know real production experiences.
Thanks

Leon Koll  on November 30th, 2007

We are using ZFS in production for half a year. 200TB of data in dozen of zpools. We learned that there is no bug-free software in this world, that’s why we’re creating snapshots on storage arrays (XIV snapshot capability is excellent). We never had a situation when the HW snapshot is needed (touch wood).

ajmaidak  on November 30th, 2007

What release of Solaris are you running? Looks like 6458218 is fixed in the S10u4 kernel patch. I can’t find reference to 6454482 anywhere in sunsolve. Is that a typo or is a non-public bug id…

Chris  on December 1st, 2007

I too have been running a 6TB zpool with ZFS on Sol.10-U3 (Sparc) for approx. a year now, (on E15K). I’ve even lost the AMS1000 array due to major looping issues, but ZFS came back up just fine. I’m also running many smaller ZFS pools on web & DB servers, both Sparc and x86 (using an Egenera BladeFrame, Sol.10 U3 also). I hope never to have issues since I’m not running any snapshots, yet! Just normal backups (NetBackup).

Have there been more problems with Update 4 with ZFS than U3?

Happy Holidays

Ross  on December 10th, 2007

Mind if I ask how you’re using ZFS there to have hit these bugs? We’re looking to purchase a couple of Thumpers soon, and I’m wondering if these bugs something we’re likely to experience?

Any idea what caused them?

Eddie  on December 11th, 2007

Well – I’ve got a 6 Tb 3511 array that my organisation has spent serious $$ on, and I’m using a Solaris 10u4 sparc box with zfs to drive it.

All was going well until a week ago. It now kernel panics twice a day. zpool status reports no errors, and zpool scrub just seems to make it crash a bit more consistently (about 50% through).

Here is the error message:

panic[cpu1]/thread=2a100a95cc0: zfs: freeing free segment (offset=423713792 size=1024)

This zpool has about 200 snapshots on it.

Somehow I get the feeling that there is something going on that sun is not telling us…

Alvaro  on January 2nd, 2008

I got hit but some bug that got me into a state that not even Sun support eng were able to fix it. I had Oracle running on top of a 60gig pool, and when oracle went nuts (actually a stored procedure) the pool got 100% usage and bam… nothing else we could do. I was running 11/06… It was a development box so no harm was done, but I am staying away from this, as appealing as it looks, not production ready :-(

Alvaro  on January 2nd, 2008

please read ‘by some bug’ …

nelson  on January 31st, 2008

we’ve hit the snapshot destroy bug, the real *busy* system causes ZFS to hang, ARC cache growing and *locking* ram, san/zfs cache flushing problem and ye ol’ standard live upgrade zones on zfs.

1 was worked around quickly and easiler
3 and 4 were /etc/system changes
5 is waiting in 04/08
2 is (currently) an IDR fix that we’re testing

next  on February 6th, 2008

the real *busy* system causes ZFS to hang – I seem to be experiencing this as well. I’m not sure what your “hang” means, in my case it looks like nothing is written/read to/from the pool for several seconds. Often it takes up to two minutes.

I hope it gets solved soon now will be available as a patch.

nelson  on February 7th, 2008

with regards to the hang…er it’s more like a deadlock. we’ve let it go and it panics cluster nodes.

on a non clustered node i’ve seen it hang for many hours (cough – overnight in fact).

during this time you can no longer log into the server but it responds to ping. applications appear to fault. it should be under bug 6513209 (well the IDR states that anyway)

Some guy  on February 17th, 2008

This is not necessarily a ZFS bug, but recently this happened with a 6TB filesystem that sees heavy use:


pool: somepool
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scrub: scrub completed with 7 errors on Tue Feb 12 17:09:21 2008
config:

DATASET OBJECT RANGE
somepool 164910 0-131072
5 132353a lvl=0 blkid=0
5 48a25df lvl=0 blkid=0
5 5ed1f75 lvl=0 blkid=19
somepool 114393905 0-131072
31 132353a lvl=0 blkid=0
31 48a25df lvl=0 blkid=0
31 4c297d7 lvl=0 blkid=0
31 4c297d7 lvl=0 blkid=16
31 5ed1f75 lvl=0 blkid=19
31 633f73f lvl=0 blkid=0
31 68d62e7 lvl=0 blkid=1


Yes, there’s no redundancy. Just one vdev on top of 6TB of dumb-as-a-rock hardware RAID 5. No, that’s not my doing. Sadly, this way of operating (zero-planning) is normal for the organization I work for.

Of course, it could be that it *is* a ZFS bug, because management has dictated that patches are NOT A PRIORITY AT ALL so this particular machine is six-months behind on kernel patches. Great place to work, huh?

matty  on February 17th, 2008

Hey Some guy.

I feel your pain, and am starting to realize more and more that you need to have the latest Solaris 10 patches applied in order to get a stable operating environment. Which Solaris 10 update (and kernel revision) are you running?

- Ryan

Raul Ortega  on March 22nd, 2008

I have 2 server x86 using openSolaris b81 and b74, quite some experimental, but have mor than 6 month withouth problem. b81 it’s a CIFS server to 4 Wireless clients and 2 Desktops and Squid Proxy restriction, and b74 only Squid server to 20 computer. ZFS = 0 Problems.
My laptop > 8 month with openSolaris = 0 Corruption. :-P