Serveo
08/07/14, 17:42
Hi,
Al enige tijd hebben we een offsite backup omgeving draaien op basis van illumian + napp-it. Een backup server in de front-end welke op CentOS draait en de volume via ISCSI connect.
Sinds een week houden we echter problemen met de disk. Deze komt keer op keer op RO als we in een map rm uitvoeren of at random. We hebben fsck, fsck.ext3, e2fsck en almede ook met een alternative superblocks. Echter blijft deze in RO terecht komen.
Uit de output van de zfs omgeving komt niks vreemds naar voren:
# zpool status -v
pool: backupstor
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
backupstor ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c0t50014EE000F3205Cd0 ONLINE 0 0 0
c0t50014EE101EA1E7Fd0 ONLINE 0 0 0
c0t50014EE101EA2C44d0 ONLINE 0 0 0
c0t50014EE1573F0F9Fd0 ONLINE 0 0 0
c0t50014EE202F6D251d0 ONLINE 0 0 0
c0t50014EE25A8DA70Dd0 ONLINE 0 0 0
c0t50014EE2ADAB754Cd0 ONLINE 0 0 0
c0t50014EE2ADE1DC4Bd0 ONLINE 0 0 0
c0t50014EE2AEB409D2d0 ONLINE 0 0 0
c0t50014EE2AEB40AF0d0 ONLINE 0 0 0
c0t50014EE2AFA9B534d0 ONLINE 0 0 0
c0t50014EE2AFD3F853d0 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
c0t5E83A97F71290563d0 ONLINE 0 0 0
c0t5E83A97FC34F8D6Bd0 ONLINE 0 0 0
cache
c0t5001517BB28A01F8d0 ONLINE 0 0 0
c0t5001517BB28B5D9Ed0 ONLINE 0 0 0
errors: No known data errors
pool: rpool
state: ONLINE
scan: resilvered 6K in 0h0m with 0 errors on Fri Dec 31 14:05:04 2004
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c3t0d0s0 ONLINE 0 0 0
c3t1d0s0 ONLINE 0 0 0
errors: No known data errors
# zpool list
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
backupstor 5.44T 4.82T 631G - 88% 1.00x ONLINE -
rpool 29.8G 14.3G 15.5G - 48% 1.00x ONLINE -
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
backupstor 4.43T 451G 69.6K /backupstor
backupstor/backups 3.99T 0 3.99T -
rpool 15.3G 14.0G 40.5K /rpool
rpool/ROOT 3.10G 14.0G 31K none
rpool/ROOT/illumian 24.8M 14.0G 2.06G legacy
rpool/ROOT/napp-it-0.8h 4.75M 14.0G 2.78G legacy
rpool/ROOT/napp-it-0.9p1 3.07G 14.0G 2.84G legacy
rpool/ROOT/pre_napp-it-0.8h 85.5K 14.0G 1.02G legacy
rpool/ROOT/pre_napp-it-0.9f1 84.5K 14.0G 2.81G legacy
rpool/ROOT/pre_napp-it-0.9p1 88.5K 14.0G 2.76G legacy
rpool/dump 11.2G 14.0G 11.2G -
rpool/swap 1.03G 15.0G 16K -
# zfs get all backupstor/backups
NAME PROPERTY VALUE SOURCE
backupstor/backups type volume -
backupstor/backups creation Mon Jul 23 13:10 2012 -
backupstor/backups used 3.99T -
backupstor/backups available 0 -
backupstor/backups referenced 3.99T -
backupstor/backups compressratio 1.00x -
backupstor/backups reservation none default
backupstor/backups volsize 2T local
backupstor/backups volblocksize 512 -
backupstor/backups checksum on default
backupstor/backups compression off default
backupstor/backups readonly off default
backupstor/backups copies 1 default
backupstor/backups refreservation none default
backupstor/backups primarycache all default
backupstor/backups secondarycache all default
backupstor/backups usedbysnapshots 0 -
backupstor/backups usedbydataset 3.99T -
backupstor/backups usedbychildren 0 -
backupstor/backups usedbyrefreservation 0 -
backupstor/backups logbias latency default
backupstor/backups dedup off default
backupstor/backups mlslabel none default
backupstor/backups sync standard default
backupstor/backups refcompressratio 1.00x -
backupstor/backups written 3.99T -
Gezien bovenstaande ben ik naar de CentOS gaan kijken:
# iscsiadm -m session
tcp: [1] x.x.x.x:3260,2 iqn.1986-03.com.sun:t-backups
# ip ro get 0.0.0.254
0.0.0.254 dev bond0.xx src 0.0.0.246
cache mtu 1500 advmss 1460 hoplimit 64
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.4.0-2 (October 7, 2008)
Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:00:00:00:00:00
Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:00:00:00:00:xx
# ls -lh /dev/mapper/
totaal 0
crw------- 1 root root 10, 60 jul 8 13:16 control
brw-rw---- 1 root disk 253, 47 jul 8 13:17 backups
# cat /proc/diskstats
253 47 dm-47 143 0 1138 663 2 0 16 24 0 687 687
# grep dm-47 /var/log/messages
Jul 8 11:09:47 localhost kernel: Aborting journal on device dm-47.
Jul 8 11:09:47 localhost kernel: EXT3-fs error (device dm-47): ext3_journal_start_sb: Detected aborted journal
Jul 8 11:14:14 localhost kernel: EXT3-fs error (device dm-47): ext3_put_super: Couldn't clean up the journal
Jul 8 11:15:52 localhost kernel: EXT3-fs warning (device dm-47): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Jul 8 11:15:52 localhost kernel: EXT3-fs warning (device dm-47): ext3_clear_journal_err: Marking fs in need of filesystem check.
Jul 8 11:15:52 localhost kernel: EXT3 FS on dm-47, internal journal
Jul 8 12:50:53 localhost kernel: Buffer I/O error on device dm-47, logical block 488474868
Jul 8 12:50:53 localhost kernel: lost page write due to I/O error on dm-47
Jul 8 13:17:02 localhost kernel: EXT3 FS on dm-47, internal journal
Jul 8 13:22:29 localhost kernel: EXT3-fs error (device dm-47): htree_dirblock_to_tree: bad entry in directory #228737064: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
Jul 8 13:22:29 localhost kernel: Aborting journal on device dm-47.
Jul 8 13:22:29 localhost kernel: EXT3-fs error (device dm-47): ext3_journal_start_sb: Detected aborted journal
Ook mannueel heb ik geprobeerd de inode / directory te verwijderen, enkel komt het FS dan in RO.
# find /mnt/backups/ -inum 228737064
Iemand toevallig iets dergelijks meegemaakt?->|
Al enige tijd hebben we een offsite backup omgeving draaien op basis van illumian + napp-it. Een backup server in de front-end welke op CentOS draait en de volume via ISCSI connect.
Sinds een week houden we echter problemen met de disk. Deze komt keer op keer op RO als we in een map rm uitvoeren of at random. We hebben fsck, fsck.ext3, e2fsck en almede ook met een alternative superblocks. Echter blijft deze in RO terecht komen.
Uit de output van de zfs omgeving komt niks vreemds naar voren:
# zpool status -v
pool: backupstor
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
backupstor ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c0t50014EE000F3205Cd0 ONLINE 0 0 0
c0t50014EE101EA1E7Fd0 ONLINE 0 0 0
c0t50014EE101EA2C44d0 ONLINE 0 0 0
c0t50014EE1573F0F9Fd0 ONLINE 0 0 0
c0t50014EE202F6D251d0 ONLINE 0 0 0
c0t50014EE25A8DA70Dd0 ONLINE 0 0 0
c0t50014EE2ADAB754Cd0 ONLINE 0 0 0
c0t50014EE2ADE1DC4Bd0 ONLINE 0 0 0
c0t50014EE2AEB409D2d0 ONLINE 0 0 0
c0t50014EE2AEB40AF0d0 ONLINE 0 0 0
c0t50014EE2AFA9B534d0 ONLINE 0 0 0
c0t50014EE2AFD3F853d0 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
c0t5E83A97F71290563d0 ONLINE 0 0 0
c0t5E83A97FC34F8D6Bd0 ONLINE 0 0 0
cache
c0t5001517BB28A01F8d0 ONLINE 0 0 0
c0t5001517BB28B5D9Ed0 ONLINE 0 0 0
errors: No known data errors
pool: rpool
state: ONLINE
scan: resilvered 6K in 0h0m with 0 errors on Fri Dec 31 14:05:04 2004
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c3t0d0s0 ONLINE 0 0 0
c3t1d0s0 ONLINE 0 0 0
errors: No known data errors
# zpool list
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
backupstor 5.44T 4.82T 631G - 88% 1.00x ONLINE -
rpool 29.8G 14.3G 15.5G - 48% 1.00x ONLINE -
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
backupstor 4.43T 451G 69.6K /backupstor
backupstor/backups 3.99T 0 3.99T -
rpool 15.3G 14.0G 40.5K /rpool
rpool/ROOT 3.10G 14.0G 31K none
rpool/ROOT/illumian 24.8M 14.0G 2.06G legacy
rpool/ROOT/napp-it-0.8h 4.75M 14.0G 2.78G legacy
rpool/ROOT/napp-it-0.9p1 3.07G 14.0G 2.84G legacy
rpool/ROOT/pre_napp-it-0.8h 85.5K 14.0G 1.02G legacy
rpool/ROOT/pre_napp-it-0.9f1 84.5K 14.0G 2.81G legacy
rpool/ROOT/pre_napp-it-0.9p1 88.5K 14.0G 2.76G legacy
rpool/dump 11.2G 14.0G 11.2G -
rpool/swap 1.03G 15.0G 16K -
# zfs get all backupstor/backups
NAME PROPERTY VALUE SOURCE
backupstor/backups type volume -
backupstor/backups creation Mon Jul 23 13:10 2012 -
backupstor/backups used 3.99T -
backupstor/backups available 0 -
backupstor/backups referenced 3.99T -
backupstor/backups compressratio 1.00x -
backupstor/backups reservation none default
backupstor/backups volsize 2T local
backupstor/backups volblocksize 512 -
backupstor/backups checksum on default
backupstor/backups compression off default
backupstor/backups readonly off default
backupstor/backups copies 1 default
backupstor/backups refreservation none default
backupstor/backups primarycache all default
backupstor/backups secondarycache all default
backupstor/backups usedbysnapshots 0 -
backupstor/backups usedbydataset 3.99T -
backupstor/backups usedbychildren 0 -
backupstor/backups usedbyrefreservation 0 -
backupstor/backups logbias latency default
backupstor/backups dedup off default
backupstor/backups mlslabel none default
backupstor/backups sync standard default
backupstor/backups refcompressratio 1.00x -
backupstor/backups written 3.99T -
Gezien bovenstaande ben ik naar de CentOS gaan kijken:
# iscsiadm -m session
tcp: [1] x.x.x.x:3260,2 iqn.1986-03.com.sun:t-backups
# ip ro get 0.0.0.254
0.0.0.254 dev bond0.xx src 0.0.0.246
cache mtu 1500 advmss 1460 hoplimit 64
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.4.0-2 (October 7, 2008)
Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:00:00:00:00:00
Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:00:00:00:00:xx
# ls -lh /dev/mapper/
totaal 0
crw------- 1 root root 10, 60 jul 8 13:16 control
brw-rw---- 1 root disk 253, 47 jul 8 13:17 backups
# cat /proc/diskstats
253 47 dm-47 143 0 1138 663 2 0 16 24 0 687 687
# grep dm-47 /var/log/messages
Jul 8 11:09:47 localhost kernel: Aborting journal on device dm-47.
Jul 8 11:09:47 localhost kernel: EXT3-fs error (device dm-47): ext3_journal_start_sb: Detected aborted journal
Jul 8 11:14:14 localhost kernel: EXT3-fs error (device dm-47): ext3_put_super: Couldn't clean up the journal
Jul 8 11:15:52 localhost kernel: EXT3-fs warning (device dm-47): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Jul 8 11:15:52 localhost kernel: EXT3-fs warning (device dm-47): ext3_clear_journal_err: Marking fs in need of filesystem check.
Jul 8 11:15:52 localhost kernel: EXT3 FS on dm-47, internal journal
Jul 8 12:50:53 localhost kernel: Buffer I/O error on device dm-47, logical block 488474868
Jul 8 12:50:53 localhost kernel: lost page write due to I/O error on dm-47
Jul 8 13:17:02 localhost kernel: EXT3 FS on dm-47, internal journal
Jul 8 13:22:29 localhost kernel: EXT3-fs error (device dm-47): htree_dirblock_to_tree: bad entry in directory #228737064: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
Jul 8 13:22:29 localhost kernel: Aborting journal on device dm-47.
Jul 8 13:22:29 localhost kernel: EXT3-fs error (device dm-47): ext3_journal_start_sb: Detected aborted journal
Ook mannueel heb ik geprobeerd de inode / directory te verwijderen, enkel komt het FS dan in RO.
# find /mnt/backups/ -inum 228737064
Iemand toevallig iets dergelijks meegemaakt?->|