Hi,

Al enige tijd hebben we een offsite backup omgeving draaien op basis van illumian + napp-it. Een backup server in de front-end welke op CentOS draait en de volume via ISCSI connect.

Sinds een week houden we echter problemen met de disk. Deze komt keer op keer op RO als we in een map rm uitvoeren of at random. We hebben fsck, fsck.ext3, e2fsck en almede ook met een alternative superblocks. Echter blijft deze in RO terecht komen.

Uit de output van de zfs omgeving komt niks vreemds naar voren:

# zpool status -v
Code:
pool: backupstor
 state: ONLINE
  scan: none requested
config:

        NAME                       STATE     READ WRITE CKSUM
        backupstor                 ONLINE       0     0     0
          raidz2-0                 ONLINE       0     0     0
            c0t50014EE000F3205Cd0  ONLINE       0     0     0
            c0t50014EE101EA1E7Fd0  ONLINE       0     0     0
            c0t50014EE101EA2C44d0  ONLINE       0     0     0
            c0t50014EE1573F0F9Fd0  ONLINE       0     0     0
            c0t50014EE202F6D251d0  ONLINE       0     0     0
            c0t50014EE25A8DA70Dd0  ONLINE       0     0     0
            c0t50014EE2ADAB754Cd0  ONLINE       0     0     0
            c0t50014EE2ADE1DC4Bd0  ONLINE       0     0     0
            c0t50014EE2AEB409D2d0  ONLINE       0     0     0
            c0t50014EE2AEB40AF0d0  ONLINE       0     0     0
            c0t50014EE2AFA9B534d0  ONLINE       0     0     0
            c0t50014EE2AFD3F853d0  ONLINE       0     0     0
        logs
          mirror-1                 ONLINE       0     0     0
            c0t5E83A97F71290563d0  ONLINE       0     0     0
            c0t5E83A97FC34F8D6Bd0  ONLINE       0     0     0
        cache
          c0t5001517BB28A01F8d0    ONLINE       0     0     0
          c0t5001517BB28B5D9Ed0    ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: resilvered 6K in 0h0m with 0 errors on Fri Dec 31 14:05:04 2004
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            c3t0d0s0  ONLINE       0     0     0
            c3t1d0s0  ONLINE       0     0     0

errors: No known data errors

# zpool list
Code:
NAME         SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
backupstor  5.44T  4.82T   631G         -    88%  1.00x  ONLINE  -
rpool       29.8G  14.3G  15.5G         -    48%  1.00x  ONLINE  -
# zfs list
Code:
NAME                           USED  AVAIL  REFER  MOUNTPOINT
backupstor                    4.43T   451G  69.6K  /backupstor
backupstor/backups        3.99T      0  3.99T  -
rpool                         15.3G  14.0G  40.5K  /rpool
rpool/ROOT                    3.10G  14.0G    31K  none
rpool/ROOT/illumian           24.8M  14.0G  2.06G  legacy
rpool/ROOT/napp-it-0.8h       4.75M  14.0G  2.78G  legacy
rpool/ROOT/napp-it-0.9p1      3.07G  14.0G  2.84G  legacy
rpool/ROOT/pre_napp-it-0.8h   85.5K  14.0G  1.02G  legacy
rpool/ROOT/pre_napp-it-0.9f1  84.5K  14.0G  2.81G  legacy
rpool/ROOT/pre_napp-it-0.9p1  88.5K  14.0G  2.76G  legacy
rpool/dump                    11.2G  14.0G  11.2G  -
rpool/swap                    1.03G  15.0G    16K  -
# zfs get all backupstor/backups
Code:
NAME                    PROPERTY              VALUE                  SOURCE
backupstor/backups  type                  volume                 -
backupstor/backups  creation              Mon Jul 23 13:10 2012  -
backupstor/backups  used                  3.99T                  -
backupstor/backups  available             0                      -
backupstor/backups  referenced            3.99T                  -
backupstor/backups  compressratio         1.00x                  -
backupstor/backups  reservation           none                   default
backupstor/backups  volsize               2T                     local
backupstor/backups  volblocksize          512                    -
backupstor/backups  checksum              on                     default
backupstor/backups  compression           off                    default
backupstor/backups  readonly              off                    default
backupstor/backups  copies                1                      default
backupstor/backups  refreservation        none                   default
backupstor/backups  primarycache          all                    default
backupstor/backups  secondarycache        all                    default
backupstor/backups  usedbysnapshots       0                      -
backupstor/backups  usedbydataset         3.99T                  -
backupstor/backups  usedbychildren        0                      -
backupstor/backups  usedbyrefreservation  0                      -
backupstor/backups  logbias               latency                default
backupstor/backups  dedup                 off                    default
backupstor/backups  mlslabel              none                   default
backupstor/backups  sync                  standard               default
backupstor/backups  refcompressratio      1.00x                  -
backupstor/backups  written               3.99T                  -
Gezien bovenstaande ben ik naar de CentOS gaan kijken:

# iscsiadm -m session
Code:
tcp: [1] x.x.x.x:3260,2 iqn.1986-03.com.sun:t-backups
# ip ro get 0.0.0.254
Code:
0.0.0.254 dev bond0.xx  src 0.0.0.246 
    cache  mtu 1500 advmss 1460 hoplimit 64
# cat /proc/net/bonding/bond0
Code:
Ethernet Channel Bonding Driver: v3.4.0-2 (October 7, 2008)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:00:00:00:00:00

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:00:00:00:00:xx
# ls -lh /dev/mapper/
Code:
totaal 0
crw------- 1 root root  10, 60 jul  8 13:16 control
brw-rw---- 1 root disk 253, 47 jul  8 13:17 backups
# cat /proc/diskstats
Code:
253   47 dm-47 143 0 1138 663 2 0 16 24 0 687 687
# grep dm-47 /var/log/messages
Code:
Jul  8 11:09:47 localhost kernel: Aborting journal on device dm-47.
Jul  8 11:09:47 localhost kernel: EXT3-fs error (device dm-47): ext3_journal_start_sb: Detected aborted journal
Jul  8 11:14:14 localhost kernel: EXT3-fs error (device dm-47): ext3_put_super: Couldn't clean up the journal
Jul  8 11:15:52 localhost kernel: EXT3-fs warning (device dm-47): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Jul  8 11:15:52 localhost kernel: EXT3-fs warning (device dm-47): ext3_clear_journal_err: Marking fs in need of filesystem check.
Jul  8 11:15:52 localhost kernel: EXT3 FS on dm-47, internal journal
Jul  8 12:50:53 localhost kernel: Buffer I/O error on device dm-47, logical block 488474868
Jul  8 12:50:53 localhost kernel: lost page write due to I/O error on dm-47
Jul  8 13:17:02 localhost kernel: EXT3 FS on dm-47, internal journal
Jul  8 13:22:29 localhost kernel: EXT3-fs error (device dm-47): htree_dirblock_to_tree: bad entry in directory #228737064: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
Jul  8 13:22:29 localhost kernel: Aborting journal on device dm-47.
Jul  8 13:22:29 localhost kernel: EXT3-fs error (device dm-47): ext3_journal_start_sb: Detected aborted journal
Ook mannueel heb ik geprobeerd de inode / directory te verwijderen, enkel komt het FS dan in RO.
# find /mnt/backups/ -inum 228737064

Iemand toevallig iets dergelijks meegemaakt?