Am 22.08.2011 23:27, schrieb Sunil Mushran:
On 08/22/2011 11:11 AM, Stefan Priebe - Profihost AG wrote:
BTW, how much memory do the nodes have?
Lock Resources: 1863 (497079)
This means that that node has created 497K lock resources
but have under 2000 cached. That could be due to deletes
or could be due to lack of memory (lock churning).
They all have 8GB of mem but only 1,5-2,5 are in use regarding free
command. What does lock churning mean and is there another setting /
value i can look at?
Its tied to the inode cache. As long as the inode is cached,
the lock resource will be cached.
Thanks for your answer but i didn't get it. It is enough "main" memory
free. Or is this to some special kind of memory? perhaps tuneable via
sysctl?
inode cache. I don't think you can pin the size of this cache.
It size varies. But we are jumping to conclusions. Right now we
don't know whether that is the problem as the numbers you sent
me were fairly random.
ok here is a new complete test with values before and after and all
bonnie details.
File creation and seq. delete drops again massively.
Stefan
Revision: 0.90
Mount Count: 0 Max Mount Count: 20
State: 0 Errors: 0
Check Interval: 0 Last Check: Wed Aug 24 06:46:33 2011
Creator OS: 0
Feature Compat: 3 backup-super strict-journal-super
Feature Incompat: 16208 sparse extended-slotmap inline-data metaecc
xattr indexed-dirs refcount discontig-bg
Tunefs Incomplete: 0
Feature RO compat: 7 unwritten usrquota grpquota
Root Blknum: 5 System Dir Blknum: 6
First Cluster Group Blknum: 3
Block Size Bits: 12 Cluster Size Bits: 12
Max Node Slots: 4
Extended Attributes Inline Size: 256
Label: ocfs2
UUID: 2B77D909E4704CFFBB1AED98361B8612
Hash: 2958246527 (0xb053427f)
DX Seed[0]: 0xfe73a451
DX Seed[1]: 0xd1189052
DX Seed[2]: 0x88ad3b12
Cluster stack: classic o2cb
Inode: 2 Mode: 00 Generation: 3341884273 (0xc7311b71)
FS Generation: 3341884273 (0xc7311b71)
CRC32: 7ce5ef8e ECC: 0a7a
Type: Unknown Attr: 0x0 Flags: Valid System Superblock
Dynamic Features: (0x0)
User: 0 (root) Group: 0 (root) Size: 0
Links: 0 Clusters: 2928672759
ctime: 0x4e548229 -- Wed Aug 24 06:46:33 2011
atime: 0x0 -- Thu Jan 1 01:00:00 1970
mtime: 0x4e548229 -- Wed Aug 24 06:46:33 2011
dtime: 0x0 -- Thu Jan 1 01:00:00 1970
ctime_nsec: 0x00000000 -- 0
atime_nsec: 0x00000000 -- 0
mtime_nsec: 0x00000000 -- 0
Refcount Block: 0
Last Extblk: 0 Orphan Slot: 0
Sub Alloc Slot: Global Sub Alloc Bit: 65535
1,1,30562612,51424252942,261375675972,5674946908498,411,52407052
Domain: 2B77D909E4704CFFBB1AED98361B8612 Key: 0xcf2f2e77 Protocol: 1.1
Thread Pid: 4344 Node: 0 State: JOINED
Number of Joins: 1 Joining Node: 255
Domain Map: 0 1
Live Map: 0 1
Lock Resources: 18 (17380228)
MLEs: 0 (15281027)
Blocking: 0 (3)
Mastery: 0 (15281024)
Migration: 0 (0)
Lists: Dirty=Empty Purge=Empty PendingASTs=Empty PendingBASTs=Empty
Purge Count: 0 Refs: 1
Dead Node: 255
Recovery Pid: 4345 Master: 255 State: INACTIVE
Recovery Map:
Recovery Node State:
Revision: 0.90
Mount Count: 0 Max Mount Count: 20
State: 0 Errors: 0
Check Interval: 0 Last Check: Wed Aug 24 06:46:33 2011
Creator OS: 0
Feature Compat: 3 backup-super strict-journal-super
Feature Incompat: 16208 sparse extended-slotmap inline-data metaecc
xattr indexed-dirs refcount discontig-bg
Tunefs Incomplete: 0
Feature RO compat: 7 unwritten usrquota grpquota
Root Blknum: 5 System Dir Blknum: 6
First Cluster Group Blknum: 3
Block Size Bits: 12 Cluster Size Bits: 12
Max Node Slots: 4
Extended Attributes Inline Size: 256
Label: ocfs2
UUID: 2B77D909E4704CFFBB1AED98361B8612
Hash: 2958246527 (0xb053427f)
DX Seed[0]: 0xfe73a451
DX Seed[1]: 0xd1189052
DX Seed[2]: 0x88ad3b12
Cluster stack: classic o2cb
Inode: 2 Mode: 00 Generation: 3341884273 (0xc7311b71)
FS Generation: 3341884273 (0xc7311b71)
CRC32: 7ce5ef8e ECC: 0a7a
Type: Unknown Attr: 0x0 Flags: Valid System Superblock
Dynamic Features: (0x0)
User: 0 (root) Group: 0 (root) Size: 0
Links: 0 Clusters: 2928672759
ctime: 0x4e548229 -- Wed Aug 24 06:46:33 2011
atime: 0x0 -- Thu Jan 1 01:00:00 1970
mtime: 0x4e548229 -- Wed Aug 24 06:46:33 2011
dtime: 0x0 -- Thu Jan 1 01:00:00 1970
ctime_nsec: 0x00000000 -- 0
atime_nsec: 0x00000000 -- 0
mtime_nsec: 0x00000000 -- 0
Refcount Block: 0
Last Extblk: 0 Orphan Slot: 0
Sub Alloc Slot: Global Sub Alloc Bit: 65535
1,1,18,47809,162693,4867302,30,47565364
Domain: 2B77D909E4704CFFBB1AED98361B8612 Key: 0xcf2f2e77 Protocol: 1.1
Thread Pid: 4344 Node: 0 State: JOINED
Number of Joins: 1 Joining Node: 255
Domain Map: 0 1
Live Map: 0 1
Lock Resources: 12 (12)
MLEs: 0 (15)
Blocking: 0 (3)
Mastery: 0 (12)
Migration: 0 (0)
Lists: Dirty=Empty Purge=Empty PendingASTs=Empty PendingBASTs=Empty
Purge Count: 0 Refs: 1
Dead Node: 255
Recovery Pid: 4345 Master: 255 State: INACTIVE
Recovery Map:
Recovery Node State:
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
server123-han 16G 441 99 113301 23 45351 6 2641 99 102175 6 1158
16
Latency 19822us 524ms 205ms 16403us 249ms 965ms
Version 1.96 ------Sequential Create------ --------Random Create--------
server123-han -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
1024:32768:0/1024 569 39 178 7 866 20 556 39 63 3 183 4
Latency 1192ms 542ms 1594ms 1079ms 625ms 4700ms
1.96,1.96,server123-han,1,1314185820,16G,,441,99,113301,23,45351,6,2641,99,102175,6,1158,16,1024,32768,,,1024,569,39,178,7,866,20,556,39,63,3,183,4,19822us,524ms,205ms,16403us,249ms,965ms,1192ms,542ms,1594ms,1079ms,625ms,4700ms
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
server123-han 16G 1715 98 117309 6 47932 4 4656 98 141380 7 796.2
24
Latency 6829us 127ms 206ms 8787us 258ms 270ms
Version 1.96 ------Sequential Create------ --------Random Create--------
server123-han -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
1024:32768:0/1024 2611 27 873 5 8528 76 2604 27 93 0 3875 51
Latency 1077ms 453ms 53421us 1235ms 329ms 1577ms
1.96,1.96,server123-han,1,1314119433,16G,,1715,98,117309,6,47932,4,4656,98,141380,7,796.2,24,1024,32768,,,1024,2611,27,873,5,8528,76,2604,27,93,0,3875,51,6829us,127ms,206ms,8787us,258ms,270ms,1077ms,453ms,53421us,1235ms,329ms,1577ms
Revision: 0.90
Mount Count: 0 Max Mount Count: 20
State: 0 Errors: 0
Check Interval: 0 Last Check: Wed Aug 24 06:46:33 2011
Creator OS: 0
Feature Compat: 3 backup-super strict-journal-super
Feature Incompat: 16208 sparse extended-slotmap inline-data metaecc
xattr indexed-dirs refcount discontig-bg
Tunefs Incomplete: 0
Feature RO compat: 7 unwritten usrquota grpquota
Root Blknum: 5 System Dir Blknum: 6
First Cluster Group Blknum: 3
Block Size Bits: 12 Cluster Size Bits: 12
Max Node Slots: 4
Extended Attributes Inline Size: 256
Label: ocfs2
UUID: 2B77D909E4704CFFBB1AED98361B8612
Hash: 2958246527 (0xb053427f)
DX Seed[0]: 0xfe73a451
DX Seed[1]: 0xd1189052
DX Seed[2]: 0x88ad3b12
Cluster stack: classic o2cb
Inode: 2 Mode: 00 Generation: 3341884273 (0xc7311b71)
FS Generation: 3341884273 (0xc7311b71)
CRC32: 7ce5ef8e ECC: 0a7a
Type: Unknown Attr: 0x0 Flags: Valid System Superblock
Dynamic Features: (0x0)
User: 0 (root) Group: 0 (root) Size: 0
Links: 0 Clusters: 2928672759
ctime: 0x4e548229 -- Wed Aug 24 06:46:33 2011
atime: 0x0 -- Thu Jan 1 01:00:00 1970
mtime: 0x4e548229 -- Wed Aug 24 06:46:33 2011
dtime: 0x0 -- Thu Jan 1 01:00:00 1970
ctime_nsec: 0x00000000 -- 0
atime_nsec: 0x00000000 -- 0
mtime_nsec: 0x00000000 -- 0
Refcount Block: 0
Last Extblk: 0 Orphan Slot: 0
Sub Alloc Slot: Global Sub Alloc Bit: 65535
Domain: 2B77D909E4704CFFBB1AED98361B8612 Key: 0xcf2f2e77 Protocol: 1.1
Thread Pid: 5511 Node: 0 State: JOINED
Number of Joins: 1 Joining Node: 255
Domain Map: 0
Live Map: 0
Lock Resources: 18 (17587050)
MLEs: 0 (15487846)
Blocking: 0 (0)
Mastery: 0 (15487846)
Migration: 0 (0)
Lists: Dirty=Empty Purge=Empty PendingASTs=Empty PendingBASTs=Empty
Purge Count: 0 Refs: 1
Dead Node: 255
Recovery Pid: 5512 Master: 255 State: INACTIVE
Recovery Map:
Recovery Node State:
Revision: 0.90
Mount Count: 0 Max Mount Count: 20
State: 0 Errors: 0
Check Interval: 0 Last Check: Wed Aug 24 06:46:33 2011
Creator OS: 0
Feature Compat: 3 backup-super strict-journal-super
Feature Incompat: 16208 sparse extended-slotmap inline-data metaecc
xattr indexed-dirs refcount discontig-bg
Tunefs Incomplete: 0
Feature RO compat: 7 unwritten usrquota grpquota
Root Blknum: 5 System Dir Blknum: 6
First Cluster Group Blknum: 3
Block Size Bits: 12 Cluster Size Bits: 12
Max Node Slots: 4
Extended Attributes Inline Size: 256
Label: ocfs2
UUID: 2B77D909E4704CFFBB1AED98361B8612
Hash: 2958246527 (0xb053427f)
DX Seed[0]: 0xfe73a451
DX Seed[1]: 0xd1189052
DX Seed[2]: 0x88ad3b12
Cluster stack: classic o2cb
Inode: 2 Mode: 00 Generation: 3341884273 (0xc7311b71)
FS Generation: 3341884273 (0xc7311b71)
CRC32: 7ce5ef8e ECC: 0a7a
Type: Unknown Attr: 0x0 Flags: Valid System Superblock
Dynamic Features: (0x0)
User: 0 (root) Group: 0 (root) Size: 0
Links: 0 Clusters: 2928672759
ctime: 0x4e548229 -- Wed Aug 24 06:46:33 2011
atime: 0x0 -- Thu Jan 1 01:00:00 1970
mtime: 0x4e548229 -- Wed Aug 24 06:46:33 2011
dtime: 0x0 -- Thu Jan 1 01:00:00 1970
ctime_nsec: 0x00000000 -- 0
atime_nsec: 0x00000000 -- 0
mtime_nsec: 0x00000000 -- 0
Refcount Block: 0
Last Extblk: 0 Orphan Slot: 0
Sub Alloc Slot: Global Sub Alloc Bit: 65535
Domain: 2B77D909E4704CFFBB1AED98361B8612 Key: 0xcf2f2e77 Protocol: 1.1
Thread Pid: 5511 Node: 0 State: JOINED
Number of Joins: 1 Joining Node: 255
Domain Map: 0
Live Map: 0
Lock Resources: 12 (12)
MLEs: 0 (12)
Blocking: 0 (0)
Mastery: 0 (12)
Migration: 0 (0)
Lists: Dirty=Empty Purge=Empty PendingASTs=Empty PendingBASTs=Empty
Purge Count: 0 Refs: 1
Dead Node: 255
Recovery Pid: 5512 Master: 255 State: INACTIVE
Recovery Map:
Recovery Node State:
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
server123-han 16G 519 99 113505 22 44504 8 2601 99 97508 6 1241 18
Latency 16386us 454ms 324ms 12653us 248ms 1364ms
Version 1.96 ------Sequential Create------ --------Random Create--------
server123-han -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
1024:32768:0/1024 3147 75 437 10 1670 23 3145 76 68 1 197 2
Latency 2277ms 505ms 1428ms 2690ms 1743ms 1919ms
1.96,1.96,server123-han,1,1314168297,16G,,519,99,113505,22,44504,8,2601,99,97508,6,1241,18,1024,32768,,,1024,3147,75,437,10,1670,23,3145,76,68,1,197,2,16386us,454ms,324ms,12653us,248ms,1364ms,2277ms,505ms,1428ms,2690ms,1743ms,1919ms
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users
