On 3/24/15 2:52 PM, Shrinand Javadekar wrote:
Hi,
I wrote a small microbenchmark for measuring the performance of
extended attributes in XFS. In the experiment, I wrote 100K files,
each with extended attributes. In one experiment, XFS was formatted
with the default inode size of 256 bytes. In the other experiment, it
was formatted with an inode size of 2048 bytes.
Here's the actual code: http://pastie.org/private/2gnr3j9p5lxzzufdypkyzq
I ran this using cProfile. The results suggest that time required for
"xattr._xattr.fsetxattr" when the inode is 256 bytes is ~4 times that
when the inode is 2048 bytes.
When the inode is 256 bytes:
ncalls tottime percall cumtime percall filename:lineno(function)
100000 3.933 0.000 3.933 0.000 {xattr._xattr.fsetxattr}
100000 4.344 0.000 4.344 0.000 {xattr._xattr.fsetxattr}
100000 4.624 0.000 4.624 0.000 {xattr._xattr.fsetxattr}
When the inode is 2048 bytes:
ncalls tottime percall cumtime percall filename:lineno(function)
100000 1.239 0.000 1.239 0.000 {xattr._xattr.fsetxattr}
100000 1.102 0.000 1.102 0.000 {xattr._xattr.fsetxattr}
100000 1.067 0.000 1.067 0.000 {xattr._xattr.fsetxattr}
I will delve into more elaborate experiments with Swift and XFS
filesystems formatted with different inode sizes. However, before
that, I wanted to check if the above values are plausible and/or get
suggestions around this microbenchmark.
The benchmark makes sense. The code (reproduced below in case that paste
goes away someday [1]) is setting the xattr "user.swift.metadata" (19
bytes) to a value of length 228, for a total of 247 bytes.
If the extended attributes will fit into the inode, then XFS will store
them there. However, if the extended attributes do not fit into the
inode, then the extended attributes are stored outside the inode in one
or more blocks. See [2] for more details.
It's important to note that *all* the extended attributes are stored in
the *same place*. If everything fits in the inode, then all extended
attributes live in the inode. If the extended attributes, taken
together, exceed the available inode space by just 1 byte, then all the
extended attributes are kicked out of the inode and off to an extents block.
Thus, with a 256-byte inode and a 247-byte payload, there's not room in
the inode with all the other stuff that lives in there for the extended
attributes, so they spill to extents. With a 2 KiB inode, there's more
than enough room.
If these values make sense, will this translate into improved Swift performance?
Maybe(TM). It depends on your workload. When you make inodes bigger,
fewer of them fit in the kernel's buffer cache, possibly resulting in
more work. On the other hand, when you make them smaller, then you
always get xattrs spilled to extents.
Also, it's affected by what your users do. Are your users storing lots
of metadata (X-Object-Meta-*) on their objects? Are you using middleware
that stores lots of sysmeta (X-Object-Sysmeta-*) on your objects? Are
you running SELinux, which uses lots of xattrs for its own purposes?
Further, I think the benchmarking script is too naive to really get at
the key improvement here: avoiding seeks. If the xattrs live in the
inode, then we get them essentially for free when we open() the file. If
the xattrs live in extents, then at least one random IO is required to
fetch them. The improvement shown here is on the order of 30
microseconds, while an extra disk seek is more like 10-20 *milliseconds*
(on spinning platters). If you really want to measure the performance
gain, you'll need to drop the buffer cache each time and fsync() when
you're done.
[1]
============
#!/usr/bin/python
import xattr
prefix = "/srv/node/r1/"
for i in range (100000):
fd = open(prefix + "file-" + str(i), 'w')
fd.write("Some text")
xattr.setxattr(fd, "user.swift.metadata",
"U.Content-Lengthq.U.166U.nameq.U'/AUTH_admin/container-test/testxattr.pyq.U.X-Object-Meta-MtimeU.1426996406.379914q.U.ETagq.U
bab3674eed1a8725793f2d2de21f50a9q.U.X-Timestampq.U.1426996448.35449U.Content-Typeq.U.text/x-pythonq.u.")
fd.flush()
fd.close()
============
[2]
http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure/tmp/en-US/html/Extended_Attributes.html
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : [email protected]
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
