Re: [CentOS] Mysql 5.6, Centos 7 and errno: 24 - Too many open files

[Todor Petkov]

On 29/04/2015 09:31 AM, Götz Reinicke - IT Koordinator wrote:

Hi,

may be somewon has a working solution and information on that:

I installed the most recent mysql community on a server and do get a 
lot

of "errno: 24 - Too many open files".

There are suggestions to increase the open_files_limit, change/add that
to /etc/security/limits.conf and modify the systemd script by hand.

Depending on how you start mysql (restart, or at systemboot time ...)
the limit get set or not.

Question: Is that may be a known bug or something?

And thats mit a question regarding mariadb or mysql.

Regards . Götz
--
Götz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reini...@filmakademie.de

Filmakademie Baden-Württemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
Staatssekretär im Ministerium für Wissenschaft,
Forschung und Kunst Baden-Württemberg

Geschäftsführer: Prof. Thomas Schadt


Hi,

can you do "lsof|grep mysql" and show us the output?

Regards,


___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] CENTOS not DoD approved

[Jason Pyeron]

> -Original Message-
> From: Jim Perrin
> Sent: Tuesday, April 28, 2015 20:45
> 
> On 04/28/2015 06:05 PM, Akemi Yagi wrote:
> > On Tue, Apr 28, 2015 at 3:10 PM, Johnny Hughes 
>  wrote:
> > 
> >> CentOS is not approved for DOD use.  In fact, CentOS is 
> not now, nor has
> >> it ever been *certified* for anything.  Certifications 
> require people to
> >> PAY to certify a product.
> >>
> >> Specifically, EAL4 Certification, a requirement for the 
> DOD, costs up to
> >> 2.5 million dollars .. see this link:
> >>
> >> 
> http://en.wikipedia.org/wiki/Evaluation_Assurance_Level#Impact
> _on_cost_and_schedule
> >>
> >> That cost would be for each main version of CentOS (2.1, 
> 3, 4, 5, 6, and
> >> 7) .. so the cost to have all 6 previous major versions 
> certified would be:
> >>
> >> 6 x $2.5 Million =  $15 Million dollars.
> >>
> >> Since CentOS is given away for free ... I can't afford to 
> pay 15 million
> >> dollars to have it EAL4 certified .. can anyone on this list?
> >>
> >> Certifications and security testing and assurance, along 
> with a Service
> >> Level Agreement for fixing bugs is why people who require 
> any of those
> >> things need to buy RHEL.
> > 
> > Incidentally, someone has just started a thread related to 
> DoD in the
> > RH community discussion session entitled, "A DoD version of RHEL - A
> > money maker for RH? Maybe!" :
> > 
> > https://access.redhat.com/comment/913243

There have already been high level conversation between DISA JIE and RH CTO 
with regards to that. The short story RH is built to the greater good of their 
customers. DoD will have to continue to apply their configuration updates per 
STIG.

> 
> There have been similar requests in the past. At one point someone on
> forge.mil was working on a rebuild which met STIG requirements, but

A good topic for another thread, we do that in our office.

> there were all sorts of issues with that. While I'm not in 
> sales, I feel
> safe in speculating that RH's sales folks work rather hard to 
> make sure
> the DOD as a whole stays happy.
> 
> Jason and Johnny are both right, because the DOD is a rather large
> entity with a stupidly complex array of regulations. What works in one
> command doesn't always fly in another even within a branch, let alone

There is a reciprocity between DAAs for ATOs. If any DAA has approved A then 
any other DAA can say ok because the other DAA said ok.

> jumping between branches.

It is at these lower levels where resistance is encountered.

E.g. we do not use X because Y.

> 
> TL;DR. Answer varies wildly on approval because the DOD is a GIANT
> organization with multiple levels of interwoven regulations, networks,
> and varied systems.
> 
> Article is a bit dated, but I don't imagine the situation has improved
> since I stopped doing Defense consulting.
> 
> http://www.wired.com/2010/10/read-em-all-pentagons-193-mind-nu
mbing-cyber-security-regs/
> 

The securing of RH is the same as securing CentOS, but I strongly suggest 
purchasing RH when used in a all MAC I/II 
(https://en.wikipedia.org/wiki/Mission_assurance) systems and for all 
production systems.

The CJCS put out a memo to treat all OSS as COTS, but the responsibility is 
still on the systems' CONOPS to address (self) support of the OSS. This is why 
you should purchase RH, for the support.

-Jason
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

[CentOS] CentOS-announce Digest, Vol 122, Issue 13

[centos-announce-request]

Send CentOS-announce mailing list submissions to
centos-annou...@centos.org

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.centos.org/mailman/listinfo/centos-announce
or, via email, send a message with subject or body 'help' to
centos-announce-requ...@centos.org

You can reach the person managing the list at
centos-announce-ow...@centos.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of CentOS-announce digest..."


Today's Topics:

   1. CEEA-2015:0913 CentOS 6 tzdata Enhancement Update (Johnny Hughes)
   2. CESA-2015:0895 Important CentOS 7 389-ds-base Security Update
  (Johnny Hughes)
   3. CEEA-2015:0913 CentOS 7 tzdata Enhancement Update (Johnny Hughes)
   4. CEEA-2015:0913 CentOS 5 tzdata Enhancement Update (Johnny Hughes)
   5. CEBA-2015:0915  CentOS 6 dracut BugFix Update (Johnny Hughes)
   6. CEBA-2015:0916  CentOS 6 libvirt BugFix Update (Johnny Hughes)


--

Message: 1
Date: Tue, 28 Apr 2015 19:18:48 +
From: Johnny Hughes 
To: centos-annou...@centos.org
Subject: [CentOS-announce] CEEA-2015:0913 CentOS 6 tzdata Enhancement
Update
Message-ID: <20150428191848.ga43...@n04.lon1.karan.org>
Content-Type: text/plain; charset=us-ascii


CentOS Errata and Enhancement Advisory 2015:0913 

Upstream details at : https://rhn.redhat.com/errata/RHEA-2015-0913.html

The following updated files have been uploaded and are currently 
syncing to the mirrors: ( sha256sum Filename ) 

i386:
17d079889c081ec38565a95ed34c0d5063b643d97d79da7f421ee423d324f547  
tzdata-2015d-1.el6.noarch.rpm
29c5ce7a9f8b50743ee0d1d665525a2110eb6d6cc9a6e86289614d1b6b34fa34  
tzdata-java-2015d-1.el6.noarch.rpm

x86_64:
17d079889c081ec38565a95ed34c0d5063b643d97d79da7f421ee423d324f547  
tzdata-2015d-1.el6.noarch.rpm
29c5ce7a9f8b50743ee0d1d665525a2110eb6d6cc9a6e86289614d1b6b34fa34  
tzdata-java-2015d-1.el6.noarch.rpm

Source:
36b196f2a6dd2f589917ae18efc38dab3b36849075569f1d0406533add310161  
tzdata-2015d-1.el6.src.rpm



-- 
Johnny Hughes
CentOS Project { http://www.centos.org/ }
irc: hughesjr, #cen...@irc.freenode.net



--

Message: 2
Date: Tue, 28 Apr 2015 21:49:18 +
From: Johnny Hughes 
To: centos-annou...@centos.org
Subject: [CentOS-announce] CESA-2015:0895 Important CentOS 7
389-ds-base Security Update
Message-ID: <20150428214918.ga59...@n04.lon1.karan.org>
Content-Type: text/plain; charset=us-ascii


CentOS Errata and Security Advisory 2015:0895 Important

Upstream details at : https://rhn.redhat.com/errata/RHSA-2015-0895.html

The following updated files have been uploaded and are currently 
syncing to the mirrors: ( sha256sum Filename ) 

x86_64:
0b695f877ea66c6ba7cb15d35f743d2cdb7d26b3c2ad7da7bf5ef716e7dd7a17  
389-ds-base-1.3.3.1-16.el7_1.x86_64.rpm
eaed7a4bf2c0822c4fee1690b818d9bb5184c3e417e6473a663d060c01c83a5d  
389-ds-base-devel-1.3.3.1-16.el7_1.x86_64.rpm
5dd1ceb2c16799fd72ee1d9d79b934bdff6294dbe7e3dc605393701d6dd0ffa1  
389-ds-base-libs-1.3.3.1-16.el7_1.x86_64.rpm

Source:
a5c1801f6621b7d79263c1df4d28afd00ff3ec2f6abc0f2faea373bcfcf70eb3  
389-ds-base-1.3.3.1-16.el7_1.src.rpm



-- 
Johnny Hughes
CentOS Project { http://www.centos.org/ }
irc: hughesjr, #cen...@irc.freenode.net



--

Message: 3
Date: Tue, 28 Apr 2015 21:49:49 +
From: Johnny Hughes 
To: centos-annou...@centos.org
Subject: [CentOS-announce] CEEA-2015:0913 CentOS 7 tzdata Enhancement
Update
Message-ID: <20150428214949.ga59...@n04.lon1.karan.org>
Content-Type: text/plain; charset=us-ascii


CentOS Errata and Enhancement Advisory 2015:0913 

Upstream details at : https://rhn.redhat.com/errata/RHEA-2015-0913.html

The following updated files have been uploaded and are currently 
syncing to the mirrors: ( sha256sum Filename ) 

x86_64:
9926c46800be11c474b10719b9935229019201c195e4debb0da4bfab5ff5ec8e  
tzdata-2015d-1.el7.noarch.rpm
95551f69c3f204b9a7c279698a69b93075f59d82866f7e929b11d8ab60a1fcff  
tzdata-java-2015d-1.el7.noarch.rpm

Source:
c733876d95f1e111fe2cb11a0628b09a10d82f539e299ef2358f11c8d868d3cf  
tzdata-2015d-1.el7.src.rpm



-- 
Johnny Hughes
CentOS Project { http://www.centos.org/ }
irc: hughesjr, #cen...@irc.freenode.net



--

Message: 4
Date: Tue, 28 Apr 2015 21:52:40 +
From: Johnny Hughes 
To: centos-annou...@centos.org
Subject: [CentOS-announce] CEEA-2015:0913 CentOS 5 tzdata Enhancement
Update
Message-ID: <20150428215240.ga4...@chakra.karan.org>
Content-Type: text/plain; charset=us-ascii


CentOS Errata and Enhancement Advisory 2015:0913 

Upstream details at : https://rhn.redhat.com/errata/RHEA-2015-0913.html

The following updated files have been uploaded and are currently 
syncing to the mirrors: ( sha256sum Filename ) 

i386:
8387fba89796124258e17bcea37cc9aa891f272170df2d4ed52b993d9e4eecce  
tzdata-2015d-1.el5.i386.rpm
cab6bde88543bd51b4ba52e7

Re: [CentOS] Mysql 5.6, Centos 7 and errno: 24 - Too many open files

[Johan Kooijman]

Gotz,

This is due to systemd, it overrules your settings. Add a file to systemd
config fixes it:

[root@mysql2 ~]# cat /etc/systemd/system/mariadb.service.d/limits.conf
[Service]
LimitNOFILE=1
LimitMEMLOCK=1


On Wed, Apr 29, 2015 at 8:31 AM, Götz Reinicke - IT Koordinator <
goetz.reini...@filmakademie.de> wrote:

> Hi,
>
> may be somewon has a working solution and information on that:
>
> I installed the most recent mysql community on a server and do get a lot
> of "errno: 24 - Too many open files".
>
> There are suggestions to increase the open_files_limit, change/add that
> to /etc/security/limits.conf and modify the systemd script by hand.
>
> Depending on how you start mysql (restart, or at systemboot time ...)
> the limit get set or not.
>
> Question: Is that may be a known bug or something?
>
> And thats mit a question regarding mariadb or mysql.
>
> Regards . Götz
> --
> Götz Reinicke
> IT-Koordinator
>
> Tel. +49 7141 969 82 420
> E-Mail goetz.reini...@filmakademie.de
>
> Filmakademie Baden-Württemberg GmbH
> Akademiehof 10
> 71638 Ludwigsburg
> www.filmakademie.de
>
> Eintragung Amtsgericht Stuttgart HRB 205016
>
> Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
> Staatssekretär im Ministerium für Wissenschaft,
> Forschung und Kunst Baden-Württemberg
>
> Geschäftsführer: Prof. Thomas Schadt
>
>
> ___
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
>


-- 
Met vriendelijke groeten / With kind regards,
Johan Kooijman
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Mysql 5.6, Centos 7 and errno: 24 - Too many open files

[Carl E. Hartung]

Hi Johan,

Does systemd also overrule /etc/my.conf?

Thx!

Carl

On Wed, 29 Apr 2015 14:58:52 +0200
Johan Kooijman wrote:

> Gotz,
> 
> This is due to systemd, it overrules your settings. Add a file to
> systemd config fixes it:
> 
> [root@mysql2 ~]# cat /etc/systemd/system/mariadb.service.d/limits.conf
> [Service]
> LimitNOFILE=1
> LimitMEMLOCK=1
> 
> 
> On Wed, Apr 29, 2015 at 8:31 AM, Götz Reinicke - IT Koordinator <
> goetz.reini...@filmakademie.de> wrote:
> 
> > Hi,
> >
> > may be somewon has a working solution and information on that:
> >
> > I installed the most recent mysql community on a server and do get
> > a lot of "errno: 24 - Too many open files".
> >
> > There are suggestions to increase the open_files_limit, change/add
> > that to /etc/security/limits.conf and modify the systemd script by
> > hand.
> >
> > Depending on how you start mysql (restart, or at systemboot
> > time ...) the limit get set or not.
> >
> > Question: Is that may be a known bug or something?
> >
> > And thats mit a question regarding mariadb or mysql.
> >
> > Regards . Götz
> > --
> > Götz Reinicke
> > IT-Koordinator
> >
> > Tel. +49 7141 969 82 420
> > E-Mail goetz.reini...@filmakademie.de
> >
> > Filmakademie Baden-Württemberg GmbH
> > Akademiehof 10
> > 71638 Ludwigsburg
> > www.filmakademie.de
> >
> > Eintragung Amtsgericht Stuttgart HRB 205016
> >
> > Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
> > Staatssekretär im Ministerium für Wissenschaft,
> > Forschung und Kunst Baden-Württemberg
> >
> > Geschäftsführer: Prof. Thomas Schadt
> >
> >
> > ___
> > CentOS mailing list
> > CentOS@centos.org
> > http://lists.centos.org/mailman/listinfo/centos
> >
> >
> 
> 

___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

[CentOS] nfs (or tcp or scheduler) changes between centos 5 and 6?

[Matt Garman]

We have a "compute cluster" of about 100 machines that do a read-only
NFS mount to a big NAS filer (a NetApp FAS6280).  The jobs running on
these boxes are analysis/simulation jobs that constantly read data off
the NAS.

We recently upgraded all these machines from CentOS 5.7 to CentOS 6.5.
We did a "piecemeal" upgrade, usually upgrading five or so machines at
a time, every few days.  We noticed improved performance on the CentOS
6 boxes.  But as the number of CentOS 6 boxes increased, we actually
saw performance on the CentOS 5 boxes decrease.  By the time we had
only a few CentOS 5 boxes left, they were performing so badly as to be
effectively worthless.

What we observed in parallel to this upgrade process was that the read
latency on our NetApp device skyrocketed.  This in turn caused all
compute jobs to actually run slower, as it seemed to move the
bottleneck from the client servers' OS to the NetApp.  This is
somewhat counter-intuitive: CentOS 6 performs faster, but actually
results in net performance loss because it creates a bottleneck on our
centralized storage.

All indications are that CentOS 6 seems to be much more "aggressive"
in how it does NFS reads.  And likewise, CentOS 5 was very "polite",
to the point that it basically got starved out by the introduction of
the 6.5 boxes.

What I'm looking for is a "deep dive" list of changes to the NFS
implementation between CentOS 5 and CentOS 6.  Or maybe this is due to
a change in the TCP stack?  Or maybe the scheduler?  We've tried a lot
of sysctl tcp tunings, various nfs mount options, anything that's
obviously different between 5 and 6... But so far we've been unable to
find the "smoking gun" that causes the obvious behavior change between
the two OS versions.

Just hoping that maybe someone else out there has seen something like
this, or can point me to some detailed documentation that might clue
me in on what to look for next.

Thanks!
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] nfs (or tcp or scheduler) changes between centos 5 and 6?

[m . roth]

Matt Garman wrote:
> We have a "compute cluster" of about 100 machines that do a read-only
> NFS mount to a big NAS filer (a NetApp FAS6280).  The jobs running on
> these boxes are analysis/simulation jobs that constantly read data off
> the NAS.
>
> We recently upgraded all these machines from CentOS 5.7 to CentOS 6.5.
> We did a "piecemeal" upgrade, usually upgrading five or so machines at
> a time, every few days.  We noticed improved performance on the CentOS
> 6 boxes.  But as the number of CentOS 6 boxes increased, we actually
> saw performance on the CentOS 5 boxes decrease.  By the time we had
> only a few CentOS 5 boxes left, they were performing so badly as to be
> effectively worthless.
>
> What we observed in parallel to this upgrade process was that the read
> latency on our NetApp device skyrocketed.  This in turn caused all
> compute jobs to actually run slower, as it seemed to move the
> bottleneck from the client servers' OS to the NetApp.  This is
> somewhat counter-intuitive: CentOS 6 performs faster, but actually
> results in net performance loss because it creates a bottleneck on our
> centralized storage.

*IF* I understand you, I've got one question: what parms are you using to
mount the storage? We had *real* performance problems when we went from 5
to 6 - as in, unzipping a 26M file to 107M, while writing to an
NFS-mounted drive, went from 30 sec or so to a *timed* 7 min. The final
answer was that once we mounted the NFS filesystem with nobarrier in fstab
instead of default, the time dropped to 35 or 40 sec again.

barrier is in 6, and tries to make writes atomic transactions; its intent
is to protect in case of things like power failure. Esp. if you're on
UPSes, nobarrier is the way to go.

   mark

___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] nfs (or tcp or scheduler) changes between centos 5 and 6?

[Devin Reade]

--On Wednesday, April 29, 2015 08:35:29 AM -0500 Matt Garman 
 wrote:



All indications are that CentOS 6 seems to be much more "aggressive"
in how it does NFS reads.  And likewise, CentOS 5 was very "polite",
to the point that it basically got starved out by the introduction of
the 6.5 boxes.


Some things come to mind as far as investigating differences; you don't
have to answer them all here; just making sure you've covered them all:

Have you looked at the client-side NFS cache?  Perhaps the C6 cache
is either disabled, has fewer resources, or is invalidating faster?
(I don't think that would explain the C5 starvation, though, unless
it's a secondary effect from retransmits, etc.)

Regarding the cache, do you have multiple mount points on a client
that resolve to the same server filesystem?  If so, do they have
different mount options?  If so, that can result in multiple caches
instead of a single disk cache.  The client cache can also be bypassed
if your application is doing direct I/O on the files.  Perhaps there
is a difference in the application between C5 and C6, including
whether or not it was just recompiled?  (If so, can you try a C5 version
on the C6 machines?)

If you determine that C6 is doing aggressive caching, does this match
the needs of your application?  That is, do you have the situation
where the client NFS layer does an aggressive read-ahead that is never
used by the application?

Are C5 and C6 using the same NFS protocol version?  How about TCP vs
UDP?  If UDP is in play, have a look at fragmentation stats under load.

Are both using the same authentication method (ie: maybe just
UID-based)?

And, like always, is DNS sane for all your clients and servers?  Everything
(including clients) has proper PTR records, consistent with A records,
et al?  DNS is so fundamental to everything that if it is out of whack
you can get far-reaching symptoms that don't seem to have anything to do
with DNS.

 has helpful information about enabling debug
output on the client end to see what is going on.  I don't know in your
situation if enabling server-side debugging is feasible.
 also has useful tuning information.

You may want to look at NFSometer and see if it can help.

Devin

___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] nfs (or tcp or scheduler) changes between centos 5 and 6?

[James Pearson]

m.r...@5-cent.us wrote:

Matt Garman wrote:


We have a "compute cluster" of about 100 machines that do a read-only
NFS mount to a big NAS filer (a NetApp FAS6280).  The jobs running on
these boxes are analysis/simulation jobs that constantly read data off
the NAS.



*IF* I understand you, I've got one question: what parms are you using to
mount the storage? We had *real* performance problems when we went from 5
to 6 - as in, unzipping a 26M file to 107M, while writing to an
NFS-mounted drive, went from 30 sec or so to a *timed* 7 min. The final
answer was that once we mounted the NFS filesystem with nobarrier in fstab
instead of default, the time dropped to 35 or 40 sec again.

barrier is in 6, and tries to make writes atomic transactions; its intent
is to protect in case of things like power failure. Esp. if you're on
UPSes, nobarrier is the way to go.


The server in this case isn't a Linux box with an ext4 file system - so 
that won't help ...


James Pearson
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] nfs (or tcp or scheduler) changes between centos 5 and 6?

[m . roth]

James Pearson wrote:
> m.r...@5-cent.us wrote:
>> Matt Garman wrote:
>>
>>>We have a "compute cluster" of about 100 machines that do a read-only
>>>NFS mount to a big NAS filer (a NetApp FAS6280).  The jobs running on
>>>these boxes are analysis/simulation jobs that constantly read data off
>>>the NAS.
>>
>> 
>> *IF* I understand you, I've got one question: what parms are you using
>> to mount the storage? We had *real* performance problems when we went from
>> 5 to 6 - as in, unzipping a 26M file to 107M, while writing to an
>> NFS-mounted drive, went from 30 sec or so to a *timed* 7 min. The final
>> answer was that once we mounted the NFS filesystem with nobarrier in
>> fstab instead of default, the time dropped to 35 or 40 sec again.
>>
>> barrier is in 6, and tries to make writes atomic transactions; its
>> intent is to protect in case of things like power failure. Esp. if
you're on
>> UPSes, nobarrier is the way to go.
>
> The server in this case isn't a Linux box with an ext4 file system - so
> that won't help ...
>
What kind of filesystem is it? I note that xfs also has barrier as a mount
option.

   mark

___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] CENTOS not DoD approved

[Akemi Yagi]

On Tue, Apr 28, 2015 at 4:05 PM, Akemi Yagi  wrote:

> Incidentally, someone has just started a thread related to DoD in the
> RH community discussion session entitled, "A DoD version of RHEL - A
> money maker for RH? Maybe!" :
>
> https://access.redhat.com/comment/913243

A new comment has been posted by a person who is "one of the ones who
writes the STIGs for Red Hat, working out of Red Hat's U.S. Public
Sector group":

https://access.redhat.com/comment/913583#comment-913583

Akemi
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] nfs (or tcp or scheduler) changes between centos 5 and 6?

[Matt Garman]

On Wed, Apr 29, 2015 at 10:36 AM, Devin Reade  wrote:
> Have you looked at the client-side NFS cache?  Perhaps the C6 cache
> is either disabled, has fewer resources, or is invalidating faster?
> (I don't think that would explain the C5 starvation, though, unless
> it's a secondary effect from retransmits, etc.)

Do you know where the NFS cache settings are specified?  I've looked
at the various nfs mount options.  Anything cache-related appears to
be the same between the two OSes, assuming I didn't miss anything.  We
did experiment with the "noac" mount option, though that had no effect
in our tests.

FWIW, we've done a tcpdump on both OSes, performing the same tasks,
and it appears that 5 actually has more "chatter".  Just looking at
packet counts, 5 has about 17% more packets than 6, for the same
workload.  I haven't dug too deep into the tcpdump files, since we
need a pretty big workload to trigger the measurable performance
discrepancy.  So the resulting pcap files are on the order of 5 GB.

> Regarding the cache, do you have multiple mount points on a client
> that resolve to the same server filesystem?  If so, do they have
> different mount options?  If so, that can result in multiple caches
> instead of a single disk cache.  The client cache can also be bypassed
> if your application is doing direct I/O on the files.  Perhaps there
> is a difference in the application between C5 and C6, including
> whether or not it was just recompiled?  (If so, can you try a C5 version
> on the C6 machines?)

No multiple mount points to the same server.

No application differences.  We're still compiling on 5, regardless of
target platform.

> If you determine that C6 is doing aggressive caching, does this match
> the needs of your application?  That is, do you have the situation
> where the client NFS layer does an aggressive read-ahead that is never
> used by the application?

That was one of our early theories.  On 6, you can adjust this via
/sys/class/bdi/X:Y/read_ahead_kb (use stat on the mountpoint to
determine X and Y).  This file doesn't exist on 5.  But we tried
increasing and decreasing it from the default (960), and didn't see
any changes.

> Are C5 and C6 using the same NFS protocol version?  How about TCP vs
> UDP?  If UDP is in play, have a look at fragmentation stats under load.

Yup, both are using tcp, protocol version 3.

> Are both using the same authentication method (ie: maybe just
> UID-based)?

Yup, sec=sys.

> And, like always, is DNS sane for all your clients and servers?  Everything
> (including clients) has proper PTR records, consistent with A records,
> et al?  DNS is so fundamental to everything that if it is out of whack
> you can get far-reaching symptoms that don't seem to have anything to do
> with DNS.

I believe so.  I wouldn't bet my life on it.  But there were certainly
no changes to our DNS before, during or since the OS upgrade.

> You may want to look at NFSometer and see if it can help.

Haven't seen that, will definitely give it a try!

Thanks for your thoughts and suggestions!
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] nfs (or tcp or scheduler) changes between centos 5 and 6?

[Matt Garman]

On Wed, Apr 29, 2015 at 10:51 AM,   wrote:
>> The server in this case isn't a Linux box with an ext4 file system - so
>> that won't help ...
>>
> What kind of filesystem is it? I note that xfs also has barrier as a mount
> option.

The server is a NetApp FAS6280.  It's using NetApp's filesystem.  I'm
almost certain it's none of the common Linux ones.  (I think they call
it WAFL IIRC.)

Either way, we do the NFS mount read-only, so write barriers don't
even come into play.  E.g., with your original example, if we unzipped
something, we'd have to write to the local disk.

Furthermore, in "low load" situations, the NetApp read latency stays
low, and the 5/6 performance is fairly similar.  It's only when the
workload gets high, and it turn this "aggressive" demand is placed on
the NetApp, that we in turn see overall decreased performance.

Thanks for the thoughts!
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Mysql 5.6, Centos 7 and errno: 24 - Too many open files

[Johan Kooijman]

Carl,

By default my.cnf has to obey the OS limits, so in this case the order is:
systemd > /etc/security/limits* > /etc/my*.

On Wed, Apr 29, 2015 at 3:22 PM, Carl E. Hartung 
wrote:

> Hi Johan,
>
> Does systemd also overrule /etc/my.conf?
>
> Thx!
>
> Carl
>
> On Wed, 29 Apr 2015 14:58:52 +0200
> Johan Kooijman wrote:
>
> > Gotz,
> >
> > This is due to systemd, it overrules your settings. Add a file to
> > systemd config fixes it:
> >
> > [root@mysql2 ~]# cat /etc/systemd/system/mariadb.service.d/limits.conf
> > [Service]
> > LimitNOFILE=1
> > LimitMEMLOCK=1
> >
> >
> > On Wed, Apr 29, 2015 at 8:31 AM, Götz Reinicke - IT Koordinator <
> > goetz.reini...@filmakademie.de> wrote:
> >
> > > Hi,
> > >
> > > may be somewon has a working solution and information on that:
> > >
> > > I installed the most recent mysql community on a server and do get
> > > a lot of "errno: 24 - Too many open files".
> > >
> > > There are suggestions to increase the open_files_limit, change/add
> > > that to /etc/security/limits.conf and modify the systemd script by
> > > hand.
> > >
> > > Depending on how you start mysql (restart, or at systemboot
> > > time ...) the limit get set or not.
> > >
> > > Question: Is that may be a known bug or something?
> > >
> > > And thats mit a question regarding mariadb or mysql.
> > >
> > > Regards . Götz
> > > --
> > > Götz Reinicke
> > > IT-Koordinator
> > >
> > > Tel. +49 7141 969 82 420
> > > E-Mail goetz.reini...@filmakademie.de
> > >
> > > Filmakademie Baden-Württemberg GmbH
> > > Akademiehof 10
> > > 71638 Ludwigsburg
> > > www.filmakademie.de
> > >
> > > Eintragung Amtsgericht Stuttgart HRB 205016
> > >
> > > Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
> > > Staatssekretär im Ministerium für Wissenschaft,
> > > Forschung und Kunst Baden-Württemberg
> > >
> > > Geschäftsführer: Prof. Thomas Schadt
> > >
> > >
> > > ___
> > > CentOS mailing list
> > > CentOS@centos.org
> > > http://lists.centos.org/mailman/listinfo/centos
> > >
> > >
> >
> >
>
> ___
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>



-- 
Met vriendelijke groeten / With kind regards,
Johan Kooijman
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

[CentOS] Cron Issues

[Matt]

I have noanacron installed on a fresh centos 7 install.

I added this too settings.

nano /etc/cron.d/0hourly

*/5 * * * * root run-parts /etc/cron.fiveminutes
*/1 * * * * root run-parts /etc/cron.minute
0,30 * * * * root run-parts /etc/cron.halfhour

and then created the directories for it.  Now I keep getting these
errors in secure log.

 pam_systemd(crond:session): Failed to create session: Did not
receive a reply. Possible causes include: the remote application did
not send a reply, the message bus security policy blocked the reply,
the reply timeout expired, or the network connection was broken.

Can anyone tell me how to resolve it?
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Cron Issues

[Eero Volotinen]

Check selinux context for directory?
30.4.2015 12.19 ap. "Matt"  kirjoitti:

> I have noanacron installed on a fresh centos 7 install.
>
> I added this too settings.
>
> nano /etc/cron.d/0hourly
>
> */5 * * * * root run-parts /etc/cron.fiveminutes
> */1 * * * * root run-parts /etc/cron.minute
> 0,30 * * * * root run-parts /etc/cron.halfhour
>
> and then created the directories for it.  Now I keep getting these
> errors in secure log.
>
>  pam_systemd(crond:session): Failed to create session: Did not
> receive a reply. Possible causes include: the remote application did
> not send a reply, the message bus security policy blocked the reply,
> the reply timeout expired, or the network connection was broken.
>
> Can anyone tell me how to resolve it?
> ___
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] Mysql 5.6, Centos 7 and errno: 24 - Too many open files

[carlh04426]

Thank you for clarifying this, Johan. Very much appreciated!

On Wed, 29 Apr 2015 22:28:00 +0200
Johan Kooijman wrote:

> Carl,
> 
> By default my.cnf has to obey the OS limits, so in this case the
> order is: systemd > /etc/security/limits* > /etc/my*.
> 
> On Wed, Apr 29, 2015 at 3:22 PM, Carl E. Hartung
>  wrote:
> 
> > Hi Johan,
> >
> > Does systemd also overrule /etc/my.conf?
> >
> > Thx!
> >
> > Carl
> >
> > On Wed, 29 Apr 2015 14:58:52 +0200
> > Johan Kooijman wrote:
> >
> > > Gotz,
> > >
> > > This is due to systemd, it overrules your settings. Add a file to
> > > systemd config fixes it:
> > >
> > > [root@mysql2 ~]#
> > > cat /etc/systemd/system/mariadb.service.d/limits.conf [Service]
> > > LimitNOFILE=1
> > > LimitMEMLOCK=1
> > >
> > >
> > > On Wed, Apr 29, 2015 at 8:31 AM, Götz Reinicke - IT Koordinator <
> > > goetz.reini...@filmakademie.de> wrote:
> > >
> > > > Hi,
> > > >
> > > > may be somewon has a working solution and information on that:
> > > >
> > > > I installed the most recent mysql community on a server and do
> > > > get a lot of "errno: 24 - Too many open files".
> > > >
> > > > There are suggestions to increase the open_files_limit,
> > > > change/add that to /etc/security/limits.conf and modify the
> > > > systemd script by hand.
> > > >
> > > > Depending on how you start mysql (restart, or at systemboot
> > > > time ...) the limit get set or not.
> > > >
> > > > Question: Is that may be a known bug or something?
> > > >
> > > > And thats mit a question regarding mariadb or mysql.
> > > >
> > > > Regards . Götz
> > > > --
> > > > Götz Reinicke
> > > > IT-Koordinator
> > > >
> > > > Tel. +49 7141 969 82 420
> > > > E-Mail goetz.reini...@filmakademie.de
> > > >
> > > > Filmakademie Baden-Württemberg GmbH
> > > > Akademiehof 10
> > > > 71638 Ludwigsburg
> > > > www.filmakademie.de
> > > >
> > > > Eintragung Amtsgericht Stuttgart HRB 205016
> > > >
> > > > Vorsitzender des Aufsichtsrats: Jürgen Walter MdL
> > > > Staatssekretär im Ministerium für Wissenschaft,
> > > > Forschung und Kunst Baden-Württemberg
> > > >
> > > > Geschäftsführer: Prof. Thomas Schadt
> > > >
> > > >
> > > > ___
> > > > CentOS mailing list
> > > > CentOS@centos.org
> > > > http://lists.centos.org/mailman/listinfo/centos
> > > >
> > > >
> > >
> > >
> >
> > ___
> > CentOS mailing list
> > CentOS@centos.org
> > http://lists.centos.org/mailman/listinfo/centos
> >
> 
> 
> 

___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] nfs (or tcp or scheduler) changes between centos 5 and 6?

[Dennis Jacobfeuerborn]

>> You may want to look at NFSometer and see if it can help.
> 
> Haven't seen that, will definitely give it a try!

Try "nfsstat -cn" on the clients to see if any particular NFS operations
occur more or less frequently on the C6 systems.

Also look at the "lookupcache" option found in "man nfs":

lookupcache=mode
Specifies how the kernel manages its cache of directory entries for a
given mount point. mode can be one of all, none, pos, or positive.  This
option is supported in kernels 2.6.28 and later.
(there is more text in the man page)

Since C5 came with 2.6.18 and C6 with 2.6.32 this might have something
to do with it.

Regards,
  Dennis

___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos