Here are some ideas:
1. What could we implement in the strategy so its not continually
open()/close() the jrb files?
a) On solaris 10, I can have 65535 file descriptors open.
b) How much memory does jrobin require from the jvm for each open jrb
file?
c) If we can't keep them all open, can we pick a strategy to keep some
# of them open all the time?
2. Are there other strategies we could use for consolidating data into fewer
jrb files?
a) ex: network devices with a fixed number of interfaces on a blade.
Instead of storing them as $NODE/$INTF/$GROUP.jrb could we restructure the
storage to put the data into fewer files? Could we have
$NODE/netif/$BLADE/$GROUP.jrb and then map the
3. Implement an add/remove datasource feature where a jrb file would
grow/shrink based on the operation? So we could then place all data with the
same RRA setups into fewer files?
a) Replace ds.properties with a ds.xml with entries like below, and
place more data inside fewer files. Also store the RRA definitions to make it
easier to add/remove datasources with the same configurations. Include limiting
number of datasources per file?
<datasource id="1" name="ifHCInOctets" group="mib2-X-interfaces" file="ds1.jrb">
<datasource id="2" name="ifHCOutOctets" group="mib2-X-interfaces"
file="ds1.jrb">
<datasource id="3" name="ifInDiscards" group="mib2-interfaces" file="ds1.jrb">
<datasource id="4" name="ifInErrors" group="mib2-interfaces" file="ds1.jrb">
<datasource id="5" name="ifInNUcastpkts" group="mib2-interfaces" file="ds1.jrb">
<datasource id="6" name="ifInOctets" group="mib2-interfaces" file="ds1.jrb">
<datasource id="7" name="ifInUcastpkts" group="mib2-interfaces" file="ds1.jrb">
<datasource id="8" name="ifOutDiscards" group="mib2-interfaces" file="ds1.jrb">
<datasource id="9" name="ifOutErrors" group="mib2-interfaces" file="ds1.jrb">
<datasource id="10" name="ifOutNUcastPkts" group="mib2-interfaces"
file="ds1.jrb">
<datasource id="11" name="ifOutOctets" group="mib2-interfaces" file="ds1.jrb">
<datasource id="12" name="ifOutUcastPkts" group="mib2-interfaces"
file="ds1.jrb">
<datasource id="13" name="locIfInPktsSec" group="cisco-router-interface"
file="ds1.jrb">
<datasource id="14" name="locIfInQueueDrops" group="cisco-router-interface"
file="ds1.jrb">
<datasource id="15" name="locIfOutPktsSec" group="cisco-router-interface"
file="ds1.jrb">
<datasource id="16" name="locIfOutQueueDrops" group="cisco-router-interface"
file="ds1.jrb">
4. On the JRobin front:
1. Reformat the file layout so you're not having to seek through the
whole file to get a grasp of how the data is laid out.
2. Place the bytes that get updated the most together.
(ArcState.accumValue, ArcState.nansteps, Robin.pointer,
Datasource[i].nanSeconds, Datasource[i].accumValue, Header. lastUpdateTime)
-----Original Message-----
From: Michael Seibold [mailto:[email protected]]
Sent: Thursday, June 09, 2011 6:05 AM
To: OpenNMS Code Development and Bugs
Subject: [opennms-devel] Antw: Re: jrobin file format
Hi Ronald,
as I'm also always seeking for improvements in this area I am very interested
in optimization of data collection.
...
> With org.jrobin.core.RrdBackendFactory=MNIO, totalOperationsPending
varied between 30,000 and 45,000. CPU load average was around 8, 21-30%
utilization.
As I understood MNIO basically buffers I/O requests to avoid seeking.
As you have everything in memory there I see no reason why MNIO should speed up
the write process.
> I’m a little dismayed that OpenNMS can’t get its queued rrd
operations down to 0 when its basically got no disk IO traffic to handle.
Well, maybe the queuing writes only when a complete "block" is filled with
data, This would explain this behaviour. As long as the block isn't full it
won't be written, even if there is no ohter i/o traffic.
> I think you might be able to fix some of this with an improved file
format, but I think for long term scalability what’s needed is to have a
RrdStrategy that opens fewer larger files, maybe 1 per node?. But after looking
at the code for OpenNMS and JRobin, I don’t think that’s an easy task since too
much smarts about the file layouts are handled by the datacollection classes.
I still hope that soon the SSD disks will be cheap AND performant enough
(number of rewrites...) to handle this without any problems.
Unfortunally I have no possibility to test this at the moment.
- Michael
Ihr starker Gesundheitspartner – die BARMER GEK.
Auch 2011 gilt: kein Zusatzbeitrag! www.barmer-gek.de
Diese Nachricht der BARMER GEK kann vertrauliche firmeninterne Informationen
enthalten. Sofern Sie nicht der beabsichtigte Empfänger sind, bitten wir Sie,
den Absender zu informieren und die Nachricht sowie deren Anhänge zu löschen.
Unzulässige Veröffentlichung, Verwendung, Verbreitung, Weiterleitung und das
Kopieren dieser Mail und ihrer verknüpften Anhänge sind nicht gestattet.
------------------------------------------------------------------------------
EditLive Enterprise is the world's most technically advanced content authoring
tool. Experience the power of Track Changes, Inline Image Editing and ensure
content is compliant with Accessibility Checking.
http://p.sf.net/sfu/ephox-dev2dev
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ
opennms-devel mailing list
To *unsubscribe* or change your subscription options, see the bottom of this
page:
https://lists.sourceforge.net/lists/listinfo/opennms-devel
This e-mail message is being sent solely for use by the intended recipient(s)
and may contain confidential information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by phone or reply by e-mail, delete the
original message and destroy all copies. Thank you.
------------------------------------------------------------------------------
EditLive Enterprise is the world's most technically advanced content
authoring tool. Experience the power of Track Changes, Inline Image
Editing and ensure content is compliant with Accessibility Checking.
http://p.sf.net/sfu/ephox-dev2dev
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ
opennms-devel mailing list
To *unsubscribe* or change your subscription options, see the bottom of this
page:
https://lists.sourceforge.net/lists/listinfo/opennms-devel
