Hi Peter, I appreciate the hierarchical nature of the data_format tag. Perhaps the example of taking two bytes was highly unfortunate given that the field is logically split in 20+12 bits.
Of course I share your view on handling unknowns in a future sFlow version and the recommendation to developers of structures. Indeed the '0xFFFFFFFF' approach can result the least impacting - except imagine somebody does sanity checks on those fields, which i guess it's why you dropped an email in the first place. As said, I was focusing only on the case in which enterprise value is 0 leaving in the hands of specific developers, enterprise value != 0, to find a way to meaningfully handle unknowns - given that likely they will be in control of both the agent and the collector. While thinking at an acceptable trick on the data_format tag, i stumbled upon the 32 bit length field just besides that. Do you see a chance of recommending carving part of that field (despite its name) for the bitmap - when enterprise value equals to zero? At this propo, I couldn't find any explicit reference to what is thought to be the maximum size of an sFlow datagram; did i miss such indication or should that be implicitely inferred? Cheers, Paolo On Sat, May 23, 2009 at 09:13:30AM -0700, Peter Phaal wrote: > It would be nice to have a general purpose "unknown" mechanism and the use > of a bitmap to indicate which elements in a structure fits well with XDR > encoding. > > However, your proposal to take the upper two bytes from the data_format > field has some problems. The sFlow version 5 data_format field is fully > defined. Your proposed change would not be backward compatible for sFlow > collectors since they would need to be altered to ignore the bits (or use > them to mask the structure). > > The data_format uniquely identifies the format of an opaque structure in > the sFlow specification. A data_format is contructed as follows: > - The most significant 20 bits correspond to the SMI Private Enterprise > Code of the entity responsible for the structure definition. A value > of zero is used to denote standard structures defined by sflow.org. > - The least significant 12 bits are a structure format number assigned > by the enterprise that should uniquely identify the the format of the > structure. > > I just checked IANA and the largest currently assigned enterprise number is > now 33572. Using the upper two bytes of the data_format would overlap with > enterprise codes, breaking vendor extensibility - an important feature of > sFlow version 5. > > I think you may have misunderstood the role of the structure number in the > data_format tag. The entire data_format element (32 bits) is treated as a > unique structure id. The allocation scheme is hierarchical, providing each > enterprise publishing structures with a full 12 bits that they can assign as > they choose (i.e. the structure number by itself does not uniquely > distinguish a structure, it must be combined with the assigning entity to > provide a unique id). > > Currently, sFlow, version 5 defines unknown fields as follows: > The following values should be used for fields that are > unknown (unless otherwise indicated in the structure > definitions). > - Unknown integer value. Use a value of 0 to indicate that > a value is unknown. > - Unknown counter. Use the maximum counter value to indicate > that the counter is not available. Within any given sFlow > session a particular counter must be always available, or > always unavailable. An available counter may temporarily > have the max value just before it rolls to zero. This is > permitted. > - Unknown string. Use the zero length empty string. */ > > Of course there is also the option of omitting a structure all together to > indicate that none of its fields are known to the agent. > > These rules work reasonably well, except for the case of the extended_switch > structure. By "otherwise indicating" the unknown values as 0xFFFFFFFF the > proposal is the smallest possible change that should be backward compatible > for existing agents and collectors. > > If an sFlow version 6 is ever defined then your suggestion of using a bit > array to provide a general unknown value mechanism would be an attractive > extension. > > As new structures are defined for sFlow version 5, adopting a convention in > which the first member of the structure is a bit array indicating any > unknown elements in the structure would be something that authors of > structures should consider if there is any possibility that elements of the > structure might not be known. > > Peter > > > -----Original Message----- > > From: Paolo Lucente [mailto:pl+l...@pmacct.net] > > Sent: Saturday, May 23, 2009 4:59 AM > > To: Peter Phaal > > Cc: sflow@sflow.org > > Subject: Re: [sFlow] Encoding unknown values in the > > extended_switchstructure > > > > Hi Peter, > > > > Not really an objection - rather a proposal for your consideration. > > > > What about bitmapping fields contained in standard elements within, > > say, the upper two bytes of the element tag? > > > > typedef struct _SFLFlow_sample_element { > > struct _SFLFlow_sample_element *nxt; > > u_int32_t tag; /* SFLFlow_type_tag */ <----- > > u_int32_t length; > > SFLFlow_type flowType; > > } SFLFlow_sample_element; > > > > ie. > > > > #define EX_SWITCH_SRC_VLAN 0x00010000 > > #define EX_SWITCH_SRC_PRIORITY 0x00020000 > > #define EX_SWITCH_DST_VLAN 0x00040000 > > #define EX_SWITCH_DST_PRIORITY 0x00080000 > > > > It would have the beauty of being backward compatible, meaning the > > collector can just ignore it and not take benefit from it, and will > > maintain intact the 32 bit field - which is not a requirement today, > > for this specific case, but you might be less lucky tomorrow trying > > to do the same trick elsewhere; so a more coherent approach. > > > > Companies developing closed extensions and thus making use of the > > upper bytes of the tag field, ie. > > > > /* enterprise = 4300 (inmon)...*/ > > SFLFLOW_EX_PROCESS = (4300 << 12) + 3 > > > > might (or will need to) use a different way to signal some fields > > are not populated within their elements. > > > > The only con of this approach is that it would put an upper bound > > to the number of fields that can be packed within an element, ie. > > 15 fields if using the upper 2 bytes (keeping out the '0x00000000' > > value), but i see you have never been a big fan of flat structures > > ( :-) ) so i wouldn't see this as a true biggie. Quickly scanning > > through the existing elements, all seem to fit this approach. > > > > Cheers, > > Paolo > > > > > > On Fri, May 22, 2009 at 11:07:55AM -0700, Peter Phaal wrote: > > > There currently isn't a satisfactory way to indicate that one or more of > > the > > > fields in the extended_switch structure is unknown to the sFlow > > exporter. > > > > > > /* Extended Switch Data */ > > > /* opaque = flow_data; enterprise = 0; format = 1001 */ > > > /* Note: For untagged ingress ports, use the assigned vlan and priority > > > of the port for the src_vlan and src_priority values. > > > For untagged egress ports, use the values for dst_vlan and > > > dst_priority that would have been placed in the 802.Q tag > > > had the egress port been a tagged member of the VLAN instead > > > of an untagged member. */ > > > > > > struct extended_switch { > > > unsigned int src_vlan; /* The 802.1Q VLAN id of incoming frame */ > > > unsigned int src_priority; /* The 802.1p priority of incoming frame > > */ > > > unsigned int dst_vlan; /* The 802.1Q VLAN id of outgoing frame */ > > > unsigned int dst_priority; /* The 802.1p priority of outgoing frame > > */ > > > } > > > > > > Since 0 is a valid value for both 802.1Q VLAN 802.1p priority it should > > not > > > be used to encode the unknown value. Rather than omit the whole > > structure > > > because one field isn't known, it makes sense to define an unambiguous > > value > > > to indicate that a field is unknown. > > > > > > Fortunately these fields have relatively small maximum values, vlan <= > > 4096 > > > and priority <= 7, so an unambiguous choice for an unknown value would > > be > > > 0xffffffff (Note: this same value that sFlow currently uses to indicate > > that > > > a 32 bit counter value is unknown). If there are no objections, I think > > it > > > would make sense to include a note to this effect on Specifications page > > on > > > sFlow.org. > > > > > > Peter
