Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

-Original Message- From: wireshark-dev-boun...@wireshark.org [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of Guy Harris Sent: den 13 maj 2013 08:11 To: Developer support list for Wireshark Subject: Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13 On May 12, 2013, at 10:

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On May 12, 2013, at 11:07 PM, Anders Broman wrote: > Guy Harris skrev 2013-05-11 19:45: >> On May 11, 2013, at 7:28 AM, Ambarisha B wrote: >> >>> I was trying to move the reassembled data from the reassembled_table to >>> frame_data. > > If the pointer to the reassembled data is stored in t

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On May 12, 2013, at 10:57 PM, Anders Broman wrote: > So either this is a bad idea, don't pursue it or everything we can read we > should be able to write in pcap-ng > as a temporary file at least. ...or if we're going to store the contents of packet *not* just generated by gluing together stu

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

Guy Harris skrev 2013-05-11 19:45: On May 11, 2013, at 7:28 AM, Ambarisha B wrote: I was trying to move the reassembled data from the reassembled_table to frame_data. If the pointer to the reassembled data is stored in the per_packet_data in frame_data the memory usage should work out to

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

Guy Harris skrev 2013-05-12 00:52: On May 2, 2013, at 9:25 AM, Anders Broman wrote: Alternatively if the file is rewritten on the first pass writing the reassembled data together with the packet holding the last fragment the reassembled data could be read together with the packet But that mi

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On Sun, May 12, 2013 at 4:18 AM, Guy Harris wrote: > I dropped the attempt in question for now. > > A from-scratch rewrite might called for here. Can this be done this summer itself? As I see there's no point in improving the current state of things (filebacked -reassembly etc), if sooner or la

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On May 2, 2013, at 9:25 AM, Anders Broman wrote: > Alternatively if the file is rewritten on the first pass writing the > reassembled data together with the packet holding the last fragment the > reassembled data could be read together with the packet > But that might have other problems and m

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On Apr 18, 2013, at 10:28 AM, Evan Huus wrote: > That hadn't even occurred to me; I was thinking more of the fact that > we don't have a separate 'head' structure for reassembly chains and > just assume certain fields are set/unset based on whether the > structure is first in the list or not. I

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On May 11, 2013, at 7:28 AM, Ambarisha B wrote: > I was trying to move the reassembled data from the reassembled_table to > frame_data. Moving stuff to frame_data is something that should only be done as a last resort if it's being done as part of a project to reduce memory usage. :-) *Every

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

Hi, I was trying to move the reassembled data from the reassembled_table to frame_data. The key to the reassembled_table is (framenum, ID). I am not able to figure out what ID is for. In a capture file, frame number itself would be unique, wouldn't it? Cheers Ambarisha ___

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

From: wireshark-dev-boun...@wireshark.org [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of Ambarisha B Sent: den 3 maj 2013 11:38 To: Developer support list for Wireshark Subject: Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13 On Thu, May 2, 2013 at 9:55 PM, Anders B

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On Thu, May 2, 2013 at 9:55 PM, Anders Broman wrote: > Hi, > > I’m not sure I get you, it currently works something like this on the > first pass: > > A packet is read from the file, the frame data structure is built and the > file pointer is preserved in fdata, a tvb is built with the pa

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

dev-boun...@wireshark.org [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of Ambarisha B Sent: den 2 maj 2013 17:33 To: Developer support list for Wireshark Subject: Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13 On Thu, May 2, 2013 at 5:17 PM, Anders Broman mailto:anders.bro...@ericsson.com>

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On Thu, May 2, 2013 at 5:17 PM, Anders Broman wrote: > > Just keep all the info in temporary files and clean up the files when > free'ing the tvb's. In this case, can we use the wiretap to deal with the > temporary files as well? > > The tvb:s only “live” in packet scope so what we need is file b

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

From: wireshark-dev-boun...@wireshark.org [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of Ambarisha B Sent: den 2 maj 2013 11:26 To: Developer support list for Wireshark Subject: Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13 >On Wed, May 1, 2013 at 9:46 PM, Ander

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On Wed, May 1, 2013 at 9:46 PM, Anders Broman wrote: > It may be problematic to obtain the fragments from the original file in > case it is gziped or if the fragments are > parts of decrypted packets so writing to a new file might be the best > option. > Agreed. Jeff suggested that we've decentl

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On 05/01/13 12:16, Anders Broman wrote: Ambarisha B skrev 2013-05-01 17:01: On Thu, Apr 25, 2013 at 7:34 PM, Jeff Morriss mailto:jeff.morriss...@gmail.com>> wrote: [...] >What will this data member contain in case of a reassembly implementation which uses file backed tvbuffs? file pointer(s)

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On 05/01/13 11:01, Ambarisha B wrote: On Thu, Apr 25, 2013 at 7:34 PM, Jeff Morriss mailto:jeff.morriss...@gmail.com>> wrote: But the dissectors create the tvb from the reassembled data. So, if we define a new type of tvbuff which has to be used, wouldn't that eff

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

Ambarisha B skrev 2013-05-01 17:01: On Thu, Apr 25, 2013 at 7:34 PM, Jeff Morriss mailto:jeff.morriss...@gmail.com>> wrote: But the dissectors create the tvb from the reassembled data. So, if we define a new type of tvbuff which has to be used, wouldn't that effe

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On Thu, Apr 25, 2013 at 7:34 PM, Jeff Morriss wrote: > But the dissectors create the tvb from the reassembled data. So, if we >> define a new type of tvbuff which has to be used, wouldn't that effect >> the dissectors? >> > > It shouldn't: dissectors should be using the TVB-accessor functions so >

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On 04/20/13 17:13, Ambarisha B wrote: Hi Anders, On Sun, Apr 21, 2013 at 2:05 AM, Anders Broman mailto:a.bro...@bredband.net>> wrote: Not necessarily if we can hide the fact in the tvb or reassembly functions. But the dissectors create the tvb from the reassembled data. So, if we defi

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On Sun, Apr 21, 2013 at 2:43 AM, Ambarisha B wrote: > I understand that feasibility could be a concern in the sense that "we cut > down memory usage without *any* loss in performance* because that would > depend on how well we can tune the cache. I was thinking that the file > backing the tvbuff

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

Hi Anders, On Sun, Apr 21, 2013 at 2:05 AM, Anders Broman wrote: > Not necessarily if we can hide the fact in the tvb or reassembly > functions. > But the dissectors create the tvb from the reassembled data. So, if we define a new type of tvbuff which has to be used, wouldn't that effect the di

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

Ambarisha B skrev 2013-04-19 23:01: Hi, On Thu, Apr 18, 2013 at 11:31 PM, Ambarisha B > wrote: I'll also see if I can get a profile from massif as Evan suggested. I tried out massif on wireshark today. I just profiled on a web-browsing session capture file

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On Apr 18, 2013, at 11:42 AM, Anders Broman wrote: > This might require redesigning the per-packet-data functionality to keep > track of the level in the packet > protocol stack as say IP might occur more than once in a packet. There might *already* be reasons for doing that. _

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

Evan Huus skrev 2013-04-18 18:28: Just throwing in some more stuff :-) - It would be nice to have a reference trace to test performance against, memory usage and execution time. - As a start of performance testing one could remove the reassembled data from it's hash table and store it in per-pa

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On Thu, Apr 18, 2013 at 9:25 PM, Jeff Morriss wrote: > The real problem (which I thought file-backed-tvbuffs might solve) would > be when dissectors have to make copies of tvbuffs in order to do, for > example, reassembly. Those copies are malloc()'d and it is believed that, > in some situation

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On Apr 18, 2013, at 10:28 AM, Evan Huus wrote: > That hadn't even occurred to me; I was thinking more of the fact that > we don't have a separate 'head' structure for reassembly chains and > just assume certain fields are set/unset based on whether the > structure is first in the list or not. I

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On Thu, Apr 18, 2013 at 1:11 PM, Guy Harris wrote: > > On Apr 18, 2013, at 9:28 AM, Evan Huus wrote: > >> - Our reassembly code is a bit of a mess anyways, as Guy's recent >> commit indicates. > > ...and that commit doesn't include another thing I started working on (but > haven't done much on y

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On Apr 18, 2013, at 9:28 AM, Evan Huus wrote: > - Our reassembly code is a bit of a mess anyways, as Guy's recent > commit indicates. ...and that commit doesn't include another thing I started working on (but haven't done much on yet): The "reassembled" hash table looks up by endpoints and an

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

A few misc notes on this topic in no particular order: - Once everything is converted to wmem (after 1.10 branches) it would be trivial to write a backend allocator that collected statistics on memory usage. - Has anybody ever tried to see if Massif (http://valgrind.org/info/tools.html#massif) gi

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

Jeff Morriss skrev 2013-04-18 17:55: On 04/15/13 10:01, Ambarisha B wrote: Hi dev, I am a final year engineering student pursuing my bachelors in Computer Science. I was going through the GSoC'13 ideas page and found "Filebacked-tvbuffs" interesting, so I looked it up. Here's a (probably not so

Re: [Wireshark-dev] Filebacked-tvbuffs : GSoC'13

On 04/15/13 10:01, Ambarisha B wrote: Hi dev, I am a final year engineering student pursuing my bachelors in Computer Science. I was going through the GSoC'13 ideas page and found "Filebacked-tvbuffs" interesting, so I looked it up. Here's a (probably not so) short summary of what I did and unde

[Wireshark-dev] Filebacked-tvbuffs : GSoC'13

Hi dev, I am a final year engineering student pursuing my bachelors in Computer Science. I was going through the GSoC'13 ideas page and found "Filebacked-tvbuffs" interesting, so I looked it up. Here's a (probably not so) short summary of what I did and understood. I'm only a novice, so if I've go