> -----Original Message----- > From: Ahmed Mansour [mailto:ahmed.mans...@nxp.com] > Sent: Thursday, February 15, 2018 7:51 PM > To: Trahe, Fiona <fiona.tr...@intel.com>; Verma, Shally > <shally.ve...@cavium.com>; dev@dpdk.org > Cc: Athreya, Narayana Prasad <narayanaprasad.athr...@cavium.com>; Gupta, > Ashish > <ashish.gu...@cavium.com>; Sahu, Sunila <sunila.s...@cavium.com>; De Lara > Guarch, Pablo > <pablo.de.lara.gua...@intel.com>; Challa, Mahipal > <mahipal.cha...@cavium.com>; Jain, Deepak K > <deepak.k.j...@intel.com>; Hemant Agrawal <hemant.agra...@nxp.com>; Roy Pledge > <roy.ple...@nxp.com>; Youri Querry <youri.querr...@nxp.com> > Subject: Re: [RFC v2] doc compression API for DPDK > > /// snip /// > >>>>> > >>>>>>>> [Fiona] I propose if BFINAL bit is detected before end of input > >>>>>>>> the decompression should stop. In this case consumed will be < > >>>>>>>> src.length. > >>>>>>>> produced will be < dst buffer size. Do we need an extra STATUS > >>>>>>>> response? > >>>>>>>> STATUS_BFINAL_DETECTED ? > >>>>>>> [Shally] @fiona, I assume you mean here decompressor stop after > >>>>>>> processing Final block right? > >>>>>> [Fiona] Yes. > >>>>>> > >>>>>> And if yes, > >>>>>>> and if it can process that final block successfully/unsuccessfully, > >>>>>>> then status could simply be > >>>>>>> SUCCESS/FAILED. > >>>>>>> I don't see need of specific return code for this use case. Just to > >>>>>>> share, in past, we have > practically > >> run into > >>>>>>> such cases with boost lib, and decompressor has simply worked this > >>>>>>> way. > >>>>>> [Fiona] I'm ok with this. > >>>>>> > >>>>>>>> Only thing I don't like this is it can impact on performance, as > >>>>>>>> normally > >>>>>>>> we can just look for STATUS == SUCCESS. Anything else should be an > >>>>>>>> exception. > >>>>>>>> Now the application would have to check for SUCCESS || > >>>>>>>> BFINAL_DETECTED every time. > >>>>>>>> Do you have a suggestion on how we should handle this? > >>>>>>>> > >>>>> [Ahmed] This makes sense. So in all cases the PMD should assume that it > >>>>> should stop as soon as a BFINAL is observed. > >>>>> > >>>>> A question. What happens ins stateful vs stateless modes when > >>>>> decompressing an op that encompasses multiple BFINALs. I assume the > >>>>> caller in that case will use the consumed=x bytes to find out how far in > >>>>> to the input is the end of the first stream and start from the next > >>>>> byte. Is this correct? > >>>> [Shally] As per my understanding, each op can be tied up to only one > >>>> stream as we have only one > >> stream pointer per op and one > >>> stream can have only one BFINAL (as stream is one complete compressed > >>> data) but looks like you're > >> suggesting a case where one op > >>> can carry multiple independent streams? and thus multiple BFINAL?! , such > >>> as, below here is op > >> pointing to more than one streams > >>>> -------------------------------------------- > >>>> op --> |stream1|stream2| |stream3| > >>>> -------------------------------------------- > >>>> > >>>> Could you confirm if I understand your question correct? > >>> [Ahmed] Correct. We found that in some storage applications the user > >>> does not know where exactly the BFINAL is. They rely on zlib software > >>> today. zlib.net software halts at the first BFINAL. Users put multiple > >>> streams in one op and rely on zlib to stop and inform them of the end > >>> location of the first stream. > >> [Shally] Then this is practically case possible on decompressor and > >> decompressor doesn't regard flush > >> flag. So in that case, I expect PMD to internally reset themselves (say in > >> case of zlib going through > cycle > >> of deflateEnd and deflateInit or deflateReset) and return with status = > >> SUCCESS with updated > produced > >> and consumed. Now in such case, if previous stream also has some footer > >> followed by start of next > >> stream, then I am not sure how PMD / lib can support that case. Have you > >> had practically run of such > >> use-case on zlib? If yes, how then such application handle it in your > >> experience? > >> I can imagine for such input zlib would return with Z_FLUSH_END after 1st > >> BFINAL is processed to the > >> user. Then application doing deflateReset() or Init-End() cycle before > >> starting with next. But if it starts > >> with input that doesn't have valid zlib header, then likely it will throw > >> an error. > >> > > [Fiona] The consumed and produced tell the Application hw much data was > > processed up to > > the end of the first deflate block encountered with a bfinal set. > > If there is data, e.g. footer after the block with bfinal, then I think it > > must be the responsibility of > > the application to know this, the PMD can't have any responsibility for > > this. > > The next op sent to the PMD must start with a valid deflate block. > [Ahmed] Agreed. This is exactly what I expected. In our case we support > gzip and zlib header/footer processing, but that does not fundamentally > change the setup. The user may have other meta data after the footer > which the PMD is not responsible for. The PMD should stop processing > depending on the mode. In raw DEFLATE, it should stop immediately. In > other modes it should stop after the footer. We also have a mode in our > PMD to simply continue decompression. In that case there cannot be > header/footer between streams in raw DEFLATE. That mode can be enabled > perhaps at the session level in the future with a session parameter at > setup time. We call it "member continue". In this mode the PMD plows > through as much of the op as possible. If it hits incorrectly setup data > then it returns what it did decompress successfully and the error code > in decompressing the data afterwards. [Fiona] Yes, these would be interesting capabilities which could be added to the API in future releases.
> > > > > >>>> Thanks > >>>> Shally > >>>> > >
