Hi Ahmed, thanks for your feedback and sorry for the slow response. Comments below.
> -----Original Message----- > From: Ahmed Mansour [mailto:ahmed.mans...@nxp.com] > Sent: Monday, December 18, 2017 9:07 PM > To: dev@dpdk.org; shally.ve...@cavium.com; Hemant Agrawal > <hemant.agra...@nxp.com> > Cc: Hemant Agrawal <hemant.agra...@nxp.com>; mahipal.cha...@cavium.com; > Trahe, Fiona > <fiona.tr...@intel.com>; narayanaprasad.athr...@cavium.com; De Lara Guarch, > Pablo > <pablo.de.lara.gua...@intel.com>; Roy Pledge <roy.ple...@nxp.com>; Youri > Querry > <youri.querr...@nxp.com> > Subject: Re: [RFC v3 0/1] Compression API in DPDK > > Hi Fiona, > > On 12/15/2017 11:16 PM, Trahe, Fiona wrote: > > > With the vast amounts of data being transported around networks and stored > > in > > storage systems, reducing data size is becoming ever more important. There > > are both software libraries and hardware devices available that provide > > compression, but no common API. This RFC proposes a compression API for > > DPDK to address this need. > > > > Features: > > • Deflate Algorithm > (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc195 > 1&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea > 1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=yglh48%2F8IuEn%2F7YCL4 > 9FlyhGyCnNRX4g4xx2WJQesFs%3D&reserved=0) > > • LZS algorithm > (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc239 > 5&data=02%7C01%7Cahmed.mansour%40nxp.com%7C76241a2796db4701ef0108d54634a70f%7C686ea > 1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636492114308805852&sdata=BlfNe3pRXpEFJE4KS8Spk8 > bJ5GWPDqADhZYoD7SKvjk%3D&reserved=0) > > • Static and Dynamic Huffman encoding. > > • Compression levels > > • Checksum generation > > • Asynchronous burst API > > • Session-based (a session contains immutable data only and is useable > > across devices) > > • stream-based to maintain state and history data for stateful flows. > > > > Note 1: Split of functionality above/below API > > When considering whether features should be supported on the API or not, the > > decision was based on the following: > > The purpose of the API is to decouple the application from the > > compute-intensive > > functions needed for compression by abstracting them under a common API. > > These > > can then be implemented by either hardware accelerators or optimised > > software > > libraries. Where features are not compute-intensive and unlikely to be > > offloaded or optimised, there’s nothing to be gained by each PMD having > > to separately implement them, and it makes more sense for them to be done > > above the API. So the following are not handled on the API and can be done > > above. > > • Prepending/appending protocol headers (gzip, zlib) > > Agreed with the notion, however the header and footer handling can be > added as an option. PMDs can support or not support each format. We > (NXP) support auto padding of gzip and zlib headers as well as DEFLATE only. [Fiona] We'd like to stabilise the API with the current functionality before adding new features. Our focus now is on delivering a v1 of the full code rather than another iteration of the RFC. The API will be experimental initially, so I think it should be easy add these later. > > • File-handling, breaking files up into packets, reassembling. > > • Synchronous API > > • Serialisation of stateful requests > > > > > Is stateful planned for next phase? During design discussions we uncovered > many API design > considerations necessary for stateful use. Chained stateful support in the > future might not be > possible without compatibility breaking > [Fiona] Stateful is covered in the v3 version. See the thread http://dpdk.org/ml/archives/dev/2017-December/084713.html Have a look at this, though maybe it would be better to hold off on a reply until the v2 version of this doc is posted by Shally. > > Note 2: The tricky question of where the API belongs > > We considered > > 1. Extending cryptodev > > 2. New acceldev APIs for device handling + compressdev APIs for data path > > 3. New acceldev for all APIs > > 4. New compressdev API > > We've gone with option 4, a compressdev API. See original RFC [1] for > > reasons. > > We explored wrapping this around a generic acceldev that would be hidden > > from the API > > but could be common to cryptodev, compressdev and other accelerators on the > > PMD interface, > > but this added complexity and indirection and didn't add enough value, so > > we've abandoned it. > > > Makes sense. compression is common enough to be attempted to be a > different device category. > > > > Opens: > > - Define structures and API for proposed hash functionality > > - Agree on stateful behaviour > > What are the the current thoughts for stateful behavior? > > > - Complete capability APIs > > A capability API is very much required as different HW/SW can have > different capabilities. [Fiona] Agreed. We have work to do to flesh this out - but expect to do in v1 of actual code rather than a further RFC iteration > > regards, > Ahmed >
