> -----Original Message----- > From: Ahmed Mansour [mailto:ahmed.mans...@nxp.com] > Sent: 19 December 2017 02:37 > To: dev@dpdk.org; Verma, Shally <shally.ve...@cavium.com>; Hemant > Agrawal <hemant.agra...@nxp.com> > Cc: Hemant Agrawal <hemant.agra...@nxp.com>; Challa, Mahipal > <mahipal.cha...@cavium.com>; fiona.tr...@intel.com; Athreya, Narayana > Prasad <narayanaprasad.athr...@cavium.com>; > pablo.de.lara.gua...@intel.com; Roy Pledge <roy.ple...@nxp.com>; Youri > Querry <youri.querr...@nxp.com> > Subject: Re: [RFC v3 0/1] Compression API in DPDK > > Hi Fiona, > > On 12/15/2017 11:16 PM, Trahe, Fiona wrote: > > > With the vast amounts of data being transported around networks and > stored in > > storage systems, reducing data size is becoming ever more important. > There > > are both software libraries and hardware devices available that provide > > compression, but no common API. This RFC proposes a compression API for > > DPDK to address this need. > > > > Features: > > • Deflate Algorithm > (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto > ols.ietf.org%2Fhtml%2Frfc1951&data=02%7C01%7Cahmed.mansour%40nxp. > com%7C76241a2796db4701ef0108d54634a70f%7C686ea1d3bc2b4c6fa92cd99c > 5c301635%7C0%7C0%7C636492114308805852&sdata=yglh48%2F8IuEn%2F7YC > L49FlyhGyCnNRX4g4xx2WJQesFs%3D&reserved=0) > > • LZS algorithm > (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto > ols.ietf.org%2Fhtml%2Frfc2395&data=02%7C01%7Cahmed.mansour%40nxp. > com%7C76241a2796db4701ef0108d54634a70f%7C686ea1d3bc2b4c6fa92cd99c > 5c301635%7C0%7C0%7C636492114308805852&sdata=BlfNe3pRXpEFJE4KS8Sp > k8bJ5GWPDqADhZYoD7SKvjk%3D&reserved=0) > > • Static and Dynamic Huffman encoding. > > • Compression levels > > • Checksum generation > > • Asynchronous burst API > > • Session-based (a session contains immutable data only and is useable > across devices) > > • stream-based to maintain state and history data for stateful flows. > > > > Note 1: Split of functionality above/below API > > When considering whether features should be supported on the API or > not, the > > decision was based on the following: > > The purpose of the API is to decouple the application from the compute- > intensive > > functions needed for compression by abstracting them under a common > API. These > > can then be implemented by either hardware accelerators or optimised > software > > libraries. Where features are not compute-intensive and unlikely to be > > offloaded or optimised, there’s nothing to be gained by each PMD > having > > to separately implement them, and it makes more sense for them to be > done > > above the API. So the following are not handled on the API and can be > done above. > > • Prepending/appending protocol headers (gzip, zlib) > > Agreed with the notion, however the header and footer handling can be > added as an option. PMDs can support or not support each format. We > (NXP) support auto padding of gzip and zlib headers as well as DEFLATE only. >
[Shally] Just my thoughts on this. For such case, ZLIB and GZIP should be added into algorithm lists. PMD which support them can generate ZLIB/GZIP styled compressed streams and app can invoke compression with algo such as ZLIB/GZIP. Thanks Shally > > • File-handling, breaking files up into packets, reassembling. > > • Synchronous API > > • Serialisation of stateful requests > > > > > Is stateful planned for next phase? During design discussions we uncovered > many API design > considerations necessary for stateful use. Chained stateful support in the > future might not be > possible without compatibility breaking > > > Note 2: The tricky question of where the API belongs > > We considered > > 1. Extending cryptodev > > 2. New acceldev APIs for device handling + compressdev APIs for data path > > 3. New acceldev for all APIs > > 4. New compressdev API > > We've gone with option 4, a compressdev API. See original RFC [1] for > reasons. > > We explored wrapping this around a generic acceldev that would be hidden > from the API > > but could be common to cryptodev, compressdev and other accelerators > on the PMD interface, > > but this added complexity and indirection and didn't add enough value, so > we've abandoned it. > > > Makes sense. compression is common enough to be attempted to be a > different device category. > > > > Opens: > > - Define structures and API for proposed hash functionality > > - Agree on stateful behaviour > > What are the the current thoughts for stateful behavior? > > > - Complete capability APIs > > A capability API is very much required as different HW/SW can have > different capabilities. > > > regards, > Ahmed >
