Hi Tony, today, I had a look at the recent sources of the PEM API implementation - i.e. https://github.com/ascarpino/jdk/commits/pem
It seems that the PEM API Tests are out of sync. They fail with JT Harness : Tests that failed java/security/PEM/PEMDecoderTest.java: Testing PEM decodings java/security/PEM/PEMEncoderTest.java: Testing PEM decoding I wanted to update the PEM keystore implementation to the latest code base. Can you tell about the planned next steps? Regards, Karl On Sat, Mar 16, 2024 at 10:36 AM Karl Scheibelhofer < karl.scheibelhofer...@gmail.com> wrote: > Hi Tony, > > find my replies inline... > > On Mon, Mar 11, 2024 at 6:13 AM Anthony Scarpino > <anthony.scarp...@oracle.com> wrote: > > > > > > > > On Mar 9, 2024, at 8:09 AM, Karl Scheibelhofer < > karl.scheibelhofer...@gmail.com> wrote: > > > > > > ... try again from from my subscribed mail account... > > > >> Hi Tony, > >> > >> in my jdk fork, I created a branch named pem-feedback-karl. > >> > >> https://github.com/KarlScheibelhofer/jdk/tree/pem-feedback-karl > >> > >> It is based on the pem branch of your jdk fork. > >> In this pem-feedback-karl branch, I did some cleanup without changing > >> the API. Your tests pass as before. > >> > >> My original pem-keystore implementation for the SUN provider is in this > branch > >> > >> https://github.com/KarlScheibelhofer/jdk/tree/pem-keystore > >> > >> It did not use the PEM API. > >> > >> In the branch > >> > >> https://github.com/KarlScheibelhofer/jdk/tree/pem-keystore-pem-api > >> > >> I modified the PEM keystore implementation to use the PEMDecoder and > PEMEncoder. > >> To make it pass all tests, I had to fix some issues with the PEM api: > >> > >> * fix missing line-breaks when encoding certificates (and CRLs) > >> * use uniform line length for PEM encoding keys and certificates > > > > > > It sounds like I did my repo update to use MimeEncoder after you had > pulled the changes. > > In which repo can i see the version using MimeEncoder that you > mentioned? i can't find this in > https://github.com/ascarpino/jdk/commits/pem > > > > >> During this work, I took some notes regarding the PEM api: > >> > >> * Consider decoupling the raw PEM encoding and decoding from > SecurityObject. > >> This would make the API usable for general purpose PEM encoding and > >> decoding, not just for keys and certificates, as it is now. > > > > > > There has been discussions about adding a generic PEM object that would > have methods to return the header, footer, and PEM text, instead of > processing into a class or KeySpec. Is there something more “general > purpose” you had it mind? > > In addition to header, footer and PEM text, for my keystore > implementation, I would need the preceding PEM explanatory text lines, > i.e. the lines before the header line > > > > >> * For this PEM keystore implementation it is essential to parse the > >> preceding explanatory text lines. > >> The PEM decoder should support this. > >> As it is now, the keystore implementation must parse the PEM objects > >> itself, including reading PEM header and footer lines. > >> Having to handle half the work in the application diminishes the > >> value of the PEMDecoder. > >> It only delegates the decoding of certificates and keys to the > PEMDecoder. > >> > >> * PEMDecoder should be able to handle or pass through unknown PEM > >> objects gracefully. > >> Otherwise the application has to check the PEM labels in advance > >> itself, which does not make sense. > > > > > > So do you not have a structured data file? I expected the application > would parse its own metadata, then when the structured code got to a PEM > tag, it would pass the InputStream to PEMDecoder. > > > > I am concerned about a generic PEM object storing an unknown amount of > application metadata and returning it back. I feel handling non-PEM should > be in the scope of the encoder/decoder. > > > No, there is no structured data file. Just a PEM file which contains > multiple PEM objects, typically certificates and private keys. It is > common practice to have related certificates and keys in a single > file. The explanatory text lines contain the name of the key, because > this is the place specified in PEM RFC to hold meta information like > this. To enable implementing a consistent Java Keystore with PEM > format, I see this necessary. At least, I found no other way for > handling key alias names as required by Java Keystore. If you know > another way, please let me know. > > The files that my keystore implementation reads and writes are just > completely PEM format, from first to last line. > > > > >> > >> * Though supporting InputStream/OutputStream for reading and writing > >> makes sense, > >> supporting Reader/Writer feels even more essential for robust > >> support for all character encodings. > >> Multi-Byte character encodings won't work with > InputStream/OutputStream. > > > > > > A Reader will read ahead, buffering the input data. I saw this when I > had `decode(Reader)` in the API. It would return the first PEM object, but > the read pointer was at the end of the file, missing the remaining PEM > objects. > > An application or a Keystore implementation gets a stream providing > multiple concatenated PEM objects that are in this keystore object. It > needs to get all the PEM objects handing in that stream, i.e. in > essence hand in the input stream and get out a list of pem objects > containing certificates, keys and their names. > > > > > Is the multi-byte characters for the keystore metadata? I don’t see how > this affects Base64. > > Try feeding in a UTF-16 encoded PEM file. I guess it won't work with > InputStream. Because the current implementation expects one character > in each byte. > > > > >> > >> * The standard line separator for PEM is "\r\n". > >> For PEM files stored in a typical linux file system, "\n" is > >> typically used, however. > >> See the output of openssl, for example. > >> Because there are still several command line utilities that do not > >> work well with "\r\n" line breaks. > >> Supporting a different line-separator than "\r\n" is a good idea in > >> my opinion. > >> Base64.getMimeEncoder also supports selecting a custom line separator. > > > > > > The standards I saw says the line separator is “\r\n”. I realize > decoders have to be more flexible because many may not follow the line > separators or cut-n-paste removes them. I think having a configurable line > separator for encoding is likely to create more incompatibility rather than > lessen for cross-platform and using with other applications. > > Yes, the PEM standard say"\r\n". But widely used tools like openssl > produce "\n". In practice, this is just the better solution. It goes > better with most other tools and version control. > I would opt for "\r\n" by default, but provide an option for "\n" only. > There must be a reason why Base64.getMimeEncoder supports custom line > separators. > > > > >> > >> * The PEMEncoder encodes the predefined SecurityObjects only. > >> There is no way to use it to PEM encoded any other type of object. > >> Consider opening a path to generic use. > > > > > > The generic PEM object I mentioned previously I think fits this case. > It would still be a SecurityObject as I don’t see value in passing in any > object. > > I will have a look at it. Can you provide a link to your implementation? > > > > >> > >> * If an application has a DER encoded certificate, it has to decode > >> and parse the certificate before > >> it can encode it using PEMEncoder. > >> This is inconvenient. > > > > > > Yeah, there isn’t an EncodedKeySpec equivalent. I’d have to think if > there is an easy way to do this without causing more problems. Given the > purpose is going between Java Objects and PEM, accepting random data isn’t > a goal. Maybe something that can be addressed with a generic PEM object. > > As a fallback simply allow passing in a byte[] that is base64 encoded, > i.e. the result of Certificate.getEncoded(). > > Thank you! > > Karl > > ... >
