[
https://issues.apache.org/jira/browse/TIKA-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129222#comment-18129222
]
ASF GitHub Bot commented on TIKA-4980:
--------------------------------------
tballison commented on PR #3370:
URL: https://github.com/apache/tika/pull/3370#issuecomment-6294563045
Feeling like github ci/cd is basically unusable with these queue times.
Github status shows that everything is green with github so this isn't an
outage.
> Extract previews from Olympus ORF, Panasonic RW2, Epson ERF, Kodak KDC and
> Leaf MOS
> -----------------------------------------------------------------------------------
>
> Key: TIKA-4980
> URL: https://issues.apache.org/jira/browse/TIKA-4980
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Dominik Schmidt
> Priority: Major
>
> RawTiffParser (TIKA-4851, TIKA-4861) extracts the camera-generated JPEG
> previews of
> TIFF-based raw formats: Nikon NEF/NRW, Sony ARW/SRF/SR2, Pentax PEF/PTX,
> Samsung SRW,
> Adobe DNG and Canon CR2. Five further TIFF containers yield nothing, each for
> a reason of
> its own, although every one of them carries a usable preview.
> h3. 1. Olympus ORF
> Detected correctly as image/x-raw-olympus by the magic entry in
> tika-mimetypes.xml, but no
> parser declares the type, and RawTiffParser would refuse the file anyway: an
> ORF carries
> its own magic number (0x4F52 for "IIRO" and "MMOR", 0x5352 for "IIRS") where
> a TIFF has
> 42, so locateJpegPreviews() throws "bad TIFF magic number".
> The previews are not in the IFD chain either. IFD0 holds the sensor image;
> the previews
> hang in the MakerNote, whose offsets count from its own first byte:
> {code}
> Exif IFD (0x8769) -> MakerNote (0x927C), header "OLYMPUS\0" + its own byte
> order marker
> 0x0100 camera thumbnail (UNDEFINED, count = length)
> 0x2020 CameraSettings sub-IFD
> 0x0101/0x0102 PreviewImageStart / PreviewImageLength
> {code}
> The byte order marker inside the MakerNote is the MakerNote's own and does
> not have to
> repeat the file's.
> h3. 2. Panasonic RW2
> Detected correctly as image/x-raw-panasonic, same two blockers: no parser
> declares the
> type, and the magic number is 0x0055 ("IIU\0"). The preview sits in the
> private tag 0x002E
> (JpgFromRaw), while the standard strip tags are placeholders (StripOffsets =
> 0xFFFFFFFF,
> StripByteCounts = 0), so a reader that trusts them finds a strip of no length
> where the
> preview should be.
> h3. 3. Epson ERF
> Not recognized as a raw at all. RawTiffDetector has no Epson entry, so the
> file is detected
> as image/tiff and handled by TiffParser. Declaring the type does not help
> either, because
> image/x-raw-epson has no sub-class-of in tika-mimetypes.xml: MimeTypes only
> accepts a
> declared type that is a descendant of what it detected, so the hint is
> dropped:
> {code}
> hint image/x-raw-epson -> detected image/tiff (hint discarded)
> hint image/x-raw-adobe -> detected image/x-raw-adobe (hint accepted: it is a
> descendant)
> {code}
> The raw IFD of an R-D1 file has Compression 32769 (Epson's own code, as
> ExifTool lists it)
> and PhotometricInterpretation 32803 (CFA), and IFD0 carries Make "SEIKO EPSON
> CORP.",
> which is what RawTiffDetector already keys on for the other vendors.
> The preview is harder to see than in the other formats, and a search for the
> JPEG SOI
> marker misses it twice over. There is no JPEG in any IFD of an ERF: the
> preview is named in
> the MakerNote, which ExifTool's Olympus.pm documents as tag 0x0280
> (PreviewImage, "found in
> ERF and JPG images from some Epson models"), its offset counts from the start
> of the file
> rather than from the MakerNote, and its first byte is not the JPEG's. All
> three R-D1
> generations write 0xEE there, where the SOI begins; the rest is a regular
> baseline JPEG that
> ends with FFD9. Put the byte back and it decodes:
> {code}
> MakerNote @708, 586 bytes, header "EPSON\0" + version, its IFD at +8, file
> byte order
> tag 0x0280 BYTE count=55150 value=9903844 -> ee d8 ff db ... -> JPEG
> 640x424
> {code}
> ExifTool restores that byte as well, without documenting it: {{exiftool -b
> -PreviewImage}}
> returns the same 55150 bytes, beginning with FFD8.
> h3. 4. Kodak KDC
> A plain little-endian TIFF that points at its preview with the standard tags,
> which
> RawTiffParser already reads for CR2 and NEF:
> {code}
> 100_3710.KDC (Kodak P880)
> IFD0 Compression 7 (JPEG), PhotometricInterpretation 6 (YCbCr)
> JPEGInterchangeFormat 12150832, JPEGInterchangeFormatLength 1677970
> Tika today: image/tiff, no embedded resources
> {code}
> Nothing has to be read differently here; the type is simply not connected to
> the parser.
> Handing the same bytes to it under a type it is bound to shows what is there:
> {code}
> as 100_3710.KDC -> image/tiff no previews
> as probe.pef -> image/x-raw-pentax THUMBNAIL 3264x2448, 1677970 bytes
> {code}
> Detection by content is a separate matter and stays unsolved: the SubIFD has
> Compression 1
> and neither PhotometricInterpretation nor BitsPerSample, so none of
> RawTiffDetector's raw
> markers fire and a KDC without a file name stays image/tiff.
> h3. 5. Leaf MOS
> A big-endian TIFF whose IFD0 has Compression 99, which is no TIFF value, and
> no preview
> tags of the standard kind. The preview sits in the Leaf data block, tag
> 0x8606, a chain of
> named blocks:
> {code}
> "PKTS" + 4-byte version + 40-byte NUL-padded name + 4-byte length + payload
> JPEG_preview_info 12 bytes "8 332 244 1"
> JPEG_preview_data 24132 bytes -> a complete baseline JPEG, 332x244
> {code}
> The bytes decode as they are stored, with no patching. Note that tag 0x8606
> is typed ASCII,
> so a reader that takes it for a string damages what it holds: {{exiftool -b
> -PreviewImage}}
> returns 24132 bytes of which 136 differ from the file, every 0x0A turned into
> 0x20, and the
> result no longer decodes. Reading the block's bytes gives the JPEG unharmed.
> h3. Proposed fix
> * RawTiffDetector: Epson by compression code 32769 and by Make "SEIKO
> EPSON"/"EPSON".
> * tika-mimetypes.xml: sub-class-of image/tiff for image/x-raw-epson,
> image/x-raw-kodak and
> image/x-raw-leaf, so the types are TIFFs for the registry, a declared type
> can refine to
> them, and the parser lookup has something to fall back on.
> * RawTiffParser: accept the ORF and RW2 magic numbers as classic TIFF
> layouts; read the
> Panasonic JpgFromRaw tag ahead of the standard tags, which an RW2 fills
> with placeholders;
> follow the Exif IFD of a classic TIFF to its MakerNote and read it by the
> name it begins
> with, the Olympus one in the byte order its own marker names and the Epson
> one at the
> offset it gives from the start of the file, restoring the SOI as the bytes
> are handed on;
> read the Leaf data block for its JPEG_preview_data; and add
> image/x-raw-olympus,
> image/x-raw-panasonic, image/x-raw-epson, image/x-raw-kodak and
> image/x-raw-leaf to the
> supported types.
> * Reading previews from three more places reshapes the walk TIKA-4851 added:
> the candidates
> of an IFD are collected before they are checked, and the per-file budget
> moves into a
> collector, so a preview from a MakerNote or a vendor block passes the same
> limits as one
> from the IFD chain.
> * The existing rule picks the resource type: the largest preview by JPEG
> length becomes the
> THUMBNAIL, the smaller ones INLINE images.
> Verified against CC0 samples from raw.pixls.us:
> {code}
> P3180165.ORF (E-M10) image/x-raw-olympus THUMBNAIL 3200x2400
> INLINE 160x120
> _7061961_copy.ORF (E-5) image/x-raw-olympus THUMBNAIL 3200x2400
> INLINE 160x120
> RAW_PANASONIC_DMC-GX-7_11 image/x-raw-panasonic THUMBNAIL 1920x1920
> Lumix_DMC-LX100_1.RW2 image/x-raw-panasonic THUMBNAIL 1920x1440
> epson-r-d1.erf image/x-raw-epson THUMBNAIL 640x424
> 55150 bytes
> epson-r-d1s.erf image/x-raw-epson THUMBNAIL 640x424
> 189141 bytes
> epson-r-d1x.erf image/x-raw-epson THUMBNAIL 640x424
> 82817 bytes
> 100_3710.KDC (P880) image/x-raw-kodak THUMBNAIL 3264x2448
> 1677970 bytes
> L_003172.mos (Aptus 22) image/x-raw-leaf THUMBNAIL 332x244
> 24132 bytes
> {code}
> Every one of those previews is byte for byte what the file holds. Make, model
> and dimensions
> are extracted for all of them, and none records a parse exception. The tests
> use hand-built
> files in the style of the existing raw test documents, each under a kilobyte.
> h3. Not covered
> Canon CRW (CIFF), Sigma X3F (FOVb) and Minolta MRW keep their previews in
> containers that
> are not TIFF; each needs a reader of its own. The TIFF raws that carry no
> JPEG whatsoever
> (Hasselblad 3FR and FFF, Kodak DCR, Sinar STI, Kodak DC120) have nothing to
> extract and keep
> answering with metadata only. Also out: the older Olympus "OLYMP\0" MakerNote
> header, Fuji
> RAF (not a TIFF container), and the vendor-level mime entries that carry no
> sub-class-of
> because they span several containers (image/x-raw-canon covers CRW, CR2 and
> CR3).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
