[
https://issues.apache.org/jira/browse/TIKA-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominik Schmidt updated TIKA-4980:
----------------------------------
Description:
RawTiffParser (TIKA-4851, TIKA-4861) extracts the camera-generated JPEG
previews of
TIFF-based raw formats: Nikon NEF/NRW, Sony ARW/SRF/SR2, Pentax PEF/PTX,
Samsung SRW,
Adobe DNG and Canon CR2. Three further TIFF-based formats yield nothing, each
for a
different reason, although all three carry a preview.
h3. 1. Olympus ORF
Detected correctly as image/x-raw-olympus by the magic entry in
tika-mimetypes.xml, but
no parser declares the type, and RawTiffParser would refuse the file anyway: an
ORF
carries its own magic number (0x4F52 for "IIRO" and "MMOR", 0x5352 for "IIRS")
where a
TIFF has 42, so locateJpegPreviews() throws "bad TIFF magic number".
The previews are not in the IFD chain either. IFD0 holds the sensor image; the
previews
hang in the MakerNote, whose offsets count from its own first byte:
{code}
Exif IFD (0x8769) -> MakerNote (0x927C), header "OLYMPUS\0" + its own byte
order marker
0x0100 camera thumbnail (UNDEFINED, count = length)
0x2020 CameraSettings sub-IFD
0x0101/0x0102 PreviewImageStart / PreviewImageLength
{code}
The byte order marker inside the MakerNote is the MakerNote's own and does not
have to
repeat the file's.
h3. 2. Panasonic RW2
Detected correctly as image/x-raw-panasonic, same two blockers: no parser
declares the
type, and the magic number is 0x0055 ("IIU\0"). The preview sits in the private
tag
0x002E (JpgFromRaw) of the IFD, while the standard strip tags are placeholders
(StripOffsets = 0xFFFFFFFF, StripByteCounts = 0), so a reader that trusts them
finds a
strip of no length where the preview should be.
h3. 3. Epson ERF
Not recognized as a raw at all. RawTiffDetector has no Epson entry, so the file
is
detected as image/tiff and handled by TiffParser. Declaring the type does not
help
either, because image/x-raw-epson has no sub-class-of in tika-mimetypes.xml:
MimeTypes
only accepts a declared type that is a descendant of what it detected, so the
hint is
dropped. Measured against an Epson R-D1 file:
{code}
hint image/x-raw-epson -> detected image/tiff (hint discarded)
hint image/x-raw-adobe -> detected image/x-raw-adobe (hint accepted: it is a
descendant)
{code}
The raw IFD of that file has Compression 32769 (Epson's own code, as ExifTool
lists it)
and PhotometricInterpretation 32803 (CFA), and IFD0 carries Make "SEIKO EPSON
CORP.",
which is what RawTiffDetector already keys on for the other vendors.
The preview is harder to see than in the other two formats, and a search for
the JPEG
SOI marker misses it twice over. There is no JPEG in any IFD of an ERF: the
preview is
named in the MakerNote, which ExifTool's Olympus.pm documents as tag 0x0280
(PreviewImage, "found in ERF and JPG images from some Epson models"), its
offset counts
from the start of the file rather than from the MakerNote, and its first byte
is not the
JPEG's. All three R-D1 generations write 0xEE there, where the SOI begins; the
rest is a
regular baseline JPEG that ends with FFD9. Put the byte back and it decodes:
{code}
MakerNote @708, 586 bytes, header "EPSON\0" + version, its IFD at +8, file byte
order
tag 0x0280 BYTE count=55150 value=9903844 -> ee d8 ff db ... -> JPEG
640x424
{code}
h3. Proposed fix
* RawTiffDetector: Epson by compression code 32769 and by Make "SEIKO
EPSON"/"EPSON".
* tika-mimetypes.xml: sub-class-of image/tiff for image/x-raw-epson, so the
type is a
TIFF for the registry, a declared type can refine to it, and the parser
lookup has
something to fall back on.
* RawTiffParser: accept the ORF and RW2 magic numbers as classic TIFF layouts;
read the
Panasonic JpgFromRaw tag before the strip tags it writes placeholders into;
read the
MakerNote for both vendors, the Olympus one in the byte order its own marker
names and
the Epson one at the offset it gives from the start of the file, restoring
the SOI as
the bytes are handed on; add image/x-raw-olympus, image/x-raw-panasonic and
image/x-raw-epson to the supported types.
* The existing rule picks the resource type: the largest preview by JPEG length
becomes
the THUMBNAIL, the smaller ones INLINE images.
Verified against CC0 samples from raw.pixls.us, which are all the ORF, RW2 and
ERF
samples it has for these cameras:
{code}
P3180165.ORF (E-M10) image/x-raw-olympus THUMBNAIL 3200x2400
INLINE 160x120
_7061961_copy.ORF (E-5) image/x-raw-olympus THUMBNAIL 3200x2400
INLINE 160x120
RAW_PANASONIC_DMC-GX-7_11 image/x-raw-panasonic THUMBNAIL 1920x1920
Lumix_DMC-LX100_1.RW2 image/x-raw-panasonic THUMBNAIL 1920x1440
epson-r-d1.erf image/x-raw-epson THUMBNAIL 640x424
55150 bytes
epson-r-d1s.erf image/x-raw-epson THUMBNAIL 640x424
189141 bytes
epson-r-d1x.erf image/x-raw-epson THUMBNAIL 640x424
82817 bytes
{code}
Make, model and dimensions are extracted for all of them, and none records a
parse
exception. The tests use hand-built files in the style of the existing raw test
documents, each under a kilobyte.
Not covered: the older Olympus "OLYMP\0" MakerNote header, Fuji RAF (not a TIFF
container), and the vendor-level mime entries that exist without a sub-class-of
because
they span several containers (image/x-raw-canon covers CRW, CR2 and CR3).
was:
RawTiffParser (TIKA-4851, TIKA-4861) extracts the camera-generated JPEG
previews of
TIFF-based raw formats: Nikon NEF/NRW, Sony ARW/SRF/SR2, Pentax PEF/PTX,
Samsung SRW,
Adobe DNG and Canon CR2. Three further TIFF-based formats yield nothing, each
for a
different reason.
h3. 1. Olympus ORF
Detected correctly as image/x-raw-olympus (magic entry in tika-mimetypes.xml),
but no
parser declares the type, and RawTiffParser would refuse the file anyway: ORF
uses its
own magic number (0x4F52 for "IIRO"/"MMOR", 0x5352 for "IIRS") where a TIFF has
42, so
locateJpegPreviews() throws "bad TIFF magic number".
On top of that the previews are not in the IFD chain. IFD0 holds the sensor
image; the
previews hang in the MakerNote, whose offsets are relative to its own first
byte:
{code}
EXIF IFD (0x8769) -> MakerNote (0x927C), header "OLYMPUS\0II", its IFD at +12
0x0100 camera thumbnail (UNDEFINED, count = length)
0x2020 CameraSettings sub-IFD
0x0101/0x0102 PreviewImageStart / PreviewImageLength
{code}
Sample (raw.pixls.us, CC0, Olympus E-M10, P3180165.ORF): a 3200x2400 preview of
918873
bytes and a 160x120 thumbnail.
h3. 2. Panasonic RW2
Detected correctly as image/x-raw-panasonic, same two blockers: no parser
declares the
type, and the magic number is 0x0055 ("IIU\0"). The preview sits in the private
tag
0x002E (JpgFromRaw) of IFD0, while the standard strip tags are placeholders
(StripOffsets = 0xFFFFFFFF, StripByteCounts = 0).
Sample (raw.pixls.us, CC0, Panasonic DMC-GX7): a 1920x1920 preview of 926720
bytes.
h3. 3. Epson ERF
Not recognized as a raw at all. RawTiffDetector has no Epson entry, so the file
is
detected as image/tiff and handled by TiffParser. Declaring the type does not
help
either, because image/x-raw-epson has no sub-class-of in tika-mimetypes.xml:
MimeTypes
only accepts a declared type that is a descendant of what it detected, so the
hint is
dropped. Measured against an Epson R-D1 file:
{code}
hint image/x-raw-epson -> detected image/tiff (hint discarded)
hint image/x-raw-adobe -> detected image/x-raw-adobe (hint accepted: it is a
descendant)
{code}
The raw IFD of that file has Compression 32769 (Epson's own code, as ExifTool
lists it)
and PhotometricInterpretation 32803 (CFA), and IFD0 carries Make "SEIKO EPSON
CORP.",
which is what RawTiffDetector already keys on for the other vendors. Note this
particular sample carries no embedded JPEG at all, so it will stay without a
thumbnail;
the detection and the typing are wrong regardless, and ERFs that do carry a
preview
cannot be reached today.
h3. Proposed fix
* RawTiffDetector: Epson by compression code 32769 and by Make "SEIKO
EPSON"/"EPSON".
* tika-mimetypes.xml: sub-class-of image/tiff for image/x-raw-epson, so the
type is a
TIFF for the registry, a declared type can refine to it, and the parser
lookup has
something to fall back to.
* RawTiffParser: accept the ORF and RW2 magic numbers as classic TIFF layouts;
read the
Panasonic JpgFromRaw tag; read the Olympus MakerNote (only the "OLYMPUS\0II"
header,
the older "OLYMP\0" one addresses differently) for the thumbnail and the
preview; add
image/x-raw-olympus, image/x-raw-panasonic and image/x-raw-epson to the
supported
types.
* The existing rule picks the resource type: the largest preview by JPEG length
becomes
the THUMBNAIL, the smaller ones INLINE images.
Verified against real files (raw.pixls.us samples, CC0):
{code}
P3180165.ORF (E-M10) image/x-raw-olympus THUMBNAIL 3200x2400 INLINE
160x120
_7061961_copy.ORF (E-5) image/x-raw-olympus THUMBNAIL 3200x2400 INLINE
160x120
RAW_PANASONIC_DMC-GX-7_11 image/x-raw-panasonic THUMBNAIL 1920x1920
Lumix_DMC-LX100_1.RW2 image/x-raw-panasonic THUMBNAIL 1920x1440
pixls-cc0-epson-rd1.erf image/x-raw-epson (no preview in the file)
{code}
Make, model and dimensions are extracted for all of them, and none records a
parse
exception. The tests use two hand-built files of 890 and 421 bytes, in the
style of the
existing raw test documents.
Not covered: the older Olympus "OLYMP\0" MakerNote header, Fuji RAF (not a TIFF
container), and the vendor-level mime entries that exist without a sub-class-of
because
they span several containers (image/x-raw-canon covers CRW, CR2 and CR3).
> Extract previews from Olympus ORF, Panasonic RW2 and Epson ERF
> --------------------------------------------------------------
>
> Key: TIKA-4980
> URL: https://issues.apache.org/jira/browse/TIKA-4980
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Reporter: Dominik Schmidt
> Priority: Major
>
> RawTiffParser (TIKA-4851, TIKA-4861) extracts the camera-generated JPEG
> previews of
> TIFF-based raw formats: Nikon NEF/NRW, Sony ARW/SRF/SR2, Pentax PEF/PTX,
> Samsung SRW,
> Adobe DNG and Canon CR2. Three further TIFF-based formats yield nothing, each
> for a
> different reason, although all three carry a preview.
> h3. 1. Olympus ORF
> Detected correctly as image/x-raw-olympus by the magic entry in
> tika-mimetypes.xml, but
> no parser declares the type, and RawTiffParser would refuse the file anyway:
> an ORF
> carries its own magic number (0x4F52 for "IIRO" and "MMOR", 0x5352 for
> "IIRS") where a
> TIFF has 42, so locateJpegPreviews() throws "bad TIFF magic number".
> The previews are not in the IFD chain either. IFD0 holds the sensor image;
> the previews
> hang in the MakerNote, whose offsets count from its own first byte:
> {code}
> Exif IFD (0x8769) -> MakerNote (0x927C), header "OLYMPUS\0" + its own byte
> order marker
> 0x0100 camera thumbnail (UNDEFINED, count = length)
> 0x2020 CameraSettings sub-IFD
> 0x0101/0x0102 PreviewImageStart / PreviewImageLength
> {code}
> The byte order marker inside the MakerNote is the MakerNote's own and does
> not have to
> repeat the file's.
> h3. 2. Panasonic RW2
> Detected correctly as image/x-raw-panasonic, same two blockers: no parser
> declares the
> type, and the magic number is 0x0055 ("IIU\0"). The preview sits in the
> private tag
> 0x002E (JpgFromRaw) of the IFD, while the standard strip tags are placeholders
> (StripOffsets = 0xFFFFFFFF, StripByteCounts = 0), so a reader that trusts
> them finds a
> strip of no length where the preview should be.
> h3. 3. Epson ERF
> Not recognized as a raw at all. RawTiffDetector has no Epson entry, so the
> file is
> detected as image/tiff and handled by TiffParser. Declaring the type does not
> help
> either, because image/x-raw-epson has no sub-class-of in tika-mimetypes.xml:
> MimeTypes
> only accepts a declared type that is a descendant of what it detected, so the
> hint is
> dropped. Measured against an Epson R-D1 file:
> {code}
> hint image/x-raw-epson -> detected image/tiff (hint discarded)
> hint image/x-raw-adobe -> detected image/x-raw-adobe (hint accepted: it is a
> descendant)
> {code}
> The raw IFD of that file has Compression 32769 (Epson's own code, as ExifTool
> lists it)
> and PhotometricInterpretation 32803 (CFA), and IFD0 carries Make "SEIKO EPSON
> CORP.",
> which is what RawTiffDetector already keys on for the other vendors.
> The preview is harder to see than in the other two formats, and a search for
> the JPEG
> SOI marker misses it twice over. There is no JPEG in any IFD of an ERF: the
> preview is
> named in the MakerNote, which ExifTool's Olympus.pm documents as tag 0x0280
> (PreviewImage, "found in ERF and JPG images from some Epson models"), its
> offset counts
> from the start of the file rather than from the MakerNote, and its first byte
> is not the
> JPEG's. All three R-D1 generations write 0xEE there, where the SOI begins;
> the rest is a
> regular baseline JPEG that ends with FFD9. Put the byte back and it decodes:
> {code}
> MakerNote @708, 586 bytes, header "EPSON\0" + version, its IFD at +8, file
> byte order
> tag 0x0280 BYTE count=55150 value=9903844 -> ee d8 ff db ... -> JPEG
> 640x424
> {code}
> h3. Proposed fix
> * RawTiffDetector: Epson by compression code 32769 and by Make "SEIKO
> EPSON"/"EPSON".
> * tika-mimetypes.xml: sub-class-of image/tiff for image/x-raw-epson, so the
> type is a
> TIFF for the registry, a declared type can refine to it, and the parser
> lookup has
> something to fall back on.
> * RawTiffParser: accept the ORF and RW2 magic numbers as classic TIFF
> layouts; read the
> Panasonic JpgFromRaw tag before the strip tags it writes placeholders into;
> read the
> MakerNote for both vendors, the Olympus one in the byte order its own
> marker names and
> the Epson one at the offset it gives from the start of the file, restoring
> the SOI as
> the bytes are handed on; add image/x-raw-olympus, image/x-raw-panasonic and
> image/x-raw-epson to the supported types.
> * The existing rule picks the resource type: the largest preview by JPEG
> length becomes
> the THUMBNAIL, the smaller ones INLINE images.
> Verified against CC0 samples from raw.pixls.us, which are all the ORF, RW2
> and ERF
> samples it has for these cameras:
> {code}
> P3180165.ORF (E-M10) image/x-raw-olympus THUMBNAIL 3200x2400
> INLINE 160x120
> _7061961_copy.ORF (E-5) image/x-raw-olympus THUMBNAIL 3200x2400
> INLINE 160x120
> RAW_PANASONIC_DMC-GX-7_11 image/x-raw-panasonic THUMBNAIL 1920x1920
> Lumix_DMC-LX100_1.RW2 image/x-raw-panasonic THUMBNAIL 1920x1440
> epson-r-d1.erf image/x-raw-epson THUMBNAIL 640x424
> 55150 bytes
> epson-r-d1s.erf image/x-raw-epson THUMBNAIL 640x424
> 189141 bytes
> epson-r-d1x.erf image/x-raw-epson THUMBNAIL 640x424
> 82817 bytes
> {code}
> Make, model and dimensions are extracted for all of them, and none records a
> parse
> exception. The tests use hand-built files in the style of the existing raw
> test
> documents, each under a kilobyte.
> Not covered: the older Olympus "OLYMP\0" MakerNote header, Fuji RAF (not a
> TIFF
> container), and the vendor-level mime entries that exist without a
> sub-class-of because
> they span several containers (image/x-raw-canon covers CRW, CR2 and CR3).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
