[
https://issues.apache.org/jira/browse/TIKA-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101802#comment-18101802
]
ASF GitHub Bot commented on TIKA-4798:
--------------------------------------
nddipiazza opened a new pull request, #2987:
URL: https://github.com/apache/tika/pull/2987
## Summary
Fixes [TIKA-4798](https://issues.apache.org/jira/browse/TIKA-4798):
`dcterms:modified` for `.msg` (Outlook MAPI) files changed between Tika 2.9.0
and 3.2.3.
## Root Cause
[TIKA-4360](https://issues.apache.org/jira/browse/TIKA-4360) (#2073, merged
into the 3.x line in Dec 2024) added code to
`OutlookExtractor.handleGeneralDates()` that unconditionally overwrites
`TikaCoreProperties.MODIFIED` (`dcterms:modified`) with the raw MAPI
`PR_LAST_MODIFICATION_TIME` property whenever it is present.
`PR_LAST_MODIFICATION_TIME` records when the `.msg` file's underlying
CFB/OLE storage was last **written to disk** (e.g. when Outlook re-saves or
exports the item as a `.msg` file) — it is not a semantic "the email content
was edited" timestamp. Since an email's content is effectively immutable once
sent, this property commonly reflects a later, unrelated storage-level event
(archival, export, re-indexing, etc.), not a real content modification.
I confirmed this against the reporter's attached sample files:
`PR_LAST_MODIFICATION_TIME` for `Sample1.msg` is `2018-10-06T00:03:21Z`,
exactly matching the "wrong" 3.2.3 value reported in the ticket, while
`msg.getMessageDate()` (delivery/submit time, used in 2.9.0) is
`2018-08-03T19:13:22Z`, matching the "expected" 2.9.0 value.
## Changes
- Removed the block in `OutlookExtractor.handleGeneralDates()` that
overwrote `dcterms:modified` with `PR_LAST_MODIFICATION_TIME`.
`dcterms:modified` now falls back to the message date (delivery/submit time)
again, matching 2.9.0 behavior.
- The raw property is **not lost** — it remains available unchanged under
`mapi:last-modification-time` (already populated separately by
`handleMessageInfo`/`LITERAL_TIME_PROPERTIES`, added in the same TIKA-4360
change).
- Added a regression assertion in `OutlookParserTest#testOutlookParsing`
using the existing `test-outlook.msg` resource, whose
`PR_LAST_MODIFICATION_TIME` diverges from its message date by about six months
— the same kind of divergence reported in the ticket.
## Review Focus Areas
- Confirm the semantics: should `dcterms:modified` reflect message date or
`PR_LAST_MODIFICATION_TIME`? This PR takes the position that message date is
the more meaningful default for email, given `PR_LAST_MODIFICATION_TIME`'s
storage-level semantics, while still exposing the raw MAPI value separately.
- Check no other code path or downstream consumer relies on the TIKA-4360
override behavior.
## Critical Files
-
`tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/OutlookExtractor.java`
-
`tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/OutlookParserTest.java`
## Testing Instructions
```
cd
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module
mvn test -Dtest=OutlookParserTest
```
Also ran the full module test suite (`mvn test`) with no new failures, and
manually parsed the reporter's attached sample `.msg` files to confirm
`dcterms:modified` now matches the value produced by Tika 2.9.0.
## Review Checklist
- [x] Root cause identified and confirmed against reporter's sample files
- [x] Existing tests pass
- [x] New regression test added
- [x] No data loss — raw MAPI property still exposed under its own key
## Potential Concerns
- This is a behavior change from 3.2.3 back to 2.9.0 semantics; any user who
started relying on the 3.x `dcterms:modified` value (intentionally or not) will
see it change again. Given this was an undocumented, likely unintended side
effect of TIKA-4360, I believe reverting to the original, more intuitive
semantics is the right call, with the raw property still available under
`mapi:last-modification-time` for anyone who wants it.
> AutoDetectParser Gets Different Value for metadata dcterms:modified from msg
> File with Tika 3.2.3
> -------------------------------------------------------------------------------------------------
>
> Key: TIKA-4798
> URL: https://issues.apache.org/jira/browse/TIKA-4798
> Project: Tika
> Issue Type: Bug
> Components: detector
> Affects Versions: 3.2.3
> Reporter: Xiaohong Yang
> Priority: Major
> Attachments: Program_and_Sample_Files.zip
>
>
> [^Program_and_Sample_Files.zip]
> We use Tika to extract metadata from msg files. We found out that the value
> of metadata dcterms:modified with Tika 3.2.3 is different from that with Tika
> 2.9.0 for some msg files.
> Following is the stand alone program that reproduces the problem with the
> attached sample files. The Java version is 21.
>
> import org.apache.tika.config.TikaConfig;
> import org.apache.tika.metadata.Metadata;
> import org.apache.tika.parser.AutoDetectParser;
> import org.apache.tika.parser.ParseContext;
> import org.xml.sax.helpers.DefaultHandler;
> import java.io.BufferedInputStream;
> import java.io.File;
> import java.io.FileInputStream;
> public class TestAutoDetectParser {
> public static void main(String args[]) {
> try {
> System.{_}out{_}.println("Start");
> File inputFile = new
> File("C:\\Users\\xyang\\Data\\testdirs\\TC321926\\Sample1.msg");
> TikaConfig config = new
> TikaConfig("C:\\Users\\xyang\\Data\\testdirs\\TC321926\\tika-config.xml");
> ParseContext context = new ParseContext();
> context.set(TikaConfig.class, config);
> Metadata metadata = new Metadata();
> try (BufferedInputStream inputStream = new
> BufferedInputStream(new FileInputStream(inputFile))) {
> new AutoDetectParser(config).parse(inputStream, new
> DefaultHandler(), metadata, context);
> }
> for (String name : metadata.names()) {
> System.{_}out{_}.println(name + ": " + metadata.get(name));
> }
> System.{_}out{_}.println("End");
> }
> catch(Exception ex) {
> ex.printStackTrace();
> }
> }
> }
>
> The metadata with Tika 2.9.0 is:
> dcterms:modified: 2018-08-03T19:13:22Z
>
> And the metadata with Tika 3.2.3 is:
> dcterms:modified: 2018-10-06T00:03:21Z
>
> Wonder if it is an improvement in 3.2.3 or a bug.
>
> Attached are 4 files:
> TestAutoDetectParser.java --- the standalone program
> tika-config.xml --- config file
> Sample1.msg --- Sample file
> Sample2.msg --- Sample file
> Sample3.msg --- Sample file
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
