This was breaking a few bots and I couldn't find you on irc so I've reverted it thusly:
echristo@jhereg ~/s/llvm-project> git llvm push Pushing 1 commit: 175b1b856ea Temporarily Revert [Modules][PCH] Hash input files content as it's breaking a few bots. Sending cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td Sending cfe/trunk/include/clang/Driver/Options.td Sending cfe/trunk/include/clang/Lex/HeaderSearchOptions.h Sending cfe/trunk/include/clang/Serialization/ASTBitCodes.h Sending cfe/trunk/include/clang/Serialization/ASTReader.h Sending cfe/trunk/lib/Driver/ToolChains/Clang.cpp Sending cfe/trunk/lib/Frontend/CompilerInstance.cpp Sending cfe/trunk/lib/Frontend/CompilerInvocation.cpp Sending cfe/trunk/lib/Serialization/ASTReader.cpp Sending cfe/trunk/lib/Serialization/ASTWriter.cpp Deleting cfe/trunk/test/Modules/validate-file-content.m Deleting cfe/trunk/test/PCH/validate-file-content.m Transmitting file data ..........done Committing transaction... Committed revision 374842. Committed 175b1b856ea to svn. Sorry for the inconvenience! -eric On Mon, Oct 14, 2019 at 3:59 PM Bruno Cardoso Lopes via cfe-commits <cfe-commits@lists.llvm.org> wrote: > > Author: bruno > Date: Mon Oct 14 16:02:03 2019 > New Revision: 374841 > > URL: http://llvm.org/viewvc/llvm-project?rev=374841&view=rev > Log: > [Modules][PCH] Hash input files content > > Summary: > When files often get touched during builds, the mtime based validation > leads to different problems in implicit modules builds, even when the > content doesn't actually change: > > - Modules only: module invalidation due to out of date files. Usually causing > rebuild traffic. > - Modules + PCH: build failures because clang cannot rebuild a module if it > comes from building a PCH. > - PCH: build failures because clang cannot rebuild a PCH in case one of the > input headers has different mtime. > > This patch proposes hashing the content of input files (headers and > module maps), which is performed during serialization time. When looking > at input files for validation, clang only computes the hash in case > there's a mtime mismatch. > > I've tested a couple of different hash algorithms availble in LLVM in > face of building modules+pch for `#import <Cocoa/Cocoa.h>`: > - `hash_code`: performace diff within the noise, total module cache increased > by 0.07%. > - `SHA1`: 5% slowdown. Haven't done real size measurements, but it'd be > BLOCK_ID+20 bytes per input file, instead of BLOCK_ID+8 bytes from > `hash_code`. > - `MD5`: 3% slowdown. Like above, but BLOCK_ID+16 bytes per input file. > > Given the numbers above, the patch uses `hash_code`. The patch also > improves invalidation error msgs to point out which type of problem the > user is facing: "mtime", "size" or "content". > > rdar://problem/29320105 > > Reviewers: dexonsmith, arphaman, rsmith, aprantl > > Subscribers: jkorous, cfe-commits, ributzka > > Tags: #clang > > Differential Revision: https://reviews.llvm.org/D67249 > > Added: > cfe/trunk/test/Modules/validate-file-content.m > cfe/trunk/test/PCH/validate-file-content.m > Modified: > cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td > cfe/trunk/include/clang/Driver/Options.td > cfe/trunk/include/clang/Lex/HeaderSearchOptions.h > cfe/trunk/include/clang/Serialization/ASTBitCodes.h > cfe/trunk/include/clang/Serialization/ASTReader.h > cfe/trunk/lib/Driver/ToolChains/Clang.cpp > cfe/trunk/lib/Frontend/CompilerInstance.cpp > cfe/trunk/lib/Frontend/CompilerInvocation.cpp > cfe/trunk/lib/Serialization/ASTReader.cpp > cfe/trunk/lib/Serialization/ASTWriter.cpp > > Modified: cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td?rev=374841&r1=374840&r2=374841&view=diff > ============================================================================== > --- cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td (original) > +++ cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td Mon Oct 14 > 16:02:03 2019 > @@ -18,13 +18,16 @@ def err_fe_pch_malformed : Error< > def err_fe_pch_malformed_block : Error< > "malformed block record in PCH file: '%0'">, DefaultFatal; > def err_fe_pch_file_modified : Error< > - "file '%0' has been modified since the precompiled header '%1' was > built">, > + "file '%0' has been modified since the precompiled header '%1' was built" > + ": %select{size|mtime|content}2 changed">, > DefaultFatal; > def err_fe_module_file_modified : Error< > - "file '%0' has been modified since the module file '%1' was built">, > + "file '%0' has been modified since the module file '%1' was built" > + ": %select{size|mtime|content}2 changed">, > DefaultFatal; > def err_fe_ast_file_modified : Error< > - "file '%0' has been modified since the AST file '%1' was built">, > + "file '%0' has been modified since the AST file '%1' was built" > + ": %select{size|mtime|content}2 changed">, > DefaultFatal; > def err_fe_pch_file_overridden : Error< > "file '%0' from the precompiled header has been overridden">; > @@ -399,6 +402,8 @@ def warn_module_uses_date_time : Warning > def err_module_no_size_mtime_for_header : Error< > "cannot emit module %0: %select{size|mtime}1 must be explicitly specified " > "for missing header file \"%2\"">; > +def err_module_unable_to_hash_content : Error< > + "failed to hash content for '%0' because memory buffer cannot be > retrieved">; > } // let CategoryName > } // let Component > > > Modified: cfe/trunk/include/clang/Driver/Options.td > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Driver/Options.td?rev=374841&r1=374840&r2=374841&view=diff > ============================================================================== > --- cfe/trunk/include/clang/Driver/Options.td (original) > +++ cfe/trunk/include/clang/Driver/Options.td Mon Oct 14 16:02:03 2019 > @@ -1368,6 +1368,28 @@ def fmodules_validate_system_headers : F > HelpText<"Validate the system headers that a module depends on when > loading the module">; > def fno_modules_validate_system_headers : Flag<["-"], > "fno-modules-validate-system-headers">, > Group<i_Group>, Flags<[DriverOption]>; > + > +def fvalidate_ast_input_files_content: > + Flag <["-"], "fvalidate-ast-input-files-content">, > + Group<f_Group>, Flags<[CC1Option]>, > + HelpText<"Compute and store the hash of input files used to build an AST." > + " Files with mismatching mtime's are considered valid" > + " if both contents is identical">; > +def fmodules_validate_input_files_content: > + Flag <["-"], "fmodules-validate-input-files-content">, > + Group<f_Group>, Flags<[DriverOption]>, > + HelpText<"Validate PCM input files based on content if mtime differs">; > +def fno_modules_validate_input_files_content: > + Flag <["-"], "fno_modules-validate-input-files-content">, > + Group<f_Group>, Flags<[DriverOption]>; > +def fpch_validate_input_files_content: > + Flag <["-"], "fpch-validate-input-files-content">, > + Group<f_Group>, Flags<[DriverOption]>, > + HelpText<"Validate PCH input files based on content if mtime differs">; > +def fno_pch_validate_input_files_content: > + Flag <["-"], "fno_pch-validate-input-files-content">, > + Group<f_Group>, Flags<[DriverOption]>; > + > def fmodules : Flag <["-"], "fmodules">, Group<f_Group>, > Flags<[DriverOption, CC1Option]>, > HelpText<"Enable the 'modules' language feature">; > > Modified: cfe/trunk/include/clang/Lex/HeaderSearchOptions.h > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Lex/HeaderSearchOptions.h?rev=374841&r1=374840&r2=374841&view=diff > ============================================================================== > --- cfe/trunk/include/clang/Lex/HeaderSearchOptions.h (original) > +++ cfe/trunk/include/clang/Lex/HeaderSearchOptions.h Mon Oct 14 16:02:03 2019 > @@ -195,6 +195,10 @@ public: > /// Whether to validate system input files when a module is loaded. > unsigned ModulesValidateSystemHeaders : 1; > > + // Whether the content of input files should be hashed and used to > + // validate consistency. > + unsigned ValidateASTInputFilesContent : 1; > + > /// Whether the module includes debug information (-gmodules). > unsigned UseDebugInfo : 1; > > @@ -208,7 +212,8 @@ public: > UseBuiltinIncludes(true), UseStandardSystemIncludes(true), > UseStandardCXXIncludes(true), UseLibcxx(false), Verbose(false), > ModulesValidateOncePerBuildSession(false), > - ModulesValidateSystemHeaders(false), UseDebugInfo(false), > + ModulesValidateSystemHeaders(false), > + ValidateASTInputFilesContent(false), UseDebugInfo(false), > ModulesValidateDiagnosticOptions(true), ModulesHashContent(false) {} > > /// AddPath - Add the \p Path path to the specified \p Group list. > > Modified: cfe/trunk/include/clang/Serialization/ASTBitCodes.h > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTBitCodes.h?rev=374841&r1=374840&r2=374841&view=diff > ============================================================================== > --- cfe/trunk/include/clang/Serialization/ASTBitCodes.h (original) > +++ cfe/trunk/include/clang/Serialization/ASTBitCodes.h Mon Oct 14 16:02:03 > 2019 > @@ -382,7 +382,10 @@ namespace serialization { > /// inside the control block. > enum InputFileRecordTypes { > /// An input file. > - INPUT_FILE = 1 > + INPUT_FILE = 1, > + > + /// The input file content hash > + INPUT_FILE_HASH > }; > > /// Record types that occur within the AST block itself. > > Modified: cfe/trunk/include/clang/Serialization/ASTReader.h > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTReader.h?rev=374841&r1=374840&r2=374841&view=diff > ============================================================================== > --- cfe/trunk/include/clang/Serialization/ASTReader.h (original) > +++ cfe/trunk/include/clang/Serialization/ASTReader.h Mon Oct 14 16:02:03 2019 > @@ -930,6 +930,9 @@ private: > /// Whether validate system input files. > bool ValidateSystemInputs; > > + /// Whether validate headers and module maps using hash based on contents. > + bool ValidateASTInputFilesContent; > + > /// Whether we are allowed to use the global module index. > bool UseGlobalIndex; > > @@ -1203,6 +1206,7 @@ private: > > struct InputFileInfo { > std::string Filename; > + uint64_t ContentHash; > off_t StoredSize; > time_t StoredTime; > bool Overridden; > @@ -1437,6 +1441,8 @@ private: > void Error(StringRef Msg) const; > void Error(unsigned DiagID, StringRef Arg1 = StringRef(), > StringRef Arg2 = StringRef()) const; > + void Error(unsigned DiagID, StringRef Arg1, StringRef Arg2, > + unsigned Select) const; > void Error(llvm::Error &&Err) const; > > public: > @@ -1485,7 +1491,9 @@ public: > StringRef isysroot = "", bool DisableValidation = false, > bool AllowASTWithCompilerErrors = false, > bool AllowConfigurationMismatch = false, > - bool ValidateSystemInputs = false, bool UseGlobalIndex = true, > + bool ValidateSystemInputs = false, > + bool ValidateASTInputFilesContent = false, > + bool UseGlobalIndex = true, > std::unique_ptr<llvm::Timer> ReadTimer = {}); > ASTReader(const ASTReader &) = delete; > ASTReader &operator=(const ASTReader &) = delete; > > Modified: cfe/trunk/lib/Driver/ToolChains/Clang.cpp > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Driver/ToolChains/Clang.cpp?rev=374841&r1=374840&r2=374841&view=diff > ============================================================================== > --- cfe/trunk/lib/Driver/ToolChains/Clang.cpp (original) > +++ cfe/trunk/lib/Driver/ToolChains/Clang.cpp Mon Oct 14 16:02:03 2019 > @@ -2777,6 +2777,10 @@ static void RenderModulesOptions(Compila > std::string("-fprebuilt-module-path=") + A->getValue())); > A->claim(); > } > + if (Args.hasFlag(options::OPT_fmodules_validate_input_files_content, > + options::OPT_fno_modules_validate_input_files_content, > + false)) > + CmdArgs.push_back("-fvalidate-ast-input-files-content"); > } > > // -fmodule-name specifies the module that is currently being built (or > @@ -4899,6 +4903,10 @@ void Clang::ConstructJob(Compilation &C, > Std && (Std->containsValue("c++2a") || > Std->containsValue("c++latest")); > RenderModulesOptions(C, D, Args, Input, Output, CmdArgs, HaveModules); > > + if (Args.hasFlag(options::OPT_fpch_validate_input_files_content, > + options::OPT_fno_pch_validate_input_files_content, false)) > + CmdArgs.push_back("-fvalidate-ast-input-files-content"); > + > Args.AddLastArg(CmdArgs, options::OPT_fexperimental_new_pass_manager, > options::OPT_fno_experimental_new_pass_manager); > > > Modified: cfe/trunk/lib/Frontend/CompilerInstance.cpp > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Frontend/CompilerInstance.cpp?rev=374841&r1=374840&r2=374841&view=diff > ============================================================================== > --- cfe/trunk/lib/Frontend/CompilerInstance.cpp (original) > +++ cfe/trunk/lib/Frontend/CompilerInstance.cpp Mon Oct 14 16:02:03 2019 > @@ -510,7 +510,8 @@ IntrusiveRefCntPtr<ASTReader> CompilerIn > PP, ModuleCache, &Context, PCHContainerRdr, Extensions, > Sysroot.empty() ? "" : Sysroot.data(), DisablePCHValidation, > AllowPCHWithCompilerErrors, /*AllowConfigurationMismatch*/ false, > - HSOpts.ModulesValidateSystemHeaders, UseGlobalModuleIndex)); > + HSOpts.ModulesValidateSystemHeaders, > HSOpts.ValidateASTInputFilesContent, > + UseGlobalModuleIndex)); > > // We need the external source to be set up before we read the AST, because > // eagerly-deserialized declarations may use it. > @@ -1492,6 +1493,7 @@ void CompilerInstance::createModuleManag > /*AllowASTWithCompilerErrors=*/false, > /*AllowConfigurationMismatch=*/false, > HSOpts.ModulesValidateSystemHeaders, > + HSOpts.ValidateASTInputFilesContent, > getFrontendOpts().UseGlobalModuleIndex, std::move(ReadTimer)); > if (hasASTConsumer()) { > ModuleManager->setDeserializationListener( > > Modified: cfe/trunk/lib/Frontend/CompilerInvocation.cpp > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Frontend/CompilerInvocation.cpp?rev=374841&r1=374840&r2=374841&view=diff > ============================================================================== > --- cfe/trunk/lib/Frontend/CompilerInvocation.cpp (original) > +++ cfe/trunk/lib/Frontend/CompilerInvocation.cpp Mon Oct 14 16:02:03 2019 > @@ -2081,6 +2081,8 @@ static void ParseHeaderSearchArgs(Header > getLastArgUInt64Value(Args, OPT_fbuild_session_timestamp, 0); > Opts.ModulesValidateSystemHeaders = > Args.hasArg(OPT_fmodules_validate_system_headers); > + Opts.ValidateASTInputFilesContent = > + Args.hasArg(OPT_fvalidate_ast_input_files_content); > if (const Arg *A = Args.getLastArg(OPT_fmodule_format_EQ)) > Opts.ModuleFormat = A->getValue(); > > > Modified: cfe/trunk/lib/Serialization/ASTReader.cpp > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTReader.cpp?rev=374841&r1=374840&r2=374841&view=diff > ============================================================================== > --- cfe/trunk/lib/Serialization/ASTReader.cpp (original) > +++ cfe/trunk/lib/Serialization/ASTReader.cpp Mon Oct 14 16:02:03 2019 > @@ -1247,6 +1247,12 @@ void ASTReader::Error(unsigned DiagID, > Diag(DiagID) << Arg1 << Arg2; > } > > +void ASTReader::Error(unsigned DiagID, StringRef Arg1, StringRef Arg2, > + unsigned Select) const { > + if (!Diags.isDiagnosticInFlight()) > + Diag(DiagID) << Arg1 << Arg2 << Select; > +} > + > void ASTReader::Error(llvm::Error &&Err) const { > Error(toString(std::move(Err))); > } > @@ -2241,6 +2247,24 @@ ASTReader::readInputFileInfo(ModuleFile > R.TopLevelModuleMap = static_cast<bool>(Record[5]); > R.Filename = Blob; > ResolveImportedPath(F, R.Filename); > + > + Expected<llvm::BitstreamEntry> MaybeEntry = Cursor.advance(); > + if (!MaybeEntry) // FIXME this drops errors on the floor. > + consumeError(MaybeEntry.takeError()); > + llvm::BitstreamEntry Entry = MaybeEntry.get(); > + assert(Entry.Kind == llvm::BitstreamEntry::Record && > + "expected record type for input file hash"); > + > + Record.clear(); > + if (Expected<unsigned> Maybe = Cursor.readRecord(Entry.ID, Record)) > + assert(static_cast<InputFileRecordTypes>(Maybe.get()) == INPUT_FILE_HASH > && > + "invalid record type for input file hash"); > + else { > + // FIXME this drops errors on the floor. > + consumeError(Maybe.takeError()); > + } > + R.ContentHash = (static_cast<uint64_t>(Record[1]) << 32) | > + static_cast<uint64_t>(Record[0]); > return R; > } > > @@ -2271,6 +2295,7 @@ InputFile ASTReader::getInputFile(Module > bool Overridden = FI.Overridden; > bool Transient = FI.Transient; > StringRef Filename = FI.Filename; > + uint64_t StoredContentHash = FI.ContentHash; > > const FileEntry *File = nullptr; > if (auto FE = FileMgr.getFile(Filename, /*OpenFile=*/false)) > @@ -2325,14 +2350,46 @@ InputFile ASTReader::getInputFile(Module > } > } > > - bool IsOutOfDate = false; > + enum ModificationType { > + Size, > + ModTime, > + Content, > + None, > + }; > + auto HasInputFileChanged = [&]() { > + if (StoredSize != File->getSize()) > + return ModificationType::Size; > + if (!DisableValidation && StoredTime && > + StoredTime != File->getModificationTime()) { > + // In case the modification time changes but not the content, > + // accept the cached file as legit. > + if (ValidateASTInputFilesContent && > + StoredContentHash != static_cast<uint64_t>(llvm::hash_code(-1))) { > + auto MemBuffOrError = FileMgr.getBufferForFile(File); > + if (!MemBuffOrError) { > + if (!Complain) > + return ModificationType::ModTime; > + std::string ErrorStr = "could not get buffer for file '"; > + ErrorStr += File->getName(); > + ErrorStr += "'"; > + Error(ErrorStr); > + return ModificationType::ModTime; > + } > + > + auto ContentHash = hash_value(MemBuffOrError.get()->getBuffer()); > + if (StoredContentHash == static_cast<uint64_t>(ContentHash)) > + return ModificationType::None; > + return ModificationType::Content; > + } > + return ModificationType::ModTime; > + } > + return ModificationType::None; > + }; > > + bool IsOutOfDate = false; > + auto FileChange = HasInputFileChanged(); > // For an overridden file, there is nothing to validate. > - if (!Overridden && // > - (StoredSize != File->getSize() || > - (StoredTime && StoredTime != File->getModificationTime() && > - !DisableValidation) > - )) { > + if (!Overridden && FileChange != ModificationType::None) { > if (Complain) { > // Build a list of the PCH imports that got us here (in reverse). > SmallVector<ModuleFile *, 4> ImportStack(1, &F); > @@ -2341,13 +2398,17 @@ InputFile ASTReader::getInputFile(Module > > // The top-level PCH is stale. > StringRef TopLevelPCHName(ImportStack.back()->FileName); > - unsigned DiagnosticKind = > moduleKindForDiagnostic(ImportStack.back()->Kind); > + unsigned DiagnosticKind = > + moduleKindForDiagnostic(ImportStack.back()->Kind); > if (DiagnosticKind == 0) > - Error(diag::err_fe_pch_file_modified, Filename, TopLevelPCHName); > + Error(diag::err_fe_pch_file_modified, Filename, TopLevelPCHName, > + (unsigned)FileChange); > else if (DiagnosticKind == 1) > - Error(diag::err_fe_module_file_modified, Filename, TopLevelPCHName); > + Error(diag::err_fe_module_file_modified, Filename, TopLevelPCHName, > + (unsigned)FileChange); > else > - Error(diag::err_fe_ast_file_modified, Filename, TopLevelPCHName); > + Error(diag::err_fe_ast_file_modified, Filename, TopLevelPCHName, > + (unsigned)FileChange); > > // Print the import stack. > if (ImportStack.size() > 1 && !Diags.isDiagnosticInFlight()) { > @@ -5192,6 +5253,8 @@ bool ASTReader::readASTFileControlBlock( > consumeError(MaybeRecordType.takeError()); > } > switch ((InputFileRecordTypes)MaybeRecordType.get()) { > + case INPUT_FILE_HASH: > + break; > case INPUT_FILE: > bool Overridden = static_cast<bool>(Record[3]); > std::string Filename = Blob; > @@ -12153,7 +12216,7 @@ ASTReader::ASTReader(Preprocessor &PP, I > StringRef isysroot, bool DisableValidation, > bool AllowASTWithCompilerErrors, > bool AllowConfigurationMismatch, bool > ValidateSystemInputs, > - bool UseGlobalIndex, > + bool ValidateASTInputFilesContent, bool UseGlobalIndex, > std::unique_ptr<llvm::Timer> ReadTimer) > : Listener(DisableValidation > ? cast<ASTReaderListener>(new SimpleASTReaderListener(PP)) > @@ -12167,6 +12230,7 @@ ASTReader::ASTReader(Preprocessor &PP, I > AllowASTWithCompilerErrors(AllowASTWithCompilerErrors), > AllowConfigurationMismatch(AllowConfigurationMismatch), > ValidateSystemInputs(ValidateSystemInputs), > + ValidateASTInputFilesContent(ValidateASTInputFilesContent), > UseGlobalIndex(UseGlobalIndex), CurrSwitchCaseStmts(&SwitchCaseStmts) { > SourceMgr.setExternalSLocEntrySource(this); > > > Modified: cfe/trunk/lib/Serialization/ASTWriter.cpp > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTWriter.cpp?rev=374841&r1=374840&r2=374841&view=diff > ============================================================================== > --- cfe/trunk/lib/Serialization/ASTWriter.cpp (original) > +++ cfe/trunk/lib/Serialization/ASTWriter.cpp Mon Oct 14 16:02:03 2019 > @@ -1099,6 +1099,7 @@ void ASTWriter::WriteBlockInfoBlock() { > > BLOCK(INPUT_FILES_BLOCK); > RECORD(INPUT_FILE); > + RECORD(INPUT_FILE_HASH); > > // AST Top-Level Block. > BLOCK(AST_BLOCK); > @@ -1764,6 +1765,7 @@ struct InputFileEntry { > bool IsTransient; > bool BufferOverridden; > bool IsTopLevelModuleMap; > + uint32_t ContentHash[2]; > }; > > } // namespace > @@ -1787,6 +1789,13 @@ void ASTWriter::WriteInputFiles(SourceMa > IFAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // File name > unsigned IFAbbrevCode = Stream.EmitAbbrev(std::move(IFAbbrev)); > > + // Create input file hash abbreviation. > + auto IFHAbbrev = std::make_shared<BitCodeAbbrev>(); > + IFHAbbrev->Add(BitCodeAbbrevOp(INPUT_FILE_HASH)); > + IFHAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); > + IFHAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); > + unsigned IFHAbbrevCode = Stream.EmitAbbrev(std::move(IFHAbbrev)); > + > // Get all ContentCache objects for files, sorted by whether the file is a > // system one or not. System files go at the back, users files at the > front. > std::deque<InputFileEntry> SortedFiles; > @@ -1810,6 +1819,25 @@ void ASTWriter::WriteInputFiles(SourceMa > Entry.BufferOverridden = Cache->BufferOverridden; > Entry.IsTopLevelModuleMap = isModuleMap(File.getFileCharacteristic()) && > File.getIncludeLoc().isInvalid(); > + > + auto ContentHash = hash_code(-1); > + if (PP->getHeaderSearchInfo() > + .getHeaderSearchOpts() > + .ValidateASTInputFilesContent) { > + auto *MemBuff = Cache->getRawBuffer(); > + if (MemBuff) > + ContentHash = hash_value(MemBuff->getBuffer()); > + else > + // FIXME: The path should be taken from the FileEntryRef. > + PP->Diag(SourceLocation(), diag::err_module_unable_to_hash_content) > + << Entry.File->getName(); > + } > + auto CH = llvm::APInt(64, ContentHash); > + Entry.ContentHash[0] = > + static_cast<uint32_t>(CH.getLoBits(32).getZExtValue()); > + Entry.ContentHash[1] = > + static_cast<uint32_t>(CH.getHiBits(32).getZExtValue()); > + > if (Entry.IsSystemFile) > SortedFiles.push_back(Entry); > else > @@ -1834,17 +1862,26 @@ void ASTWriter::WriteInputFiles(SourceMa > > // Emit size/modification time for this file. > // And whether this file was overridden. > - RecordData::value_type Record[] = { > - INPUT_FILE, > - InputFileOffsets.size(), > - (uint64_t)Entry.File->getSize(), > - (uint64_t)getTimestampForOutput(Entry.File), > - Entry.BufferOverridden, > - Entry.IsTransient, > - Entry.IsTopLevelModuleMap}; > + { > + RecordData::value_type Record[] = { > + INPUT_FILE, > + InputFileOffsets.size(), > + (uint64_t)Entry.File->getSize(), > + (uint64_t)getTimestampForOutput(Entry.File), > + Entry.BufferOverridden, > + Entry.IsTransient, > + Entry.IsTopLevelModuleMap}; > + > + // FIXME: The path should be taken from the FileEntryRef. > + EmitRecordWithPath(IFAbbrevCode, Record, Entry.File->getName()); > + } > > - // FIXME: The path should be taken from the FileEntryRef. > - EmitRecordWithPath(IFAbbrevCode, Record, Entry.File->getName()); > + // Emit content hash for this file. > + { > + RecordData::value_type Record[] = {INPUT_FILE_HASH, > Entry.ContentHash[0], > + Entry.ContentHash[1]}; > + Stream.EmitRecordWithAbbrev(IFHAbbrevCode, Record); > + } > } > > Stream.ExitBlock(); > > Added: cfe/trunk/test/Modules/validate-file-content.m > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/validate-file-content.m?rev=374841&view=auto > ============================================================================== > --- cfe/trunk/test/Modules/validate-file-content.m (added) > +++ cfe/trunk/test/Modules/validate-file-content.m Mon Oct 14 16:02:03 2019 > @@ -0,0 +1,33 @@ > +// REQUIRES: shell > +// > +// Check driver works > +// RUN: %clang -fmodules -fsyntax-only > -fmodules-validate-input-files-content %s -### 2>&1 | FileCheck > --check-prefix=CHECK-CC1 %s > +// CHECK-CC1: -fvalidate-ast-input-files-content > +// > +// PCH+Modules: Test that a mtime mismatch without content change is fine > +// RUN: rm -rf %t > +// RUN: mkdir %t > +// RUN: echo '// m.h' > %t/m.h > +// RUN: echo '#include "m.h"' > %t/a.h > +// RUN: echo 'module m { header "m.h" }' > %t/module.modulemap > +// RUN: %clang_cc1 -emit-pch -fmodules-cache-path=%t/cache -fmodules > -fimplicit-module-maps -o %t/a.pch -I %t -x objective-c-header %t/a.h > -fvalidate-ast-input-files-content > +// RUN: touch -m -a -t 202901010000 %t/m.h > +// RUN: %clang_cc1 -fsyntax-only -fmodules-cache-path=%t/cache -fmodules > -fimplicit-module-maps -I %t -include-pch %t/a.pch %s -verify > -fvalidate-ast-input-files-content > +// > +// PCH+Modules: Test that a mtime mismatch with content change > +// RUN: rm -rf %t > +// RUN: mkdir %t > +// RUN: echo '// m.h' > %t/m.h > +// RUN: echo '#include "m.h"' > %t/a.h > +// RUN: echo 'module m { header "m.h" }' > %t/module.modulemap > +// RUN: %clang_cc1 -emit-pch -fmodules-cache-path=%t/cache -fmodules > -fimplicit-module-maps -o %t/a.pch -I %t -x objective-c-header %t/a.h > -fvalidate-ast-input-files-content > +// RUN: echo '// m.x' > %t/m.h > +// RUN: touch -m -a -t 202901010000 %t/m.h > +// RUN: not %clang_cc1 -fsyntax-only -fmodules-cache-path=%t/cache -fmodules > -fimplicit-module-maps -I %t -include-pch %t/a.pch %s > -fvalidate-ast-input-files-content 2> %t/stderr > +// RUN: FileCheck %s < %t/stderr > +// > +// CHECK: file '[[M_H:.*[/\\]m\.h]]' has been modified since the precompiled > header '[[A_PCH:.*/a\.pch]]' was built: content changed > +// CHECK: '[[M_H]]' required by '[[M_PCM:.*[/\\]m.*\.pcm]]' > +// CHECK: '[[M_PCM]]' required by '[[A_PCH]]' > +// CHECK: please rebuild precompiled header '[[A_PCH]]' > +// expected-no-diagnostics > > Added: cfe/trunk/test/PCH/validate-file-content.m > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/test/PCH/validate-file-content.m?rev=374841&view=auto > ============================================================================== > --- cfe/trunk/test/PCH/validate-file-content.m (added) > +++ cfe/trunk/test/PCH/validate-file-content.m Mon Oct 14 16:02:03 2019 > @@ -0,0 +1,29 @@ > +// REQUIRES: shell > +// > +// Check driver works > +// RUN: %clang -x objective-c-header -fsyntax-only > -fpch-validate-input-files-content %t/a.h -### 2>&1 | FileCheck > --check-prefix=CHECK-CC1 %s > +// CHECK-CC1: -fvalidate-ast-input-files-content > +// > +// PCH only: Test that a mtime mismatch without content change is fine > +// RUN: rm -rf %t > +// RUN: mkdir %t > +// RUN: echo '// m.h' > %t/m.h > +// RUN: echo '#include "m.h"' > %t/a.h > +// RUN: %clang_cc1 -emit-pch -o %t/a.pch -I %t -x objective-c-header %t/a.h > -fvalidate-ast-input-files-content > +// RUN: touch -m -a -t 202901010000 %t/m.h > +// RUN: %clang_cc1 -fsyntax-only -I %t -include-pch %t/a.pch %s -verify > -fvalidate-ast-input-files-content > +// > +// PCH only: Test that a mtime mismatch with content change > +// RUN: rm -rf %t > +// RUN: mkdir %t > +// RUN: echo '// m.h' > %t/m.h > +// RUN: echo '#include "m.h"' > %t/a.h > +// RUN: %clang_cc1 -emit-pch -o %t/a.pch -I %t -x objective-c-header %t/a.h > -fvalidate-ast-input-files-content > +// RUN: echo '// m.x' > %t/m.h > +// RUN: touch -m -a -t 202901010000 %t/m.h > +// RUN: not %clang_cc1 -fsyntax-only -I %t -include-pch %t/a.pch %s > -fvalidate-ast-input-files-content 2> %t/stderr > +// RUN: FileCheck %s < %t/stderr > +// > +// CHECK: file '[[M_H:.*[/\\]m\.h]]' has been modified since the precompiled > header '[[A_PCH:.*/a\.pch]]' was built: content changed > +// CHECK: please rebuild precompiled header '[[A_PCH]]' > +// expected-no-diagnostics > > > _______________________________________________ > cfe-commits mailing list > cfe-commits@lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
