Dear all, Attached please find an updated base64 embedding patch. This patch does not write original filename information to the .lyx file so the filename in .lyx file issue is resolved. The code is even simpler than before because of the removal of the link and lookup function. (Andre: neither QTextStream and QDataStream should be used, QFile.write() is enough. :-)
The new file format is:
\begin_layout Standard
\begin_inset Graphics
embedded E1632621729.png
\end_inset
\begin_embedded_files
\file_map
\end_file_map
\embed E1632621729.png
iVBORw0KGgoAAAANSUhEUgAAABgAAAAYEAYAAACw5+G7AAAABmJLR0T///////8JWPfcAAAA
CXBIWXMAAABIAAAASABGyWs+AAA
\end_embed
\end_embedded_files
At this stage, the file_map section will be empty and all embedded
files will be anonymous. This section is put there to avoid later
lyx2lyx hassles.
What do I mean by later?
Technically speaking, it is perfectly OK to keep embedded files
anonymous, just like how they exist in MS/Word or OOffice. However,
keeping original filepaths in .lyx file leads to a few convenient
features such as 'un-embed an inset' and 'extract all embedded files'.
More importantly, this allows easy sharing of external files across
different .lyx documents, which is very important for my work.
I proposed the use of a BufferParam::'keep embedded files anonymous'
flag to control the output of this section. If this flag is turned on
by default, no filename information will be leaked by default. For an
advanced user who happens to work with shared, out of tree files on
different machines, he can turn off this flag and take things under
his control. This option is very easy to implement and I personally
prefer this method.
If the above method is not good enough, Andre's encryption suggestion
makes most sense to me, because it can apply not only to this
particular problem, but also to the general problem of handling
confidential information that we may encounter later. I do not have a
complete working scenario right now, but it can be something like:
1. The information is encrypted with a password.
2. The recipient of the document can work with it without knowing such
information, or 'unlock' such information with a password.
I would propose the following key-handling procedure:
1. A user can use a global lyxrc password for all his document (and on
all his working machines). If this password is set, it will be
automatically used to encrypt and decrypt all documents, unless s/he
intentionally sets a different password for a document, or needs a
different password to decrypt a file.
2. A user can use a document-specific password in the document
settings dialog, or use a password to decrypt a document in this
dialog.
Anyway, the filename in .lyx file issue is resolved now so the only
'problem' with my proposal is the 'individual embedding' feature,
which I will address in another email.
Cheers,
Bo
Index: development/scons/scons_manifest.py
===================================================================
--- development/scons/scons_manifest.py (revision 24712)
+++ development/scons/scons_manifest.py (working copy)
@@ -57,6 +57,7 @@
Dimension.h
DispatchResult.h
DocIterator.h
+ EmbeddedFiles.h
Encoding.h
ErrorList.h
Exporter.h
@@ -161,6 +162,7 @@
CutAndPaste.cpp
DepTable.cpp
DocIterator.cpp
+ EmbeddedFiles.cpp
Encoding.cpp
ErrorList.cpp
Exporter.cpp
Index: src/insets/InsetGraphicsParams.h
===================================================================
--- src/insets/InsetGraphicsParams.h (revision 24712)
+++ src/insets/InsetGraphicsParams.h (working copy)
@@ -76,7 +76,7 @@
/// Buffer is needed to figure out if a figure is embedded.
void Write(std::ostream & os, Buffer const & buf) const;
/// If the token belongs to our parameters, read it.
- bool Read(Lexer & lex, std::string const & token, std::string const & bufpath);
+ bool Read(Lexer & lex, std::string const & token, Buffer const & buf);
/// convert
// Only a subset of InsetGraphicsParams is needed for display purposes.
// This function also interrogates lyxrc to ascertain whether
Index: src/insets/InsetGraphics.cpp
===================================================================
--- src/insets/InsetGraphics.cpp (revision 24712)
+++ src/insets/InsetGraphics.cpp (working copy)
@@ -125,7 +125,7 @@
}
-void readInsetGraphics(Lexer & lex, string const & bufpath,
+void readInsetGraphics(Lexer & lex, Buffer const & buf,
InsetGraphicsParams & params)
{
bool finished = false;
@@ -142,7 +142,7 @@
if (token == "\\end_inset") {
finished = true;
} else {
- if (!params.Read(lex, token, bufpath))
+ if (!params.Read(lex, token, buf))
lyxerr << "Unknown token, "
<< token
<< ", skipping."
@@ -275,7 +275,7 @@
{
lex.setContext("InsetGraphics::read");
//lex >> "Graphics";
- readInsetGraphics(lex, buffer().filePath(), params_);
+ readInsetGraphics(lex, buffer(), params_);
graphic_->update(params().as_grfxParams());
}
@@ -935,7 +935,7 @@
lex.setContext("InsetGraphics::string2params");
lex >> "graphics";
params = InsetGraphicsParams();
- readInsetGraphics(lex, buffer.filePath(), params);
+ readInsetGraphics(lex, buffer, params);
}
Index: src/insets/InsetGraphicsParams.cpp
===================================================================
--- src/insets/InsetGraphicsParams.cpp (revision 24712)
+++ src/insets/InsetGraphicsParams.cpp (working copy)
@@ -13,10 +13,11 @@
#include "InsetGraphicsParams.h"
+#include "Buffer.h"
+#include "EmbeddedFiles.h"
#include "LyX.h" // for use_gui
#include "Lexer.h"
#include "LyXRC.h"
-#include "Buffer.h"
#include "graphics/GraphicsParams.h"
#include "graphics/GraphicsTypes.h"
@@ -140,8 +141,14 @@
void InsetGraphicsParams::Write(ostream & os, Buffer const & buffer) const
{
// Do not write the default values
- if (!filename.empty())
- os << "\tfilename " << filename.outputFilename(buffer.filePath()) << '\n';
+ if (!filename.empty()) {
+ if (filename.isRelativeTo(buffer.temppath())) {
+ os << "\tembedded " << filename.onlyFileName() << '\n';
+ // this file will be embedded in the .lyx file
+ buffer.embeddedFiles().registerFile(filename);
+ } else
+ os << "\tfilename " << filename.outputFilename(buffer.filePath()) << '\n';
+ }
if (lyxscale != 100)
os << "\tlyxscale " << lyxscale << '\n';
if (display != graphics::DefaultDisplay)
@@ -182,11 +189,15 @@
}
-bool InsetGraphicsParams::Read(Lexer & lex, string const & token, string const & bufpath)
+bool InsetGraphicsParams::Read(Lexer & lex, string const & token,
+ Buffer const & buf)
{
if (token == "filename") {
lex.eatLine();
- filename.set(lex.getString(), bufpath);
+ filename.set(lex.getString(), buf.filePath());
+ } else if (token == "embedded") {
+ lex.eatLine();
+ filename.set(lex.getString(), buf.temppath());
} else if (token == "lyxscale") {
lex.next();
lyxscale = lex.getInteger();
Index: src/EmbeddedFiles.cpp
===================================================================
--- src/EmbeddedFiles.cpp (revision 0)
+++ src/EmbeddedFiles.cpp (revision 0)
@@ -0,0 +1,221 @@
+// -*- C++ -*-
+/**
+ * \file EmbeddedFiles.cpp
+ * This file is part of LyX, the document processor.
+ * Licence details can be found in the file COPYING.
+ *
+ * \author Bo Peng
+ *
+ * Full author contact details are available in file CREDITS.
+ */
+
+#include <config.h>
+
+
+#include "EmbeddedFiles.h"
+
+#include "support/lassert.h"
+#include "support/convert.h"
+#include "support/debug.h"
+#include "support/FileName.h"
+#include "support/filetools.h"
+#include "support/gettext.h"
+
+
+#include <iostream>
+#include <vector>
+#include <map>
+
+using namespace std;
+
+namespace lyx {
+
+using support::FileName;
+using support::DocFileName;
+using support::getExtension;
+
+struct EmbeddedFiles::Impl
+{
+ Impl() : temp_path_(), file_map_(), file_list_() {}
+
+ /// buffer temporary path
+ string temp_path_;
+ // A map between external file and embedded (randomly named) files
+ map<DocFileName, DocFileName> file_map_;
+ /// a list of files that will be written to .lyx file
+ mutable vector<string> file_list_;
+};
+
+
+EmbeddedFiles::EmbeddedFiles()
+ : d(new Impl())
+{
+}
+
+
+EmbeddedFiles::~EmbeddedFiles()
+{
+ delete d;
+}
+
+
+void EmbeddedFiles::setTempPath(std::string const & path)
+{
+ d->temp_path_ = path;
+}
+
+
+DocFileName EmbeddedFiles::embed(DocFileName const & file)
+{
+ map<DocFileName, DocFileName>::iterator it = d->file_map_.find(file);
+
+ // NOTE: There are two possibilities, file already been inserted
+ // or not. This implementation choose to return the existing
+ // embedded file. That is to say,
+ // a. \c file will have a single embedded version. The content of the
+ // last time this file is inserted will be the content for all instances.
+ // b. All embedded file will have a single source.
+ //
+ // I choose this implementation because if we simply do
+ // d->file_map_[file] = new_temp_name
+ // the old embedded file will lose its source filename. This also avoid
+ // multiple embedded copies of the same file if a file is inserted
+ // several times.
+ FileName res;
+ if (it == d->file_map_.end()) {
+ string ext = getExtension(file.absFilename());
+ while (true) {
+ string num = convert<string>(rand());
+ res.set(d->temp_path_ + "/E" + num + "." + ext);
+ if (!res.exists())
+ break;
+ }
+ d->file_map_[file] = res;
+ } else {
+ res = it->second;
+ }
+
+ if (file.exists())
+ file.copyTo(res);
+
+ LYXERR(Debug::FILES, "Copy file " << file.absFilename() << " to "
+ << res.absFilename());
+
+ return res;
+}
+
+
+void EmbeddedFiles::clearWriteList() const
+{
+ d->file_list_.clear();
+}
+
+
+void EmbeddedFiles::registerFile(DocFileName const & file) const
+{
+ d->file_list_.push_back(file.onlyFileName());
+}
+
+
+bool EmbeddedFiles::read(Lexer & lex, ErrorList & errorList)
+{
+ int has_embed_section = false;
+
+ while (lex.isOK()) {
+ string token;
+ lex >> token;
+
+ if (token.empty())
+ continue;
+
+ if (token == "\\begin_embedded_files") {
+ has_embed_section = true;
+ continue;
+ } else if (token == "\\end_embedded_files") {
+ if (!has_embed_section){
+ docstring const s = _("\\begin_embedded_files is missing");
+ errorList.push_back(ErrorItem(_("Document embedded file error"),
+ s, -1, 0, 0));
+ }
+ return !has_embed_section;
+ } else if (token == "\\end_document") {
+ lex.pushToken(token);
+ if (has_embed_section) {
+ docstring const s = _("\\end_embedded_files is missing");
+ errorList.push_back(ErrorItem(_("Document embedded file error"),
+ s, -1, 0, 0));
+ }
+ return has_embed_section;
+ } else if (token == "\\file_map") {
+ lex.eatLine();
+
+ string file_map;
+ string tmp;
+ while (true) {
+ lex.next();
+ tmp = lex.getString();
+ if (tmp == "\\end_file_map" || tmp.empty())
+ break;
+ file_map += tmp;
+ }
+ // currently, file_map should be empty. It will be used to fill
+ // the file_map_ structure.
+ } else if (token == "\\embed") {
+ lex.eatLine();
+ DocFileName file;
+ file.set(lex.getString(), d->temp_path_);
+
+ string content;
+ string tmp;
+ while (true) {
+ lex.next();
+ tmp = lex.getString();
+ if (tmp == "\\end_embed" || tmp.empty())
+ break;
+ content += tmp;
+ }
+ LYXERR(Debug::FILES, "Creating embedded file " << file.absFilename()
+ << " from content '" << content << "'" << endl);
+ file.decodeFrom(content);
+ }
+ }
+ return false;
+}
+
+
+void EmbeddedFiles::write(ostream & ofs)
+{
+ if (d->file_list_.empty())
+ return;
+
+ // write map between embedded and external file.
+ //
+ ofs << "\n\\file_map\n";
+ // This is currently empty. An encrypted text is likely to be written here.
+ ofs << "\n\\end_file_map\n";
+
+ // sort embedded files so that change of inset order will not lead
+ // to massive change of .lyx file.
+ std::sort(d->file_list_.begin(), d->file_list_.end());
+
+ std::vector<string>::const_iterator it = d->file_list_.begin();
+ std::vector<string>::const_iterator it_end = d->file_list_.end();
+
+ DocFileName file;
+ for (; it != it_end; ++it) {
+ file.set(*it, d->temp_path_);
+ // embedded version should always exist
+ LASSERT(file.exists(), /**/);
+ ofs << "\n\\embed " << *it << "\n";
+ // encode returns a long string so I need to add a few newlines.
+ string content = file.encode();
+ int const CHARS_PER_LINE = 72;
+ // This can be made safer.
+ for (int line = 0; line <= content.size()/CHARS_PER_LINE; ++line)
+ ofs << content.substr(line * CHARS_PER_LINE, CHARS_PER_LINE) << "\n";
+ ofs << "\\end_embed\n";
+ }
+}
+
+
+} // namespace lyx
Index: src/Buffer.h
===================================================================
--- src/Buffer.h (revision 24712)
+++ src/Buffer.h (working copy)
@@ -27,6 +27,7 @@
class BiblioInfo;
class BufferParams;
class DocIterator;
+class EmbeddedFiles;
class ErrorItem;
class ErrorList;
class FuncRequest;
@@ -410,6 +411,8 @@
/// the document contents.
TocBackend & tocBackend() const;
+ EmbeddedFiles & embeddedFiles() const;
+
///
Undo & undo();
Index: src/Text.cpp
===================================================================
--- src/Text.cpp (revision 24712)
+++ src/Text.cpp (working copy)
@@ -1256,6 +1256,11 @@
if (token == "\\begin_body")
continue;
+ if (token == "\\begin_embedded_files") {
+ lex.pushToken(token);
+ return false;
+ }
+
if (token == "\\end_document")
return false;
Index: src/frontends/qt4/GuiGraphics.cpp
===================================================================
--- src/frontends/qt4/GuiGraphics.cpp (revision 24712)
+++ src/frontends/qt4/GuiGraphics.cpp (working copy)
@@ -16,6 +16,8 @@
#include "GuiGraphics.h"
+#include "Buffer.h"
+#include "EmbeddedFiles.h"
#include "LengthCombo.h"
#include "Length.h"
#include "LyXRC.h"
@@ -162,6 +164,8 @@
//graphics pane
connect(filename, SIGNAL(textChanged(const QString &)),
this, SLOT(change_adaptor()));
+ connect(embedCB, SIGNAL(toggled(bool)),
+ this, SLOT(change_adaptor()));
connect(WidthCB, SIGNAL( clicked()),
this, SLOT(change_adaptor()));
connect(HeightCB, SIGNAL( clicked()),
@@ -459,7 +463,11 @@
//lyxerr << bufferFilepath();
string const name =
igp.filename.outputFilename(fromqstr(bufferFilepath()));
- filename->setText(toqstr(name));
+
+ bool embedded = igp.filename.isRelativeTo(buffer().temppath());
+ embedCB->setCheckState(embedded ? Qt::Checked : Qt::Unchecked);
+ // do not display embedded filename
+ filename->setText(embedded ? qt_("*****") : toqstr(name));
// set the bounding box values
if (igp.bb.empty()) {
@@ -598,7 +606,13 @@
{
InsetGraphicsParams & igp = params_;
- igp.filename.set(fromqstr(filename->text()), fromqstr(bufferFilepath()));
+ DocFileName file;
+ file.set(fromqstr(filename->text()), fromqstr(bufferFilepath()));
+ bool embedded = embedCB->checkState() == Qt::Checked;
+ if (embedded && file.exists())
+ igp.filename = buffer().embeddedFiles().embed(file);
+ else
+ igp.filename = file;
// the bb section
igp.bb.erase();
Index: src/frontends/qt4/ui/GraphicsUi.ui
===================================================================
--- src/frontends/qt4/ui/GraphicsUi.ui (revision 24712)
+++ src/frontends/qt4/ui/GraphicsUi.ui (working copy)
@@ -142,6 +142,13 @@
<property name="spacing" >
<number>6</number>
</property>
+ <item row="0" column="4" >
+ <widget class="QCheckBox" name="embedCB" >
+ <property name="text" >
+ <string>E&mbed</string>
+ </property>
+ </widget>
+ </item>
<item row="0" column="2" >
<widget class="QPushButton" name="browsePB" >
<property name="toolTip" >
@@ -757,6 +764,7 @@
<tabstop>tabWidget</tabstop>
<tabstop>filename</tabstop>
<tabstop>browsePB</tabstop>
+ <tabstop>embedCB</tabstop>
<tabstop>scaleCB</tabstop>
<tabstop>Scale</tabstop>
<tabstop>WidthCB</tabstop>
Index: src/Makefile.am
===================================================================
--- src/Makefile.am (revision 24712)
+++ src/Makefile.am (working copy)
@@ -104,6 +104,7 @@
CutAndPaste.cpp \
DepTable.cpp \
DocIterator.cpp \
+ EmbeddedFiles.cpp \
Encoding.cpp \
ErrorList.cpp \
Exporter.cpp \
@@ -200,6 +201,7 @@
DepTable.h \
DispatchResult.h \
DocIterator.h \
+ EmbeddedFiles.h \
Encoding.h \
ErrorList.h \
Exporter.h \
Index: src/support/FileName.cpp
===================================================================
--- src/support/FileName.cpp (revision 24712)
+++ src/support/FileName.cpp (working copy)
@@ -21,6 +21,7 @@
#include "support/Package.h"
#include "support/qstring_helpers.h"
+#include <QByteArray>
#include <QDateTime>
#include <QDir>
#include <QFile>
@@ -554,6 +555,48 @@
}
+bool FileName::isRelativeTo(string const & path) const
+{
+ return prefixIs(absFilename(), path);
+}
+
+
+string const FileName::encode() const
+{
+ if (!isReadableFile()) {
+ LYXERR0("File '" << *this << "' is not redable!");
+ return string();
+ }
+
+ QFile file(d->fi.absoluteFilePath());
+ if (!file.open(QIODevice::ReadOnly)) {
+ LYXERR0("File '" << *this
+ << "' could not be opened in read only mode!");
+ return string();
+ }
+
+ QByteArray text = file.readAll();
+ file.close();
+
+ // NOTE: we do not care about the encoding of the file here.
+ return text.toBase64().data();
+}
+
+
+void FileName::decodeFrom(string const & encoded) const
+{
+ QFile file(d->fi.absoluteFilePath());
+ if (!file.open(QFile::WriteOnly | QFile::Truncate)) {
+ LYXERR0("File '" << *this
+ << "' could not be opened in write only mode!");
+ return;
+ }
+
+ QByteArray text(encoded.c_str());
+ file.write(QByteArray::fromBase64(text));
+}
+
+
docstring FileName::displayName(int threshold) const
{
return makeDisplayPath(absFilename(), threshold);
Index: src/support/FileName.h
===================================================================
--- src/support/FileName.h (revision 24712)
+++ src/support/FileName.h (working copy)
@@ -177,6 +177,15 @@
docstring const absoluteFilePath() const;
+ /// \return true if this file is under directory path
+ bool isRelativeTo(std::string const & path) const;
+
+ /// encode the file to a string use base64 encoding
+ std::string const encode() const;
+
+ /// decode a file from base64 encoded string
+ void decodeFrom(std::string const & encoded) const;
+
private:
///
struct Private;
Index: src/EmbeddedFiles.h
===================================================================
--- src/EmbeddedFiles.h (revision 0)
+++ src/EmbeddedFiles.h (revision 0)
@@ -0,0 +1,72 @@
+// -*- C++ -*-
+/**
+ * \file EmbeddedFiles.h
+ * This file is part of LyX, the document processor.
+ * Licence details can be found in the file COPYING.
+ *
+ * \author Bo Peng
+ *
+ * Full author contact details are available in file CREDITS.
+ */
+
+#ifndef EMBEDDED_FILES_H
+#define EMBEDDED_FILES_H
+
+#include "ErrorList.h"
+#include "Lexer.h"
+
+#include "support/strfwd.h"
+
+namespace lyx {
+
+class ErrorList;
+
+namespace support {
+class DocFileName;
+class FileName;
+}
+
+
+class EmbeddedFiles {
+public:
+ ///
+ EmbeddedFiles();
+ ~EmbeddedFiles();
+
+ /// embedded files are put under the temp_path of a buffer
+ void setTempPath(std::string const & temp_path);
+
+ /** Copy \c file to a random new file under the buffer temporary directory.
+ *
+ * \param file external file to be embedded
+ * \result an embedded file with random filename
+ */
+ support::DocFileName embed(support::DocFileName const & file);
+
+ /** register file as writable to .lyx file
+ * \param file embedded file that will be written to .lyx file
+ */
+ void registerFile(support::DocFileName const & file) const;
+
+ /// clear 'write to .lyx file' list, this should be done before
+ /// write a .lyx file.
+ void clearWriteList() const;
+
+ /** read encoded embedded files and create them under the buffer tempoary directory
+ * \return \c false if file is not completely read
+ */
+ bool read(Lexer & lex, ErrorList & errorList);
+
+ /// write embedded files in base64 encoding
+ void write(std::ostream & ofs);
+
+private:
+
+ class Impl;
+ Impl * const d;
+};
+
+
+} // namespace lyx
+
+#endif // EMBEDDED_FILES
Index: src/Buffer.cpp
===================================================================
--- src/Buffer.cpp (revision 24712)
+++ src/Buffer.cpp (working copy)
@@ -25,6 +25,7 @@
#include "Converter.h"
#include "Counters.h"
#include "DocIterator.h"
+#include "EmbeddedFiles.h"
#include "Encoding.h"
#include "ErrorList.h"
#include "Exporter.h"
@@ -168,6 +169,9 @@
///
mutable TocBackend toc_backend;
+ ///
+ mutable EmbeddedFiles embedded_files;
+
/// macro tables
typedef pair<DocIterator, MacroData> ScopeMacro;
typedef map<DocIterator, ScopeMacro> PositionScopeMacroMap;
@@ -236,10 +240,11 @@
Buffer::Impl::Impl(Buffer & parent, FileName const & file, bool readonly_)
: parent_buffer(0), lyx_clean(true), bak_clean(true), unnamed(false),
read_only(readonly_), filename(file), file_fully_loaded(false),
- toc_backend(&parent), macro_lock(false), timestamp_(0),
- checksum_(0), wa_(0), undo_(parent)
+ toc_backend(&parent), embedded_files(), macro_lock(false),
+ timestamp_(0), checksum_(0), wa_(0), undo_(parent)
{
temppath = createBufferTmpDir();
+ embedded_files.setTempPath(temppath.absFilename());
lyxvc.setBuffer(&parent);
if (use_gui)
wa_ = new frontend::WorkAreaManager;
@@ -364,6 +369,12 @@
}
+EmbeddedFiles & Buffer::embeddedFiles() const
+{
+ return d->embedded_files;
+}
+
+
Undo & Buffer::undo()
{
return d->undo_;
@@ -561,8 +572,10 @@
}
// read main text
- bool const res = text().read(*this, lex, errorList, &(d->inset));
-
+ bool res = text().read(*this, lex, errorList, &(d->inset));
+ // read embedded files
+ res = res || embeddedFiles().read(lex, errorList);
+
updateMacros();
updateMacroInstances();
return res;
@@ -873,6 +886,11 @@
ofs.imbue(locale::classic());
#endif
+ // clear write list. Each inset, when being written to the .lyx file,
+ // can optionally register some files to be embedded, i.e., appended
+ // in base64 format to the .lyx file.
+ embeddedFiles().clearWriteList();
+
// The top of the file should not be written by params().
// write out a comment in the top of the file
@@ -903,6 +921,10 @@
text().write(*this, ofs);
ofs << "\n\\end_body\n";
+ ofs << "\n\\begin_embedded_files\n";
+ embeddedFiles().write(ofs);
+ ofs << "\n\\end_embedded_files\n\n";
+
// Write marker that shows file is complete
ofs << "\\end_document" << endl;
embed.lyx
Description: application/lyx
