>> +def identity_copy(sbox):
>> + "copy UTF-8 svn:* props identically"
>> + orig_lc_all = locale.setlocale(locale.LC_ALL)
>> + other_locales = [ "English.1252", "German.1252", "French.1252",
>> "en_US.ISO-8859-1", "en_GB.ISO-8859-1", "de_DE.ISO-8859-1" ]
>> + for other_locale in other_locales:
>> + try:
>> + locale.setlocale(locale.LC_ALL, other_locale)
>> + break
>> + except:
>> + pass
>
> Don't you need to check that at least one of the six locales was
> set successfully?
My bad. Yes I should.
Attached are the corrected patch and log message. The new patch
incorporates this fix:
https://github.com/dtrebbien/subversion/commit/190f876b52626be6b30fe4e5a311c113fd87e589
[[[
Add a command line option (--source-prop-encoding) to the svnsync init, sync,
and copy-revprops subcommands that allows the user to specify the character
encoding of translatable properties from the source repository. This is needed
to allow svnsync to sync some older Subversion repositories that have
properties that were not encoded in UTF-8.
As discussed at:
http://thread.gmane.org/gmane.comp.version-control.subversion.user/100020
http://thread.gmane.org/gmane.comp.version-control.subversion.devel/122518
http://thread.gmane.org/gmane.comp.version-control.subversion.devel/122550
http://thread.gmane.org/gmane.comp.version-control.subversion.devel/124731
http://thread.gmane.org/gmane.comp.version-control.subversion.devel/125057
http://thread.gmane.org/gmane.comp.version-control.subversion.devel/125783
Most of the changes exist to pass the argument to --source-prop-encoding through
to the functions that need it (mainly normalize_string() in
subversion/svnsync/sync.c).
* subversion/svnsync/main.c
(svnsync__opt) Add svnsync_opt_source_encoding.
(svnsync_cmd_table): Add svnsync_opt_source_encoding to the list of
acceptable options for the init, sync, and copy-revprops subcommands.
(svnsync_options): Add a description of the --source-prop-encoding option.
(opt_baton_t, subcommand_baton_t): Add the SOURCE_PROP_ENCODING field.
(log_properties_normalized): Document the QUIET parameter.
(copy_revprops): Add the SOURCE_PROP_ENCODING parameter. Pass it through to
svnsync_normalize_revprops().
(make_subcommand_baton): Set the SOURCE_PROP_ENCODING field of the resulting
subcommand_baton_t object to the value of SOURCE_PROP_ENCODING from the
opt_baton_t object.
(do_initialize, do_synchronize, do_copy_revprops, replay_rev_started,
replay_rev_finished): Pass SOURCE_PROP_ENCODING to svnsync_* functions and
copy_revprops().
(main): Handle the case when the command line option is
--source-prop-encoding. Set the SOURCE_PROP_ENCODING field of the
opt_baton_t object to either OPT_ARG or NULL.
* subversion/svnsync/sync.c
(normalize_string): Add the SOURCE_PROP_ENCODING parameter. Always call
svn_subst_translate_string2(). Switch to the "two pools" (result & scratch
pools) pattern.
(svnsync_normalize_revprops): Add the SOURCE_PROP_ENCODING parameter. Pass the
value through to normalize_string().
(edit_baton_t): Add the SOURCE_PROP_ENCODING field.
(change_file_prop, change_dir_prop): Pass SOURCE_PROP_ENCODING from the edit
baton to normalize_string().
(svnsync_get_sync_editor): Add the SOURCE_PROP_ENCODING parameter.
* subversion/svnsync/sync.h
(svnsync_normalize_revprops): Add the SOURCE_PROP_ENCODING parameter. Update
the documentation of the function.
(svnsync_get_sync_editor): Add the SOURCE_PROP_ENCODING parameter. Update the
documentation of the function.
* subversion/tests/cmdline/svnrdump_tests.py
(copy_bad_line_endings2_dump): New test case.
(test_list): Add copy_bad_line_endings2_dump.
* subversion/tests/cmdline/svnsync_tests.py
(run_sync): Add the SOURCE_PROP_ENCODING parameter. Build up the command line
arguments to `svnsync synchronize`.
(run_copy_revprops): Add the SOURCE_PROP_ENCODING parameter. Build up the
command line arguments to `svnsync copy-revprops`.
(run_init): Add the SOURCE_PROP_ENCODING parameter. Build up the command line
arguments to `svnsync initialize`.
(setup_and_sync): Add the SOURCE_PROP_ENCODING parameter. Pass the value
through to run_init(), run_sync(), and run_copy_revprops().
(run_test): Add the SOURCE_PROP_ENCODING parameter. Pass the value through to
setup_and_sync().
(copy_bad_line_endings2): New test case.
(copy_bad_encoding): New test case.
(identity_copy): New test case.
(test_list): Add copy_bad_line_endings2, copy_bad_encoding, and identity_copy.
* subversion/tests/cmdline/svnrsync_tests_data/copy-bad-line-endings2.dump
A dump of a repository with the following features:
1. The log message (`svn:log` revision property) of revision 1 has CRLF line
endings.
2. The log message of revision 2 has CR line endings.
3. Revision 3 introduces an `svn:ignore` node property with CRLF line endings.
4. Revision 4 introduces a custom node property, `x:related-to`, with CRLF
line endings.
* subversion/tests/cmdline/svnrsync_tests_data/
copy-bad-line-endings2.expected.dump
A dump of the expected result of using svnrdump to dump
copy-bad-line-endings2.dump.
* subversion/tests/cmdline/svnsync_tests_data/copy-bad-encoding.dump
A dump of a repository with the following feature: The log message and an
`svn:ignore` node prop of revision 1 are encoded in ISO-8859-3.
* subversion/tests/cmdline/svnsync_tests_data/copy-bad-encoding.expected.dump
A dump of the expected result of using svnsync to sync copy-bad-encoding.dump
with `--source-prop-encoding ISO-8859-3`.
* subversion/tests/cmdline/svnsync_tests_data/copy-bad-line-endings2.dump
Copy of ../svnrsync_tests_data/copy-bad-line-endings2.dump.
* subversion/tests/cmdline/svnsync_tests_data/
copy-bad-line-endings2.expected.dump
A dump of the expected result of using svnsync to sync
copy-bad-line-endings2.dump.
]]]
Index: subversion/svnsync/sync.h
===================================================================
--- subversion/svnsync/sync.h (revision 1070224)
+++ subversion/svnsync/sync.h (working copy)
@@ -33,15 +33,20 @@ extern "C" {
#include "svn_delta.h"
-/* Normalize the line ending style of the values of properties in REV_PROPS
- * that "need translation" (according to svn_prop_needs_translation(),
- * currently all svn:* props) so that they contain only LF (\n) line endings.
- * The number of properties that needed normalization is returned in
+/* Normalize the encoding and line ending style of the values of properties
+ * in REV_PROPS that "need translation" (according to
+ * svn_prop_needs_translation(), which is currently all svn:* props) so that
+ * they are encoded in UTF-8 and contain only LF (\n) line endings.
+ *
+ * The number of properties that needed line ending normalization is returned in
* *NORMALIZED_COUNT.
+ *
+ * No re-encoding is performed if SOURCE_PROP_ENCODING is NULL.
*/
svn_error_t *
svnsync_normalize_revprops(apr_hash_t *rev_props,
int *normalized_count,
+ const char *source_prop_encoding,
apr_pool_t *pool);
@@ -51,15 +56,21 @@ svnsync_normalize_revprops(apr_hash_t *rev_props,
* the commit. TO_URL is the URL of the root of the repository into
* which the commit is being made.
*
+ * If SOURCE_PROP_ENCODING is NULL, then property values are presumed to be
+ * encoded in UTF-8 and are not re-encoded. Otherwise, the property values are
+ * presumed to be encoded in SOURCE_PROP_ENCODING, and are normalized to UTF-8.
+ *
* As the sync editor encounters property values, it might see the need to
- * normalize them (to LF line endings). Each carried out normalization adds 1
- * to the *NORMALIZED_NODE_PROPS_COUNTER (for notification).
+ * normalize them (re-encode and/or change to LF line endings). Each carried-out
+ * line ending normalization adds 1 to the *NORMALIZED_NODE_PROPS_COUNTER
+ * (for notification).
*/
svn_error_t *
svnsync_get_sync_editor(const svn_delta_editor_t *wrapped_editor,
void *wrapped_edit_baton,
svn_revnum_t base_revision,
const char *to_url,
+ const char *source_prop_encoding,
svn_boolean_t quiet,
const svn_delta_editor_t **editor,
void **edit_baton,
Index: subversion/svnsync/main.c
===================================================================
--- subversion/svnsync/main.c (revision 1070224)
+++ subversion/svnsync/main.c (working copy)
@@ -61,6 +61,7 @@ enum svnsync__opt {
svnsync_opt_sync_password,
svnsync_opt_config_dir,
svnsync_opt_config_options,
+ svnsync_opt_source_prop_encoding,
svnsync_opt_disable_locking,
svnsync_opt_version,
svnsync_opt_trust_server_cert,
@@ -105,8 +106,9 @@ static const svn_opt_subcommand_desc2_t svnsync_cm
"the destination repository by any method other than 'svnsync'.\n"
"In other words, the destination repository should be a read-only\n"
"mirror of the source repository.\n"),
- { SVNSYNC_OPTS_DEFAULT, 'q', svnsync_opt_allow_non_empty,
- svnsync_opt_disable_locking, svnsync_opt_steal_lock } },
+ { SVNSYNC_OPTS_DEFAULT, svnsync_opt_source_prop_encoding, 'q',
+ svnsync_opt_allow_non_empty, svnsync_opt_disable_locking,
+ svnsync_opt_steal_lock } },
{ "synchronize", synchronize_cmd, { "sync" },
N_("usage: svnsync synchronize DEST_URL [SOURCE_URL]\n"
"\n"
@@ -118,8 +120,8 @@ static const svn_opt_subcommand_desc2_t svnsync_cm
"source URL. Specifying SOURCE_URL is recommended in particular\n"
"if untrusted users/administrators may have write access to the\n"
"DEST_URL repository.\n"),
- { SVNSYNC_OPTS_DEFAULT, 'q', svnsync_opt_disable_locking,
- svnsync_opt_steal_lock } },
+ { SVNSYNC_OPTS_DEFAULT, svnsync_opt_source_prop_encoding, 'q',
+ svnsync_opt_disable_locking, svnsync_opt_steal_lock } },
{ "copy-revprops", copy_revprops_cmd, { 0 },
N_("usage:\n"
"\n"
@@ -139,8 +141,8 @@ static const svn_opt_subcommand_desc2_t svnsync_cm
"DEST_URL repository.\n"
"\n"
"Form 2 is deprecated syntax, equivalent to specifying \"-rREV[:REV2]\".\n"),
- { SVNSYNC_OPTS_DEFAULT, 'q', 'r', svnsync_opt_disable_locking,
- svnsync_opt_steal_lock } },
+ { SVNSYNC_OPTS_DEFAULT, svnsync_opt_source_prop_encoding, 'q', 'r',
+ svnsync_opt_disable_locking, svnsync_opt_steal_lock } },
{ "info", info_cmd, { 0 },
N_("usage: svnsync info DEST_URL\n"
"\n"
@@ -203,6 +205,12 @@ static const apr_getopt_option_t svnsync_options[]
"For example:\n"
" "
" servers:global:http-library=serf")},
+ {"source-prop-encoding", svnsync_opt_source_prop_encoding, 1,
+ N_("convert translatable properties from encoding ARG\n"
+ " "
+ "to UTF-8. If not specified, then properties are\n"
+ " "
+ "presumed to be encoded in UTF-8.")},
{"disable-locking", svnsync_opt_disable_locking, 0,
N_("Disable built-in locking. Use of this option can\n"
" "
@@ -238,6 +246,7 @@ typedef struct opt_baton_t {
const char *sync_password;
const char *config_dir;
apr_hash_t *config;
+ const char *source_prop_encoding;
svn_boolean_t disable_locking;
svn_boolean_t steal_lock;
svn_boolean_t quiet;
@@ -364,6 +373,9 @@ typedef struct subcommand_baton_t {
svn_boolean_t allow_non_empty;
const char *to_url;
+ /* initialize, synchronize, and copy-revprops only */
+ const char *source_prop_encoding;
+
/* initialize only */
const char *from_url;
@@ -613,8 +625,11 @@ log_properties_normalized(int normalized_rev_props
* If SYNC is TRUE, then properties on the destination revision that
* do not exist on the source revision will be removed.
*
+ * If QUIET is FALSE, then log_properties_copied() is called to log that
+ * properties were copied for revision REV.
+ *
* Make sure the values of svn:* revision properties use only LF (\n)
- * lineending style, correcting their values as necessary. The number
+ * line ending style, correcting their values as necessary. The number
* of properties that were normalized is returned in *NORMALIZED_COUNT.
*/
static svn_error_t *
@@ -623,6 +638,7 @@ copy_revprops(svn_ra_session_t *from_session,
svn_revnum_t rev,
svn_boolean_t sync,
svn_boolean_t quiet,
+ const char *source_prop_encoding,
int *normalized_count,
apr_pool_t *pool)
{
@@ -638,9 +654,10 @@ copy_revprops(svn_ra_session_t *from_session,
/* Get the list of revision properties on REV of SOURCE. */
SVN_ERR(svn_ra_rev_proplist(from_session, rev, &rev_props, subpool));
- /* If necessary, normalize line ending style, and return the count
- of changes in int *NORMALIZED_COUNT. */
- SVN_ERR(svnsync_normalize_revprops(rev_props, normalized_count, pool));
+ /* If necessary, normalize encoding and line ending style and return the count
+ of EOL-normalized properties in int *NORMALIZED_COUNT. */
+ SVN_ERR(svnsync_normalize_revprops(rev_props, normalized_count,
+ source_prop_encoding, pool));
/* Copy all but the svn:svnsync properties. */
SVN_ERR(write_revprops(&filtered_count, to_session, rev, rev_props, pool));
@@ -681,6 +698,7 @@ make_subcommand_baton(opt_baton_t *opt_baton,
b->quiet = opt_baton->quiet;
b->allow_non_empty = opt_baton->allow_non_empty;
b->to_url = to_url;
+ b->source_prop_encoding = opt_baton->source_prop_encoding;
b->from_url = from_url;
b->start_rev = start_rev;
b->end_rev = end_rev;
@@ -785,8 +803,9 @@ do_initialize(svn_ra_session_t *to_session,
LATEST is not 0, this really serves merely aesthetic and
informational purposes, keeping the output of this command
consistent while allowing folks to see what the latest revision is. */
- SVN_ERR(copy_revprops(from_session, to_session, latest, FALSE,
- baton->quiet, &normalized_rev_props_count, pool));
+ SVN_ERR(copy_revprops(from_session, to_session, latest, FALSE, baton->quiet,
+ baton->source_prop_encoding, &normalized_rev_props_count,
+ pool));
SVN_ERR(log_properties_normalized(normalized_rev_props_count, 0, pool));
@@ -929,7 +948,7 @@ open_target_session(svn_ra_session_t **target_sess
return SVN_NO_ERROR;
}
-/* Replay baton, used during sychnronization. */
+/* Replay baton, used during synchronization. */
typedef struct replay_baton_t {
svn_ra_session_t *from_session;
svn_ra_session_t *to_session;
@@ -1063,9 +1082,11 @@ replay_rev_started(svn_revnum_t revision,
apr_hash_set(filtered, SVN_PROP_REVISION_LOG, APR_HASH_KEY_STRING,
svn_string_create("", pool));
- /* If necessary, normalize line ending style, and add the number
- of changes to the overall count in the replay baton. */
- SVN_ERR(svnsync_normalize_revprops(filtered, &normalized_count, pool));
+ /* If necessary, normalize encoding and line ending style. Add the number
+ of properties that required EOL normalization to the overall count
+ in the replay baton. */
+ SVN_ERR(svnsync_normalize_revprops(filtered, &normalized_count,
+ rb->sb->source_prop_encoding, pool));
rb->normalized_rev_props_count += normalized_count;
SVN_ERR(svn_ra_get_commit_editor3(rb->to_session, &commit_editor,
@@ -1078,8 +1099,8 @@ replay_rev_started(svn_revnum_t revision,
over the RA interface, so we need an editor that's smart enough
to filter those out for us. */
SVN_ERR(svnsync_get_sync_editor(commit_editor, commit_baton, revision - 1,
- rb->sb->to_url, rb->sb->quiet,
- &sync_editor, &sync_baton,
+ rb->sb->to_url, rb->sb->source_prop_encoding,
+ rb->sb->quiet, &sync_editor, &sync_baton,
&(rb->normalized_node_props_count), pool));
SVN_ERR(svn_delta_get_cancellation_editor(check_cancel, NULL,
@@ -1133,9 +1154,10 @@ replay_rev_finished(svn_revnum_t revision,
: filter_exclude_log),
subpool);
- /* If necessary, normalize line ending style, and add the number
- of changes to the overall count in the replay baton. */
- SVN_ERR(svnsync_normalize_revprops(filtered, &normalized_count, pool));
+ /* If necessary, normalize encoding and line ending style, and add the number
+ of EOL-normalized properties to the overall count in the replay baton. */
+ SVN_ERR(svnsync_normalize_revprops(filtered, &normalized_count,
+ rb->sb->source_prop_encoding, pool));
rb->normalized_rev_props_count += normalized_count;
SVN_ERR(write_revprops(&filtered_count, rb->to_session, revision, filtered,
@@ -1239,10 +1261,9 @@ do_synchronize(svn_ra_session_t *to_session,
{
if (copying > last_merged)
{
- SVN_ERR(copy_revprops(from_session, to_session,
- to_latest, TRUE, baton->quiet,
- &normalized_rev_props_count,
- pool));
+ SVN_ERR(copy_revprops(from_session, to_session, to_latest, TRUE,
+ baton->quiet, baton->source_prop_encoding,
+ &normalized_rev_props_count, pool));
last_merged = copying;
last_merged_rev = svn_string_create
(apr_psprintf(pool, "%ld", last_merged), pool);
@@ -1405,8 +1426,9 @@ do_copy_revprops(svn_ra_session_t *to_session,
{
int normalized_count;
SVN_ERR(check_cancel(NULL));
- SVN_ERR(copy_revprops(from_session, to_session, i, TRUE,
- baton->quiet, &normalized_count, pool));
+ SVN_ERR(copy_revprops(from_session, to_session, i, TRUE, baton->quiet,
+ baton->source_prop_encoding, &normalized_count,
+ pool));
normalized_rev_props_count += normalized_count;
}
@@ -1708,6 +1730,7 @@ main(int argc, const char *argv[])
const char *username = NULL, *source_username = NULL, *sync_username = NULL;
const char *password = NULL, *source_password = NULL, *sync_password = NULL;
apr_array_header_t *config_options = NULL;
+ const char *source_prop_encoding = NULL;
apr_allocator_t *allocator;
if (svn_cmdline_init("svnsync", stderr) != EXIT_SUCCESS)
@@ -1832,6 +1855,11 @@ main(int argc, const char *argv[])
return svn_cmdline_handle_exit_error(err, pool, "svnsync: ");
break;
+ case svnsync_opt_source_prop_encoding:
+ opt_err = svn_utf_cstring_to_utf8(&source_prop_encoding, opt_arg,
+ pool);
+ break;
+
case svnsync_opt_disable_locking:
opt_baton.disable_locking = TRUE;
break;
@@ -2033,6 +2061,8 @@ main(int argc, const char *argv[])
config = apr_hash_get(opt_baton.config, SVN_CONFIG_CATEGORY_CONFIG,
APR_HASH_KEY_STRING);
+ opt_baton.source_prop_encoding = source_prop_encoding;
+
apr_signal(SIGINT, signal_handler);
#ifdef SIGBREAK
Index: subversion/svnsync/sync.c
===================================================================
--- subversion/svnsync/sync.c (revision 1070224)
+++ subversion/svnsync/sync.c (working copy)
@@ -45,18 +45,27 @@
#include <apr_uuid.h>
-/* Normalize the line ending style of *STR, so that it contains only
- * LF (\n) line endings. After return, *STR may point at a new
- * svn_string_t* allocated from POOL.
+/* Normalize the encoding and line ending style of *STR, so that it contains
+ * only LF (\n) line endings and is encoded in UTF-8. After return, *STR may
+ * point at a new svn_string_t* allocated in RESULT_POOL.
*
- * *WAS_NORMALIZED is set to TRUE when *STR needed to be normalized,
- * and to FALSE if *STR remains unchanged.
+ * If SOURCE_PROP_ENCODING is NULL, then *STR is presumed to be encoded in
+ * UTF-8.
+ *
+ * *WAS_NORMALIZED is set to TRUE when *STR needed line ending normalization.
+ * Otherwise it is set to FALSE.
+ *
+ * SCRATCH_POOL is used for temporary allocations.
*/
static svn_error_t *
normalize_string(const svn_string_t **str,
svn_boolean_t *was_normalized,
- apr_pool_t *pool)
+ const char *source_prop_encoding,
+ apr_pool_t *result_pool,
+ apr_pool_t *scratch_pool)
{
+ svn_string_t *new_str;
+
*was_normalized = FALSE;
if (*str == NULL)
@@ -64,33 +73,33 @@ normalize_string(const svn_string_t **str,
SVN_ERR_ASSERT((*str)->data != NULL);
- /* Detect inconsistent line ending style simply by looking
- for carriage return (\r) characters. */
- if (strchr((*str)->data, '\r') != NULL)
- {
- /* Found some. Normalize. */
- const char* cstring = NULL;
- SVN_ERR(svn_subst_translate_cstring2((*str)->data, &cstring,
- "\n", TRUE,
- NULL, FALSE,
- pool));
- *str = svn_string_create(cstring, pool);
- *was_normalized = TRUE;
- }
+ if (source_prop_encoding == NULL)
+ source_prop_encoding = "UTF-8";
+ new_str = NULL;
+ SVN_ERR(svn_subst_translate_string2(&new_str, NULL, was_normalized,
+ *str, source_prop_encoding, TRUE,
+ result_pool, scratch_pool));
+ *str = new_str;
+
return SVN_NO_ERROR;
}
-/* Normalize the line ending style of the values of properties in REV_PROPS
- * that "need translation" (according to svn_prop_needs_translation(),
- * currently all svn:* props) so that they contain only LF (\n) line endings.
- * The number of properties that needed normalization is returned in
+/* Normalize the encoding and line ending style of the values of properties
+ * in REV_PROPS that "need translation" (according to
+ * svn_prop_needs_translation(), which is currently all svn:* props) so that
+ * they are encoded in UTF-8 and contain only LF (\n) line endings.
+ *
+ * The number of properties that needed line ending normalization is returned in
* *NORMALIZED_COUNT.
+ *
+ * No re-encoding is performed if SOURCE_PROP_ENCODING is NULL.
*/
svn_error_t *
svnsync_normalize_revprops(apr_hash_t *rev_props,
int *normalized_count,
+ const char *source_prop_encoding,
apr_pool_t *pool)
{
apr_hash_index_t *hi;
@@ -106,14 +115,14 @@ svnsync_normalize_revprops(apr_hash_t *rev_props,
if (svn_prop_needs_translation(propname))
{
svn_boolean_t was_normalized;
- SVN_ERR(normalize_string(&propval, &was_normalized, pool));
+ SVN_ERR(normalize_string(&propval, &was_normalized,
+ source_prop_encoding, pool, pool));
+
+ /* Replace the existing prop value. */
+ apr_hash_set(rev_props, propname, APR_HASH_KEY_STRING, propval);
+
if (was_normalized)
- {
- /* Replace the existing prop value. */
- apr_hash_set(rev_props, propname, APR_HASH_KEY_STRING, propval);
- /* And count this. */
- (*normalized_count)++;
- }
+ (*normalized_count)++; /* Count it. */
}
}
return SVN_NO_ERROR;
@@ -141,6 +150,7 @@ typedef struct edit_baton_t {
const svn_delta_editor_t *wrapped_editor;
void *wrapped_edit_baton;
const char *to_url; /* URL we're copying into, for correct copyfrom URLs */
+ const char *source_prop_encoding;
svn_boolean_t called_open_root;
svn_boolean_t got_textdeltas;
svn_revnum_t base_revision;
@@ -407,7 +417,8 @@ change_file_prop(void *file_baton,
if (svn_prop_needs_translation(name))
{
svn_boolean_t was_normalized;
- SVN_ERR(normalize_string(&value, &was_normalized, pool));
+ SVN_ERR(normalize_string(&value, &was_normalized,
+ eb->source_prop_encoding, pool, pool));
if (was_normalized)
(*(eb->normalized_node_props_counter))++;
}
@@ -505,7 +516,8 @@ change_dir_prop(void *dir_baton,
if (svn_prop_needs_translation(name))
{
svn_boolean_t was_normalized;
- SVN_ERR(normalize_string(&value, &was_normalized, pool));
+ SVN_ERR(normalize_string(&value, &was_normalized, eb->source_prop_encoding,
+ pool, pool));
if (was_normalized)
(*(eb->normalized_node_props_counter))++;
}
@@ -572,6 +584,7 @@ svnsync_get_sync_editor(const svn_delta_editor_t *
void *wrapped_edit_baton,
svn_revnum_t base_revision,
const char *to_url,
+ const char *source_prop_encoding,
svn_boolean_t quiet,
const svn_delta_editor_t **editor,
void **edit_baton,
@@ -602,6 +615,7 @@ svnsync_get_sync_editor(const svn_delta_editor_t *
eb->wrapped_edit_baton = wrapped_edit_baton;
eb->base_revision = base_revision;
eb->to_url = to_url;
+ eb->source_prop_encoding = source_prop_encoding;
eb->quiet = quiet;
eb->normalized_node_props_counter = normalized_node_props_counter;
@@ -629,4 +643,3 @@ svnsync_get_sync_editor(const svn_delta_editor_t *
return SVN_NO_ERROR;
}
-
Index: subversion/tests/cmdline/svnrdump_tests.py
===================================================================
--- subversion/tests/cmdline/svnrdump_tests.py (revision 1070224)
+++ subversion/tests/cmdline/svnrdump_tests.py (working copy)
@@ -305,11 +305,17 @@ def url_encoding_load(sbox):
run_load_test(sbox, "url-encoding-bug.dump")
def copy_bad_line_endings_dump(sbox):
- "dump: inconsistent line endings in svn:props"
+ "dump: inconsistent line endings in svn:* props"
run_dump_test(sbox, "copy-bad-line-endings.dump",
expected_dumpfile_name="copy-bad-line-endings.expected.dump",
bypass_prop_validation=True)
+def copy_bad_line_endings2_dump(sbox):
+ "dump: non-LF line endings in svn:* props"
+ run_dump_test(sbox, "copy-bad-line-endings2.dump",
+ expected_dumpfile_name="copy-bad-line-endings2.expected.dump",
+ bypass_prop_validation=True)
+
def commit_a_copy_of_root_dump(sbox):
"dump: commit a copy of root"
run_dump_test(sbox, "repo-with-copy-of-root-dir.dump")
@@ -365,6 +371,7 @@ test_list = [ None,
move_and_modify_in_the_same_revision_dump,
move_and_modify_in_the_same_revision_load,
copy_bad_line_endings_dump,
+ copy_bad_line_endings2_dump,
commit_a_copy_of_root_dump,
commit_a_copy_of_root_load,
descend_into_replace_dump,
Index: subversion/tests/cmdline/svnsync_tests.py
===================================================================
--- subversion/tests/cmdline/svnsync_tests.py (revision 1070224)
+++ subversion/tests/cmdline/svnsync_tests.py (working copy)
@@ -27,6 +27,9 @@
# General modules
import sys, os
+# Test suite-specific modules
+import locale
+
# Our testing module
import svntest
from svntest.verify import SVNUnexpectedStdout, SVNUnexpectedStderr
@@ -57,18 +60,22 @@ def build_repos(sbox):
svntest.main.create_repos(sbox.repo_dir)
-def run_sync(url, source_url=None, expected_error=None):
+def run_sync(url, source_url=None, expected_error=None,
+ source_prop_encoding=None):
"Synchronize the mirror repository with the master"
if source_url is not None:
- exit_code, output, errput = svntest.main.run_svnsync(
- "synchronize", url, source_url,
+ args = ["synchronize", url, source_url,
"--username", svntest.main.wc_author,
- "--password", svntest.main.wc_passwd)
+ "--password", svntest.main.wc_passwd]
else: # Allow testing of old source-URL-less syntax
- exit_code, output, errput = svntest.main.run_svnsync(
- "synchronize", url,
+ args = ["synchronize", url,
"--username", svntest.main.wc_author,
- "--password", svntest.main.wc_passwd)
+ "--password", svntest.main.wc_passwd]
+ if source_prop_encoding:
+ args.append("--source-prop-encoding")
+ args.append(source_prop_encoding)
+
+ exit_code, output, errput = svntest.main.run_svnsync(*args)
if errput:
if expected_error is None:
raise SVNUnexpectedStderr(errput)
@@ -83,12 +90,17 @@ def build_repos(sbox):
# should be: ['Committed revision 1.\n', 'Committed revision 2.\n']
raise SVNUnexpectedStdout("Missing stdout")
-def run_copy_revprops(url, source_url, expected_error=None):
+def run_copy_revprops(url, source_url, expected_error=None,
+ source_prop_encoding=None):
"Copy revprops to the mirror repository from the master"
- exit_code, output, errput = svntest.main.run_svnsync(
- "copy-revprops", url, source_url,
+ args = ["copy-revprops", url, source_url,
"--username", svntest.main.wc_author,
- "--password", svntest.main.wc_passwd)
+ "--password", svntest.main.wc_passwd]
+ if source_prop_encoding:
+ args.append("--source-prop-encoding")
+ args.append(source_prop_encoding)
+
+ exit_code, output, errput = svntest.main.run_svnsync(*args)
if errput:
if expected_error is None:
raise SVNUnexpectedStderr(errput)
@@ -104,12 +116,16 @@ def build_repos(sbox):
# 'Copied properties for revision 2.\n']
raise SVNUnexpectedStdout("Missing stdout")
-def run_init(dst_url, src_url):
+def run_init(dst_url, src_url, source_prop_encoding=None):
"Initialize the mirror repository from the master"
- exit_code, output, errput = svntest.main.run_svnsync(
- "initialize", dst_url, src_url,
+ args = ["initialize", dst_url, src_url,
"--username", svntest.main.wc_author,
- "--password", svntest.main.wc_passwd)
+ "--password", svntest.main.wc_passwd]
+ if source_prop_encoding:
+ args.append("--source-prop-encoding")
+ args.append(source_prop_encoding)
+
+ exit_code, output, errput = svntest.main.run_svnsync(*args)
if errput:
raise SVNUnexpectedStderr(errput)
if output != ['Copied properties for revision 0.\n']:
@@ -139,7 +155,7 @@ def run_info(url, expected_error=None):
def setup_and_sync(sbox, dump_file_contents, subdir=None,
- bypass_prop_validation=False):
+ bypass_prop_validation=False, source_prop_encoding=None):
"""Create a repository for SBOX, load it with DUMP_FILE_CONTENTS, then create a mirror repository and sync it with SBOX. Return the mirror sandbox."""
# Create the empty master repository.
@@ -165,10 +181,12 @@ def setup_and_sync(sbox, dump_file_contents, subdi
repo_url = sbox.repo_url
if subdir:
repo_url = repo_url + subdir
- run_init(dest_sbox.repo_url, repo_url)
+ run_init(dest_sbox.repo_url, repo_url, source_prop_encoding)
- run_sync(dest_sbox.repo_url, repo_url)
- run_copy_revprops(dest_sbox.repo_url, repo_url)
+ run_sync(dest_sbox.repo_url, repo_url,
+ source_prop_encoding=source_prop_encoding)
+ run_copy_revprops(dest_sbox.repo_url, repo_url,
+ source_prop_encoding=source_prop_encoding)
return dest_sbox
@@ -190,7 +208,7 @@ def verify_mirror(dest_sbox, exp_dump_file_content
"Dump files", "DUMP", exp_dump_file_contents, dest_dump)
def run_test(sbox, dump_file_name, subdir=None, exp_dump_file_name=None,
- bypass_prop_validation=False):
+ bypass_prop_validation=False, source_prop_encoding=None):
"""Load a dump file, sync repositories, and compare contents with the original
or another dump file."""
@@ -204,7 +222,7 @@ or another dump file."""
'rb').readlines()
dest_sbox = setup_and_sync(sbox, master_dumpfile_contents, subdir,
- bypass_prop_validation)
+ bypass_prop_validation, source_prop_encoding)
# Compare the dump produced by the mirror repository with either the original
# dump file (used to create the master repository) or another specified dump
@@ -794,6 +812,40 @@ def copy_bad_line_endings(sbox):
exp_dump_file_name="copy-bad-line-endings.expected.dump",
bypass_prop_validation=True)
+def copy_bad_line_endings2(sbox):
+ "copy with non-LF line endings in svn:* props"
+ run_test(sbox, "copy-bad-line-endings2.dump",
+ exp_dump_file_name="copy-bad-line-endings2.expected.dump",
+ bypass_prop_validation=True)
+
+def copy_bad_encoding(sbox):
+ "copy and reencode non-UTF-8 svn:* props"
+ run_test(sbox, "copy-bad-encoding.dump",
+ exp_dump_file_name="copy-bad-encoding.expected.dump",
+ bypass_prop_validation=True, source_prop_encoding="ISO-8859-3")
+
+def identity_copy(sbox):
+ "copy UTF-8 svn:* props identically"
+ orig_lc_all = locale.setlocale(locale.LC_ALL)
+ other_locales = [ "English.1252", "German.1252", "French.1252",
+ "en_US.ISO-8859-1", "en_GB.ISO-8859-1", "de_DE.ISO-8859-1",
+ "en_US.ISO8859-1", "en_GB.ISO8859-1", "de_DE.ISO8859-1" ]
+ for other_locale in other_locales:
+ try:
+ locale.setlocale(locale.LC_ALL, other_locale)
+ break
+ except:
+ pass
+ if locale.setlocale(locale.LC_ALL) != other_locale:
+ raise svntest.Skip
+
+ try:
+ run_test(sbox, "copy-bad-encoding.expected.dump",
+ exp_dump_file_name="copy-bad-encoding.expected.dump",
+ bypass_prop_validation=True)
+ finally:
+ locale.setlocale(locale.LC_ALL, orig_lc_all)
+
#----------------------------------------------------------------------
def delete_svn_props(sbox):
@@ -912,6 +964,9 @@ test_list = [ None,
info_synchronized,
info_not_synchronized,
copy_bad_line_endings,
+ copy_bad_line_endings2,
+ copy_bad_encoding,
+ identity_copy,
delete_svn_props,
commit_a_copy_of_root,
descend_into_replace,
