Greetings all, just writing to let you all know the present status of
repomerge and a problem that we have encountered that you're likely to run
into later as well.
== Executive summary ==
* We probably have to rewrite git history to fix problems caused by the
repomerge script. This will break existing checkouts.
* Contributors: Please hold off on moving PRs.
The steps we have to take to correct the repository may mean that the PRs
will need to be migrated again (or they may just need to be rebased. We'll
test and let you know). We've disabled prmover.pythonanywhere.com for now
so no more prs are migrated.
* git log and git bisect are currently broken in the ansible/ansible devel
branch.
* Checkouts since repomerge (Yesterday, December 6, 2016 UTC) may need to
use some extra git flags to pull afterwards. We'll test and let you know
once we've fixed the history.
* The Ansible Core Team is working through the fixed merging strategies to
figure out which of them is needed to get things working..
== Detail ==
Yesterday (Tuesday, December 6, 2016) we ran a script which merged the
repositories, we performed some minimal checking, and then pushed the
branch to the devel branch on github.
Overnight someone commented that the history (via git log) didn't appear to
work quite right (the history of commits existed as shown by git blame but
git log --follow FILENAME was unable to find most of the commits).
Further testing showed that git bisect was also having difficulties with
the migrated repository: git bisect seemed to get stuck on some commits
instead of jumping to a new commit after tagging something as good or bad.
It also jumped far back into history instead of another commit within the
initial good/bad range.
Today (Wednesday, December 7, 2016) we've diagnosed the following problems
with the migrated repository:
* The log problems are caused because the merge script imported the files
at locations relative to the top of the module repos. At those locations,
git was unable to determine that the files should share a common history
with the files in their final locations (even when --follow was used). We
prototyped a fix for that which rewrote the file path prior to importing
the modules and found that we could fix git log --follow with that.
* The bisect problem was more mysterious for a while. Eventually we
realized that the merge script was interleaving the state of the
ansible/ansible tree with the state of the ansible-modules-core and
ansible-modules-extras tree. That means that at any given checkout, we
might have all of the ansible/ansible files and none of the modules or all
of ansible-modules-core and none of ansible/ansible or the same for
ansible-modules-extras. When trying to git bisect this tree, we then would
run across commits that we couldn't test accurately because it didn't have
all of the files necessary to test.
The latter problem doesn't appear to be fixable without removing these
commits. So we're probably going to need to reset the tree to before the
repomerge apply a fixed method of merging, and then git push --force the
rewritten tree to github. This will invalidate checkouts of the
ansible/ansible repository made after the repomerge:
"""
commit 38b621590311643a6c5867249415cddffc2400b9
Author: James Cammarata <j...@sngx.net>
Date: Tue Dec 6 10:57:39 2016 -0500
Das merge
"""
At the moment we're exploring two options as a fixed merge strategy. Both
start by resetting to the commit prior to the repomerge commit.
* The first option will be to simply copy the module files into the new
location in the ansible/ansible tree at this point. One commit will then
commit all of hte current modules. This strategy will mean that trying to
find history for the modules will require going back to the
ansible-modules-core and ansible-modules-extras repositories from time to
time.
* The second option is to try exporting patches from the
ansible-modules-core and ansible-modules-extras repositories (either via
git format-patch or git cherry-pick) and then reapplying those to a new
tree inside of ansible/ansible. This would preserve history but is more
work to implement. We're currently trying to find out if this is viable or
not.
On Tue, Nov 29, 2016 at 1:34 PM, James Cammarata <jcammar...@ansible.com>
wrote:
> Hi all, just wanted to announce/remind people about the upcoming repo
> merge.
>
> The Plan:
> -------------------------------------------------
>
> Next week, we'll be re-merging the module repositories back into
> ansible/ansible on GitHub. The two module repos will be essentially locked,
> though they will be kept in place for the existing 2.1 and 2.2
> dependencies. Once 2.2 moves out of official support (in about a year),
> we'll completely remove those repositories. Until then, any issues/PRs
> opened there will be auto-closed with a note to open it on ansible/ansible.
>
> Why Are We Doing This (Again...)?
> -------------------------------------------------
>
> For those who've been using Ansible long enough, you know that originally
> we started with a single repository. The original intention of the core vs.
> extras split was that core would be better supported/tested/etc. Extras
> would have been a bit more of a "wild-west" for modules, to allow new
> modules to make it into the distribution more quickly. Unfortunately this
> never really worked out, as well as the following:
>
> 1. Many modules in the core repo were also essentially "grand-fathered"
> in, despite not having a good set of tests or dedicated maintainers from
> the community.
> 2. The time in queue for modules to be merged into extras was not really
> any different from the time to merge modules into core.
> 3. The split introduced a few other problems for contributors such as
> having to submit multiple related PRs for modules with tests, or for those
> which rely on action plugins.
> 4. git submodules are notoriously complicated, even for contributors with
> decent git experience. The constant need to update git submodule pointers
> for devel and each stable branch can lead to testing surprises and really
> buys us nothing in terms of flexibility.
> 5. Users can already be confused about where to open issues, especially
> when the problem appears to be with a module but is actually an action
> plugin (ie. template) or something more fundamental like includes. Having
> everything back in one repo makes it easier to link issues, and you're
> always sure to open a bug report in the right place.
>
> Issues and PRs:
> -------------------------------------------------
>
> We're working on creating a tool which will allow users to migrate their
> issues from ansible-modules-core and ansible-modules-extras, details of
> that will be available next week as well. PRs will be a bit more tricky,
> though we are working on a script/process to make it easier for users to
> move their PRs from the existing repos into the ansible/ansible code base
> post-merge, however in most cases PRs will need to be resubmitted or left
> in the respective module repository until we can revisit them ourselves.
>
> Metadata - Support/Ownership and Module Status:
> ----------------------------------------------------------------------
>
> As part of this move, we will be introducing module metadata, which will
> contain a couple of pieces of information regarding modules:
>
> 1. Support Status: This field indicates who supports the module, whether
> it's the core team, the community, the person who wrote it, or if it is an
> abandoned module which is not receiving regular updates. The Ansible team
> has gone through the list of modules and we have marked about 100 of them
> as "Core Supported", meaning a member of the Ansible core team should be
> actively fixing bugs on those modules. The vast majority of the rest will
> be community supported. This is not really a change from the status quo,
> this just makes it clearer.
> 2. Module Status: This field indicates how well supported that module may
> be. This generally applies to the maturity of the module code however, not
> necessarily its bug status.
>
> The documentation pages for modules will be updated to reflect the above
> information as well, so that users can evaluate the status of a module
> before committing to using it in playbooks and roles.
>
> Summary:
> -------------------------------------------------
>
> For the vast majority of Ansible end-users, we expect this change to have
> little impact. The impact on developers will definitely be higher, though
> we're really trying to make it much smoother than when we made the original
> split.
>
> If you have any questions or concerns regarding this, please be sure to
> let us know ASAP.
>
> Thanks!
>
> James Cammarata
>
> Ansible Lead/Sr. Principal Software Engineer
> Ansible by Red Hat
> twitter: @thejimic, github: jimi-c
>
> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-devel+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Ansible Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-devel+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
