866260 - cut over build/* repos to the new vcs-sync system

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Hal Wine [:hwine] use NI!]

The original configs for the hg -> git for build/* repos are non optimal, and the instructions are also that way.

Two things to fix:
 - git repos should be created with '--bare' option (configure_repo script and docs)
 - git push default s/b +refs/heads/*:+refs/tags/* or mirror

Comment 1

[Aki Sasaki (not active)]

https://github.com/escapewindow/mozharness/compare/vcs_multi_repo has my wip for build/* repos.

Comment 2

[Aki Sasaki (not active)]

http://people.mozilla.org/~hwine/tmp/vcs2vcs/ has Hal's legacy system docs
https://wiki.mozilla.org/ReleaseEngineering/VCSSync/HowTo are my new system docs

Comment 3

[Pete Moore [:pmoore][:pete]]

Work in progress here: https://github.com/petemoore/build-mozharness/tree/bug866260

Comment 4

[Aki Sasaki (not active)]

Attachment: mozharness patch

I needed this patch to get the build/* repos converting properly.

Comment 5

[Aki Sasaki (not active)]

Attachment: build-repos.py

This was my build repos config file.

Comment 6

[Pete Moore [:pmoore][:pete]]

Attachment: Mozharness patch

Hi Aki,

This code has been tested in github-sync2, pushing to https://github.com/petermoore/build-* repos.

Some specific points:

1) There is a cron set up to run the sync process every minute under my user account. Eventually this should be moved to vcs2vcs user.

Here is the cron definition:

[[e-mail]3 ~]$ sudo cat /etc/cron.d/build-repos-vcs-sync 
MAILTO="[e-mail]"
PATH="/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/pmoore/bin"

# Every minute
* * * * * pmoore /home/pmoore/vcs_sync/run_build_repos_sync.sh

MAILTO=root

2) I have installed lockfile executable under /home/pmoore/bin/lockfile
3) I created a directory /home/pmoore/vcs_sync
4) The sync script:

[[e-mail]3 ~]$ cat /home/pmoore/vcs_sync/run_build_repos_sync.sh
#!/bin/bash -e
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
cd "$(dirname "${0}")"
exec > build-repos-vcs-sync.log 2>&1

function log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') pid-$$ $*"
}
log "Acquiring lock"
~/bin/lockfile -s60 -r5 build-repos-vcs-sync.lock
trap "rm -f $PWD/build-repos-vcs-sync.lock" EXIT

# Get mozharness updated / checked out and working
log "Updating mozharness"
(cd mozharness; timeout 20 git pull)
log "Running vcs_sync.py"
python mozharness/scripts/vcs-sync/vcs_sync.py -c mozharness/configs/vcs_sync/build-repos.py
# Touch our timestamp file so nagios can check if we're fresh (not currently implemented)
touch build-repos-vcs-sync.stamp
log "Done"

5) Mozharness checked out with my user, by running:

git clone git://github.com/petemoore/build-mozharness /home/pmoore/vcs_sync/mozharness

6) For pushing to github account, ssh key-pair stored is stored here:
/home/pmoore/.ssh/github_mozilla_rsa
/home/pmoore/.ssh/github_mozilla_rsa.pub

This will need to be switched to a key pair that works with the official mozilla github account, when we switch over.

7) The config and log files are currently being uploaded to:

asasaki@github-sync2:/home/asasaki/upload/build-repos-upload

I stole your key-pair to do this, and they are stored here:
/home/pmoore/vcs_sync/ssh_keys/id_rsa
/home/pmoore/vcs_sync/ssh_keys/id_rsa.pub

8) It is necessary one time to manually ssh onto github-sync2 (localhost) and one time to issue a git command interactively on the console to github account, in order to get the two server's host keys added to the user's ssh known_hosts file, before running for the first time

Regarding the remaining anomalies between the https://github.com/petermoore/build-* repos (new vcs sync results) and the https://github.com/mozilla/build-* repos (currently legacy vcs sync results), I will post a separate attachment of details.

Comment 7

[Pete Moore [:pmoore][:pete]]

regarding point 5 above, I also switched to the bug866260 branch of that repo...

cd /home/pmoore/vcs_sync/mozharness
git branch -t bug866260 origin/bug866260
git checkout bug866260
git pull

Comment 8

[Pete Moore [:pmoore][:pete]]

Attachment: Same patch as before, but removed .pydevproject IDE file

Whoops, I see my IDE project file got caught up in the mix - have removed from patch...

Comment 9

[Pete Moore [:pmoore][:pete]]

Attachment: Current differences in target repos of legacy vcs2vcs system and new vcs2vcs system for build-* repos

I still need to look at these myself, but for purposes of transparency, here are the current set of anomalies I see between the legacy system and the new system.

Several factors could be involved:
1) legacy system is also pushing closed mercurial branches
2) repos can be marginally out-of-sync with each other, if between the last time the legacy system ran and the new system ran there were commits or tags applied
3) there could be legacy branches/tags that were deleted, but had already got pushed to github - i believe the legacy system (and the new system) do not clean up branches/tags that get removed

Therefore this list of anomalies is not necessarily a reason for concern, but I will still investigate the reasons for the differences, to make sure everything is working as we wish it to.

Comment 10

[Pete Moore [:pmoore][:pete]]

Last comment in this thread(!): please note I am only using http://pypi.pub.build.mozilla.org/pub pip binary store, not http://pypi.pvt.build.mozilla.org/pub, to protect against the latter not being available from within the DMZ.

Happy reviewing! =)

Pete

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 8375040 [details]
Current differences in target repos of legacy vcs2vcs system and new vcs2vcs system for build-* repos

> build-buildbot-configs
> ======================
> 4,5c4,5
> < e42e0136332d8d3bb27e497231fb1aecccbb7a5a    refs/heads/production

https://github.com/mozilla/build-buildbot-configs/commit/e42e0136332d8d3bb27e497231fb1aecccbb7a5a exists, so this is probably a "production is newer than your conversion" thing.

> < 1ae1cb640f78a15f9ee1898d5f21addd1d720735    refs/heads/seamonkey-production

This does *not* exist on build-buildbot-configs: https://github.com/mozilla/build-buildbot-configs/commit/1ae1cb640f78a15f9ee1898d5f21addd1d720735
It's possible that there's a SHA mismatch here.

So.  This is a matter of both

a) does anyone care if we blow away the existing repos and populate it with new SHAs? Especially the seamonkey-production branch, which probably only Callek cares about (and I don't think he uses github for his buildbot-configs work) and
b) do we want to get you the practice of dealing with SHA divergence, since l10n and gecko.git will require the SHAs match exactly?

If the answer to those above two questions is 'no', then it doesn't matter.  If either is 'yes' to the point where we are ok with delaying the build/* rollout til this is dealt with, you'll probably need to work with Hal to figure this out.

https://bugzilla.mozilla.org/show_bug.cgi?id=847727#c40 and later comments show what I did when I hit SHA divergence in beagle (gecko-dev, gecko-projects).  This will at least be partially different from what you're experiencing, because a lot of the issues I hit were in git-filter-branch and CVS prepending (which we're not doing in build/*), but it has a lot of hints about how to debug, like https://bugzilla.mozilla.org/show_bug.cgi?id=847727#c43 and the bottom of https://bugzilla.mozilla.org/show_bug.cgi?id=847727#c48


My vote: warn everyone the SHAs may change for seamonkey-production, and push with -f.
L10n will get overwritten and will probably not diverge; gecko.git likely shouldn't diverge from gecko-projects and gecko-dev since they're so similar.  This exercise may be wasted time.

> 20a21,22
> > 3b773a196fdd85207ad12c9144001743d63abee9    refs/tags/FENNEC_10_0_6esr_BUILD1
> > 3b773a196fdd85207ad12c9144001743d63abee9    refs/tags/FENNEC_10_0_6esr_RELEASE
<snip>

I'm still guessing these were on old heads.  Again, we can make this an exercise of completeness for practice's sake, or we can Ask to see if any users of the github repos care about these old tags and ignore 'em.  Pretty sure the build/* repos are on github for ease of development and not for production accuracy.

> build-buildbotcustom
> ====================

<snip>

So look at the github/mozilla/build-REPO/commit/SHA to make sure your shas exist in each repo as I did above.
If so, we're probably good.  If not, as above.  Missing tags in other repos as above.

Comment 12

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #11)
> My vote: warn everyone the SHAs may change for seamonkey-production, and
> push with -f.

That is, push with -f the initial time.  Whether we push with -f on an ongoing basis is a topic of discussion for build/* (likely; we're doing so for gecko-projects already iirc).  We absolutely do not want to push with -f for gecko.git ever, and not on an ongoing basis for l10n.

Comment 13

[Hal Wine [:hwine] use NI!]

Comment on attachment 8375040 [details]
Current differences in target repos of legacy vcs2vcs system and new vcs2vcs system for build-* repos

I agree with Aki on most points, but I think understanding the root cause is worth some time.

Just double checking since this is on your laptop -- how do the versions of python, hg, and hg-git compare to production?

Another very useful check would be to compare the mapfile contents from the hg directories. If they aren't equivalent, one should be a proper subset of the other (the older a proper subset of the newer). You can grab the mapfile from gd0 for buildbot-configs repo -- let me know if you need a hand with that.

What puzzles me most is that it is older tags that are off, but newer are okay. Once the hash chain diverges, it should not fix itself. Which would require that each of these tags is a head. But that's not the case according to 'hg heads -t' (for buildbot-configs at least).

Final note - if we do decide to just overwrite the current repos, we'll need to coordinate with other folks who have forked the repos on github. Mostly team members, iirc.

Comment 14

[Aki Sasaki (not active)]

(In reply to Hal Wine [:hwine] (use needinfo) from comment #13)
> Just double checking since this is on your laptop -- how do the versions of
> python, hg, and hg-git compare to production?

They're using my gecko-{dev,projects} fork for hg-git, as opposed to the current build/* conversions which are probably a different version.

If it's all the same, I'd prefer to keep all new vcs-sync on the same version of hg-git (and other modules).  If we for some reason have a requirement to keep the same build/* shas, we may have to use a different hg-git version for those repos only.

Comment 15

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #14)
> (In reply to Hal Wine [:hwine] (use needinfo) from comment #13)
> > Just double checking since this is on your laptop -- how do the versions of
> > python, hg, and hg-git compare to production?
> 
> They're using my gecko-{dev,projects} fork for hg-git, as opposed to the
> current build/* conversions which are probably a different version.
> 
> If it's all the same, I'd prefer to keep all new vcs-sync on the same
> version of hg-git (and other modules).  If we for some reason have a
> requirement to keep the same build/* shas, we may have to use a different
> hg-git version for those repos only.

Unstated, so stating for clarity: we cannot change the hg-git fork we're using on gecko-{dev,projects} without introducing unneeded risk for SHA divergence and potentially removing our ability to recreate the history from scratch if needed in the future, so we either keep the fork for all new vcs-sync conversions, or we use a different hg-git for build/* only.  Changing the gecko-{dev,projects} hg-git is not an option.

Comment 16

[Aki Sasaki (not active)]

Comment on attachment 8375030 [details] [diff] [review]
Same patch as before, but removed .pydevproject IDE file

r=me with caveats.  If you prefer to resubmit for re-review, feel free.  I think I'm being clear with my comments, but feel free to ask.

> diff --git a/configs/vcs_sync/build-repos.py b/configs/vcs_sync/build-repos.py

I diffed this against https://bug866260.bugzilla.mozilla.org/attachment.cgi?id=8372441 for sanity.

> +for repo in build_repos:
> +    conversion_repos.append({
> +        "repo": "https://hg.mozilla.org/build/%s" % repo,
> +        "repo_name": "build-%s" % repo,
> +        "conversion_dir": "build-%s" % repo,
> +        "targets": [{
> +            "target_dest": "build-%s/.git" % repo,
> +            "vcs": "git",
> +            "test_push": True,

If we decide to push -f, you can set "force_push": True here like in https://bug866260.bugzilla.mozilla.org/attachment.cgi?id=8372441

> +        }, {
> +            "target_dest": "build-%s-github" % repo,
> +        }],
> +        "vcs": "hg",
> +

Nit: trailing whitespace.

> +    remote_targets["build-%s-github" % repo] = {
> +        "repo": "[e-mail]:mozilla/build-%s.git" % repo,
> +        "ssh_key": "~/.ssh/github_mozilla_rsa",

On vcssync1 and 2, this key is named ~/.ssh/releng-github-id_rsa
When we start pushing to the /mozilla/ github repos (in production), let's use the same name.

> +config = {
> +    "log_name": "build-repos",
> +    "log_max_rotate": 99,
> +    "job_name": "build-repos",
> +    "env": {
> +        "PATH": "%(PATH)s:/usr/libexec/git-core",
> +    },
> +    "conversion_repos": conversion_repos,
> +    "remote_targets": remote_targets,
> +    "virtualenv_modules": [
> +        "bottle==0.11.6",
> +        "dulwich==0.9.0",
> +        "ordereddict==1.1",
> +        "hg-git==0.4.0-moz2",
> +        "mapper==0.1",
> +        "mercurial==2.6.3",
> +        "mozfile==0.9",
> +        "mozinfo==0.5",
> +        "mozprocess==0.11",
> +    ],
> +    "find_links": [
> +        "http://pypi.pub.build.mozilla.org/pub"
> +    ],
> +    "pip_index": False,
> +
> +    "upload_config": [{
> +        "ssh_key": "/home/pmoore/vcs_sync/ssh_keys/id_rsa",
> +        "ssh_user": "asasaki",
> +        "remote_host": "github-sync2",
> +        "remote_path": "/home/asasaki/upload/build-repos-upload",
> +    }],

This should change for production.  Either we care about mapfile publishing, or we don't.  If not, remove upload_config; if we do, we need the upload location to be somewhere visible to the public.

> +
> +    "default_notify_from": "vcs2vcs@%s" % hostname,
> +    "notify_config": [{
> +        "to": "[e-mail]",

release+vcs2vcs@, please

> +        "failure_only": False,
> +        "skip_empty_messages": True,
> +    }],

If you want more verbose info from the email for the initial rollout, you can add yourself to this list with "skip_empty_messages": False.  You'll get an email almost every minute, so make sure your filters are set up.  This lets you see what's going on a lot more clearly, though, if you take the time to parse it all.  Purely optional.

> +
> +    # Disallow sharing, since we want pristine .hg and .git directories.
> +    "vcs_share_base": None,
> +    "hg_share_base": None,
> +

Nit: trailing whitespace.

> +    # any hg command line options
> +    "hg_options": (
> +        "--config",
> +        "web.cacerts=/etc/pki/tls/certs/ca-bundle.crt"
> +    )
> +}
> \ No newline at end of file

You'll need to add the combined_mapfile if you want to add this to mapper, but that's not a hard requirement for rollout.

> diff --git a/scripts/vcs-sync/vcs_sync.py b/scripts/vcs-sync/vcs_sync.py

Because you're making changes to this script, there's a chance that you'll affect gecko-dev and gecko-projects (and the not-running l10n and gecko-git) conversions with these changes.  If gecko-dev goes sideways that's going to be a large political Event.  I'm trying to keep an eye towards that in this review, though.

You'll need to make further changes to this script in the future, so please keep in mind that if you make changes, you may need to update all of http://hg.mozilla.org/build/mozharness/file/f3e61d3d1ee2/configs/vcs_sync to match.  Any changes you're worried about would be best if you do at least some testing in the various configs -- I realize this matrix will grow, but hopefully soon you'll have build/*, l10n, and gecko-git figured out in staging, so it shouldn't be too difficult to do a quick test run.

> index 8d2ed8c..c9eb54a 100755
> --- a/scripts/vcs-sync/vcs_sync.py
> +++ b/scripts/vcs-sync/vcs_sync.py
> @@ -55,20 +55,21 @@ class HgGitScript(VirtualenvMixin, TooltoolMixin, TransferMixin, VCSSyncScript):
>      mapfile_binary_search = None
>      all_repos = None
>      successful_repos = []
> -    config_options = [[
> -        ["--no-check-incoming", ],
> -        {"action": "store_false",
> -         "dest": "check_incoming",
> -         "default": True,
> -         "help": "Don't check for incoming changesets"
> -         }
> -    ]]
> +    config_options = [
> +        [["--no-check-incoming"], {
> +            "action": "store_false",
> +            "dest": "check_incoming",
> +            "default": True,
> +            "help": "Don't check for incoming changesets"
> +        }]
> +    ]

Looks like a whitespace-only change.  I think both sets are PEP 8 compliant, so no-op.

> @@ -314,7 +316,7 @@ intree=1
>          """ Update a stage repo.
>              See update_stage_mirror() for a description of the stage repos.
>              """
> -        hg = self.query_exe('hg', return_type='list')
> +        hg = self._query_hg_exe()
>          dirs = self.query_abs_dirs()
>          repo_name = repo_config['repo_name']
>          source_dest = os.path.join(dirs['abs_source_dir'],
> @@ -436,7 +438,7 @@ intree=1
>              self.fatal("No conversion_dir for %s!" % repo_config['repo_name'])
>          source_dir = os.path.join(dirs['abs_source_dir'], repo_config['repo_name'])
>          git = self.query_exe('git', return_type='list')
> -        hg = self.query_exe('hg', return_type='list')
> +        hg = self._query_hg_exe()
>          return_status = ''
>          for target_config in repo_config['targets']:
>              test_push = False
> @@ -512,7 +514,7 @@ intree=1
>                              continue
>                          tag_name = tag_parts[0]
>                          for regex in regex_list:
> -                            if regex.search(tag_name) is not None:
> +                            if tag_name != 'tip' and regex.search(tag_name) is not None:
>                                  refs_list += ['+refs/tags/%s:refs/tags/%s' % (tag_name, tag_name)]
>                                  continue
>                  error_msg = "%s: Can't push %s to %s!\n" % (repo_config['repo_name'], conversion_dir, target_name)
> @@ -605,6 +607,19 @@ intree=1
>              create_parent_dir=True
>          )
>
> +    def _query_hg_exe(self):
> +        """Returns the hg executable command as a list
> +        """
> +        # If "hg" is set in "exes" section of config use that.
> +        # If not, get path from self.query_virtualenv_path() method
> +        # (respects --work-dir and --venv-path and --virtualenv-path).
> +        exe_command = self.query_exe('hg', return_type="list", default=[os.path.join(self.query_virtualenv_path(), "bin", "hg")])
> +

Nit: trailing whitespace.

> +        # possible additional command line options can be specified in "hg_options" of self.config
> +        hg_options = self.config.get("hg_options", ())
> +        exe_command.extend(hg_options)

You're doing quite a bit of combining tuples and lists.  This works, but I'm wondering if it's intentional.  This looks like it won't affect gecko-dev or gecko-projects negatively.

The net effect of this change shouldn't really matter for production, and introduces some risk.  If you decide not to go through with this, that works.  If you decide it's better (I agree it is, just like limiting risk for gecko-{dev,projects}) please keep an eye on those two as well when you roll this out.

> +    def list_repos(self):
> +        repos = self.query_all_repos()
> +        self.info(pprint.pformat(repos))
> +

Nit: trailing whitespace.


The notify_config must change.  The upload_config should change; the trailing whitespaces are an affront to all things good ;-)  The key name is good hygiene.  I think the vcs_sync.py changes are fairly low risk.

I'll let Hal be the arbiter on whether/when this rolls out :)

Comment 17

[Hal Wine [:hwine] use NI!]

(In reply to Aki Sasaki [:aki] from comment #14)
> (In reply to Hal Wine [:hwine] (use needinfo) from comment #13)
> > Just double checking since this is on your laptop -- how do the versions of
> > python, hg, and hg-git compare to production?
> 
> They're using my gecko-{dev,projects} fork for hg-git, as opposed to the
> current build/* conversions which are probably a different version.

I know that's the intent - I just wanted to verify that _is_ what is installed on Pete's laptop.

> 
> If it's all the same, I'd prefer to keep all new vcs-sync on the same
> version of hg-git (and other modules).  If we for some reason have a
> requirement to keep the same build/* shas, we may have to use a different
> hg-git version for those repos only.

Absolutely! The new version is the production version! Split versions would be a nightmare.

My point is both the legacy production system and vcs-sync production system, while using different versions do produce the same shas for a very complex set of repositories. The different one is sitting on a laptop -- I'm going to suspect that first. :)

Comment 18

[Aki Sasaki (not active)]

[19:51]	<aki>	hwine: vcs_sync.py builds its own venv with the specific versions of the modules, so pete's probably using the same modules as new vcs-sync production
[19:51]	<aki>	but sure, worth a look
[19:51]	<hwine>	but same python?

I think this latest conversion was done on gd2, so we shouldn't see wildly divergent stuff aiui.

[19:51]	<hwine>	and, to be clear, we're in 100% agreement on not changing gecko*.git
[19:57]	<hwine>	aki: not being familiar with things, I'm not sure I see the benefit of the change to self._query_hg_exe -- it looks like changed functionality, not a straight refactor. What am I missing?

Also, I want to be sufficiently paranoid but not overly paranoid here.
The change is a good one: it allows you to specify a base_work_dir and work_dir that aren't os.getcwd() and 'build', which are the defaults.  The change doesn't block rollout, since assuming the workdir is ./build works just fine for gecko-{dev,projects}.  The change also at least looks harmless to gecko-{dev,projects}, so I'm inclined to leave it.

I want to be paranoid enough to limit risk, but I also want Pete to feel like he has ownership and a comfort level here; I'm certain you'll do just fine as his sense of paranoia, Hal :)

Comment 19

[Pete Moore [:pmoore][:pete]]

Many thanks guys for all your comments! And to think I was worried I was providing too much information! :)

So - a lot to process here, but a first few key points:

1) Aki you are right - this is running in gd2, *not* on my laptop. The hg-git version is pulled in by pip and defined in the code, placed in a sandbox (a local python virtualenv) so should not in any case be affected by the local environment. The git version used is the system git version.

2) I agree about balancing the risk of a change with the improvements a change might bring. In the case of the new self._query_hg_exe() method - this has been thoroughly tested in gd2, and if it doesn't work correctly, it would not return a valid hg exe with the hg-git plugin, and therefore the entire process would break - which in theory should not do any damage. This is not a change which would introduce risk several iterations after going live - it would present itself on the very first iteration - therefore this should have extremely low risk of impacting the target repos.

Comment 20

[Pete Moore [:pmoore][:pete]]

Hal, what is your opinion on force-push for build-* repos?

I think I prefer not to use this option as a risk limitation, at least initially; but I think Aki is in favour - curious what your opinion is.

I have a former colleague that happened to have some git push automation set up for the Hudson open source project, and a few months ago, his tool hit an issue and got outdated revisions in his working dir, and he rolled back over 100 open source repos back to an outdated state because his automation was using the --force-push option. The risk on the other side is that there is a genuine non-fast-forward commit intentionally done and then vcs2vcs stops working. For me this seems lower risk as I can't imagine anyone would want to do a non-fast-forward commit in hg, as it could impact others in the team that had the repo checked out, and in such a case, not only the vcs2vcs system would have a problem, but probably many of our team when we went to update our working dirs.

Also I think it could be beneficial to have a consistent policy across projects (and we won't be doing this for gecko-dev, l10n etc) since then we are hitting the same lines of code, and hopefully introducing less variation between the various processes.

But I'm curious what you think!

For your reference, here is the "disaster" thread that occurred in jenkins when --force-push was used in the automation: https://groups.google.com/forum/#!topic/jenkinsci-dev/-myjRIPcVwU

I think the native git mechanism of not allowing non-fast-forward commits (unless the --force option is used) offers at least some protection if the system breaks, suddenly hashes change etc - it just gets paused, rather than potentially corrupting a repository with bad (historic) shas or wrong heads etc.

Comment 21

[Aki Sasaki (not active)]

fyi, gecko-projects is using force push, since project branches get reset etc., and we're very clear that the SLA for gecko-projects is lower than gecko-dev.  (we reserve the right to reset gecko-projects if needed.)  Feel free to be more paranoid than me, though, as long as it doesn't leave us moving too slowly :)

Comment 22

[Pete Moore [:pmoore][:pete]]

I can go either way - I'll let Hal take the deciding vote. :)

Comment 23

[Pete Moore [:pmoore][:pete]]

You think it is ok to add .pydevproject (my IDE project file) to .hgignore - so that I don't need to keep excluding it from my patches (or worse still accidentally include it), or is it preferred not to include IDE files in .hgignore / .gitignore ?

Comment 24

[Aki Sasaki (not active)]

(In reply to Pete Moore [:pete][:pmoore] from comment #23)
> You think it is ok to add .pydevproject (my IDE project file) to .hgignore -
> so that I don't need to keep excluding it from my patches (or worse still
> accidentally include it), or is it preferred not to include IDE files in
> .hgignore / .gitignore ?

Yes, feel free to add it.
Also, if .project doesn't make sense to have in-tree, we can |hg rm| it and add to .hgignore.

Comment 25

[Hal Wine [:hwine] use NI!]

(In reply to Pete Moore [:pete][:pmoore] from comment #22)
> I can go either way - I'll let Hal take the deciding vote. :)

We have several distinct classes of repositories that we convert:
 - those that are directly included in shipped code
 - those that are 'mirrored'
 - all others (this could be subdivided in the future if need be)

For the "shipped code" repos, we should never allow history to be re-written. This can (and is) enforced by proper setting of RoR server config. That effectively disables --force, and means we shouldn't even attempt it.

For the "mirror" repos, if the RoR changes, the downstream repos should change as well. So --force is correct (and save a bunch of tracking new branches and tags and ...). If the downstream diverges from the RoR that's a bug in conversion, and end users can detect that by reference to the RoR.

For all others, it's negotiable, and we should do what makes sense for the business purpose of the converted repository. Hence gecko-projects is forced and gecko-dev is not. We just have to make sure we have the agreement documented somewhere.

Note that our conversion system never touches the RoR, and the RoR (not downstream conversions) is the SSOT (single source of truth). That allows for quick problem detection, and a clear recovery path.

In all cases, any converted repository should (must) be completely re-constructable from the RoR and the conversion code. That means we have a very tight configuration management on the tooling. This gives us a "disaster recovery" path, and minimizes the chances for 'oops'.

Comment 26

[Pete Moore [:pmoore][:pete]]

Attachment: Updated Mozharness patch

Based on all previous feedback, an updated patch to incorporate all discussed changes...

Comment 27

[Aki Sasaki (not active)]

Comment on attachment 8375885 [details] [diff] [review]
Updated Mozharness patch

>+for repo in build_repos:
>+    conversion_repos.append({
>+        "repo": "https://hg.mozilla.org/build/%s" % repo,
>+        "repo_name": "build-%s" % repo,
>+        "conversion_dir": "build-%s" % repo,
>+        "targets": [{
>+            "target_dest": "build-%s/.git" % repo,
>+            "vcs": "git",
>+            "test_push": True,
>+            "force_push": True
>+        }, {
>+            "target_dest": "build-%s-github" % repo,
>+            "force_push": True
>+        }],

The test_push is really there to make sure we don't do a non-ffwd push afterwards.
Since we're force_push'ing, we can get rid of the first target, and leave

>+        "targets": [{
>+            "target_dest": "build-%s-github" % repo,
>+            "force_push": True
>+        }],

Your patch as-is isn't harmful, but will be slower and use more disk.

Thanks!

Comment 28

[Hal Wine [:hwine] use NI!]

Comment on attachment 8375885 [details] [diff] [review]
Updated Mozharness patch

Review of attachment 8375885 [details] [diff] [review]:
-----------------------------------------------------------------

Okay - my question is a just that, although if the modules aren't used, I'm fine with not having them in there. :)

::: configs/vcs_sync/build-repos.py
@@ +71,5 @@
> +        "bottle==0.11.6",
> +        "dulwich==0.9.0",
> +        "ordereddict==1.1",
> +        "hg-git==0.4.0-moz2",
> +        "mapper==0.1",

sorry - this may have been answered already, but why include the mapper & bottle code here? I'm not aware of the use case for the mapper with these repos.

Comment 29

[Pete Moore [:pmoore][:pete]]

Hey Hal,

The only reason is consistency with the other vcs2vcs services. Ideally in a future iteration I would like to pull this config out and move it directly into the vcs_sync module, since it is identical across all vcs2vcs configs, and probably should remain so. If I pull those out now, it will start diverging from the other sync services. Also having it there means mapper it is simpler to include map files for build/* repos in the future.

Pete

Comment 30

[Hal Wine [:hwine] use NI!]

Fine - that's reasonable. It's always a tradeoff on how configs are inherited. 

I think sendmail used to have this issue to.... maybe with some m4 macros, we could .... ;)

Comment 31

[Pete Moore [:pmoore][:pete]]

Checking diffs between old and new vcs...

build-buildbot-configs/production
=================================
RoR:    http://hg.mozilla.org/build/buildbot-configs/rev/7b470f6c4b62
Legacy: https://github.com/mozilla/build-buildbot-configs/tree/production
New:    https://github.com/petermoore/build-buildbot-configs/tree/production

Looks like "New" is consistent with "RoR" but Legacy is out-of-date

Comment 32

[Pete Moore [:pmoore][:pete]]

Same with:

build-buildbot-configs/seamonkey-production
===========================================
RoR:    http://hg.mozilla.org/build/buildbot-configs/rev/603b016e0501
Legacy: https://github.com/mozilla/build-buildbot-configs/tree/seamonkey-production
New:    https://github.com/petermoore/build-buildbot-configs/tree/seamonkey-production

Comment 33

[Pete Moore [:pmoore][:pete]]

Same with:

build-buildbotcustom
====================
6,7c6,7
< 6f5815cfd1f60dfe20d53821cdf7d58dd9c51357	refs/heads/production-0.8
< 22ab9f61a16461006efb8c9fa86c843e2e07f483	refs/heads/seamonkey-production
---
> 95dbc648e6a246632e1a963fb16e046e385a4d8b	refs/heads/production-0.8
> 02bfdccd6bb6b369d4aa2d5dec7c2e0806b60c94	refs/heads/seamonkey-production

Comment 34

[Pete Moore [:pmoore][:pete]]

Same with:

build-puppet
============
3c3
< 2f54f7e4e878850af8a794091562030c76bfe79a	refs/heads/production
---
> 0b93477f135cc204abeea5f592fd26256e188eaf	refs/heads/production

Comment 35

[Pete Moore [:pmoore][:pete]]

2 issues here:


build-mozharness
================
1c1
< 22c487845402714312c3fee89368171b4d0749b6	HEAD
---
> f76ac3f29a1b8ae7db640b797cb431f3cbfc8e79	HEAD
3,4c3,4
< 22c487845402714312c3fee89368171b4d0749b6	refs/heads/master
< 3c94ec1b8c2624b2e88950541b314c1b1b12050b	refs/heads/production
---
> f76ac3f29a1b8ae7db640b797cb431f3cbfc8e79	refs/heads/master
> 3751792c34930eaea889721bbe50c565b9c223a6	refs/heads/production

1) production branch difference: same as examples above - legacy vcs is out-of-date
2) master branch difference: was just slightly out-of-sync due to when this report ran; now however, both repos have same master - so a non-issue

Comment 36

[Pete Moore [:pmoore][:pete]]

build-mozharness
================
728a729
> 840f812ec84c349fecb8c4acfaa67b0ebb9b314d	refs/tags/FIREFOX_23_0_1_BUILD2
761a763,764
> b9972d1c1ad21accaf3ad4572426ca37be36a279	refs/tags/FIREFOX_24_0b3_BUILD1
> b9972d1c1ad21accaf3ad4572426ca37be36a279	refs/tags/FIREFOX_24_0b3_RELEASE


These 3 tags are no longer in the RoR: http://hg.mozilla.org/build/mozharness/tags
They are listed in the legacy git repo, not the new git repo.
So looks like they were created at some point, sync'd to git legacy repo and then deleted in RoR. So it is right they are not in new git repo - it is a non-issue.

Comment 37

[Pete Moore [:pmoore][:pete]]

build-buildbot-configs
======================
20a21,22
> 3b773a196fdd85207ad12c9144001743d63abee9	refs/tags/FENNEC_10_0_6esr_BUILD1
> 3b773a196fdd85207ad12c9144001743d63abee9	refs/tags/FENNEC_10_0_6esr_RELEASE
64a67,132
> d5f268fa57d44dbc6de7c5afc375214f96e58788	refs/tags/FENNEC_13_0b2_BUILD1
> 8aa49ffd790b8ca0fb914aedcdb62abdcf962086	refs/tags/FENNEC_13_0b2_BUILD2
> a85487918295d66da65560e475ca69b9b02bd69e	refs/tags/FENNEC_13_0b2_BUILD3
> a85487918295d66da65560e475ca69b9b02bd69e	refs/tags/FENNEC_13_0b2_RELEASE
> 0b63bfeb2d21d80a603b5c5c95d2a2b682925fe8	refs/tags/FENNEC_13_0b3_BUILD1
> 0b63bfeb2d21d80a603b5c5c95d2a2b682925fe8	refs/tags/FENNEC_13_0b3_RELEASE
> bd59c2323bce0367c773c77acda2f740004f80ca	refs/tags/FENNEC_14_0_1_BUILD1
> 1705844f368f545dee94f74633d7c37cdd8eb024	refs/tags/FENNEC_14_0_1_BUILD2
> 1705844f368f545dee94f74633d7c37cdd8eb024	refs/tags/FENNEC_14_0_1_RELEASE
> 0b6ec632e32e5b2f80557aea9db33dac2abcc783	refs/tags/FENNEC_14_0_2_BUILD1
> ed4755eec32490a676bc122fbd742f2840189e4a	refs/tags/FENNEC_14_0_2_BUILD2
> ed4755eec32490a676bc122fbd742f2840189e4a	refs/tags/FENNEC_14_0_2_RELEASE
> e4c9ecbabcad5bcede93b312a4dc58f85466a7f3	refs/tags/FENNEC_14_0_BUILD1
> 3e3c636797e8ed8a7c3bb80d6c5d75a182cb01a0	refs/tags/FENNEC_14_0_BUILD2
> 3e3c636797e8ed8a7c3bb80d6c5d75a182cb01a0	refs/tags/FENNEC_14_0_RELEASE
> 6190915ec1b3b9186f253965a6b3a53768b6b563	refs/tags/FENNEC_14_0b10_BUILD1
> 2f7d3d88ddb4130f9e3079a336f7e2e770003e7c	refs/tags/FENNEC_14_0b10_BUILD2
> 2f7d3d88ddb4130f9e3079a336f7e2e770003e7c	refs/tags/FENNEC_14_0b10_RELEASE
> 429dab933600eb479149ecf5e23c42c530465546	refs/tags/FENNEC_14_0b11_BUILD1
> 99f5256fef09b9e9d6bb4951eb22410268ea5f02	refs/tags/FENNEC_14_0b11_RELEASE
> ab27c50824c18915a6f357dbc6334a1ec3588095	refs/tags/FENNEC_14_0b12_BUILD1
> ab27c50824c18915a6f357dbc6334a1ec3588095	refs/tags/FENNEC_14_0b12_RELEASE
> 859b8e4425173a5a910c5152b36cabd6b2f45369	refs/tags/FENNEC_14_0b1_BUILD1
> 2b2de4b08d664e94e01e80588db3c13bc3cec842	refs/tags/FENNEC_14_0b1_BUILD2
> 5e4465a9bbe1cd63fd89db2036eb0b314e23396c	refs/tags/FENNEC_14_0b1_BUILD3
> 5e4465a9bbe1cd63fd89db2036eb0b314e23396c	refs/tags/FENNEC_14_0b1_RELEASE
> a85487918295d66da65560e475ca69b9b02bd69e	refs/tags/FENNEC_14_0b2_BUILD1
> a85487918295d66da65560e475ca69b9b02bd69e	refs/tags/FENNEC_14_0b2_RELEASE
> 19fc46576cfc25168b3072553698502969fcbd87	refs/tags/FENNEC_14_0b3_BUILD1
> 0d27fb05926088dec36fe25281d08dc5351cff5c	refs/tags/FENNEC_14_0b3_BUILD2
> 0d27fb05926088dec36fe25281d08dc5351cff5c	refs/tags/FENNEC_14_0b3_RELEASE
> 8c319b63243a6d2c8dcfbe7ab2706293b35741b9	refs/tags/FENNEC_14_0b4_BUILD1
> 8c319b63243a6d2c8dcfbe7ab2706293b35741b9	refs/tags/FENNEC_14_0b4_RELEASE
> c64ff3a9bb0bbb0f5a2c536e3f8c2ca259151f35	refs/tags/FENNEC_14_0b5_BUILD1
> c64ff3a9bb0bbb0f5a2c536e3f8c2ca259151f35	refs/tags/FENNEC_14_0b5_RELEASE
> 7ce8220fa48879a42b98f6ffc7e7dc4d15b5f737	refs/tags/FENNEC_14_0b6_BUILD1
> 7ce8220fa48879a42b98f6ffc7e7dc4d15b5f737	refs/tags/FENNEC_14_0b6_RELEASE
> 0bda7c54c987d9f1fc264a763fdb7f01e5c42e04	refs/tags/FENNEC_14_0b7_BUILD1
> a15f85183bbc5348aa6424ce4482afdb6f3fa9d3	refs/tags/FENNEC_14_0b7_BUILD2
> 47667f241ddf1827d4eedaf815f456ac89b10ba0	refs/tags/FENNEC_14_0b7_BUILD3
> 47667f241ddf1827d4eedaf815f456ac89b10ba0	refs/tags/FENNEC_14_0b7_RELEASE
> f8024b93cea30c0aa828271a28cb8461c4a291f2	refs/tags/FENNEC_14_0b8_BUILD1
> ac10567d61cafea8da44eadc08fa72a869fab154	refs/tags/FENNEC_14_0b8_BUILD2
> a4b485cdc364c343dfcc0b7fb33a267f1337082a	refs/tags/FENNEC_14_0b8_BUILD3
> 847f148526bd517426ff5b2c82c71823ccf2fb41	refs/tags/FENNEC_14_0b8_BUILD4
> 847f148526bd517426ff5b2c82c71823ccf2fb41	refs/tags/FENNEC_14_0b8_RELEASE
> 722380b964056ccb8c33ce09f349e47540c8e9f7	refs/tags/FENNEC_15_0_1_BUILD1
> cd8870475b0e6fb3bf732c7ce74c1fb99a40b2cb	refs/tags/FENNEC_15_0_1_BUILD2
> cd8870475b0e6fb3bf732c7ce74c1fb99a40b2cb	refs/tags/FENNEC_15_0_1_RELEASE
> fec02d5080b34302c5c3b9821cb41d56ba70ad17	refs/tags/FENNEC_15_0_BUILD1
> fec02d5080b34302c5c3b9821cb41d56ba70ad17	refs/tags/FENNEC_15_0_RELEASE
> 743a5278046e275db50d7cd765ec0e8b8921385d	refs/tags/FENNEC_15_0b1_BUILD1
> ce29a28009ea01813668936f0d735609789d1db3	refs/tags/FENNEC_15_0b1_BUILD2
> ce29a28009ea01813668936f0d735609789d1db3	refs/tags/FENNEC_15_0b1_RELEASE
> 0d3f761796840ab50b6d75ca57b07b3d1e97bc46	refs/tags/FENNEC_15_0b2_BUILD1
> 0d3f761796840ab50b6d75ca57b07b3d1e97bc46	refs/tags/FENNEC_15_0b2_RELEASE
> d7bfeee5e8eaf8bc0a7af6ad0c20d26886eb1b85	refs/tags/FENNEC_15_0b3_BUILD1
> d7bfeee5e8eaf8bc0a7af6ad0c20d26886eb1b85	refs/tags/FENNEC_15_0b3_RELEASE
> f4f750eb4796712d03d38cff85da90b53de31d52	refs/tags/FENNEC_15_0b4_BUILD1
> 8ac5e1398859211975d84173b8de0c5fbeb65fb2	refs/tags/FENNEC_15_0b4_RELEASE
> 976d9dd6333ce736a00bb876fb34db974150c069	refs/tags/FENNEC_15_0b5_BUILD1
> 976d9dd6333ce736a00bb876fb34db974150c069	refs/tags/FENNEC_15_0b5_RELEASE
> 6523c1094c0af2941ea8573d1062a4ec348dbf2d	refs/tags/FENNEC_15_0b6_BUILD1
> 6523c1094c0af2941ea8573d1062a4ec348dbf2d	refs/tags/FENNEC_15_0b6_RELEASE
> a0b755f3991f4c2a24c8f329952b26b08606fb5b	refs/tags/FENNEC_15_0b7_BUILD1
> a0b755f3991f4c2a24c8f329952b26b08606fb5b	refs/tags/FENNEC_15_0b7_RELEASE
70a139,147
> 34423629d30ec0675e6bccfda9cf1915a2f410a6	refs/tags/FENNEC_16_0b1_BUILD1
> 34423629d30ec0675e6bccfda9cf1915a2f410a6	refs/tags/FENNEC_16_0b1_RELEASE
> 3807a1b5f7d05d9996c8e7b42cb1702bd5a58822	refs/tags/FENNEC_16_0b2_BUILD1
> cd8870475b0e6fb3bf732c7ce74c1fb99a40b2cb	refs/tags/FENNEC_16_0b2_BUILD2
> cd8870475b0e6fb3bf732c7ce74c1fb99a40b2cb	refs/tags/FENNEC_16_0b2_RELEASE
> cfb82608ace7349aea61e23a064de2271c6a4e75	refs/tags/FENNEC_16_0b3_BUILD1
> cfb82608ace7349aea61e23a064de2271c6a4e75	refs/tags/FENNEC_16_0b3_RELEASE
> f744aeb4ce4581abf54b7bd2a5831fd7ac03ecec	refs/tags/FENNEC_16_0b4_BUILD1
> f744aeb4ce4581abf54b7bd2a5831fd7ac03ecec	refs/tags/FENNEC_16_0b4_RELEASE
394a472,475
> 3b773a196fdd85207ad12c9144001743d63abee9	refs/tags/FIREFOX_10_0_6esr_BUILD1
> 3b773a196fdd85207ad12c9144001743d63abee9	refs/tags/FIREFOX_10_0_6esr_RELEASE
> 73e2bfd293a6ab402cdff99b97bf7491b69828fa	refs/tags/FIREFOX_10_0_7esr_BUILD1
> 73e2bfd293a6ab402cdff99b97bf7491b69828fa	refs/tags/FIREFOX_10_0_7esr_RELEASE
450a532,537
> 1be7b0084a89b1a808f35bcfa8440c79ef4d399a	refs/tags/FIREFOX_13_0_1_BUILD1
> 1be7b0084a89b1a808f35bcfa8440c79ef4d399a	refs/tags/FIREFOX_13_0_1_RELEASE
> 482aa8e9e3d27569a12f854f410043beb4785411	refs/tags/FIREFOX_13_0_2_BUILD1
> 482aa8e9e3d27569a12f854f410043beb4785411	refs/tags/FIREFOX_13_0_2_RELEASE
> d0352399badced0f7366ff10fc726aa562cb7400	refs/tags/FIREFOX_13_0_BUILD1
> d0352399badced0f7366ff10fc726aa562cb7400	refs/tags/FIREFOX_13_0_RELEASE
452a540,543
> d5f268fa57d44dbc6de7c5afc375214f96e58788	refs/tags/FIREFOX_13_0b2_BUILD1
> d5f268fa57d44dbc6de7c5afc375214f96e58788	refs/tags/FIREFOX_13_0b2_RELEASE
> 0b63bfeb2d21d80a603b5c5c95d2a2b682925fe8	refs/tags/FIREFOX_13_0b3_BUILD1
> 0b63bfeb2d21d80a603b5c5c95d2a2b682925fe8	refs/tags/FIREFOX_13_0b3_RELEASE
454a546,584
> 754be03d2e78391ed22a962890d59a7917f93292	refs/tags/FIREFOX_13_0b5_BUILD1
> 754be03d2e78391ed22a962890d59a7917f93292	refs/tags/FIREFOX_13_0b5_RELEASE
> 217481a0b9aaf3a0c5919fd4d5cc6e228a987060	refs/tags/FIREFOX_13_0b6_BUILD1
> 217481a0b9aaf3a0c5919fd4d5cc6e228a987060	refs/tags/FIREFOX_13_0b6_RELEASE
> 7f8e7445bd82120fb78719dfae88027a5f259279	refs/tags/FIREFOX_13_0b7_BUILD1
> 7f8e7445bd82120fb78719dfae88027a5f259279	refs/tags/FIREFOX_13_0b7_RELEASE
> 602b7372e6e9483f359f64ac4649a56febca21f1	refs/tags/FIREFOX_14_0_1_BUILD1
> 602b7372e6e9483f359f64ac4649a56febca21f1	refs/tags/FIREFOX_14_0_1_RELEASE
> 6190915ec1b3b9186f253965a6b3a53768b6b563	refs/tags/FIREFOX_14_0b10_BUILD1
> 6190915ec1b3b9186f253965a6b3a53768b6b563	refs/tags/FIREFOX_14_0b10_RELEASE
> b8128f44820f7783c325cf39a89e71a0907885c4	refs/tags/FIREFOX_14_0b11_BUILD1
> 34610252380dcef129984a284862bde8cdcfaea4	refs/tags/FIREFOX_14_0b11_RELEASE
> ab27c50824c18915a6f357dbc6334a1ec3588095	refs/tags/FIREFOX_14_0b12_BUILD1
> ab27c50824c18915a6f357dbc6334a1ec3588095	refs/tags/FIREFOX_14_0b12_RELEASE
> 3a9e9e3e1e1e95a792924481529366d140b4d787	refs/tags/FIREFOX_14_0b6_BUILD1
> eaf680d9867581cd60e53790d5bdc6c424f516c2	refs/tags/FIREFOX_14_0b6_BUILD2
> eaf680d9867581cd60e53790d5bdc6c424f516c2	refs/tags/FIREFOX_14_0b6_RELEASE
> 0bda7c54c987d9f1fc264a763fdb7f01e5c42e04	refs/tags/FIREFOX_14_0b7_BUILD1
> 0bda7c54c987d9f1fc264a763fdb7f01e5c42e04	refs/tags/FIREFOX_14_0b7_RELEASE
> f8024b93cea30c0aa828271a28cb8461c4a291f2	refs/tags/FIREFOX_14_0b8_BUILD1
> f8024b93cea30c0aa828271a28cb8461c4a291f2	refs/tags/FIREFOX_14_0b8_RELEASE
> 47e4ffb3eb0f60bda0aa086ac236489050d410c0	refs/tags/FIREFOX_14_0b9_BUILD1
> 47e4ffb3eb0f60bda0aa086ac236489050d410c0	refs/tags/FIREFOX_14_0b9_RELEASE
> 722380b964056ccb8c33ce09f349e47540c8e9f7	refs/tags/FIREFOX_15_0_1_BUILD1
> 722380b964056ccb8c33ce09f349e47540c8e9f7	refs/tags/FIREFOX_15_0_1_RELEASE
> fec02d5080b34302c5c3b9821cb41d56ba70ad17	refs/tags/FIREFOX_15_0_BUILD1
> fec02d5080b34302c5c3b9821cb41d56ba70ad17	refs/tags/FIREFOX_15_0_RELEASE
> 743a5278046e275db50d7cd765ec0e8b8921385d	refs/tags/FIREFOX_15_0b1_BUILD1
> 743a5278046e275db50d7cd765ec0e8b8921385d	refs/tags/FIREFOX_15_0b1_RELEASE
> 0d3f761796840ab50b6d75ca57b07b3d1e97bc46	refs/tags/FIREFOX_15_0b2_BUILD1
> 0d3f761796840ab50b6d75ca57b07b3d1e97bc46	refs/tags/FIREFOX_15_0b2_RELEASE
> 5eb20c5be11283a60628cdd43d7a723bacc7155b	refs/tags/FIREFOX_15_0b3_BUILD1
> 5eb20c5be11283a60628cdd43d7a723bacc7155b	refs/tags/FIREFOX_15_0b3_RELEASE
> 19a7f82e6c44c8da28e99e5ea1cd91c4df487363	refs/tags/FIREFOX_15_0b4_BUILD1
> a51b7bbbc1f943fff4dbd6b7d1505f93ed070a04	refs/tags/FIREFOX_15_0b4_RELEASE
> 976d9dd6333ce736a00bb876fb34db974150c069	refs/tags/FIREFOX_15_0b5_BUILD1
> 976d9dd6333ce736a00bb876fb34db974150c069	refs/tags/FIREFOX_15_0b5_RELEASE
> 5be08f8f0023cdd381a02b522d51704901077763	refs/tags/FIREFOX_15_0b6_BUILD1
> 5be08f8f0023cdd381a02b522d51704901077763	refs/tags/FIREFOX_15_0b6_RELEASE
460a591,598
> 34423629d30ec0675e6bccfda9cf1915a2f410a6	refs/tags/FIREFOX_16_0b1_BUILD1
> 34423629d30ec0675e6bccfda9cf1915a2f410a6	refs/tags/FIREFOX_16_0b1_RELEASE
> 3807a1b5f7d05d9996c8e7b42cb1702bd5a58822	refs/tags/FIREFOX_16_0b2_BUILD1
> 3807a1b5f7d05d9996c8e7b42cb1702bd5a58822	refs/tags/FIREFOX_16_0b2_RELEASE
> cfb82608ace7349aea61e23a064de2271c6a4e75	refs/tags/FIREFOX_16_0b3_BUILD1
> cfb82608ace7349aea61e23a064de2271c6a4e75	refs/tags/FIREFOX_16_0b3_RELEASE
> f744aeb4ce4581abf54b7bd2a5831fd7ac03ecec	refs/tags/FIREFOX_16_0b4_BUILD1
> f744aeb4ce4581abf54b7bd2a5831fd7ac03ecec	refs/tags/FIREFOX_16_0b4_RELEASE
736a875
> b2c8ad50dbf12071fb781e11ee10057d9e1f4064	refs/tags/FIREFOX_28_b01_RELEASE,_FENNEC_28_b01_RELEASE
1022a1162,1165
> c1a06ce73e81822f34e160daf35fbbfc290a2ae0	refs/tags/THUNDERBIRD_10_0_5esr_BUILD1
> 40d90499a72e92cc4e7e392b377db4fd11245c9d	refs/tags/THUNDERBIRD_10_0_5esr_BUILD2
> ca3ac124249508e76330ddcccd42ff663ae73f9e	refs/tags/THUNDERBIRD_10_0_5esr_BUILD3
> ca3ac124249508e76330ddcccd42ff663ae73f9e	refs/tags/THUNDERBIRD_10_0_5esr_RELEASE
1024a1168,1169
> 2c75dce2d45857ce3fdff6e6f997778c9827ee72	refs/tags/THUNDERBIRD_10_0_7esr_BUILD1
> 2c75dce2d45857ce3fdff6e6f997778c9827ee72	refs/tags/THUNDERBIRD_10_0_7esr_RELEASE
1043a1189,1192
> c9c83b662412d435326747324610f3a5fc944406	refs/tags/THUNDERBIRD_13_0_1_BUILD1
> c9c83b662412d435326747324610f3a5fc944406	refs/tags/THUNDERBIRD_13_0_1_RELEASE
> e703933d22831f580c56880eb55892b05b27420d	refs/tags/THUNDERBIRD_13_0_BUILD1
> e703933d22831f580c56880eb55892b05b27420d	refs/tags/THUNDERBIRD_13_0_RELEASE
1045a1195,1204
> ec7d56da6aeeb24d34f16ad1627afcc4e80c0fac	refs/tags/THUNDERBIRD_13_0b2_BUILD1
> ec7d56da6aeeb24d34f16ad1627afcc4e80c0fac	refs/tags/THUNDERBIRD_13_0b2_RELEASE
> 8b04e6403778681cecb4f858fd4806a7cd39576d	refs/tags/THUNDERBIRD_13_0b3_BUILD1
> 8b04e6403778681cecb4f858fd4806a7cd39576d	refs/tags/THUNDERBIRD_13_0b3_RELEASE
> 7638731c033136c3d15d049a1cdabe4ba157dfd2	refs/tags/THUNDERBIRD_13_0b4_BUILD1
> 7638731c033136c3d15d049a1cdabe4ba157dfd2	refs/tags/THUNDERBIRD_13_0b4_RELEASE
> d9d0ae68680b86739221fc2bd032926b187f5912	refs/tags/THUNDERBIRD_14.0b1_BUILD1
> d2cbdf25e1c2b8c4d9f47df208f2f9e5322914a4	refs/tags/THUNDERBIRD_14.0b1_RELEASE
> 7da519ca52165237692d9197ff7084c4746c7faa	refs/tags/THUNDERBIRD_14_0_BUILD1
> 7da519ca52165237692d9197ff7084c4746c7faa	refs/tags/THUNDERBIRD_14_0_RELEASE
1047a1207,1208
> 7c4408a2b46184760a9a44b606322160fe6ee8e0	refs/tags/THUNDERBIRD_14_0b2_BUILD1
> 7c4408a2b46184760a9a44b606322160fe6ee8e0	refs/tags/THUNDERBIRD_14_0b2_RELEASE
1049a1211,1228
> beb095746caa0be9a1c9bfc27d87ff42a054af15	refs/tags/THUNDERBIRD_14_0b4_BUILD1
> beb095746caa0be9a1c9bfc27d87ff42a054af15	refs/tags/THUNDERBIRD_14_0b4_RELEASE
> 429795b0baa3c7fe96b1bd8306a75aa7cb3f80db	refs/tags/THUNDERBIRD_14_0b5_BUILD1
> 429795b0baa3c7fe96b1bd8306a75aa7cb3f80db	refs/tags/THUNDERBIRD_14_0b5_RELEASE
> e6b3827d350e392c3fd269adee760e5aa454dad9	refs/tags/THUNDERBIRD_15_0_1_BUILD1
> e6b3827d350e392c3fd269adee760e5aa454dad9	refs/tags/THUNDERBIRD_15_0_1_RELEASE
> e6cff8294c13122e66b3ec6a6ee78e86815dcf0d	refs/tags/THUNDERBIRD_15_0_BUILD1
> e6cff8294c13122e66b3ec6a6ee78e86815dcf0d	refs/tags/THUNDERBIRD_15_0_RELEASE
> 1e92a322000a802b26f679f4d840ec466bfc258f	refs/tags/THUNDERBIRD_15_0b1_BUILD1
> 1e92a322000a802b26f679f4d840ec466bfc258f	refs/tags/THUNDERBIRD_15_0b1_RELEASE
> d4b0a923785323d0ee7a54e959c900f1225e5c61	refs/tags/THUNDERBIRD_15_0b2_BUILD1
> d4b0a923785323d0ee7a54e959c900f1225e5c61	refs/tags/THUNDERBIRD_15_0b2_RELEASE
> b87b69945b8ea2a803f4e296a45643f55b2ee6c6	refs/tags/THUNDERBIRD_15_0b3_BUILD1
> b87b69945b8ea2a803f4e296a45643f55b2ee6c6	refs/tags/THUNDERBIRD_15_0b3_RELEASE
> 7670f4da339916c7924355c836213d86219581c4	refs/tags/THUNDERBIRD_15_0b4_BUILD1
> 7670f4da339916c7924355c836213d86219581c4	refs/tags/THUNDERBIRD_15_0b4_RELEASE
> c09ee53f293deec7025a5879e9bc27f26ebb05b3	refs/tags/THUNDERBIRD_15_0b5_BUILD1
> c09ee53f293deec7025a5879e9bc27f26ebb05b3	refs/tags/THUNDERBIRD_15_0b5_RELEASE
1055a1235,1238
> f199217a1c2c1950ba04e48aea63901c57c709bc	refs/tags/THUNDERBIRD_16_0b1_BUILD1
> f199217a1c2c1950ba04e48aea63901c57c709bc	refs/tags/THUNDERBIRD_16_0b1_RELEASE
> fe98f9ed43fd15f0c635a70da022998b7dce87d0	refs/tags/THUNDERBIRD_16_0b2_BUILD1
> fe98f9ed43fd15f0c635a70da022998b7dce87d0	refs/tags/THUNDERBIRD_16_0b2_RELEASE

Same as above:
These tags are no longer in the RoR: http://hg.mozilla.org/build/buildbot-configs/tags
They are listed in the legacy git repo, not the new git repo.
So looks like they were created at some point, sync'd to git legacy repo and then deleted in RoR. So it is right they are not in new git repo - it is a non-issue.

Comment 38

[Pete Moore [:pmoore][:pete]]

To summarise comments 31-37 - in each case the problem was an issue with legacy vcs system being slightly behind, having stalled at some point, or having pushed tags that were later removed from the hg RoR but were not cleaned up in the target git repo.

Therefore there are no mismatched SHAs, and the generated repos are correct.

I would therefore like to roll out on Monday (17 Feb)...

Thunderbirds are go!! :)

Pete

Comment 39

[Pete Moore [:pmoore][:pete]]

Attachment: Add project IDE files to .hgignore

Comment 40

[Pete Moore [:pmoore][:pete]]

Attachment: Disable build-* jobs that have been converted

Comment 41

[Pete Moore [:pmoore][:pete]]

Attachment: Remove configs for disabled jobs

Comment 42

[Pete Moore [:pmoore][:pete]]

Hi Hal,

I've added three patches for disabling the jobs in the legacy vcs2vcs system:

1) "Add project IDE files to .hgignore" - this is just a nice-to-have which adds my IDE files to .hgignore, to prevent me accidentally committing them! (in comment 24 above, Aki approved this for the mozharness repo, so I wanted to see if I can them to repo-sync-configs repo too).

2) "Disable build-* jobs that have been converted" - this is the patch to pull the conversion jobs out of the job04_cmds file - this would be the first thing to deploy in the turn-off - ideally which we can deploy today

3) "Remove configs for disabled jobs" - this is the patch for permanently removing the job definitions from the old system - maybe we can deploy this patch at a later time, when the new system has proven itself to be worthy. Also happy to deploy it today, as we can always revert the commit if we need the files back.

Any questions let me know!

Speak later,
Pete

Comment 43

[Hal Wine [:hwine] use NI!]

Comment on attachment 8377164 [details] [diff] [review]
Disable build-* jobs that have been converted

Review of attachment 8377164 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 44

[Hal Wine [:hwine] use NI!]

Comment on attachment 8377167 [details] [diff] [review]
Remove configs for disabled jobs

Review of attachment 8377167 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with one change below;

::: build-buildbot-configs/config
@@ -4,5 @@
>  	bare = false
>  	logallrefupdates = true
> -[remote "github"]
> -	url = git+ssh://[e-mail]/mozilla/build-buildbot-configs.git
> -	fetch = +refs/heads/*:refs/remotes/github/*

Let's leave this -- it does no harm, as it's not referenced.

A couple of the legacy setup scripts assume both directions are minimally configured to allow conversion. Conversion is only performed if required.

Comment 45

[Hal Wine [:hwine] use NI!]

Comment on attachment 8377163 [details] [diff] [review]
Add project IDE files to .hgignore

Personally, I prefer handling these via a personal hgignore file, as described here: http://stackoverflow.com/questions/12783198/local-mercurial-ignore-file

Comment 46

[Pete Moore [:pmoore][:pete]]

Repositories converted over to new vcs system:

https://hg.mozilla.org/build/mozharness/rev/c84a7b30ebf7
https://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-configs/rev/6ffb5df561c3

I have checked that new commits are getting pushed to github, and they are. The logs look good, the crontab is working, the email notifications are working, and the legacy jobs for build/* were successfully disabled.

In addition, I have checked beagle conversion running on [e-mail] and this continues to work, without errors, after the mozharness changes.

Lastly, I have compared the results in hg.m.o vs prod github and staging github vs prod github and the results are identical, except for the tags that got added to production at some point in the past, that are not present in staging, as expected.

Therefore it looks like all processes are functioning correctly, so I will now close this ticket.

Job done. :)

Pete