[FIXED] How to mirror one GitLab instance to another, while applying some simple rewrite rules along the way?

Issue

We're faced with the following usecase: an external vendor is developing a project, which is composed of a number of repos (some 40) in their GitLab instance. We'd like to become a mirror (slave) of their GitLab (master). However, we also need to apply some simple rewriting rules on our side (replacing some URLs and identifiers, which can be done with a simple script). What would be the best way to approach this?

I've been able to setup push-based mirroring using Project Access Tokens, which works like a charm. It doesn't allow me to apply any rewriting rules, though.

It would also work for us to have, for each of the vendor branches (or just the master branch) in each of their 40 repos, a shadow branch in the vendor repo with the rewritten code. I.e., rewriting those URLs/identifiers on the vendor side would be no issue. How to setup this in a conflict-free way with git hooks, e.g. on every push? Or do you have a better idea?

Thanks!

Solution

I managed to solve it using Project Access Tokens and running the rewrite script and git merge and git push inside a GitLab CI/CD job (on the vendor side). Something along the lines of:

# setup git credentials
print "https://${GIT_USER}:${GIT_PASSWORD}@${GIT_HOST}\n" >> /root/.git-credentials
git config --global credentials.helper 'store --file=/root/.git-credentials'

# clone the slave (our) repo (works because of the credentials set above)
git clone slave-repo-url

# checkout the commit branch in question (might already exist or not)
git checkout -B branch --track origin/branch || git checkout -b branch

# add a new remote (vendor)
git remote add --tags vendor vendor-repo-url

# pull from the vendor branch
# note: this uses a recursive merge strategy but this is fine
git pull --no-edit -X theirs vendor branch

# rewrite the files; create the rewrite commit
# ...

# push the changes back to the origin (our slave)
git push --tags --set-upstream origin branch

This is an automatic solution which works for any branch, and also for the edge cases (new repo, new branch etc.). Tags are covered. It requires you to create the corresponding slave repo beforehand, and to configure a Project Access Token (with write_repository permission) inside of it ($GIT_PASSWORD).

Answered By - stf

Answer Checked By - David Goodson (PHPFixing Volunteer)