Request for merging "core-updates" branch

Let's see where we are with the branch currently.

Thanks,

Steve / Futurile

retitle 70456 Request for merging "core-updates" branch

thanks

Hey,

Thanks for raising this issue Steve, given the branch has been going for

around 9 months (since [1]) now, I think it's well overdue to start

looking at building and merging it.

1: https://lists.gnu.org/archive/html/guix-commits/2023-07/msg00332.html

I pushed a single commit plus a merge from master today, and that was

pretty difficult. There was plenty of conflicts, and I probably have

resolved some wrongly, and there's potentially some things that Git

didn't raise as conflicts but might have broken with merging in master.

I'm also really confused by what commits appear to be on the branch,

take 12b15585a75062f3fba09d82861c6fae9a7743b2 which appears to be one

core-updates, but it's a duplicate of

e2a7c227dea5b361e2ebdbba24b923d1922a79d0 which was pushed to

master. Same with this commit 28d14130953d868d4848540d9de8e1ae4a01a467,

which is different to f29f80c194d0c534a92354b2bc19022a9b70ecf8 on

master.

Putting aside the functional changes on core-updates, it's doesn't seem

good to merge these seemingly duplicate commits on to master. I'm not

sure how this happened though, or how to fix it.

Chris

Christopher Baines <mail@cbaines.net> writes:

I've worked out at least when these two werid commits turned up on

core-updates.

12b15585a7 is mentioned here:

https://lists.gnu.org/archive/html/guix-commits/2023-09/msg00955.html

and 28d1413095 is mentioned here:

https://lists.gnu.org/archive/html/guix-commits/2024-03/msg00381.html

With the changes last month in March, I was going to suggest deleting

the branch and then re-creating from f205179ed2 and trying to re-apply

the changes that should be on core-updates, while avoiding any

"duplicate" commits. However, I'm not even sure where to being with the

~5000 commits pushed in September, at least one of them is a duplicate

of a commit on master, but I'm not sure how many of the other ~5000 are.

For comparison, I did a merge of master in to core-updates today, and

this is what it shows up like on guix-commits:

https://lists.gnu.org/archive/html/guix-commits/2024-04/msg01209.html

There are only two new revisions, the ed update I pushed, and the merge

commit, which is what a merge should look like as far as I'm aware.

Ludovic Courtès <ludo@gnu.org> writes:

I haven't really been following core-updates, but I have had a look

since there's a request to merge it now [1].

I'm really concerned by the commits on the branch though, assuming I'm

using Git right, there are 6351 commits on the branch:

git log --pretty=oneline core-updates ^master | wc -l

Somehow, I think there's been a couple of pushes of commits to

core-updates that have partially duplicated lots of commits from master,

I put some more details in:

https://issues.guix.gnu.org/70456#3

I think keeping the Git commit history clean and representative is

really important, so to me at least this means core-updates can't be

merged to master in it's current form, even if the changes overall from

these 6351 commits are reasonable.

I'm really not sure how to move forward though, I had a go at trying to

rebuild the branch without introducing the thousands of duplicate

commits and that produced a branch with 765 commits over master, which

still seems a lot, but a big improvement over 6351:

https://git.cbaines.net/guix/log/?h=chris-core-updates-no-duplicates-attempt

That was really hard going though, as there's plenty of merge conflicts

along the way, and I'm pretty sure I solved some of them

incorrectly. The resulting branch also differs from core-updates.

Maybe someone with more time, care and attention could do a better job,

but it might be more worthwhile just starting fresh and rather than

trying to produce a like for like branch just without the thousands of

duplicate commits, effectively manually rebase the branch (without the

duplicate commits) on master and try to get the commits in to a usable

state.

Any ideas?

Thanks,

Chris

Hi,

Christopher Baines <mail@cbaines.net> writes:

I think probably what happened is that in the middle of a merge of

master -> core-updates (which entails sometimes painful conflicts

resolution), a new commit pushed to core-updates, and to be able to push

the resulting local branch (including the thousands of commits from the

merge commit) got rebased on the remote core-updates.

Perhaps another merge commit appeared on the remote around the same

time, which would explain the duplicates.

While I agree it's messy to have 5000 of duplicated commits, I'm not

sure attempting to rewrite the branch, which has seen a lot of original

commits, is a good idea (it'd be easy to have some good commits fall

into cracks, leading to lost of work).

I'd rather we take this experience as a strong reminding that rebasing

merge commits should be avoided at all costs (git already issues a

warning, IIRC). As you suggested, the next time a situation like this

happens (locally prepared merge commit with new commits made to the

remote branch), merging the remote into the local branch is probably a

nicer solution.

--

Thanks,

Maxim

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

I think it's important to weigh up the cost and risks associated with

either merging these commits, or somehow avoiding doing so. I think the

potential impact is more than just a bit of messy Git history.

Assuming we merge core-updates without doing anything about these

duplicate commits, and taking the cwltool package as a semi-random

example, if you do:

git log -p gnu/packages/bioinformatics.scm

You're going to see two commits for the update to 3.1.20240112164112,

that's maybe confusing, but not a big issue I guess since they look the

same, just different hashes.

But say you're looking at the Git history because you want that specific

version of cwltool and you're going to use guix time-machine or an

inferior looking at that revision. Well, it's a lucky dip. If you pick

the original master commit, you're in luck, you'll probably get

substitutes for cwltool. But if you pick the other seemingly identical

commit, you're effectively checking out core-updates as it was last

month and the chance of substitutes is much less likely. I also can't

really think how you'd work out which commit is best to use once

core-updates is merged? The easiest way would probably be to check the

signature, but that will only work most of the time.

This isn't a new issue, it's already problematic for substitute

availability to use intermediate commits (commits that weren't directly

pointed to by master). But there are over 1000 packages who's versions

are being changed on core-updates currently, or at least it looks like

this because of the duplicate commits, and if I'm correct about how

people are using the git history to find commits for specific versions

of packages, then having these duplicates in the Git history for master

forever more is going to catch people out for as long as those versions

remain relevant.

Hi Christopher,

Christopher Baines <mail@cbaines.net> writes:

I also think Git commit history is important, but in this case I weigh

the value of removing ~5000 duplicated rust commits against the risks of

resolving merge conflicts wrong or forgetting commits upon attempting to

recreate the branch from scratch lower than the benefit.

Given the little attention core-updates is currently receiving, I doubt

someone is willing to put the effort to recreate the branch from scratch

to clean its git history; at least speaking for myself I'd rather spend

the little hack time I have to work on it toward getting it finalized.

I believe how these duplicates came to exist was probably two separate

master -> core-updates merge commits, with one of them ending up being

rebased on top of the other, probably so that it could be pushed.

Perhaps we could capture in our contribution guidelines that rebasing a

merge commit should never be done to keep the history clean, and that in

a situation where:

1. a merge has been prepared locally (with conflicts resolved and all)

2. a new commit has appeared on the remote branch

the solution should be to merge the remote branch into the local one

instead of rebasing the local one on the remote one (as is usually

done). Disclaimer: I haven't actually tried this suggested approach,

which should be done before documenting it, if there's a consensus to do

so.

In other words, I suggest we document what *not* to do to avoid

repeating the same mistake in the future, and move on.

--

Thanks,

Maxim

block 70456 with 67973

thanks

block 70456 with 45885

thanks

block 70456 with 40316

thanks

block 70456 with 68270

thanks

Hi Christopher,

Christopher Baines <mail@cbaines.net> writes:

[...]

I trust the 'newest' (appearing first in 'git log --grep='cwltool:

Update') would yield the commit having substitutes?

If so, the inconvenience is somewhat mitigated, as long as you know to

use the newest of duplicated commits.

--

Thanks,

Maxim

block 70456 with 46442

thanks

block 70456 with 70537

thanks

block 70456 with 39415

thanks

emacs has a script gitmerge.el, it can skip some commit when merge with different merge rule (ours), maybe can make life easier:

--

?????????????