pkg server for current/arm64 stopped ? [main-armv7 on ampere2, elapsed so far: 651:21:56]

[Philip Paeps]

void <void_at_f-m.fm> wrote on

Not sure where to post this..
The last bulk build for arm64 appears to have happened around
mid-March on ampere2. Is it broken?

main-armv7 building is broken and the last completed build
was the one started on Mon, 19 Feb 2024 12:32:10 GMT. It
gets stuck making no progress until manually forced to stop,
pd5512ae7b8c6_s75464941dc 34472 12282 (+9196) 107 (+77) 4753
([phone] ([phone] parallel_build: Fri, 22 Mar 2024 11:05:01
GMT 651:21:56
p43e3af5f5763_sf5f08e41aa 19809 5919 (+3126) 137 (+100) 5363
([phone] (+522) 6995 parallel_build: Wed, 28 Feb 2024 15:46:14
GMT 359:42:14 ampere2
ampere2 alternates between trying to build main-arm64 and main-armv7,
so main-armv7 being stuck blocks main-arm64 from building.
http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=pd5512ae7b8c6_s75464941dc
It is not random which packages are building when this happens.
http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=p43e3af5f5763_sf5f08e41aa
http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=pe9c9c73181b5_sbd45bbe440
My guess is that FreeBSD has something that broken after bd45bbe440
that was broken as of f5f08e41aa and was still broken at 75464941dc .

I'll kill the build on ampere2 again. Thanks for the nudge.

We don't really have good monitoring for this. Also: builds should time
out after 36 hours. The fact that this one does not is a bug in itself.

Philip [hat: clusteradm]


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

Post by Philip Paeps

void <void_at_f-m.fm> wrote on

Not sure where to post this..
The last bulk build for arm64 appears to have happened around
mid-March on ampere2. Is it broken?

main-armv7 building is broken and the last completed build
was the one started on Mon, 19 Feb 2024 12:32:10 GMT. It
gets stuck making no progress until manually forced to stop,
pd5512ae7b8c6_s75464941dc 34472 12282 (+9196) 107 (+77) 4753 ([phone] ([phone] parallel_build: Fri, 22 Mar 2024 11:05:01 GMT 651:21:56
p43e3af5f5763_sf5f08e41aa 19809 5919 (+3126) 137 (+100) 5363 ([phone] (+522) 6995 parallel_build: Wed, 28 Feb 2024 15:46:14 GMT 359:42:14 ampere2
ampere2 alternates between trying to build main-arm64 and main-armv7, so main-armv7 being stuck blocks main-arm64 from building.
http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=pd5512ae7b8c6_s75464941dc
http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=p43e3af5f5763_sf5f08e41aa
http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=pe9c9c73181b5_sbd45bbe440
My guess is that FreeBSD has something that broken after bd45bbe440
that was broken as of f5f08e41aa and was still broken at 75464941dc .

I'll kill the build on ampere2 again. Thanks for the nudge.
We don't really have good monitoring for this. Also: builds should time out after 36 hours. The fact that this one does not is a bug in itself.
Philip [hat: clusteradm]

I'll note that I've never managed to replicate the problem for
building for armv7 on aarch64. But my context never has the
likes of:

QUOTE
Host OSVERSION: 1500006
Jail OSVERSION: 1500015
. .
!!! Jail is newer than host. (Jail: 1500015, Host: 1500006) !!!
!!! This is not supported. !!!
!!! Host kernel must be same or newer than jail. !!!
!!! Expect build failures. !!!
END QUOTE

but always has the two OSVERSION's the same, such as:

Host OSVERSION: 1500015
Jail OSVERSION: 1500015

or, recently,

Host OSVERSION: 1500018
Jail OSVERSION: 1500018

My bulk runs do go through the sequence where the hangups
have repeated for main-armv7 on ampere2.

I wonder what would happen if "Host OSVERSION" was updated
(modernized) to match the modern "Jail OSVERSION" that would
be used?



===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

void <void_at_f-m.fm> wrote on

Not sure where to post this..
The last bulk build for arm64 appears to have happened around
mid-March on ampere2. Is it broken?

main-armv7 building is broken and the last completed build
was the one started on Mon, 19 Feb 2024 12:32:10 GMT. It
gets stuck making no progress until manually forced to stop,
[...]
My guess is that FreeBSD has something that broken after bd45bbe440
that was broken as of f5f08e41aa and was still broken at 75464941dc .

Failing . . .
Host OSVERSION: 1500006
Jail OSVERSION: 1500014

I have finished a package builder refresh this morning. All our builder hosts (except PowerPC - I don't touch those) are now on main-n269671-feabaf8d5389 (OSVERSION 1500018).
ampere1 successfully finished its 140releng-armv7-quarterly build, so it looks like the problem with stuck builds was limited to ampere2 building main-armv7. I'll keep a close eye on this one when it starts its next build.

I see that main-armv7 started.

It queued only 31935 instead of the prior 34528 (or more): it is doing an
incremental build instead of a full build. For example, pkg was not built
but instead the prior build is in use. Thus bad results from the prior
build might be involved in this new build.

I'd recommend forcing a full "poudriere bulk -c -a" that does a from-scratch
build for the purposes of the main-armv7 test.

===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de