DAOS-17344 build: Add dependencies as submodules #16162

jolivier23 · 2025-03-27T05:33:43Z

As a step to building all RPMs in DAOS repo, Add dependencies as submodules. Eventually, we may want these to be snapshots.

Tested that it builds locally

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Add CopyRetriever as alternative to GitRepoRetriever. Use it for submodule option at first. Add support for internal, static library Add patch support to rpm source rpm See PR#16134 for example Signed-off-by: Jeff Olivier <jeffolivier@google.com>

github-actions · 2025-03-27T05:34:03Z

Ticket title is 'Better support for internal components'
Status is 'Open'
https://daosio.atlassian.net/browse/DAOS-17344

As a step to building all RPMs in DAOS repo, Add dependencies as submodules. Eventually, we may want these to be snapshots. Tested that it builds locally Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

daosbuild1 · 2025-04-02T17:38:09Z

Test stage Python Bandit check completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-16162/15/execution/node/135/log

daosbuild1 · 2025-04-02T22:36:47Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-16162/15/execution/node/274/log

Add backoffs to retries of pool and container rsvc operations. Move tse_task_reinit calls to the end of each completion callback, in order to reduce the chance of new data races between a task and its reinitialized instance. Signed-off-by: Li Wei <liwei@hpe.com>

Change-Id: I0817d2aaae3bf6f52d71661d6f7ec271f59c55c7 Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

daosbuild1 · 2025-04-14T23:32:18Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-16162/40/testReport/

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

daosbuild1 · 2025-04-15T17:16:14Z

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-16162/41/testReport/

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

daosbuild1 · 2025-04-16T07:14:25Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-16162/42/display/redirect

knard38

Overall LGTM

Jenkinsfile

knard38 · 2025-04-16T08:54:13Z

deps/prepare_submodule.sh

+  field_name=$2
+
+  # shellcheck disable=SC1090,SC2283
+  source <(grep = <(grep -A10 "\[${field_name}\]" "${DAOS_ROOT}/utils/build.config"))


To not depend on the number of line:

Suggested change

source <(grep = <(grep -A10 "\[${field_name}\]" "${DAOS_ROOT}/utils/build.config"))

source <(awk -v RS= "/\[${field_name}\]/" "${DAOS_ROOT}/utils/build.config" | tail -n+2)

daosbuild1 · 2025-04-16T13:59:29Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-16162/42/display/redirect

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

jolivier23 · 2025-04-17T23:29:02Z

No real changes, had to resolve merge conflict on daos_build.py

daosbuild1 · 2025-04-18T22:36:47Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-16162/45/execution/node/1471/log

DAOS-17344 build: Enhance SCons options

54937f3

Add CopyRetriever as alternative to GitRepoRetriever. Use it for submodule option at first. Add support for internal, static library Add patch support to rpm source rpm See PR#16134 for example Signed-off-by: Jeff Olivier <jeffolivier@google.com>

jolivier23 force-pushed the jvolivie/sub_deps branch 3 times, most recently from 3476c13 to 405d208 Compare March 27, 2025 05:51

DAOS-17344 build: Add dependencies as submodules

3f3141b

As a step to building all RPMs in DAOS repo, Add dependencies as submodules. Eventually, we may want these to be snapshots. Tested that it builds locally Signed-off-by: Jeff Olivier <jeffolivier@google.com>

jolivier23 force-pushed the jvolivie/sub_deps branch from 405d208 to 3f3141b Compare March 27, 2025 05:57

jolivier23 added 7 commits March 27, 2025 00:02

fix pylint issue

f40a625

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Merge branch 'jvolivie/fix_copy' into jvolivie/sub_deps

70bad8f

Fix scripts

98fb972

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Don't copy patches already in right place

d65c5a8

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Add argobots patches and fix script

d5b9c54

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Fix isort linting issue

93fdf2a

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Get the submodules

ef33152

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

jolivier23 force-pushed the jvolivie/sub_deps branch from c06eee6 to ef33152 Compare March 27, 2025 16:59

jolivier23 added 3 commits March 27, 2025 14:02

Fix spdk issue

b92318d

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Merge branch 'jvolivie/fix_copy' into jvolivie/sub_deps

8ae8f96

Extra commit after merge

45100d1

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

jolivier23 mentioned this pull request Mar 27, 2025

DAOS-17344 build: Enhance SCons options #16157

Merged

6 tasks

jolivier23 added 3 commits March 28, 2025 12:04

Fix archive script

a74e861

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Fix archive script

a05ae66

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Merge branch 'jvolivie/fix_copy' into jvolivie/sub_deps

fce0ae4

liw and others added 6 commits April 3, 2025 12:27

Fix build issue

457582b

Change-Id: I0817d2aaae3bf6f52d71661d6f7ec271f59c55c7 Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Merge remote-tracking branch 'comm/master' into jvolivie/fix_copy

dc364fe

Extra commit

8b13f7a

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Merge branch 'jvolivie/fix_copy' into jvolivie/sub_deps

f73565f

Empty commit

b492a19

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

Not sure why more logs are used

f804a1f

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

jolivier23 requested review from johannlombardi and knard38 April 15, 2025 14:57

daltonbohning previously approved these changes Apr 15, 2025

View reviewed changes

Do recursive update in daos_build test

c70b58a

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

jolivier23 dismissed daltonbohning’s stale review via c70b58a April 16, 2025 04:04

jolivier23 added 2 commits April 15, 2025 22:04

Merge remote-tracking branch 'comm/master' into jvolivie/sub_deps

d843e47

Empty commit

bb14766

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

knard38 previously approved these changes Apr 16, 2025

View reviewed changes

Review suggestions

5b90e18

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

jolivier23 dismissed knard38’s stale review via 5b90e18 April 16, 2025 17:33

jolivier23 requested review from daltonbohning and knard38 April 16, 2025 18:56

daltonbohning previously approved these changes Apr 16, 2025

View reviewed changes

knard38 previously approved these changes Apr 17, 2025

View reviewed changes

Merge remote-tracking branch 'comm/master' into jvolivie/sub_deps

295f814

Signed-off-by: Jeff Olivier <jeffolivier@google.com>

jolivier23 dismissed stale reviews from knard38 and daltonbohning via 295f814 April 17, 2025 23:24

jolivier23 requested review from daltonbohning and knard38 April 17, 2025 23:25

daltonbohning approved these changes Apr 17, 2025

View reviewed changes

knard38 approved these changes Apr 18, 2025

View reviewed changes

jolivier23 merged commit 9ec84fc into master Apr 19, 2025
60 of 64 checks passed

jolivier23 deleted the jvolivie/sub_deps branch April 19, 2025 17:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-17344 build: Add dependencies as submodules #16162

DAOS-17344 build: Add dependencies as submodules #16162

jolivier23 commented Mar 27, 2025 •

edited

Loading

github-actions bot commented Mar 27, 2025

daosbuild1 commented Apr 2, 2025

daosbuild1 commented Apr 2, 2025

daosbuild1 commented Apr 14, 2025

daosbuild1 commented Apr 15, 2025

daosbuild1 commented Apr 16, 2025

knard38 left a comment

knard38 Apr 16, 2025

jolivier23 Apr 16, 2025

daosbuild1 commented Apr 16, 2025

jolivier23 commented Apr 17, 2025

daosbuild1 commented Apr 18, 2025

	source <(grep = <(grep -A10 "\[${field_name}\]" "${DAOS_ROOT}/utils/build.config"))
	source <(awk -v RS= "/\[${field_name}\]/" "${DAOS_ROOT}/utils/build.config" \| tail -n+2)

DAOS-17344 build: Add dependencies as submodules #16162

DAOS-17344 build: Add dependencies as submodules #16162

Conversation

jolivier23 commented Mar 27, 2025 • edited Loading

Steps for the author:

After all prior steps are complete:

github-actions bot commented Mar 27, 2025

daosbuild1 commented Apr 2, 2025

daosbuild1 commented Apr 2, 2025

daosbuild1 commented Apr 14, 2025

daosbuild1 commented Apr 15, 2025

daosbuild1 commented Apr 16, 2025

knard38 left a comment

Choose a reason for hiding this comment

knard38 Apr 16, 2025

Choose a reason for hiding this comment

jolivier23 Apr 16, 2025

Choose a reason for hiding this comment

daosbuild1 commented Apr 16, 2025

jolivier23 commented Apr 17, 2025

daosbuild1 commented Apr 18, 2025

jolivier23 commented Mar 27, 2025 •

edited

Loading