[cli] compile benchmarks with `--release` flag #272

ZenVoich · 2025-03-27T11:33:59Z

fixes #270

Summary by CodeRabbit

New Features
- Introduced a new configuration option allowing users to choose between "Debug" and "Release" build profiles.
- Enhanced the benchmarking and configuration processes with improved flexibility in specifying and retrieving build profile settings.

…field in dfx.json

coderabbitai · 2025-03-27T11:34:07Z

Walkthrough

The changes extend configuration capabilities in the benchmarking tool by introducing a new build profile option, allowing users to choose between "Debug" and "Release". The bench command now imports and utilizes a public readDfxJson function to retrieve configuration data from dfx.json, and the default options are updated accordingly. Additionally, command-line arguments are adjusted based on the specified profile, ensuring that the appropriate moc flags are applied during benchmarking.

Changes

File(s)	Change Summary
`cli/commands/bench.ts`	Imported `readDfxJson` from `../mops.js`. Added a new `profile` property (`'Debug'
`cli/commands/watch/parseDfxJson.ts`	Added a new `profile` property (`'Debug'

Sequence Diagram(s)

sequenceDiagram
  participant U as User
  participant B as bench()
  participant R as readDfxJson()
  participant M as getMocArgs()

  U->>B: Execute bench command
  B->>R: Retrieve configuration from dfx.json
  R-->>B: Return configuration (includes profile)
  B->>B: Set default options (profile: 'Release' if undefined)
  B->>M: Generate moc arguments based on profile
  M-->>B: Return flag (--debug or --release)
  B->>U: Run benchmarking process with proper flags

Assessment against linked issues

Objective	Addressed	Explanation
Ensure benchmarks use the release flag by default (#270)	✅

Poem

I'm a happy rabbit, code hopping all day,
With a "Release" flag leading the way.
I munch on configs, so crisp and neat,
Debug or release—oh, what a treat!
My whiskers twitch with every new change,
Hopping through code is wonderfully strange.
Cheers to the updates in every range!

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 5184ee4 and 3e9062f.

📒 Files selected for processing (2)

cli/commands/bench.ts (4 hunks)
cli/commands/watch/parseDfxJson.ts (2 hunks)

🧰 Additional context used

🧬 Code Definitions (1)

cli/commands/bench.ts (2)

cli/mops.ts (2)

readConfig (172-198)

readDfxJson (229-241)

cli/commands/watch/parseDfxJson.ts (1)

readDfxJson (45-51)

⏰ Context from checks skipped due to timeout of 90000ms (4)

GitHub Check: node 20, moc 0.11.1, ic-mops@1.0.0
GitHub Check: ubuntu-latest, node 20, moc 0.11.1, ./cli/dist
GitHub Check: macos-latest, node 20, moc 0.11.1, ./cli/dist
GitHub Check: Run Benchmarks and Comment

🔇 Additional comments (7)

cli/commands/watch/parseDfxJson.ts (2)

8-8: Good addition to the DfxConfig type

Adding a type-safe profile property with only 'Debug' or 'Release' options provides proper type checking for this configuration setting.

45-45:
✅ Verification successful

Function now properly exported

Making the readDfxJson function exportable is appropriate since it's now being used in the bench command.

🏁 Script executed:
#!/bin/bash
# Check if there are any other readDfxJson functions in the codebase that might conflict
rg -A 2 "function readDfxJson" --type ts
Length of output: 431
Review Approved: Exported Function Verified

The readDfxJson function in cli/commands/watch/parseDfxJson.ts is now correctly exported, which aligns with its intended use in the bench command. The additional definition in cli/mops.ts exists in a separate module and does not conflict with this change.
cli/commands/bench.ts (5)

52-52: Good addition to BenchOptions type

The addition of the profile property maintains type consistency with the DfxConfig type.

57-57: Using readDfxJson to get profile configuration

This integration allows the benchmarking tool to respect the profile setting from dfx.json.

70-70: Good default profile selection

Setting 'Release' as the default profile when not specified in dfx.json is appropriate for benchmarking, as it ensures optimized builds by default.

197-202: Appropriate compiler flags applied based on profile

The conditional logic correctly applies the --debug or --release flags based on the selected profile, ensuring the compiler optimizes the code appropriately for benchmarking.

15-15:
❓ Verification inconclusive

Possible import confusion with readDfxJson

It appears you're importing readDfxJson from ../mops.js, but based on the code usage, you're utilizing functionality from the parseDfxJson.ts implementation. This could lead to unexpected behavior if these implementations differ.

🏁 Script executed:
#!/bin/bash
# Let's check the implementation of readDfxJson in mops.js
rg -A 15 "export function readDfxJson" --type ts cli/mops.js
Length of output: 114

I'm not seeing the expected implementation in cli/mops.js. To further verify whether the correct implementation of readDfxJson is imported (or if it should come from another file like parseDfxJson.ts), please run a broader search across the repository. For example, you can try running the following script to list any files named mops.js and to search for the definition of readDfxJson:

Once we have these results, we'll be able to verify if the import path is incorrect or if there’s indeed a mismatch between the implementation used and the intended one.

🏁 Script executed:
#!/bin/bash
echo "Searching for file 'mops.js':"
fd -i "mops.js"
echo "Searching for any occurrence of 'readDfxJson':"
rg -n "readDfxJson" .
Length of output: 689
Import Source Ambiguity for readDfxJson in bench.ts

It appears that in cli/commands/bench.ts (line 15), you're importing readDfxJson from ../mops.js, which resolves to the implementation in cli/mops.ts (compiled to JS). However, our search shows another implementation of readDfxJson in cli/commands/watch/parseDfxJson.ts with a notably different type signature. This raises a concern that you might be unintentionally mixing implementations, which could lead to unexpected behavior.

Confirm whether the implementation in cli/mops.ts is the intended one when calling readDfxJson in bench.ts.

If the more strongly typed version from parseDfxJson.ts was intended, please update the import path accordingly.

✨ Finishing Touches

📝 Docstrings were successfully generated. (🔄 Check again to generate docstrings again)

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai plan to trigger planning for file edits and PR creation.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

github-actions · 2025-03-27T11:37:21Z

Benchmark Results

Benchmark files:
• bench/1-buffer-vector-add.bench.mo
• bench/2-vector-buffer-add.bench.mo
• bench/array.bench.mo
• bench/removeLast.bench.mo
• bench/stable-memory.bench.mo

Add (second)

Add items one-by-one (second)

Instructions

	10	10000	1000000
Buffer	10_589 $({\color{green}-22.08\%})$	6_204_478 $({\color{gray}0\%})$	573_081_816 $({\color{gray}0\%})$
Vector	15_416 $({\color{red}+35.04\%})$	4_909_875 $({\color{gray}0\%})$	468_132_811 $({\color{gray}0\%})$

Heap

	10	10000	1000000
Buffer	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
Vector	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$

Garbage Collection

	10	10000	1000000
Buffer	1.09 KiB $({\color{gray}0\%})$	143.28 KiB $({\color{gray}0\%})$	12.02 MiB $({\color{gray}0\%})$
Vector	1.09 KiB $({\color{gray}0\%})$	45.65 KiB $({\color{gray}0\%})$	3.86 MiB $({\color{gray}0\%})$

Add

Add items one-by-one

Instructions

	10	10000	1000000
Vector	15_416 $({\color{gray}0\%})$	4_910_243 $({\color{gray}0\%})$	468_155_259 $({\color{gray}0\%})$
Buffer	10_589 $({\color{gray}0\%})$	6_203_374 $({\color{gray}0\%})$	573_078_872 $({\color{gray}0\%})$

Heap

	10	10000	1000000
Vector	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
Buffer	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$

Garbage Collection

	10	10000	1000000
Vector	1.09 KiB $({\color{gray}0\%})$	45.65 KiB $({\color{gray}0\%})$	3.86 MiB $({\color{gray}0\%})$
Buffer	1.09 KiB $({\color{gray}0\%})$	143.28 KiB $({\color{gray}0\%})$	12.02 MiB $({\color{gray}0\%})$

Array

arr arr

Instructions

	100k x1	reset1	100k x3	reset2	100k x4	reset3
Array	15_002_649 $({\color{gray}0\%})$	4_455 $({\color{gray}0\%})$	30_004_139 $({\color{gray}0\%})$	5_073 $({\color{gray}0\%})$	60_005_196 $({\color{gray}0\%})$	5_691 $({\color{gray}0\%})$

Heap

	100k x1	reset1	100k x3	reset2	100k x4	reset3
Array	390.9 KiB $({\color{gray}0\%})$	-390.37 KiB $({\color{gray}0\%})$	390.9 KiB $({\color{gray}0\%})$	-390.37 KiB $({\color{gray}0\%})$	390.9 KiB $({\color{gray}0\%})$	-390.37 KiB $({\color{gray}0\%})$

Garbage Collection

	100k x1	reset1	100k x3	reset2	100k x4	reset3
Array	360 B $({\color{gray}0\%})$	390.97 KiB $({\color{gray}0\%})$	391 KiB $({\color{gray}0\%})$	390.97 KiB $({\color{gray}0\%})$	1.14 MiB $({\color{gray}0\%})$	390.97 KiB $({\color{gray}0\%})$

Remove items using `removeLast`

Vector and buffer are initialized with 100k items and then 70k items are removed one-by-one.

Instructions

	remove 70k
Vector	32_210_297 $({\color{gray}0\%})$
Buffer	31_716_422 $({\color{gray}0\%})$

Heap

	remove 70k
Vector	-136.8 KiB $({\color{gray}0\%})$
Buffer	-269.76 KiB $({\color{gray}0\%})$

Garbage Collection

	remove 70k
Vector	139.45 KiB $({\color{gray}0\%})$
Buffer	540.43 KiB $({\color{gray}0\%})$

Stable Memory and Region

Grow Region and store blobs in it

Instructions

	Region (fill 1/100)	Region (fill 1/50)	StableMemory
10 pages	65_117_422 $({\color{gray}0\%})$	260_557_814 $({\color{gray}0\%})$	2_861 $({\color{gray}0\%})$
100 pages	1_302_325_766 $({\color{gray}0\%})$	2_604_593_980 $({\color{gray}0\%})$	2_954 $({\color{gray}0\%})$
256 pages	3_334_080_226 $({\color{gray}0\%})$	6_668_025_672 $({\color{gray}0\%})$	3_704 $({\color{gray}0\%})$

Heap

	Region (fill 1/100)	Region (fill 1/50)	StableMemory
10 pages	296 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	276 B $({\color{gray}0\%})$
100 pages	308 B $({\color{gray}0\%})$	308 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
256 pages	308 B $({\color{gray}0\%})$	308 B $({\color{gray}0\%})$	276 B $({\color{gray}0\%})$

Garbage Collection

	Region (fill 1/100)	Region (fill 1/50)	StableMemory
10 pages	2.38 MiB $({\color{gray}0\%})$	9.51 MiB $({\color{gray}0\%})$	336 B $({\color{gray}0\%})$
100 pages	47.55 MiB $({\color{gray}0\%})$	95.1 MiB $({\color{gray}0\%})$	340 B $({\color{gray}0\%})$
256 pages	121.73 MiB $({\color{gray}0\%})$	243.45 MiB $({\color{gray}0\%})$	340 B $({\color{gray}0\%})$

Stable Memory

	Region (fill 1/100)	Region (fill 1/50)	StableMemory
10 pages	8 MiB $({\color{gray}0\%})$	8 MiB $({\color{gray}0\%})$	8 MiB $({\color{gray}0\%})$
100 pages	8 MiB $({\color{gray}0\%})$	8 MiB $({\color{gray}0\%})$	0 B $({\color{gray}0\%})$
256 pages	16 MiB $({\color{gray}0\%})$	16 MiB $({\color{gray}0\%})$	16 MiB $({\color{gray}0\%})$

@ZenVoich

Docstrings generation was requested by @ZenVoich. * #272 (comment) The following files were modified: * `cli/commands/bench.ts` * `cli/commands/watch/parseDfxJson.ts`

coderabbitai · 2025-04-02T07:05:08Z

Note

Generated docstrings for this pull request at #273

[cli] compile benchmarks with --release flag and respect profile …

3e9062f

…field in dfx.json

coderabbitai bot added a commit that referenced this pull request Apr 2, 2025

📝 Add docstrings to bench-profile

4805d77

Docstrings generation was requested by @ZenVoich. * #272 (comment) The following files were modified: * `cli/commands/bench.ts` * `cli/commands/watch/parseDfxJson.ts`

coderabbitai bot mentioned this pull request Apr 2, 2025

📝 Add docstrings to bench-profile #273

Closed

ZenVoich merged commit e22b3a8 into main Apr 2, 2025
32 checks passed

ZenVoich deleted the bench-profile branch April 2, 2025 07:09

This was referenced Apr 8, 2025

Fix regression in mops bench without dfx.json file #277

Merged

[cli] bench: collapsible benchmark results in CI output #278

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[cli] compile benchmarks with `--release` flag #272

[cli] compile benchmarks with `--release` flag #272

ZenVoich commented Mar 27, 2025 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Mar 27, 2025 •

edited

Loading

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

github-actions bot commented Mar 27, 2025

coderabbitai bot commented Apr 2, 2025

[cli] compile benchmarks with --release flag #272

[cli] compile benchmarks with --release flag #272

Conversation

ZenVoich commented Mar 27, 2025 • edited by coderabbitai bot Loading

Summary by CodeRabbit

coderabbitai bot commented Mar 27, 2025 • edited Loading

Walkthrough

Changes

Sequence Diagram(s)

Assessment against linked issues

Poem

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

github-actions bot commented Mar 27, 2025

Benchmark Results

Add (second)

Add

Array

Remove items using removeLast

Stable Memory and Region

coderabbitai bot commented Apr 2, 2025

[cli] compile benchmarks with `--release` flag #272

[cli] compile benchmarks with `--release` flag #272

ZenVoich commented Mar 27, 2025 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Mar 27, 2025 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

Remove items using `removeLast`