Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporterbatcher] Update batcher to support serialized bytes based batching. #12641

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

sfc-gh-sili
Copy link
Contributor

@sfc-gh-sili sfc-gh-sili commented Mar 16, 2025

Description

This PR

  1. Updates batcher to use Size(sizerType) instead of itemsCount()
  2. For logs, traces, and metrics, it also makes sure that the cachedSize is associated with the sizerType.

This will be followed by https://github.com/open-telemetry/opentelemetry-collector/pull/12635/files which enables turning on this feature from configuration.

Link to tracking issue

#3262

Testing

Documentation

Copy link

codecov bot commented Mar 16, 2025

Codecov Report

Attention: Patch coverage is 85.04673% with 16 lines in your changes missing coverage. Please review.

Project coverage is 91.46%. Comparing base (eb00c82) to head (ca7fc03).

Files with missing lines Patch % Lines
exporter/exporterhelper/logs.go 87.50% 3 Missing ⚠️
exporter/exporterhelper/metrics.go 87.50% 3 Missing ⚠️
exporter/exporterhelper/traces.go 87.50% 3 Missing ⚠️
...xporter/exporterhelper/xexporterhelper/profiles.go 57.14% 3 Missing ⚠️
...orterhelper/internal/queuebatch/default_batcher.go 83.33% 2 Missing ⚠️
...ter/exporterhelper/internal/requesttest/request.go 0.00% 2 Missing ⚠️

❌ Your patch check has failed because the patch coverage (85.04%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #12641      +/-   ##
==========================================
- Coverage   91.50%   91.46%   -0.05%     
==========================================
  Files         480      480              
  Lines       26464    26522      +58     
==========================================
+ Hits        24217    24259      +42     
- Misses       1778     1794      +16     
  Partials      469      469              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@@ -31,6 +33,10 @@ type Request interface {
MergeSplit(context.Context, exporterbatcher.SizeConfig, Request) ([]Request, error)
}

type Sizer interface {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

@@ -27,14 +28,14 @@ var (
type logsRequest struct {
ld plog.Logs
pusher consumer.ConsumeLogsFunc
cachedSize int
cachedSize map[exporterbatcher.SizerType]int
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to have 2 values than this map which needs heap allocation and lookup is more expensive.

case exporterbatcher.SizerTypeBytes:
sz = &sizer.LogsBytesSizer{}
default:
return -1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this design seems a bit off to return -1 here. At least document this behavior.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed -1 to 0 and documented the behavior. Alternative is to panic if the sizer is unknown but I am reluctant to have any panic in the code

@github-actions github-actions bot requested a review from bogdandrutu March 18, 2025 00:22
@sfc-gh-sili sfc-gh-sili force-pushed the sili-fix branch 3 times, most recently from 435e943 to a642804 Compare March 18, 2025 01:25
@sfc-gh-sili sfc-gh-sili changed the title [exporterbatcher] Update batcher to use Size() instead of itemsCount() [exporterbatcher] Update batcher to support serialized bytes based batching. Mar 18, 2025
@sfc-gh-sili sfc-gh-sili marked this pull request as draft March 24, 2025 18:59
@sfc-gh-sili sfc-gh-sili force-pushed the sili-fix branch 4 times, most recently from 8b6b343 to 0ea6532 Compare March 26, 2025 00:32
@sfc-gh-sili sfc-gh-sili marked this pull request as ready for review March 26, 2025 00:32
@sfc-gh-sili sfc-gh-sili force-pushed the sili-fix branch 2 times, most recently from ca7fc03 to c8e1657 Compare March 26, 2025 23:07
@sfc-gh-sili sfc-gh-sili marked this pull request as draft March 26, 2025 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants