Run Modulo Tests

[skiz]

🤔 What's the problem you're trying to solve?

I want to run an indeterminate number of tests in parallel across multiple test runners without tagging individual tests.

✨ What's your proposed solution?

Add a modulo and target option that deterministically selects every Nth scenario to run.

This should also work the same across all instances using the same seed.

I would pass this in as an environment variable to the configuration options.

r1$ SEED=1234 MOD=3 INT=1 make test.... (scenarios 1,4,7,10)
r2$ SEED=1234 MOD=3 INT=2 make test... (scenarios 2,5,8,11)
r3$ SEED=1234 MOD=3 INT=3 make test... (scenarios 3,6,9,12)

⛏ Have you considered any alternatives or workarounds?

I can do this with tags, but forgetting or typo on a tag will ignore the test completely.

📚 Any additional context?

No response

[skiz]

I have made a PR for this feature: #678

[vearutop]

Hi, thank you for this contribution!

I'm not sure godog API extension is needed for such a behavior. The problem seems a bit too specialized and uncommon to me.

I think needed behavior can be achieved with better flexibility using existing "before scenario" hooks and godog.ErrSkip.

For example (based on https://github.com/cucumber/godog/blob/main/_examples/godogs/godogs_test.go):

var (
	scenarioIdx = 0
	mod         = 3
	target      = 2
)

func InitializeScenario(ctx *godog.ScenarioContext) {
	ctx.Before(func(ctx context.Context, sc *godog.Scenario) (context.Context, error) {
		scenarioIdx++

		if scenarioIdx%mod != target {
			return nil, godog.ErrSkip
		}

		return ctx, nil
	})

...

Would such approach work in you case?

[skiz]

Thanks for taking a look and getting back to me.

Parallel test support is generally the natural progression for any test environment when one needs to horizontally scale the number of tests while maintaining an acceptable run time in CI.
So naturally, I assumed adding support was the best way to go.

It looks like this approach will suffice as an easy workaround for that support.