Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Work stealing mappers #29

Open
romange opened this issue Nov 25, 2019 · 1 comment
Open

Work stealing mappers #29

romange opened this issue Nov 25, 2019 · 1 comment
Assignees

Comments

@romange
Copy link
Owner

romange commented Nov 25, 2019

Currently each mapper executor fiber processes its file fully. This is simple and efficient approach but it creates sometimes a long tail latency when some mappers has finished but others are still processing their workload.

  1. Make sure the mappers incoming queue is short, possibly of length 1.
  2. When mappers finished they will register to take some work from other mappers.
  3. Need to think how we maintain "begin_shard/process/end_shard" flow with stealing mappers.
@romange romange self-assigned this Nov 25, 2019
@romange
Copy link
Owner Author

romange commented Nov 25, 2019

When we do this we could implement pprint as mrprint

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant