You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently each mapper executor fiber processes its file fully. This is simple and efficient approach but it creates sometimes a long tail latency when some mappers has finished but others are still processing their workload.
Make sure the mappers incoming queue is short, possibly of length 1.
When mappers finished they will register to take some work from other mappers.
Need to think how we maintain "begin_shard/process/end_shard" flow with stealing mappers.
The text was updated successfully, but these errors were encountered:
Currently each mapper executor fiber processes its file fully. This is simple and efficient approach but it creates sometimes a long tail latency when some mappers has finished but others are still processing their workload.
The text was updated successfully, but these errors were encountered: