Fabian had described the two configuration parameters:
The parameters are:
- flink.num.sourceTasks to specify the parallelism of source tasks.
- flink.num.shuffleTasks to specify the parallelism of all shuffling tasks (GroupBy, CoGroup).
I tried setting these in the conf that’s passed to the Flink FlowConnector, but they don’t seem to be working as expected.
I know that something is happening, as the first time I tried this (setting flink.num.sourceTasks to 40, and flink.num.shuffleTasks to 20) I forgot to remove the -p 20 parameter from the CLI.
This caused the job to fail, with an error about not having enough slots.
When I removed the -p 20 parameter, then the job runs, but I see everything using 40 slots (source/map tasks and groups)
Fabian had described the two configuration parameters:
I tried setting these in the conf that’s passed to the Flink FlowConnector, but they don’t seem to be working as expected.
I know that something is happening, as the first time I tried this (setting
flink.num.sourceTasksto 40, andflink.num.shuffleTasksto 20) I forgot to remove the-p 20parameter from the CLI.This caused the job to fail, with an error about not having enough slots.
When I removed the
-p 20parameter, then the job runs, but I see everything using 40 slots (source/map tasks and groups)