Deadlock possible between BalancedThreadSched and RatedUnqueue

[mihaibrodschi]

For elements which have both a task and a timer which schedule each other (such as RatedUnqueue), if we use BalancedThreadScheduler, it will move the task to the least busy thread (T2), but the timer won't be moved (remains on T1). In this case, this deadlock can occur:

Thread 1
RouterThread::driver
 |_ TimerSet::run_timers -> locks the _timer_lock for T1
     |_ TimerSet::run_one_timer
         |_ BalancedThreadSched::run_timer
             |_ RouterThread::scheduled_tasks -> wants to check the scheduled tasks for T2
                 |_ RouterThread::block_tasks
                     |_ waits for T2's _task_blocker to become >= 0.

Thread 2
RouterThread::driver
 |_ RouterThread::driver_lock_tasks -> sets T2's _task_blocker to -1
 |  (...)
 |_ RouterThread::run_tasks
     |_ BandwidthRatedUnqueue::run_task
         |_ Timer::schedule_after
             |_ Timer::schedule_at_steady -> gets the TimerSet this timer belongs to (which is T1's).
                 |_ TimerSet::lock_timers -> waits to lock the _timer_lock for T1. DEADLOCK

A potential solution would be to ensure that the timer is always moved to the same thread as the task.

[tbarbette]

Is it something you ran into? Do you have a test case to demonstrate?

In practice though I would recommend using the RSS++ solution instead of the thread balancer.

[mihaibrodschi]

Yes, I ran into this with a simple setup involving two instances of my application communicating through click. I'll see if I can reproduce it using dpdk-testpmd.
I used the thread balancer because it looked easy. Will check out RSS++.

[tbarbette]

The thing is this bug comes straight from Click, I haven't used or modified those parts (scheduling and threadbalancer) myself so it's hard to help here. You can try to post that on kohler/click repo as it's not related to FastClick parts. Maybe Eddie will like this bug and solve it, happens from time to time :)