Forecast based on gluonts deepstate model is slow when it is run on Linux OS

[gisram369]

Hi,
It is observed that the GluonTS deep state time series model is performing differently in different operating systems.
The prediction takes around 3 seconds when it is run over docker container on Mac OS (Quad-core, Physical VM, intel i7 @2.3 GHz); this function takes around 8 seconds when it is run over docker container on Linux OS (4 CPU only, VM, intel Broadwell @2.19 GHz); and the same function takes around 20 seconds when it is run over Kubernetes cluster (allocated with 16 CPUs, VM, BM.standard2.52).
When monitored the CPU / memory during the time, it is observed that the program is trying to consume all the allocated CPUs while minimally taking the memory and page files.
I Ran cProfile on the Linux environment and attached is the short output from the profiler.
I understand that mxnet comes with different libraries to support the different OS. Is there any known issue around the mxnet performance? Please suggest how I can nail down the performance issue.

About environment:

• python 3.7
• gluonts 0.6.1
• mxnet 1.6.0

cProfile_output_github_query.txt

[leezu]

Have you tried setting OMP_NUM_THREADS=1

[gisram369]

Thanks for the suggestion. Noticed significant improvement in performance in the Kubernetes cluster (from 20 seconds to 8 seconds) after configuring OMP_NUM_THREADS as suggested. Now the cluster is working in line with the other virtual machine. If you have any other suggestions to improve the performance, please let me know. We want to match the ~ performance of 3 seconds that Mac is delivering.

Thank you