Network blocking for direct read_pdf calls on URL

[mxufc29]

Hi all - I’m the creator/maintainer of nbainjuries, a package that pulls the data from the NBA's injury reports on a real time basis. I'm relying on tabula-py as a core part of the data extraction process. Recently I noticed that there have been some server-side blocks when calling read_pdf() directly on a live URL (the request just hangs and timeouts). Here is a more detailed description if helpful.

This is more a network request issue than an intrinsic issue of tabula-py. But was wondering if anyone else has encountered this in their usage/workflows, and could suggest any creative or clean workarounds?

[chezou]

Thanks for building a great package on top of tabula-py!

Regarding with the issue, read_pdf() accepts user_agent option when it tries to download to temporary space.
https://tabula-py.readthedocs.io/en/latest/tabula.html#tabula.io.read_pdf

I think that helps to workaround the issue. Can you try it on?

[mxufc29]

Some follow-up considerations/questions:

• Looks like when URL is passed in read_pdf(), the req_headers are created using the User-Agent param only; is it worth considering making it possible to pass in other header params (Connection, Accept, etc)?
• Any value in using requests module instead of the urllib module for the backend URL requests in file_util?

[chezou]

• If User-Agent is passed into read_pdf, urllib.urlopen() is called with "_create_request(path_or_buffer, user_agent)" passed in.
• If not, path_or_buffer is passed directly into urllib.urlopen(), which uses the default urllib.request headers, normally something like "Python-urllib/3.10", which many web servers could reject.
  Is my understanding correct to you?

Yes, that's correct understanding.

Looks like when URL is passed in read_pdf(), the req_headers are created using the User-Agent param only; is it worth considering making it possible to pass in other header params (Connection, Accept, etc)?

If you want additional params, that should be totally fine.

Any value in using requests module instead of the urllib module for the backend URL requests in file_util?

The reason I'm stick with urllib is it's a standard library. Adding additional dependency to a package level may cause dependency conflicts with other packages, and wanted to keep less dependency. Also, requests is kinda maintenance mode, so I don't think it's good time to use it (actually, I used requests before).

[mxufc29]

Great thanks for confirming! I just ran with a custom User-Agent param included in read_pdf call, looks like it is working at least for now!

If you want additional params, that should be totally fine.

Ok sure -- I meant would you be open to generalizing "user_agent" to something like "request_headers"? That way users can have option to pass in a full set of headers (more params) if they desire. Below is the snippet to change if so, happy to submit PR if helpful.

def _create_request(path_or_buffer: str, user_agent: str) -> Request: 
req_headers = {"User-Agent": user_agent} 
return Request(path_or_buffer, headers=req_headers)

The reason I'm stick with urllib is it's a standard library. Adding additional dependency to a package level may cause dependency conflicts with other packages, and wanted to keep less dependency.

Got you that makes sense, was just wondering.

[chezou]

would you be open to generalizing "user_agent" to something like "request_headers"?

That sounds reasonable to me. As a backward compatibility, I still want to keep user_agent argument, but I'm fine to allow request headers argument as well.

[mxufc29]

Sounds good. I'll try to find time to submit a PR sometime this week, but no need to wait for my submission if you're able to do it sooner.