[EUCLIDPCR-2075] update get_product_list get_scientific_product_list …

[Enriquerzrz]

• On Euclid removed unnecessary CAST(... AS text) from array columns in get_product_list, get_scientific_product_list and __get_tile_catalogue_list, as TAP 10.x now returns proper arrays.

Jira: EUCLIDPCR-2075
CC: @esdc-esac-esa-int

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.18%. Comparing base (de661e4) to head (45928dc).
⚠️ Report is 25 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3599      +/-   ##
==========================================
- Coverage   73.21%   73.18%   -0.03%     
==========================================
  Files         226      226              
  Lines       21053    21030      -23     
==========================================
- Hits        15414    15391      -23     
  Misses       5639     5639              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bsipocz]

There are now some actual conflicts after #3601 got merged. Could you please rebase and resolve the conflict?

[Enriquerzrz]

Hello,

We have been investigating this issue and ran the following test:

@pytest.mark.remote_data
def test():
from astroquery.esa.euclid import conf, Euclid, EuclidClass

conf.ENVIRONMENTS['DEV'] = {'url_server': 'https://easdev.esac.esa.int/', 'main_table': 'catalogue.mer_catalogue',
                            'main_table_ra_column': 'right_ascension', 'main_table_dec_column': 'declination',
                            'data_set_release_part1': 'data_set_release_part1',
                            'data_set_release_part2': 'data_set_release_part2',
                            'data_set_release_part3': 'data_set_release_part3'}

euclid = EuclidClass(environment='DEV', verbose=True)

euclid.login(user='xxxxxxxx', password='xxxxxxxxxxx)

euclid.get_product_list(observation_id='2373', product_type='dpdMerFinalCatalog')

When executing it, we get the following error:

FAILED tests/test_euclid_remote.py::test - 
astropy.io.votable.exceptions.E01: None:?:?: E01: Invalid size specifier '100x' 
for a char field (in field 'file_name_list')

From our analysis (with @cosmoJFH ), this seems to be related to how astropy handles arrays of strings when generating VOTables. It looks like a potential bug or limitation in the library rather than an issue in our implementation.

Given this, we think it would be safer not to merge these changes yet until we better understand the root cause.

For that reason, we are marking this PR as Draft to block the merge for now.

@bsipocz , do you have any insight into this?

Thanks!

[bsipocz]

@Enriquerzrz - is this a new test you try to add? I don't see any related failures when running the remote tests locally.

That being said, something gone wrong with the rebase, unrelated commits are now picked up as well as there is a merge commit. I'm happy to rebase and then merge though, let me know the answer about the test

[bsipocz]

The diff looks good, tests pass for me when I run them locally, but let's hold up until we see if the traceback mentioned in the comment is coming from an old test or something not yet committed.

[bsipocz]

This is all enternal cleanup, so it doesn't need the changelog. I'll clean up the changelog file before release.

[Enriquerzrz]

Hello @bsipocz

The test included in the comment has not been merged. It is a local test that we added afterwards when my colleague identified the issue, but it is not part of the branch.

[bsipocz]

To me it looks like that the response for this query is empty. So maybe that case needs to be covered in the code?

The regression is an uncommitted test, so let's see it fixed before merging this.

[cosmoJFH]

We carried out the following test. First, we saved the VOTable obtained using the get_product_list method: (
7a9ab7cd-63eb-11f1-921d-1423f21597f1-TDEV-result.zip
). We then tried to open the votable in order to reproduce the error:

    from astropy.table import Table

    file_path = '7a9ab7cd-63eb-11f1-921d-1423f21597f1-TDEV-result.vot'

    try:
        print()
        print('*'*120)
        table = Table.read(file_path, format="votable")
        print(table)
        print(table.colnames)
        print('*'*120)
        print()

    except Exception as e:
        print(e)


7a9ab7cd-63eb-11f1-921d-1423f21597f1-TDEV-result.vot:?:?: E01: Invalid size specifier '100x' for a char field (in field 'file_name_list')

The error is described in https://docs.astropy.org/en/latest/io/votable/api_exceptions.html#e01-invalid-size-specifier-x-for-a-char-unicode-field-in-field-y

Also see astropy/astropy#17098