copy to partitioned parquet files saves partition column name in file #118

Activity

1. shashideore

   added

   needs triage

   on Oct 12, 2025
2. paultiq commented on Oct 12, 2025

   

   paultiq

   on Oct 12, 2025

   Contributor

   More actions

   It's just complaining because "id" and "year" are the only columns, and they're both partitioned... therefore there's no columns left to write.

   Try this instead, with an added "name" column:

   import duckdb
   import pandas as pd
   
   print(duckdb.__version__)
   duckdb.sql("CREATE OR REPLACE TABLE orders (id string, year INTEGER, name VARCHAR);")
   # insert some sample data
   duckdb.sql("INSERT INTO orders VALUES ('A', 2022, 'Taco');")
   duckdb.sql("INSERT INTO orders VALUES ('B', 2022, 'Hot Dog');")
   duckdb.sql("INSERT INTO orders VALUES ('C', 2023, 'Ice Cream');")
   duckdb.sql("COPY orders TO 'sample.parquet' (FORMAT PARQUET, PARTITION_BY (id, year), OVERWRITE_OR_IGNORE 1);")
   
   df = pd.read_parquet("sample.parquet")
3. paultiq commented on Oct 12, 2025

   

   paultiq

   on Oct 12, 2025

   Last edited by paultiq

   Contributor

   More actions

   * The code above is the original code from duckdb/duckdb#12147, which you referenced.

   For your code:

   import duckdb
   import pandas as pd
   
   duckdb.execute("""
                  COPY (
   SELECT 'Male' gender, r FROM range(10) t(r)
   )
   TO 'mytabledir'
   ( FORMAT PARQUET,
   PARTITION_BY (GENDER),
   WRITE_PARTITION_COLUMNS FALSE,
   COMPRESSION snappy
   )
                  """).df()

   Reading the created file

   duckdb.execute("""
   select * from read_parquet('mytabledir/gender=Male/data_0.parquet', hive_partitioning=False)
                  """).df()

   Yields the expected

   	r
   0	0
   1	1
   2	2
   3	3
   4	4
   5	5
   6	6
   7	7
   8	8
   9	9

   👍React with 👍1Reacted by Shashikant Deore
4. shashideore commented on Oct 12, 2025

   

   shashideore

   on Oct 12, 2025

   Author

   More actions

   @paultiq , Thank you for quick response !
   I was passing the full path of the file without hive_partitioning parameter , I assumed the partition column is retuned from the file.
   however it seems default for hive_partitioning is true while reading and that's why it was returning partition column.

   Thanks again for your response!
5. shashideore

   closed this as completedon Oct 12, 2025