Error with keras models containing a Concat layer #1708

Activity

1. Theodlz commented on Apr 3, 2025

   

   Theodlz

   on Apr 3, 2025

   Last edited by Theodlz

   Author

   More actions

   I should add that:

   import onnx
   onnx_model = onnx.load("acai.onnx")  # Load the ONNX model
   onnx.checker.check_model(onnx_model)  # Check that the model is well formed
   print("ONNX model is valid.")

   runs fine, indicating that onnx considers the model as valid.
2. Theodlz commented on Apr 3, 2025

   

   Theodlz

   on Apr 3, 2025

   Author

   More actions

   Ok so looking at the onnx graph, I see that the inputs of the model are:

   input {
     name: "features"
     type {
       tensor_type {
         elem_type: 1
         shape {
           dim {
             dim_param: "unk__64"
           }
           dim {
             dim_value: 25
           }
         }
       }
     }
   }
   input {
     name: "triplets"
     type {
       tensor_type {
         elem_type: 1
         shape {
           dim {
             dim_param: "unk__65"
           }
           dim {
             dim_value: 63
           }
           dim {
             dim_value: 63
           }
           dim {
             dim_value: 3
           }
         }
       }
     }
   }
   

   Could it be that tract doesn't deal well with multimodal inputs, and is confused as to why the 2 inputs found in onnx have different dims (not sure why they should?).
3. Theodlz commented on Apr 3, 2025

   

   Theodlz

   on Apr 3, 2025

   Last edited by Theodlz

   Author

   More actions

   Also running it with the onnxruntime works just fine, which makes me think that this is indeed an issue with tract itself:

   import numpy as np
   import onnxruntime as ort
   # Create an ONNX Runtime session
   ort_session = ort.InferenceSession("acai.onnx")
   # Prepare input data for the model
   input_images = np.random.random((1, 63, 63, 3)).astype(np.float32)
   input_metadata = np.random.random((1, 25)).astype(np.float32)
   
   # run the model
   ort_inputs = {
       ort_session.get_inputs()[0].name: input_metadata, # The first input is the metadata
       ort_session.get_inputs()[1].name: input_images # The second input is the images
   }
   ort_output = ort_session.run(
       output_names=[ort_session.get_outputs()[0].name], 
       input_feed=ort_inputs
   )
   print(f"ONNX Runtime Output: {ort_output}")
4. kali commented on Apr 4, 2025

   

   kali

   on Apr 4, 2025

   Collaborator

   More actions

   Hello, can you give me access to the model somehow ?
5. Theodlz commented on Apr 4, 2025

   

   Theodlz

   on Apr 4, 2025

   Author

   More actions

   Hey @kali! Sure thing, I linked the h5 above but let me send the onnx version. Here it is:

   acai_h.onnx.txt

   PS: Sorry I had to add another extension at the end for it to upload, you might have to rename it
6. Theodlz commented on Apr 4, 2025

   

   Theodlz

   on Apr 4, 2025

   Last edited by Theodlz

   Author

   More actions

   BTW I should mention that since I opened that issue, I got it to work with pykeio's ort crate. Validating that the issue doesn't come from the model being invalid or something like that.
7. kali commented on Apr 6, 2025

   

   kali

   on Apr 6, 2025

   Collaborator

   More actions

   Alright, so it's a matter of tract being stricter than most training frameworks. We expect to be able to make sense of the dimensions, including when they are "dynamic". tract uses "symbols" and expressions with them to describe the missing bits in the various tensor dimensions, and they must be valid. Other framework are more lenient.

   

   Here we have a model that uses two different "dim_param" that should be equal for the model to make sense (they're some form of batch dimension). There is actually a third one in the output that will also conflict. To make tract happy with your model, you can override the input shapes, and ignore the output shape (as tract will compute the output shape) like below.

   

   Now translating this to rust code looks like this:

        let model = tract_onnx::onnx().model_for_path("acai_h.onnx").unwrap();
       let b = model.sym("b");
       let model = model
           .with_input_fact(0, f32::fact(&[b.to_dim(), 25.to_dim()]).into())
           .unwrap()
           .with_input_fact(
               1,
               f32::fact(&[b.to_dim(), 63.to_dim(), 63.to_dim(), 3.to_dim()]).into(),
           )
           .unwrap()
           .with_output_fact(0, Default::default())
           .unwrap()
           .into_optimized()
           .unwrap();

   Note that if you don't need batching, you could probably get Keras to not use dynamic dimensions at all and get a "clean" model out of the box.
8. kali commented on Apr 6, 2025

   

   kali

   on Apr 6, 2025

   Collaborator

   More actions

   Moving this to a discussion as it is a recurring question.
9. sonos

   locked and limited conversation to collaborators on Apr 6, 2025
10. kali

    converted this issue into a discussion #1711 on Apr 6, 2025