Edit the unit test for notebookC by Hellio1006 · Pull Request #338 · biocore/kl-metapool

Hellio1006 · 2025-10-15T13:18:32Z

No description provided.

AmandaBirmingham

Thanks for putting this work into a PR! I have provided comments on some misunderstandings and asked for clarification on a few points. Also, I have allowed the continuous integration to run, which identified a number of lint issues in the code (click on the red-icon item under "Some checks were not successful" on the PR's page to see the details); these are easy-to-fix formatting issues.

AmandaBirmingham · 2025-10-30T21:29:04Z

@@ -0,0 +1,38 @@
+[Header],,,,,,,,


This file was not previously in source control. Where does it come from and why do you feel it should be added?

AmandaBirmingham · 2025-10-30T21:30:28Z

   "outputs": [],
   "source": [
    "## INPUT\n",
+    "# when paramaters does not exist use the variable value, and if exhists, use the key in the test dic.\n",


Typos in the comment :)

"paramaters" -> "parameters"
"does" -> "do
"exhists" -> "exist"

Fix typos in the comment.

AmandaBirmingham · 2025-10-30T21:37:38Z

   "metadata": {},
-   "outputs": [],
+   "outputs": [


Unfortunately, github is not very good at diff-ing jupyter notebook files. To accommodate this limitation, please open this notebook, clear all outputs, stop the kernel, save the file, and then re-commit it. This will remove a lot of the contents that are confusing the diff (but will leave any code changes you made intact, so that they can be viewed by a reviewer).

Rerun the notebook.

AmandaBirmingham · 2025-10-30T21:43:08Z

 Investigator Name,Enter the investigator name (optional),,,,,,,,,,
 Project Name,TestProjA_10002_1_2_3_4_10to1dilution,,,,,,,,,,
-Date,2025-04-01,,,,,,,,,,
+Date,2025-09-04,,,,,,,,,,


This change, while little, is an indication of a drawback of the current testing method. Some of the files that are produced by the notebook (the kind called "samplesheet" files) include the date on which they were produced--it will be different every day that they are run! This means that our method of just checking line-by-line for perfect matches will always fail with this kind of file. It wasn't an issue for notebook D, because notebook D doesn't produce that kind of output file ... but notebook C produces three of them! This suggests that the testing approach used by notebook D is not robust enough to work for all the other notebooks.

yes, thanks for pointing out this!
I have revised the test logic to verify the output file names and their locations, rather than performing a line-by-line content comparison.

AmandaBirmingham · 2025-10-30T21:48:22Z

+2.Comparing the contents of that file to a known "expected" version of the file.
+'''
+
+# imports are executed


This comment is not useful since a programmer reading the code will know this already from seeing the import statements. The same is true for many of the comments below ("define constants variable", "class definition is executed", etc). If you add comments like this to help yourself as you are becoming familiar with new code, you will want to take them out before committing to repository.

Thank you for the feedback! I have removed all the comments.

AmandaBirmingham · 2025-10-30T23:58:56Z

+                os.chdir(prev_cwd)
+
+            # collect produced data files under tmp/test_output (skip .ipynb)
+            produced_files = [


This is an interesting elaboration that I hadn't thought of--checking if the code emitted any unexpected files as well as checking that it did create the expected ones. If we were going this direction, I think it would not make sense to look only for unexpected .txt/.tsv/.csv files, though. I think instead one would want to exclude all expected files (including the ipynb) and all hidden files and then report anything ELSE, regardless of extension!

Oh wow! Thanks for pointing that out! I have updated the test to be much more strict. Instead of just looking for specific data extensions, I now:
1.Whitelisted all expected files (including the executed .ipynb).
2.Filtered out hidden system files (starting with .).
3.Flagged anything ELSE as a failure.

This ensures the notebook workspace stays perfectly clean and doesn't produce any unexpected side-effect files.

AmandaBirmingham · 2025-10-31T00:00:54Z

+            }
+
+            # helpful once: see what was actually produced
+            print("Produced files:", sorted(p.name for p in produced_files))


This sort of statement is useful while debugging but would want to remove it for production code (even production unit tests :D )

oh okay! i will!! Thanks for letting me know!

AmandaBirmingham · 2025-10-31T00:03:36Z

+                expected_fp = self._find_expected_path(name)
+
+                # guard: prevent self-compare if paths somehow coincide
+                if Path(expected_fp).resolve() == produced_fp.resolve():


Interesting ... I think it would never have occurred to me to test for this :) While I still think it is easier to just pass in the directories explicitly, I congratulate you on identifying an unusual edge case in the code ecosystem you have set up!

AmandaBirmingham · 2025-10-31T00:04:44Z

+                        "Ensure notebook wrote into tmp and goldens remain in repo."
+                    )
+
+                self._assert_text_files_equal(produced_fp, Path(expected_fp))


Note that currently the test will ALWAYS fail due to the date situation in the samplesheet files

Yes, thanks for pointing that out! I've updated the logic for Test C. I revised the test to verify the output file names and locations instead of performing a line-by-line content comparison, making it more robust against dynamic data like dates.

AmandaBirmingham · 2025-10-31T00:14:22Z

+            )
+
+
+            run_params = {


Although you have provided run parameters here and are passing them into the notebook in the execute_notebook command, these parameters will never be used. The reason is all the lines in notebook C that set these parameters--as the first example, the cell that reads

## INPUT current_set_id = "col19to24"

Your code will set all parameters in the top cell, but then they will be OVERWRITTEN by the individual statements throughout the code that currently set inputs. As we discussed during our meeting, in order for this test approach to work, it will be necessary to comment out the code lines in notebook C that set the inputs. This will be fine--the users can comment them back in when they go to run the code (at which point they will need to enter their own particular input values anyway).

Thank you for the detailed feedback! You're absolutely right. I realized that if those parameters are redefined in subsequent cells, the values injected by Papermill would be overwritten.
As we discussed, I have commented out the hardcoded input lines throughout Notebook C and tagged the top input cell as a 'parameters' cell. This ensures Papermill correctly overrides the default values during the test execution.

Describe what you changed

6dfe413

AmandaBirmingham reviewed Oct 31, 2025

View reviewed changes

Add automated test for Tellseq notebook D and Tellseq notebook C

e6dc6b3

Uh oh!

Conversation

Hellio1006 commented Oct 15, 2025

Uh oh!

AmandaBirmingham left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmandaBirmingham Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AmandaBirmingham Oct 31, 2025 •

edited

Loading