Use pytest's plain asserts

[Jelleas]

pytest has built-in support for rather detailed assert messages. Allowing the testwriter to just use plain asserts:

assert 1 == 2

instead of

if 1 == 2:
     raise check50.Failure("1 does not equal 2")

To do this pytest rewrites part of the code and as it turns out that functionality is reasonably separate from pytests' core. With inspiration from their own test suite:

echo "assert 1 == 2" > bar.py

import ast
from _pytest.assertion.rewrite import rewrite_asserts

def rewrite(src: str) -> ast.Module:
    tree = ast.parse(src)
    rewrite_asserts(tree, src.encode())
    return tree

src = open("bar.py").read()
mod = rewrite(src)
code = compile(mod, "<test>", "exec")
namespace = {}
exec(code, namespace)

Just a thought, but perhaps it's worth exploring?

[cmlsharp]

How do they generate the error messages?

We can always add "AssertionError" to the list of exceptions we catch in the check decorator and do a manual conversion fo check50.Failure. This wouldn't require rewriting the ast. The question is what the error message should be. I mean, you can include error messages in asserts like assert 1 == 2, "one does not equal two", but what if they don't include a human readable error message? Obviously this is technically possible now since someone could write raise check50.Failure(""), but this seems easier to do with the assert syntax.

I guess I'm not necessarily opposed to this, but I'm also not sure it adds much.

[Jelleas]

The default assert is rather useless. It provides nothing to produce a check50.Failure from:

try:
     assert 1 == 2
except AssertionError as e:
     print(e.args) # prints ()

pytest however puts the actual assertion into e.args, but also allows you to intercept this via _pytest.assertion.util._reprcompare (the implementation behind https://docs.pytest.org/en/latest/assert.html#defining-your-own-explanation-for-failed-assertions)

Allowing you to do the following:

import ast
from _pytest.assertion.rewrite import rewrite_asserts
from _pytest.assertion import util

def custom_assert_handler(left, right, op):
    print(f"left: {left}, right: {right}, op:{op}")

util._reprcompare = custom_assert_handler

def rewrite(src: str) -> ast.Module:
    tree = ast.parse(src)
    rewrite_asserts(tree, src.encode())
    return tree

src = open("bar.py").read()
mod = rewrite(src)
code = compile(mod, "<test>", "exec")
namespace = {}
exec(code, namespace)

Through this we could quite reasonably intercept different operators and types to provide student-friendly output, without putting all the burden on the checkwriter. Some low hanging fruit perhaps:

assert 1 == 2 => check50.Mismatch(1, 2)
assert 1 in [1,2,3] => check50.Missing(1, [1,2,3])

[Jelleas]

If curious, check out this project: https://pypi.org/project/dessert/

This provides pytest's assert rewrites as a standalone feature. Quick demo:

bar.py

def answer_to_life():
    return "fortyone"

def test():
    assert answer_to_life() == "fortytwo"

foo.py

import dessert

with dessert.rewrite_assertions_context():
    import bar
    bar.test()

$ python3 foo.py
Traceback (most recent call last):
  File "/Users/jelle/foo.py", line 5, in <module>
    test_code.test()
  File "/Users/jelle/bar.py", line 5, in test
    assert answer_to_life() == "fortytwo"
AssertionError: assert 'fortyone' == 'fortytwo'
 +  where 'fortyone' = answer_to_life()

Here's the selling point, you can hook into the assertion error and the generation of the error message. Allowing you to rewrite or raise another error:

bar.py

def answer_to_life():
    return "fortyone"

def test():
    assert "two" in answer_to_life()

foo.py

from typing import Any
import dessert

def explain_compare(op: str, left: Any, right: Any) -> str | None:
    if op == "==":
        return f":( Expected {left}, not {right}"
    if op == "in":
        return f":( Did not find {left} in {right}"
    return f":( {left} {op} {right}"

with dessert.rewrite_assertions_context():
    dessert.util._reprcompare = explain_compare
    import bar
    bar.test()

$ python3 foo.py
Traceback (most recent call last):
  File "/Users/jelle/foo.py", line 14, in <module>
    bar.test()
  File "/Users/jelle/bar.py", line 5, in test
    assert "two" in answer_to_life()
AssertionError: assert :( Did not find two in fortyone

[ivanharvard]

I tried to add a feature that somewhat simulates this. See #361.

The biggest issue I encountered was parsing the conditional to raise an inferred check50 exception based on the condition. My first idea was to take any conditional, for instance:

a and b or not c and d or not e and f and not g or h == i

and use some library to find the condition that would be evaluated first, then wrap it in parentheses. Then, do this for the second condition, and so on, until there is only one condition left. Say, for the sake of brevity, that the condition was evaluated as so:

(a and (b or (not c) and ...)))))

Notice how there's now a left and right side of the condition. The left would be a and the right would be (b or (not c) and ...))))), with the operator being and. You can now just evaluate the entire conditional by recursively evaluating the left and right side until you find a subcondition that evaluates to False, in which you raise an exception inferring solely on this one condition.

However, Python's syntax allows for some conditions to be difficult to map to a left and a right. For instance:

assert a
assert not a
assert True if a else False
assert a < b < c

It wasn't incredibly difficult to implement inference checks on simple conditions (e.g. x in y, x == y, x > y, etc.), but I found it pretty difficult to generalize it, and so most cases wouldn't be captured (which leads to unpredictable exception raises for those who did not read the documentation).

On the bright side, changing the default assertion error message is a one-line change, so if we would like to include the values of the variables in the asserted condition (as pytest does), it would be quite simple!

[Jelleas]

@ivanharvard Looks cool! Surprisingly little code through the ast.NodeTransformer

One thought from my end, common binary expressions map directly to the intent of check50 Exceptions. Why not make that the default in these specific cases?

assert "hello world" == "hello world!" # The == keyword => check50.Mismatch
assert "hello" in "hello world! # The in keyword => check50.Missing 

If the expression is more complex, or doesn't use one of these operators, then just check50.Failure. This is in line with pytest that also stops being helpful beyond binary expressions. Plus asserts should be kept simple as they are intended to be debug-able :)

If for some reason, Mismatch or Missing is not helpful, you could always escape via:

assert "hello world" == "hello world!", check50.Failure("double check for missing punctuation!")
assert "hello world" == "hello world!", "double check for missing punctuation!"

[ivanharvard]

@Jelleas Thanks for the advice and feedback! I probably was overthinking it; if binary expressions are good enough for pytest, it's good enough for check50. Feel free to check out the updated PR if you'd like.