This definitely works fine on Linux (though using avidemux3_cli makes more sense here, eliminates also the need for --quit). Are you sure you are in a location where you have write permissions? Does
/Applications/Avidemux2.6.app/Contents/MacOS/Avidemux2.6.app --output-format mp4 --quit --run project.py --save ~/output.mp4
make any difference?
edit: in other words, are you aware of your location in the terminal? "./output.mp4" places output.mp4 into the current directory, "~/output.mp4" places it into your home directory.