QuantLib-Python is a SWIG based wrapper around the C++ Quantlib, the python bindings are generated by SWIG.
You could develop your custom payoffs in C++ using some of QuantLibs functionality (e.g day count conventions, yield curve generators etc) and then expose this functionality (your new functions/class member functions) using swig. The swig definitions file my only require one line of code to do this for each function or member function.
Or you could write your custom payoffs in python and use the QuantLib-Python for any standard analytics to reduce reinventing the wheel.
In either case, you would have to install/maintain: boost, QuantLib and QuantLib-Python.