I'm a little unclear as to how this fits in with libraries like PyBind11 or nanobind? It seems like those libraries would need to be rewritten (or new libraries with the same goals created) in order to use this in the same way?
tangentially related question: is there something as simple as luajit's ffi for python? as in: give it a c header, load the shared library, it simply makes structs usable and functions callable.
It would be interesting to see benchmarks comparing HPy extensions to equivalent Cython/pybind11 implementations in terms of performance and development time.
Yeah, sure, I mean, how many people write C to write an end-user Python module. There's stuff that genuinely wraps C libraries or predates higher level language wrappers, like numpy or matplotlib, but how many new modules are actually themselves written in C?
I would guess also that HPy would replace the includes of `Python.h` that pybind11 et al make in order to bind to CPython, and so existing extensions should be easier to port?
Is my understanding correct that this would provide version agnostic python bindings? Currently, I am building a version of my bindings separately for each version (e.g. building and linking with python 3.7, 3.8, etc.). While automated, it still makes CI/CD take quite a long time.
I wouldn't recommend ccache (or sccache) in CI unless you really need it. They are not 100% reliable, and any time you save from caching will be more than lost debugging the weird failures you get when they go wrong.
python has one of the most fractured development ecosystems of any moderately used language. i’m pretty convinced python is a language that attracts poor development practices and magnifies them due to its flexibility. the people who love it don’t understand the extreme flexibility makes it fragile at scale and are willing to put up with its annoyances in an almost stockholm syndrome way
> most fractured development ecosystems of any moderately used language
Can you elaborate? What's done wrong with Python and right with other "moderately used language" ?
For start, C/C++ doesn't even have an official ecosystem. For Java or Golang, it looks better only because the "ecosystem" does not always include native extensions like cgo or JNI. Once you add them the complexity were no better than Python's
I think any programming language with a lot of popularity attracts poor development practices. Simply because a lot of programmers don't actually know the underlying processes of what they build. The flip-side of this is that freedom and flexibility also gives you a lot of control. Yes, it's very easy to write bad Python. In fact it's probably one of Python's weaknesses as you point out. If you're going to iterate over a bunch of elements, you probably expect your language standard libraries to do it in an efficient way, and Python doesn't necessarily do that. What you gain by this flexibility (and arguably sometimes poor design) is that it's also possible to write really good Python and tailor it exactly to your needs. I think Python scales rather well in fact. Django is a good example, as it's a massive workhorse for a lot of the web (Instagram still uses their own version of it as one example). It does so sort of anonymously similar to how PHP and Ruby do it outside of the hype circle, but it does it.
One of the advantages Python has, even when it's bad, is that it's often "good enough". 95% of the software which gets written is never really going to need to be extremely efficient. I would argue that in 2024 Go is actually the perfect combination of the good stuff from both Python and C. But those things aren't necessarily easy to get into if you're not familiar with something like memory management, (maybe strict typing?), explicit error handling and the differences between an interpreted and compiled language.
Anyway I don't think Python is anymore annoying than any other language. The freedom it gives you needs to be reigned in and if you don't then you'll end up with a mess. A mess which is probably perfectly fine.
The reason is a popularity not a technical one. It's inevitable to get a diverse interest to improve different parts of ecosystem by different parties.
Unless it was done at the very beginning, I doubt it would have been even possible because the current C API is the remnant from that very first public version.
I'm a little unclear as to how this fits in with libraries like PyBind11 or nanobind? It seems like those libraries would need to be rewritten (or new libraries with the same goals created) in order to use this in the same way?
tangentially related question: is there something as simple as luajit's ffi for python? as in: give it a c header, load the shared library, it simply makes structs usable and functions callable.
Yeah, cffi.
It would be interesting to see benchmarks comparing HPy extensions to equivalent Cython/pybind11 implementations in terms of performance and development time.
Looks very cool.
How many new extensions are written in C these days? I was under the impression it's mostly things like Boost Python, pybind or PyO3.
for C++ 11+, pybind11 > Boost.Python
for C++ 17+, nanobind > pybind11 (both created by the same developer)
">" meaning generally better, as described at https://nanobind.readthedocs.io/en/latest/why.html
PyO3 is bindings to the C API, so if you're using PyO3 you're still using the C API even if you're not actually writing C.
Yeah, sure, I mean, how many people write C to write an end-user Python module. There's stuff that genuinely wraps C libraries or predates higher level language wrappers, like numpy or matplotlib, but how many new modules are actually themselves written in C?
There's also Cython.
I would guess also that HPy would replace the includes of `Python.h` that pybind11 et al make in order to bind to CPython, and so existing extensions should be easier to port?
Is my understanding correct that this would provide version agnostic python bindings? Currently, I am building a version of my bindings separately for each version (e.g. building and linking with python 3.7, 3.8, etc.). While automated, it still makes CI/CD take quite a long time.
Cpython also has a limited stable abi and cp3X-abi3 wheels are compatible across multiple versions of Python.
https://docs.python.org/3/c-api/stable.html
You can already build a single wheel as long as you only target cpython, if your needs fit with the limited / stable abi (abi3).
While pypy and graal have API support they don't have abi / abi3 support, so they still have to be built on their own (and per version I think).
I believe so, but it would presumably depend on what features you use.
While automated, it still makes CI/CD take quite a long time
See about using ccache -- https://ccache.dev/
I wouldn't recommend ccache (or sccache) in CI unless you really need it. They are not 100% reliable, and any time you save from caching will be more than lost debugging the weird failures you get when they go wrong.
please provide evidence for this assertion.
Imagine how different the Python ecosystem could be, if this was done 20 years ago.
python has one of the most fractured development ecosystems of any moderately used language. i’m pretty convinced python is a language that attracts poor development practices and magnifies them due to its flexibility. the people who love it don’t understand the extreme flexibility makes it fragile at scale and are willing to put up with its annoyances in an almost stockholm syndrome way
> most fractured development ecosystems of any moderately used language
Can you elaborate? What's done wrong with Python and right with other "moderately used language" ?
For start, C/C++ doesn't even have an official ecosystem. For Java or Golang, it looks better only because the "ecosystem" does not always include native extensions like cgo or JNI. Once you add them the complexity were no better than Python's
I think any programming language with a lot of popularity attracts poor development practices. Simply because a lot of programmers don't actually know the underlying processes of what they build. The flip-side of this is that freedom and flexibility also gives you a lot of control. Yes, it's very easy to write bad Python. In fact it's probably one of Python's weaknesses as you point out. If you're going to iterate over a bunch of elements, you probably expect your language standard libraries to do it in an efficient way, and Python doesn't necessarily do that. What you gain by this flexibility (and arguably sometimes poor design) is that it's also possible to write really good Python and tailor it exactly to your needs. I think Python scales rather well in fact. Django is a good example, as it's a massive workhorse for a lot of the web (Instagram still uses their own version of it as one example). It does so sort of anonymously similar to how PHP and Ruby do it outside of the hype circle, but it does it.
One of the advantages Python has, even when it's bad, is that it's often "good enough". 95% of the software which gets written is never really going to need to be extremely efficient. I would argue that in 2024 Go is actually the perfect combination of the good stuff from both Python and C. But those things aren't necessarily easy to get into if you're not familiar with something like memory management, (maybe strict typing?), explicit error handling and the differences between an interpreted and compiled language.
Anyway I don't think Python is anymore annoying than any other language. The freedom it gives you needs to be reigned in and if you don't then you'll end up with a mess. A mess which is probably perfectly fine.
> a language that attracts poor development practices
I agree, but note there’s another way to frame it: “python can be used by people who aren’t professional software developers”.
It’s also fractured because it has such a massive user base that use it for very different applications with very different priorities.
The reason is a popularity not a technical one. It's inevitable to get a diverse interest to improve different parts of ecosystem by different parties.
C/C++ is more fractured.
While Python is fractured, it is nowhere near problems of C ecosystems.
Unless it was done at the very beginning, I doubt it would have been even possible because the current C API is the remnant from that very first public version.