The best way to Make Your Python Packages Actually Quick with Rust | by Isaac Harris-Holt | Could, 2023
Python is… gradual. This isn’t a revelation. A number of dynamic languages are. In actual fact, Python is so gradual that many authors of performance-critical Python packages have turned to a different language — C. However C is just not enjoyable, and C has sufficient foot weapons to cripple a centipede.
Introducing Rust.
Rust is a memory-efficient language with no runtime or rubbish collector. It’s extremely quick, tremendous dependable, and has a very nice neighborhood round it. Oh, and it’s additionally tremendous simple to embed into your Python code due to glorious instruments like PyO3 and maturin.
Sound thrilling? Nice! As a result of I’m about to indicate you learn how to create a Python package deal in Rust step-by-step. And in case you don’t know any Rust, don’t fear — we’re not going to be doing something too loopy, so it’s best to nonetheless be capable to observe alongside. Are you prepared? Let’s oxidise this snake.
Pre-requisites
Earlier than we get began, you’re going to want to put in Rust in your machine. You are able to do that by heading to rustup.rs and following the directions there. I’d additionally advocate making a digital surroundings that you should use for testing your Rust package deal.
Script overview
Right here’s a script that, given a quantity n, will calculate the nth Fibonacci quantity 100 occasions and time how lengthy it takes to take action.
This can be a very naive, completely unoptimised operate, and there are many methods to make this quicker utilizing Python alone, however I’m not going to be going into these right this moment. As a substitute, we’re going to take this code and use it to create a Python package deal in Rust
Maturin setup
Step one is to put in maturin, which is a construct system for constructing and publishing Rust crates as Python packages. You are able to do that with pip set up maturin.
Subsequent, create a listing on your package deal. I’ve known as mine fibbers. The ultimate setup step is to run maturin init out of your new listing. At this level, you’ll be prompted to pick out which Rust bindings to make use of. Choose pyo3.
Now, in case you check out your fibbers listing, you’ll see just a few recordsdata. Maturin has created some config recordsdata for us, specifically a Cargo.toml and pyproject.toml. The Cargo.toml file is configuration for Rust’s construct device, cargo, and incorporates some default metadata concerning the package deal, some construct choices and a dependency for pyo3. The pyproject.toml file is pretty normal, nevertheless it’s set to make use of maturin because the construct backend.
Maturin can even create a GitHub Actions workflow for releasing your package deal. It’s a small factor, however makes life so a lot simpler whenever you’re sustaining an open supply mission. The file we principally care about, nevertheless, is the lib.rs file within the src listing.
Right here’s an summary of the ensuing file construction.
fibbers/
├── .github/
│ └── workflows/
│ └── CI.yml
├── .gitignore
├── Cargo.toml
├── pyproject.toml
└── src/
└── lib.rs
Writing the Rust
Maturin has already created the scaffold of a Python module for us utilizing the PyO3 bindings we talked about earlier.
The principle components of this code are this sum_as_string operate, which is marked with the pyfunction attribute, and the fibbers operate, which represents our Python module. All of the fibbers operate is admittedly doing is registering our sum_as_string operate with our fibbers module.
If we put in this now, we’d be capable to name fibbers.sum_as_string() from Python, and it might all work as anticipated.
Nevertheless, what I’m going to do first is substitute the sum_as_string operate with our fib operate.
This has precisely the identical implementation because the Python we wrote earlier — it takes in a optimistic unsigned integer n and returns the nth Fibonacci quantity. I’ve additionally registered our new operate with the fibbers module, so we’re good to go!
Benchmarking our operate
To put in our fibbers package deal, all we’ve got to do is run maturin developin our terminal. This can obtain and compile our Rust package deal and set up it into our digital surroundings.
Now, again in our fib.py file, we are able to import fibbers, print out the results of fibbers.fib() after which add a timeit case for our Rust implementation.
If we run this now for the tenth Fibonacci quantity, you may see that our Rust operate is about 5 occasions quicker than Python, regardless of the very fact we’re utilizing an an identical implementation!
If we run for the twentieth and thirtieth fib numbers, we are able to see that Rust will get as much as being about 15 occasions quicker than Python.
However what if I advised you that we’re not even at most pace?
You see, by default, maturin developwill construct the dev model of your Rust crate, which can forego many optimisations to scale back compile time, which means this system isn’t working as quick because it may. If we head again into our fibbers listing and run maturin developonce more, however this time with the --release flag, we’ll get the optimised production-ready model of our binary.
If we now benchmark our thirtieth fib quantity, we see that Rust now offers us a whopping 40 occasions pace enchancment over Python!
Rust limitations
Nevertheless, we do have an issue with our Rust implementation. If we attempt to get the fiftieth Fibonacci quantity utilizing fibbers.fib(), you’ll see that we really hit an overflow error and get a special reply to Python.
It is because, not like Python, Rust has fixed-size integers, and a 32-bit integer isn’t giant sufficient to carry our fiftieth Fibonacci quantity.
We will get round this by altering the kind in our Rust operate from u32 to u64, however that may use extra reminiscence and won’t be supported on each machine. We may additionally resolve it through the use of a crate like num_bigint, however that’s outdoors the scope of this text.
One other small limitation is that there’s some overhead to utilizing the PyO3 bindings. You may see that right here the place I’m simply getting the first Fibonacci quantity, and Python is definitely quicker than Rust due to this overhead.
Issues to recollect
The numbers on this article weren’t recorded on an ideal machine. The benchmarks had been run on my private machine, and will not replicate real-world issues. Please take care with micro-benchmarks like this one usually, as they’re usually imperfect and emulate many points of actual world applications.
