The Value of Benchmarking – Performance of Python and WebSocket in Electric Car Charging Systems

“In our customer case, we have an Open Charge Point Protocol (OCPP) system where the central system and charging stations communicate with each other over WebSocket,” says Kärkkäinen, an AWS and Python specialist.

WebSocket is a protocol, which enables a bi-directional low overhead communications channel between client and server. It is used in various solutions where permanent frequent or infrequent communication is necessary and where backend systems need to send data to clients without waiting for any client requests. This can be compared to the http protocol, familiar to everyone using a browser, which always demands a client request.

“In the OCPP system we are using Python language both in our cloud backend running in AWS, as well as in our embedded system,” Kärkkäinen explains. “Python is fast in development but not necessarily in execution as such. Python is so vastly popular and loved because of its clean code syntax, rapid application development, and rich ecosystem of libraries and developers,” Kärkkäinen says.

Ville Kärkkäinen wanted to find out how well Python really performs by doing a WebSocket benchmark. It could give some new insight how optimal the communications in electric car charging stations and the central system will be.

“I conducted a benchmark comparing a few Python WebSocket libraries in contrast with selected Node.js, Java and Rust libraries as part of a bigger stress testing effort. Furthermore, I was interested to see would ASGI server (Asynchronous Server Gateway Interface) with WebSocket library be a viable option to enable workload scaling locally as well as in the cloud environment,” Kärkkäinen tells.

He points out that it is quite difficult to ascertain how much the benchmark tested the performance of the given WebSocket server and how much depends on the features of the given language.

“Definitely, it is an unfair game to compare compiled and interpreted languages in terms of pure execution speed. Same goes for comparing servers running multiple processes versus the one doing all its work within one thread. But that’s life. Competition is rarely fair and so within given constraints, only results matter,” Ville Kärkkäinen explains.

Kärkkäinen remarks that the value of benchmarking a single piece of technology, such as WebSocket libraries, is not only in the concrete end results with nice metrics and graphs. It also provides better understanding of rough thresholds on how far you can push some component. Based on that insight, a developer should try to accommodate the architecture and ensure that those thresholds are not exceeded.

“But benchmarking is also an interesting learning experience. What is benchmarked might behave surprisingly and against your expectations when put under stress. These surprises are something that make you ask valid questions. Can I rely on this technology? Should I prepare to optimize, and if so where and how? Or are the results so bad that it’s time to look for alternative options. And guess how you find those alternative options? Exactly, by benchmarking!” Kärkkäinen says.

So, how did his benchmark go and what were his conclusions? Since he is a big fan of Python, he was delighted to find out that Python with uWebSockets-library was the fastest performer.

“On average, it was 67% faster than websockets-library, which we currently use in our OCPP system. However, the websockets-library showed reliable performance, and given the AWS architecture where we run our OCPP system, the results did not raise immediate concerns. Although speed is important, it is still only one aspect of quality,” Kärkkäinen emphasizes.

Concerning the electric car charging station system, the WebSocket library’s performance is important. Despite this, it is not the first and most likely bottleneck for couple of reasons. First, the OCPP central system reacts to messages sent by the charging station, which then performs some business logic in the backend, typically involving database access or accessing other AWS services, and then responds to the client.

“This messaging is quite infrequent, and transaction correctness is much more important than response speed. Second, we utilize AWS infrastructure both for ensuring our system is available and that it scales up and down based on the actual load,” Ville Kärkkäinen concludes.