Approaches to reducing the CI run times. Measure, optimize, leverage the hidden resources, and more. — We’ve all been there: send another git push to a pull request and wait for all the triggered CI checks to finish during 30, 40, 60 minutes. Multitask or read some r/programming meanwhile. So annoying (having to multitask, that is). Can we do better? Let’s investigate. Below is a typical…

Some of the tricks we used to speed up SELECT-s in PostgreSQL: LEFT JOIN with redundant conditions, VALUES, extended statistics, primary key type conversion, CLUSTER, pg_hint_plan + bonus — There are numerous posts about PostgreSQL optimization: the rabbit hole is deep. When I began coding the analytics backend several years ago, I already had experience with other DBs such as MySQL and SQL Server. Still, I never had to focus on the performance so much. The previous projects I…

We developed an analytics HTTP API in Python. This is how we discovered what to improve to serve requests faster — In my previous post, I detailed a few code tricks to improve backend performance. How did I know where to focus and what to optimize, though? Indeed, joining Cython and other low-level gizmos to the party should have solid reasoning. I work at Athenian. Athenian offers a SaaS that helps…

Some of the tricks we used to speed up calls to our analytics API written in Python: played with asyncio, messed with SQLAlchemy, hacked deep in asyncpg, rewrote parts in Cython, found better data structures, replaced some pandas with pure numpy. — Python code optimization may seem easy or hard depending on the performance target. If the target is “best effort”, carefully choosing the algorithm and applying well-known common practices is usually enough. If the target is dictated by the UX, you have to go down a few abstraction layers and hack…

The empirical laws of how long it takes to review and release a pull request depending on its size. We inferred them from 100,000 pull requests to commercial closed-source repositories belonging to our software development analytics SaaS clients. — I like to read posts that describe software development best practices. It’s fun to compare your own experience with other people’s. I admit that I don’t always follow them since so many authors like to position their findings as flawless and unexceptionable. My 15 years of software engineering have taught…

My experience of deploying Google’s Kubernetes ML toolkit on physical servers with multiple GPUs — Hardware I’ve got 3 standard Supermicro towers with 256GB RAM, an SSD, 5 HDDs, and 4 GPUs each. Ethernet connects them to the “controller” Dell server with access to the internet and is supposed to gate SSH connections to the cluster. I name the towers by native cities of the team’s…

Data-driven algorithm design using Python and linear programming on a billion Git commit signatures and more. — I’ve recently had to solve an interesting problem: given two unordered lists with real people names, match identities in between. Looks easy, right? Sort both lists, and you are done. Alas, my problem is about two independent lists of people names taken from separate and independent information sources — GitHub…

Google’s Coral project has recently gone out of beta. According to the benchmarks, Coral devices provide excellent neural network inference acceleration for DIY makers. Those devices ground on the specialized Tensor Processing Unit ASIC (Edge TPU), which proved to be somewhat tricky to work with, but the enforced limitations and…