PaSh: Scaling out Shell Programs, Automatically
Unix / Linux shell programming is ubiquitous, partly due to the simplicity in which it allows combining third-party components (commands) written in any programming language. Unfortunately, this language-agnostic composition hinders automated scaleout of shell programs, often forcing developers that deal with massive datasets to manually rewrite shell programs and their commands in other languages or frameworks designed to scale.
In this talk I will present PaSh, a system for automatically scaling out shell programs. PaSh combines a just-in-time compiler that blends static pre-processing with dynamic interposition, a high-level annotation framework for capturing partial command specifications, and a collection of high-performance runtime primitives that support the execution of parallel and distributed shell programs. PaSh achieves order-of-magnitude speedups on unmodified shell programs, all while remaining virtually indistinguishable from (and requiring no modifications to) an underlying shell interpreter like Bash. PaSh is worked on by several institutions, has received multiple awards, and is open-source software available by the Linux Foundation.
Nikos Vasilakis is an Assistant Professor of Computer Science at Brown University. His research encompasses systems, programming languages, and security and has been recognized by several distinguished paper awards. His current focus is on automatically transforming systems to add new capabilities such as parallelism, distribution, and security—against a variety of threat models. Nikos is also a co-founder of RequireSecurity; the chair of the Technical Steering Committee behind PaSh, a shell-script optimization system hosted by the Linux Foundation; and a Research Affiliate at MIT CSAIL. https://nikos.vasilak.is