Related Presentations

 

GPU Acceleration of Automatic Differentiation in C++ with Clad, ACAT 2021, I Ifrim, 30 November 2021, VideoLink to slides

Automatic Differentiation (AD) is instrumental for science and industry. It is a tool to evaluate the derivative of a function specified through a computer program. The range of AD application domain spans from Machine Learning to Robotics to High Energy Physics. Computing gradients with the help of AD is guaranteed to be more precise than the numerical alternative and have at most a constant factor (4) more arithmetical operations compared to the original function. Moreover, AD applications to domain problems typically are computationally bound. They are often limited by the computational requirements of high-dimensional transformation parameters and thus can greatly benefit from parallel implementations on graphics processing units (GPUs).

Clad aims to enable differentiable analysis for C/C++ and CUDA and is a compiler-assisted AD tool available both as a compiler extension and in ROOT. Moreover, Clad works as a compiler plugin extending the Clang compiler; as a plugin extending the interactive interpreter Cling; and as a Jupyter kernel extension based on xeus-cling.

In this talk, we demonstrate the advantages of parallel gradient computations on graphics processing units (GPUs) with Clad. We explain how to bring forth a new layer of optimisation and a proportional speed up by extending the usage of Clad for CUDA. The gradients of well-behaved C++ functions can be automatically executed on a GPU. Thus, across the spectrum of fields, researchers can reuse their existing models and have workloads scheduled on parallel processors without the need to optimize their computational kernels. The library can be easily integrated into existing frameworks or used interactively, and provides optimal performance. Furthermore, we demonstrate the achieved application performance improvements, including (~10x) in ROOT histogram fitting and corresponding performance gains from offloading to GPUs.

 

Enabling Interactive C++ with Clang, LLVM Developers’ Meeting 2021, V Vassilev, 17 November 2021, VideoLink to slides

The design of LLVM and Clang enables them to be used as libraries, and has led to the creation of an entire compiler-assisted ecosystem of tools. The relatively friendly codebase of Clang and advancements in the JIT infrastructure in LLVM further enable research into different methods for processing C++ by blurring the boundary between compile time and runtime. Challenges include incremental compilation and fitting compile/link time optimizations into a more dynamic environment.

Incremental compilation pipelines process code chunk-by-chunk by building an ever-growing translation unit. Code is then lowered into the LLVM IR and subsequently run by the LLVM JIT. Such a pipeline allows creation of efficient interpreters. The interpreter enables interactive exploration and makes the C++ language more user friendly. The incremental compilation mode is used by the interactive C++ interpreter, Cling, initially developed to enable interactive high-energy physics analysis in a C++ environment. Cling is now used for interactive development in Jupyter Notebooks (via xeus-cling), dynamic python bindings (via cppyy) and interactive CUDA development.

In this talk we dive into the details of implementing incremental compilation with Clang. We outline a path forward for Clang-Repl which is built with the experience gained in Cling and is now available in mainstream llvm. We describe how the new Orc JIT infrastructure allows us to naturally perform static optimizations at runtime, and enables linker voodoo to make the compiler/interpreter boundaries disappear. We explain the potential of having a compiler-as-a-service architecture in the context of automatic language interoperability for Python and beyond.

 

Enabling Interactive C++ with Clang, 24th EuroAD Workshop 2021, I Ifrim, 4 November 2021, Link to slides

Clad enables automatic differentiation (AD) for C++ algorithms through source-to-source transformation. It is based on LLVM compiler infrastructure and as a Clang compiler plugin. Different from other tools, Clad manipulates the high-level code representation (the AST) rather than implementing its own C++ parser and does not require modifications to existing code bases. This methodology is both easier to adopt and potentially more performant than other approaches. Having full access to the Clang compiler’s internals means that Clad is able to follow the high-level semantics of algorithms and can perform domain-specific optimisations; automatically generate code (re-targeting C++) on accelerator hardware with appropriate scheduling; and has a direct connection to compiler diagnostics engine and thus producing precise and expressive diagnostics positioned at desired source locations.

In this talk, we showcase the above mentioned advantages through examples and outline Clad’s features, applications and support extensions. We describe the challenges coming from supporting automatic differentiation of broader C++ and present how Clad can compute derivatives of functions, member functions, functors and lambda expressions. We show the newly added support of array differentiation which provides the basis utility for CUDA support and parallelisation of gradient computation. Moreover, we will demo different interactive use-cases of Clad, either within a Jupyter environment as a kernel extension based on xeus-cling or within a gpu-cpu environment where the gradient computation can be accelerated through gpu-code produced by Clad and run with the help of the Cling interpreter.

 

Interactive C++ for Data Science, CppCon21, V Vassilev, 27 October 2021, Link to slides

C++ is used for many numerically intensive applications. A combination of performance and solid backward compatibility has led to its use for many research software codes over the past 20 years. Despite its power, C++ is often seen as difficult to learn and not well suited with rapid application development. The long edit-compile-run cycle is a large impediment to exploration and prototyping during development.

Cling has emerged as a recognized capability that enables interactivity, dynamic interoperability and rapid prototyping capabilities for C++ developers. Cling is an interactive C++ interpreter, built on top of the Clang and LLVM compiler infrastructure. The interpreter enables interactive exploration and makes the C++ language more welcoming for research. Cling supports the full C++ feature set including the use of templates, lambdas, and virtual inheritance.Cling’s field of origin is the field of high energy physics where it facilitates the processing of scientific data. The interpreter was an essential part of the software tools of the LHC experimental program and was part of the software used to detect the gravitational waves of the LIGO experiment. Interactive C++ has proven to be useful for other communities. The Cling ecosystem includes dynamic bindings tools to languages including python, D and Julia (cppyy); C++ in Jupyter Notebooks (xeus-cling); interactive CUDA; and automatic differentiation on the fly (clad).

This talk outlines key properties of interactive C++ such as execution results, entity redefinition, error recovery and code undo. It demonstrates the capability enabled by an interactive C++ platform in the context of data science. For example, the use of eval-style programming, C++ in Jupyter notebooks and CUDA C++. We talk about design and implementation challenges and go beyond just interpreting C++. We showcase template instantiation on demand, language interoperability on demand and bridging compiled and interpreted code. We show how to easily build new capabilities using the Cling infrastructure through developing an automatic differentiation plugin for C++ and CUDA.

 

Differentiable Programming in C++, CppCon21, W Moses, V Vassilev, 26 October 2021, VideoLink to slides

Mathematical derivatives are vital components of many computing algorithms including: neural networks, numerical optimization, Bayesian inference, nonlinear equation solvers, physics simulations, sensitivity analysis, and nonlinear inverse problems. Derivatives track the rate of change of an output parameter with respect to an input parameter, such as how much reducing an individuals’ carbon footprint will impact the Earth’s temperature. Derivatives (and generalizations such as gradients, jacobians, hessians, etc) allow us to explore the properties of a function and better describe the underlying process as a whole. In recent years, the use of gradient-based optimizations such as training neural networks have become widespread, leading to many languages making differentiation a first-class citizen.

Derivatives can be computed numerically, but unfortunately the accumulation of floating-point errors and high-computational complexity presents several challenges. These problems become worse with higher order derivatives and more parameters to differentiate.

Many derivative-based algorithms require gradients, or the computation of the derivative of an output parameter with respect to many input parameters. As such, developing techniques for computing gradients that are scalable in the number of input parameters is crucial for the performance of such algorithms. This paper describes a broad set of domains where scalable derivative computations are essential. We make an overview of the major techniques in computing derivatives, and finally, we introduce the flagman of computational differential calculus – algorithmic (also known as automatic) differentiation (AD). AD makes clever use of the ‘nice’ mathematical properties of the chained rule and generative programming to solve the scalability issues by inverting the dependence on the number of input variables to the number of output variables. AD provides mechanisms to augment the regular function computation with instructions calculating its derivatives.

Differentiable programming is a programming paradigm in which the programs can be differentiated throughout, usually via automatic differentiation. This talk introduces the differentiable programming paradigm in terms of C++. It shows its applications in science as applicable for data science and industry. The authors make an overview of the existing tools in the area and the two common implementation approaches – template metaprogramming and custom parsers. We demonstrate implementation pros and cons and propose compiler toolchain-based implementation either in clang AST or LLVM IR. We would like to briefly outline our current efforts in standardization of that feature.

 

Full list of presentations

GPU Acceleration of Automatic Differentiation in C++ with Clad
I Ifrim at the ACAT 2021 (30 November 2021) VideoLink to slides

Enabling Interactive C++ with Clang
V Vassilev at the LLVM Developers’ Meeting 2021 (17 November 2021) VideoLink to slides

Enabling Interactive C++ with Clang
I Ifrim at the 24th EuroAD Workshop 2021 (4 November 2021) Link to slides

Interactive C++ for Data Science
V Vassilev at the CppCon21 (27 October 2021) Link to slides

Differentiable Programming in C++
W Moses, V Vassilev at the CppCon21 (26 October 2021) VideoLink to slides

Add support for differentiating functors
P Arora at the IRIS-HEP GSoC presentations meeting (1 September 2021) presentation, Poster

Utilise Second Order Derivatives from Clad in ROOT
B Kundu at the IRIS-HEP GSoC presentations meeting (1 September 2021) Link to slides

Add Numerical Differentiation Support To Clad
G Singh at the IRIS-HEP GSoC presentations meeting (1 September 2021) Link to slides

GPU Acceleration of Automatic Differentiation in C++ with Clad
I Ifrim at the IRIS-HEP topical meeting (21 June 2021) Link to slides

Floating point error estimation using Clad – Final Report
G Singh at the IRIS-HEP topical meeting (21 June 2021) Link to slides

Floating point error estimation using Clad – Project Roadmap
G Singh at the Onboarding meetup (15 December 2020) pdf, pptx

Adding CUDA® Support to Cling: JIT Compile to GPUs
S Ehrig at the 2020 LLVM workshop (8 October 2020) Link to slides and video

Incremental Compilation Support in Clang
V Vassilev at the 2020 LLVM workshop (7 October 2020) Link to poster

Error estimates of floating-point numbers and Jacobian matrix computation in Clad
V Vassilev at the 2020 LLVM workshop (7 October 2020) Link to poster

Enabling C++ Modules for ROOT on Windows
V Garg at the IRIS-HEP topical meeting (8 September 2020) Link to slides

Modernizing Boost in CMSSW
L Camolezi at the IRIS-HEP topical meeting (2 September 2020) Link to slides

Clad – Automatic Differentiation in C++ and Clang
V Vassilev at the 23rd Euro AD workshop (11 August 2020) Link to talk

CaaS slide for 2020 NSF CSSI PI meeting
D Lange at the 2020 NSF CSSI meeting, Seattle, WA (13 February 2020) Link to slides

CaaS poster for 2020 NSF CSSI PI meeting
D Lange at the 2020 NSF CSSI meeting, Seattle, WA (13 February 2020) Link to poster

Automatic Differentiation in C++
V Vassilev and M Foco (NVIDIA) at the Prague 2020 ISO C++ WG21 meeting (10 February 2020) Link to slides

C++ Modules in ROOT and Beyond
O Shadura at the 2019 CHEP International Conference (7 November 2019) Link to slides

Automatic Differentiation in ROOT
O Shadura at the Computing in High Energy Physics (CHEP 2019, Adeliade) (5 November 2019) Link to slides

Clad - Clang plugin for Automatic Differentiation
J Qiu at the IRIS-HEP Topical meeting (21 August 2019) Link to slides

Implementation of GlobalModuleIndex in ROOT and Cling
A Raghunandan at the IRIS-HEP Topical meeting (21 August 2019) Link to slides

Clad: the automatic differentiation plugin for Clang
A Efremov at the IRIS-HEP topical meeting (29 May 2019) Link to slides

Migrating large codebases to C++ Modules
Y Takahashi at the 19th International Workshop on Advanced Computing and Analysis Techniques in Physics Research (ACAT 2019 Saas-Fee) (13 March 2019) Link to slides

DIANA-HEP Final Presentation Runtime C++ modules
Y Takahashi at the DIANA-HEP Topical meeting (18 February 2019) Link to slides

Future of ROOT runtime C++ modules
Y Takahashi at the ROOT Users Workshop (Sarajevo) (12 September 2018) Link to slides

Optimizing Frameworks Performance Using C++ Modules-Aware ROOT
Y Takahashi at the Computing in High Energy Physics (CHEP 2018, Sofia) (10 July 2018) Link to poster

C++ Modules in ROOT
R Isemann at the DIANA-HEP topical meeting (4 December 2017) Link to slides

ROOT C++ Modules
R Isemann at the DIANA-HEP topical meeting (17 July 2017) Link to slides