HPC Carpentry

HPC Carpentry Elections: 2025

2025-07-01T00:00:00+00:00

Democratic elections for the HPC Carpentry Steering Committee successfully completed on 1 July 2025, with 10 ballots cast. The Steering Committee members are:

Alan O’Cais

Currently the technical manager of MultiXscale, an EuroHPC centre of Excellence. Previously acted as the Software Manager of E-CAM, an EU Centre of Excellence in computing applications, while at Juelich Supercomputing Centre (JSC). Also previously managed the user software and application environment on the multi-PFlop JURECA hybrid cluster system at JSC. Between 2010 and 2015, was primarily focused within the LinkSCEEM-2 project helping to develop a Virtual Research Community in Computational Science in the Eastern Mediterranean region.

Maintainer of EasyBuild and EESSI, contributor to a number of open source projects (in particular Magic Castle, which is relevant for HPC Carpentry).

Annajiat Alim Rasel

Annajiat serves as a senior lecturer and coordinator of undergraduate programs at Computer Science and Engineering within the School of Data and Sciences, the largest school at BRAC University (BRACU), Bangladesh, under the umbrella of the world’s largest NGO, BRAC. He is continuously learning from the students and colleagues. He is grateful for having co-taught courses on problem-solving, parallel, distributed, and high-performance computing (HPC), natural language interfaces and processing, data science, modeling and simulation, machine vision, ethical hacking, etc., along with interdisciplinary workshops, training events, being a coach for APAC HPC-AI Competition, ACM ICPC, etc. With kind contributions by HPC Carpentry community members like Alan O’Cais (multiXscale & CECAM), Benson Muite (Kichakato Kizito), Wirawan Purwanto (ODU), M. Abdur Rahman (BRACU), Md. Intekhabul Hafiz (BRACU), and many others, he was able to co-teach the HPC-related pilot workshops that had a profound impact on inspiring and assisting underrepresented and underserved researchers and learners. The Carpentries community has been a source of immense growth for him. He is thankful to have had the opportunity to learn from everyone while serving the global community.

He has served as an instructor, instructor trainer, mentor, curriculum advisor, member of the executive council (EC, now transitioned into the board of directors), instructor trainers leadership committee, code of conduct committee (CoCc), instructor development committee, steering committee for HPC carpentry curriculum development, organizing committee for CarpentryCon, etc. Having joined the CoCc in 2022 and then the EC, he had the opportunity to serve both groups as an EC liaison to the CoCc during 2022–2023. He returned as a CoCc member in 2024 to continue the commitment of fostering a safe and inclusive learning environment. Looking ahead, he is passionate about bridging the digital divide, particularly through ICT4D, with a focus on ICT4ED. He believes in the power of open science, open educational resources, free/libre open-source software, tools, and good practices for research, development, engineering, and education. This includes promoting diversity, democracy, transparency, localization, accessibility, and other aspects of FAIR and CARE principles. Together, he dreams to ensure a safer community of practice, empower the community members, and build a larger Carpentries community in Bangladesh, Asia, and worldwide.

Andrew Reid

I operate a small HPC cluster at NIST, and my ulterior motive for all of this is getting better users for my cluster and others. I feel very strongly that the wider availability of HPC resources has not been matched by a greater understanding of how to use them, and that this is an important skills gap in many fields, and might be most acute for small research teams or informally-assembled clusters, although this is hard to measure. I’m a huge fan of the open-source continuous-improvement model of the Carpentries broadly, and look forward to bringing that spirit to the HPC world.

Trevor Keller

I assist Andrew with a research cluster and data storage at NIST, and have a strong interest in helping scientists overcome barriers to using HPC resources. I have served on the HPC Carpentry Steering Committee since its inception, and enjoy preparing massive PRs to implement sweeping changes to our lessons. I am excited for the Lesson Program Incubation process to advance, and look forward to incorporating more workshop feedback to improve our offerings.

HPC Carpentry at LLNL

2024-08-13T00:00:00+00:00

HPC Carpentry at LLNL

In the first week of June, 2024, instructors from HPC Carpentry taught our full workflow workshop for the first time. Over a four-day stint at Lawrence Livermore National Laboratory, we delivered this content not once, but twice!

It was immensely rewarding to see all this material come together in one place. Traveling to teach in person, while not without hiccups, was extremely worthwhile. We believe we served our learners pretty well, and we learned a few lessons relevant to future workshops.

Workshop Structure

Each workshop ran over two days. On the first day, we did the Unix Shell intro lesson from Software Carpentry in the morning, and our own HPC Intro lesson in the afternoon. On the second day, we did a variant of the workflow lesson, adapted for the Maestro workflow tool (rather than Snakemake), because it is developed and used at LLNL.

The instructor team consisted of Andrew Reid and Trevor Keller from the HPC Carpentry steering committee, and Jane Herriman from LLNL, along with helpers from the LLNL community.

While split-terminal tools exist, we used vanilla tmux with two terminals attached to the same session. This allowed the instructors to type on their own laptop while referencing the lesson webpage and selectively sharing the terminal. Learners followed along on the enhanced terminal displayed at the front of the room. Note: to “scroll up” in tmux, press Ctrl+b, [, then arrow-key around.

Maestro

Maestro is a capable workflow engine, and one we would not have explored had Jane not ported the Snakemake lesson so expertly. Maestro favors reproducibility, running every step of the task from scratch at every invocation. This is a significant difference from Snakemake which, like Make, does not re-execute completed “targets.” A significant benefit of Maestro is that the tool does not persist while jobs execute: it generates and submits native Slurm jobs, with tooling in place to check the status of running workflows. This is much more HPC-compatible, for large-scale or time-consuming jobs.

Learners

Learners had a range of backgrounds, from undergraduate bio-informatics students to experienced Linux HPC users. The lessons generally went at a slightly faster pace than expected, without leaving anyone behind. This was in part because access to LLNL’s system Ruby was by means of pre-authorized RSA tokens, removing a lot of the friction from the initial connection process that has been time-consuming in other versions of the workshop. The instructors live-coded plenty of mistakes, opening discussions on some interesting tangential topics. LLNL runs a pool of “login nodes” per HPC system, rather than a single machine, which made for interesting, early discussion of networked filesystems. The sheer number of nodes also made the output of sinfo tricky to comprehend at-a-glance, which is awesome.

Lesson Feedback

One major take-away is that the workflow lesson in particular is vulnerable to learners losing the thread if they miss a step. This lesson, in either its Maestro or Snakemake version, builds up an increasingly sophisticated workflow specification file, incrementally demonstrating workflow concepts in the context of the tool. Consequently, a learner who misses a step and falls behind can find themselves unable to recover, since the remainder of the lesson builds on precisely the content that was missed. The Workflow lesson differs in this respect from the Shell and HPC intro lessons, where later steps can better stand on their own.

The solution to this, which we already started to implement for the second workshop, was to have a shared online notepad with “checkpoint” versions of the file, to which learners can refer if they fall behind, with helpers bridging the content gap for them. Also, LLNL supports and uses the give tool, allowing users to easily pass files around: it’s nifty!

The hands-on Carpentries approach proved itself once again, building muscle memory and vocabulary in learners, who could then move on to their LLNL summer research projects with greater confidence in their ability to productively use the shared high-performance computing resources.

For the project, it was confirmation that the HPC User workshop can work, including the valuable feedback about checkpoint files and a shared notepad. We look forward to teaching this workshop more, and getting it out of beta status and into our main curriculum.

HPC Carpentry Enters Lesson Program Incubation

2024-07-24T00:00:00+00:00

Cross-posted, with minor changes of voice, from The Carpentries Blog post: HPC Carpentry Enters Lesson Program Incubation.

We are thrilled to announce that HPC Carpentry has entered Lesson Program Incubation with The Carpentries. This is another big step towards the project joining Data Carpentry, Library Carpentry and Software Carpentry as The Carpentries Lesson Program.

Toby Hodges, The Carpentries Director of Curriculum, said:

I am delighted to see HPC Carpentry move a big step closer to joining The Carpentries. I attended my first HPC Carpentry workshop in Dublin in 2018 – the day before the first ever CarpentryCon! – and have been enthusiastically following them ever since. I know from previous roles how challenging it can be for researchers to get to grips with high-performance computing environments, and the positive impact good-quality training in these skills can have on the scale and reproducibility of their work. So I find the prospect of adding HPC Carpentry workshops to the list of training offered by The Carpentries really exciting. The project is also supported by an inspiring community that shares the values of our own: adopting the project as an official Lesson Program and merging our two communities will strengthen both groups.

Alan O’Cais, a member of the HPC Carpentry Steering Committee said:

I organised that first workshop in Dublin in 2018! Even before then, there were others in the community working hard for a Carpentries direction for HPC training. Given all those incremental efforts of different people over the years, it’s really satisfying to see the tree they planted and nurtured start to blossom in the way they hoped. Of course this is only the beginning, but we are now in a place where we can really address the pedagogical and technical challenges that an HPC Carpentry Lesson Program raises. I look forward to a time when any lingering sense of eliteness around HPC is gone, and it is seen for what it should really be: just another tool in our learners’ box!

How You Can Get Involved

Help with Workshops

During Lesson Program incubation, the HPC Carpentry community will continue to teach and gather feedback on their workshops. You can help with this!

If you are interested in teaching an HPC Carpentry workshop to your local community, hosting a workshop taught by other Instructors, or getting involved in an existing workshop as an Instructor or Helper, HPC Carpentry wants to hear from you!

Contribute to Lessons

HPC Carpentry is looking to expand its community of contributors to the existing lesson projects. Many members of The Carpentries community will have expertise and experience to bring to the project’s lessons. During the Incubation phase, HPC Carpentry will be establishing and polishing its core curriculum, and you can help to guide the ongoing development of the lesson program.

Tell Your Friends!

Do you know somebody who may be interested in contributing to HPC Carpentry, or who could benefit from learning the skills it teaches? If so, tell them about it and share the information below about how they can learn more and get involved.

Contact: maintainers-hpc@lists.carpentries.org
Website: hpc-carpentry.org
Community Calls: 11:00 & 21:00 UTC on the first & third Thursdays of every month
- Find these calls in the Community Calendar

More About HPC Carpentry

As computational power has exploded, and with it the size of problems that can be solved computationally, High Performance Computing (HPC) has become fundamental science lab equipment. Not using HPC is likely to put you at a disadvantage, but new users are often unable to engage because of the complexity of HPC resources and the lack of formal curricula to teach HPC skills. The mission of HPC Carpentry is to lower the barrier to entry to HPC operations for a wide range of users, so that more learners can benefit from the increasing availability of increasingly sophisticated computer systems.

The core curriculum of HPC Carpentry bridges the gap between using a laptop and wielding a supercomputer through 3 lessons:

the Software Carpentry Unix Shell lesson,
an Introduction to High Performance Computing lesson, and,
a lesson on HPC Workflow Management (with Snakemake)

More about the Incubation Process

After several years of development and informal communication between the organisations, HPC Carpentry applied for and was invited to enter the Incubation process to formally join The Carpentries. Incubation is the second of three phases in a process that onboards new projects into The Carpentries. During this phase, a new Lesson Program has 18 months to demonstrate high-quality curriculum, demand for workshops, a healthy community of contributors, and structures and policies that can align with The Carpentries existing governance. We’re excited to take this step towards onboarding another Lesson Program into The Carpentries and invite you to support the efforts of HPC Carpentry as they continue to build out their community and lesson infrastructure!

Learn more about the Lesson Program Incubation process in the Community Handbook.

The BoF at SC21

2021-12-14T00:00:00+00:00

The BoF at SC21

The HPC Carpentry team hosted a “Birds of a Feather” session at the SC21 conference. This was a hybrid conference, and the BoF sessions were done via Zoom, with Q&A done via the Sli.do tool. Andrew Reid was on-site and hosted the on-site portion from the podium, while Trevor Keller, Annajiat Alim Rasel, Alan O’Cais, and Wirawan Purwanto were on-line, monitoring the Zoom and Sli.do questions and keeping the session flowing for all participants.

The primary purpose of this was to try to reconnect with the HPC user community, and gather feedback and guidance on how to prioritize our efforts moving forward. The BoF was a success on all counts: 20 people attended in-person in St. Louis, and 40 people attended via Zoom, with good, sometimes vibrant participation from both “factions” 😇 in the Q&A.

We imagine that the SC21 audience is probably somewhat more performance- and hardware-aware than the Carpentries community at large, and possibly also biased towards facility operators, rather than novice users. The feedback we gained is clearly valuable; nevertheless, we need to keep this intrinsic bias in mind while we evaluate and act on this input.

There were a number of survey questions planned in advance, which were presented via the polling functionality in the Sli.do. This information was captured and is available in the coordination repo. The principal results are that attendees would like to see, in descending order of preference: Future lessons on containers; Python and MPI; Dask and Julia; Ability to mix and match lesson content; Constructing half-day lessons that use the templating capability to adapting lessons to permanent on-premise facilities.

After the planned polling questions were done, a more general discussion ensued, largely captured in a CodiMD document.

A number of attendees were interested in how to give feedback on their experience with the lessons, and seemed uncertain what feedback we’re looking for. The answer is that we are mainly looking for actionable feedback¹ that helps improve the lessons for the community. The main mechanism by which we imagine this happening is through instructor and learner comments via poll (to be written), Slack, and GitHub issues. At the end of the lesson, following usual Carpentries practice, organizers should ask learners for positive and negative feedback, on green or red sticky notes, and conduct a post-workshop survey. This info, particularly the red stickies, is what we’re looking for, and ideally would be translated into issues on the appropriate lesson repository.

We will be digesting the feedback received over the next few weeks, and look forward to working through the generated issues over the coming months to ensure our lessons benefit from this excellent community participation.

Highlights

One way HPC Carpentry can add value, beyond the lessons themselves, is as a clearing house for HPC educational resources generally.
The lessons and resources we choose to focus on are a signal to the community about our priorities, so we should take some care in selecting these.
MPI, implemented with low-level programming languages (C, C++, Fortran), is still the most commonly-used framework for parallel programming. It’s appropriate for HPC Carpentry to develop lessons that focus on this foundational material.
New users are increasingly unfamiliar with the command line and file system hierarchy. Extra instructional effort may be needed to help bridge this gap, including the effort of developing GUI-focused lessons based on Open OnDemand or JupyterHub.
Some users still struggle with the difference in character between distributed and shared HPC resources, and more familiar laptop or workstation resources. HPC is not just a big laptop.
Interest in cloud-based HPC resources is surprisingly high, driven by people with possibly-transient HPC workloads but without institutional HPC facilities.

“Actionable feedback” may seem flip, but we mean it sincerely: comments that include some tangible suggestion for improvement are tremendously helpful.

Applying lesson development design principles

2020-11-12T00:00:00+00:00

A Workshop with CodeRefinery

Together with CodeRefinery (1), we helped to organise a variant of an instructor training workshop for the EU HPC Centres of Excellence (2) with a clear focus on HPC related material and teaching. We discussed lesson design a number of times during this workshop. This blog post is about a few interesting things we realised during those discussions - most notably using Bloom’s Taxonomy to formulate learning objectives.

The motivation to look at this was mostly spurred by Greg Wilson’s great book, “Teaching Tech together”, (3). Here, Greg defines:

Objectives vs. Outcomes

A learning objective is what a lesson strives to achieve. A learning outcome is what it actually achieves, i.e. what learners actually take away. The role of summative assessment is therefore to compare learning outcomes with learning objectives.

So we want to describe what we want to teach and what we want our learners to pick up. Typically, each Carpentries’ episode encodes this in the yaml frontmatter part of every lesson episode. For example, the shell-novice defines the following objectives for the lesson module “Navigating Files and Directories”:

objectives:
- "Explain the similarities and differences between a file and a directory."
- "Translate an absolute path into a relative path and vice versa."
- "Construct absolute and relative paths that identify specific files and directories."
- "Use options and arguments to change the behaviour of a shell command"
- "Demonstrate the use of tab completion, and explain its advantages."

A guide to formulating learning objectives

Bloom’s taxonomy and other taxonomies offer a structure or guideline, to help a lesson designer and teacher to formulate these learning objectives. Bloom’s taxonomy itself comprises of 6 concepts which can be paired with verbs that help guide the process.

Remembering: Exhibit memory of previously learned material by recalling facts, terms, basic concepts, and answers. (recognize, list, describe, name, find)

Understanding: Demonstrate understanding of facts and ideas by organizing, comparing, translating, interpreting, giving descriptions, and stating main ideas. (interpret, summarize, paraphrase, classify, explain)

Applying: Solve new problems by applying acquired knowledge, facts, techniques and rules in a different way. (build, identify, use, plan, select)

Analyzing: Examine and break information into parts by identifying motives or causes; make inferences and find evidence to support generalizations. (compare, contrast, simplify)

Evaluating: Present and defend opinions by making judgments about information, validity of ideas, or quality of work based on a set of criteria. (check, choose, critique, prove, rate)

Creating: Compile information together in a different way by combining elements in a new pattern or proposing alternative solutions. (design, construct, improve, adapt, maximize, solve)

An HPC inspired exercise

During the aforementioned workshop, I (Peter) asked all participants to imagine they want to teach learners about scp or ssh (participants were able to choose). The workshop participants were split in teams and asked to formulate learning objectives guided by Bloom’s Taxonomy for the topic they chose.

As a teacher, I expected the native English speakers to complete this task with ease and the non-native speakers to struggle. The following is directly copied from the workshops HackMD pad.

Remembering

ssh: name the command and list it’s parameters

scp: Name the basic structure of a scp call: command source destination

ssh: Can the student explain what does the ssh acronym stand for?

Understanding

ssh: explain what the command does

scp: Paraphrase the similarity of SCP to CP - across physical limits of computers.

Applying

ssh: use the command to log in to a new cluster / server

scp: Use SCP to transfer a set of files or folders

Analyzing

ssh: compare the local environment to the remote environment (to contrast source and destination machine)

scp: Compare : compare to rsync and ability to know when to to use which (simplify this!)

Evaluating

scp: Rate approaches to share scientific data (cloud versus shared folder on parallel filesystem versus central [ftp] server)

ssh: choose whether to use ssh to do a given task

Creating

ssh: solve a security problem using ssh

scp: Design an approach to share scientific data to more than 3 destinations (on a daily basis).

ssh: Can the student modify the following ssh command to be able to log-in directly into the directory /home/dir1?: ssh username@login.hpc.xxx.xx.uk

How to use “Bloom’s Taxonomy”

It looked like all participants struggled with this task irrespective of their English speaking capabilities. Learners asked me how I would approach this and I provided two hints:

Categories in Bloom’s taxonomy build upon each other. Take this prioritised list: Remembering, Understanding, Applying, Analyzing, Evaluating, Creating. If learners don’t remember the content that was taught, it will be hard for them to exhibit understanding. For learners to apply content, e.g. to new examples or domains, they first need to understand the content taught. I told my participants that I try to balance these categories mostly by the context of the course I am designing. For example, in the context of carpentries’ workshops, the volume to teach is often quite large and the time to dive into details is hardly ever available. Hence, I would limit myself to “Remembering, Understanding, Applying” for designing and organising my lessons. Anything in the higher categories, “Analyzing, Evaluating, Creating” I often consider a bonus.
I typically judge my formulation of learning objects by how amenable they are to create assessments from. In other words, a learning objective in the very first category of the taxonomy Remember like “At the end of this module, learners remember how to use SCP.” is way too broad of a formulation to help guide me in creating lessons and produce tangible assessments. I’d rather suggest to put it like so: “At the end of this module, learners recognize the use of scp for transferring single files or entire folders.” This formulation lends itself to create two assessments or exercises.

A testimony of a participant

For me (Alan), I realised that I had never really considered Bloom’s taxonomy beyond an abstract sense. When it was time to leverage it during the exercise to set lesson objectives, I struggled to apply it in a practical way. It raised questions for me about what level(s) of the taxonomy we are supposed to target in lesson episodes. After the workshop, and for this blogpost, I read up on this a little and came across a post from the University of Arkansas (4) that helped me to grasp the key points in the use of the taxonomy:

You set Lesson Objectives for the overall lesson that target a particular level of the taxonomy. For HPC Carpentry, this is most likely the “Applying” level.
In your lesson episodes you should follow the principle (trivially adapted from (3))

Use Bloom’s Taxonomy to make sure that the verbs you choose for your (episode) level objectives build up to the level of the verb that is in the (lesson) level objective. The (episode) level verbs can be below or equal to the (lesson) level verb, but they CANNOT be higher in level. For example, your (lesson) level verb might be an Applying level verb, “illustrate.” Your (episode) level verbs can be from any Bloom’s level that is equal or below this level (applying, understanding, or remembering).

A summary and a request

Bloom’s taxonomy appears high level at first sight. Once you sit down and try to employ it, a lesson designer can be guided by their use to more effectively and better formulate learning goals. The resulting goals are expected to offer more tangible guidance once you design the teaching material in question.

How do you feel about Bloom’s taxonomy and how we put it to use for creating learning goals for scp and ssh? Feel free to leave a comment below or join the discussion on the HPC Carpentry Slack channel (5) (to access the channel you first need to join the Carpentries Slack organisation).

HPC Carpentry Task Force

2020-09-25T00:00:00+00:00

In the fall of 2020, following successful engagement at the Carpentries@Home conference, a number of HPC Carpentry contributors got together and formed an “HPC Carpentry Task Force”, with an expected lifetime of about three months, with the goal of identifying and solving the issues which stand between the current state of the lesson, and incorporating our material into the mainstream Carpentries infrastructure, though the incubator mechanism

An initial meeting was held on Sept. 16, 2020.

Moving forward, the plan is to coordinate several co-working sessions where volunteers can identify and clear remaining issues with the various repositories, and also to have higher-level coordination meetings, approximately monthly, where issue priorities can be discussed, and tasks requiring higher-level coordination can be undertaken.

Discussion and coordination takes place on the HPC Carpentry Slack channel, The Carpentries’ HPC mailing list, and through issues on the coordination repository on Github.

All are welcome!

HPC Carpentry at CarpentryCon @ Home 2020

2020-07-13T00:00:00+00:00

Some of the HPC Carpentry team are very keen to bring HPC Carpentry into the carpentries as an incubator project (1) and eventually become a member of the Carpentries (or the Carpentry Lab (2) at least). To start this, they’ve proposed a breakout session for carpentrycon at home (3) which will take place on July 20, 2020.

Besides sampling the community for expectations and hopes, we would love to propose a way forward to bring the community together and focus on hpc-intro (4). Key points of this are:

propose a github manager who will coordinate reviews, reviewers and issues
propose building a set of task forces to tackle particular issues, and asking for volunteers
suggest to hold co-working hours (biweekly for 60-90 minutes via zoom) to establish continuous work on the material

With this message, we call for volunteers to help shape the session (we need at least 2 more people since we plan to have 4 breakout rooms during the session). Please step forward and reply to the open issue in our repo connected to the proposal (5) or to the proposal itself (6) or email us at maintainers-hpc@lists.carpentries.org (can you feel how keen we are to get you involved!).