One of our goals is to provide a set of practices that are easy to adopt and have proven effective in many computational and data-intensive (CDI) research settings to research software engineers (RSEs) and also Domain Scientists.
Best Practices address the root causes of poor software development. They are a set of empirically proven approaches to software development which when used they can improve the results. They are called “best practices” not because we can precisely quantify their value but rather they produce results that are superior to those achieved by other procedures or methods or because it has become a standard way of doing things or they are observed to be commonly used by successful organizations.
The section below introduces the identified best practices for CDI research involving applications that are compute-intensive (spending most execution time on computational algorithms) and/or data-intensive (manipulating large volumes of data). Two broad categories of best practices for supporting CDI research have emerged in the Xpert Network activities: (1) best practices related to project collaborations and (2) best practices for software development. The second category is covered well in the software engineering literature. This article considers only those software development practices that have been expressed by the Xperts as different or particularly important for CDI research. Of particular interest are practices that are actionable. In some cases, the action lies in being aware of something.
Please notice that our goal is to not only present the developed best practices but also to get your feedback on these practices. We are interested to know about the impact of applying these practices to your research. Therefore we have created a mechanism for you to vote on the impact of each practice based on your experiences and/or observations right after introducing each practice. Please remember to click the “Vote” button after sharing your experience with each practice. To read more detailed information about each practice, simply expand the practice by clicking on its title or the “+” sign next to the practice title. To share your comments, suggestions, or concerns, please use the comment section located at the end of the page. Alternatively, you can reach out to us via email at email@example.com. If you require further clarification or assistance regarding the survey questions, we can schedule a Zoom meeting to go through them together.
For a detailed description of these best practices, please visit this paper.
This video offers a concise overview of the Xpert Network, outlining its purpose and highlighting the identified best practices.
(1) best practices related to project collaboration
BP1- Diversity of Xpert Backgrounds
Be aware of different backgrounds Xperts may bring into a team; configure training so that less-familiar best practices can be acquired as needed. Members of Xpert groups may come from domain sciences, from computer science backgrounds, or from environments where they learned to assist CDI science teams through training on the job. For example, working with Unix commands, parallel programming models, and version control may be second nature to members with computer science backgrounds, whereas these very skills may be critical best practices to acquire by Xperts with the domain-science background. Vice versa, Xperts who have a domain degree are often well aware of the terminology gap, which facilitates communication with new science collaborators. Training for Xperts should accommodate these differences, allowing the trainees to focus on best practices they are not familiar with.
BP2- Understanding the Academic Environment
Be aware of the academic reward system and activities to increase academic standing. An issue for Xperts that is essentially absent in a general software engineering context is the importance of understanding the academic environment. This issue is especially relevant for software engineers with a background in the industry, where hierarchical organization, and the overriding goal of creating a reliable product as rapidly as possible, are the norm. Understanding the academic reward system and the many side activities that researchers may get engaged in to maintain the academic standing of the science team will influence project decisions. In fact, some of the best practices need to be understood from this viewpoint, such as the issues of documentation and testing.
BP3- Breadth of Xpert Skills Needed
Prepare for skills needed beyond your current expertise, by networking with other Xperts. Modern CDI applications draw from a broad range of technologies, such as computing paradigms, programming languages, architectures, and algorithms. What’s more, this range keeps evolving. For example, a new application may demand expertise in machine learning techniques. Individual Xperts and those in small teams, serving a large CDI research community, will unlikely cover the needed skills. Besides, they may need to perform tasks across the entire software life cycle and be involved in project management. It is important to be able to recognize such situations and seek external advice. Maintaining contacts with other Xpert teams, who may be consulted when needed, is highly advisable.
BP4- Collaborative Assistance Between Xperts and Domain scientists
Help propel new projects through short-term, close collaboration with domain scientists. A recommended form of interaction between Xperts and CDI domain scientists is through collaborative assistance. For a period of one to several months, Xperts work side-by-side (physically or virtually) with the domain scientists whose project they support. During the collaboration, the Xpert takes care of computer-engineering issues that arise, while the domain researcher resolves the problems requiring application science expertise. This form of joint work allows issues that fall into the competency of the collaborator to be addressed immediately, significantly reducing, or even avoiding, the need for domain researchers and Xperts to get trained on each others’ knowledge and skills at project begin. Over the course of the joint work, the collaborators tend to pick up sufficient knowledge of each others’ skills and terminology. In particular, the domain scientist becomes familiar with the software development processes and tools, allowing them to continue the work independently. Vice versa, the Xpert will have acquired sufficient knowledge of the domain problem, allowing them to effectively provide help remotely, even after moving on to other projects. This model has been pioneered by XSEDE’s ECSS group and was applied successfully by other Xpert teams.
BP5- Overcoming the Terminology Gap Between Computer and Domain Sciences
Carefully identify and resolve terminology gaps. Keep vocabulary to the essentials. Explain using many examples. The gap between computer-science and domain-science lingo is an often mentioned issue in Xpert Network discussions. The challenge can be big if the same term is used by both collaborators, but with different meanings. Participants reported significant confusion and even incorrect project executions due to this challenge. Awareness of the issue and patience in trying to understand the collaborators’ viewpoints is critical. It was pointed out that the gap is wider than in a general software engineering context when trying to understand a customer, as science terminology tends to use rich vocabularies. Keeping the vocabulary to the essentials and investing time to explain new terms is key to successful collaboration. Using many examples and frequent feedback from both sides will help bridge this gap.
(2) best practices for software development
BP6- Understanding the Domain Problem and Developing a Project Plan
Devote substantial time to understanding the domain problem, turning possibly vague ideas into a feasible solution approach, and developing a project plan. Many Xpert Network participants highlighted the importance of the first contact with the supported domain scientists and the approach taken to understand problems and develop solutions. In addition to awareness of the terminology gap, the relevance of investing time in understanding the goals and the functional as well as non-functional requirements was emphasized. Showing patience in the process is critical. The domain researcher needs to be helped to transform an initially often vague idea into a concrete plan. Developing specific requirements for the computational application and the underlying system is essential. Devote sufficient time. One challenge for the Xpert is to introduce the researcher to the possibilities of the to-be-created or improved software implementation and the underlying system while introducing only a small number of new terms. Recommendations include the use of many concrete examples and an inverted pyramid approach. The latter describes a possible solution in initially very few, high-level terms, which are then successively refined in discussions.
The situation is again related to that of software engineers with their customers, two important differences being:
- the potentially large terminology gap mentioned above, and
- the collaborative assistance situation, which often enables the inverted pyramid approach: As Xpert and domain researcher work side by side, refinements of high-level ideas can more easily happen over time, permitting the collaborators to invest their initial effort in defining the tip of the pyramid.
BP7- Prioritize Functional Requirements
Carefully vet all requirements by the application’s end-users and prioritize aggressively. The dilemma of a large number of desirable features but only a short project duration can be big in scientific software. Xperts need to learn to tell the difference between essential and desirable requirements users may have. Essential features should be strictly prioritized. Scientific software is also special in that requirements can change rapidly, as new insights of a research project emerge. Frequent reassessment of the requirements and priorities is needed.
BP8- Source Code Management and Version Control
Make use of source code management and version control systems to track your software’s evolution. An Xpert with a software engineering background is well aware of the importance of software version control. At the same time, this importance needs to be advertised to domain experts, as pointed out strongly by the Xpert network participants.
Many successful science software applications had their origin in a “toy program” written by a graduate student. It got gradually expanded by several authors, caught the attention of a wide audience, and ended up becoming an important research tool. Without version tracking, the history (origin, authors, relationship of features, and specific extensions) may no longer be known, making it difficult to extend further and obtain needed documentation. Learning version control methods and tools will not only help overcome these difficulties; it will also increase the productivity of the software developer as soon as the application exceeds the boundaries of a small program. What’s more, source code management tools greatly facilitate collaborative software development, enable software roll-back to a previous, well-defined state, and help developers start a new branch of the software. The latter can be important if, say, two graduate students want to add their own, separate feature sets.
BP9- Issue Tracking
Track origin as well as implementation status of requirements and bug reports. Issue tracking is important in most mid-size and large-size software development. Two points make it a particular concern in scientific applications. The first is the belief that research projects — often of a three-year duration — will remain small and easy to oversee, obviating the need for issue tracking. The second is that developers (typically graduate students and postdoctoral researchers) tend to change often, losing important project memory. The reality is that, even within three years, remembering what feature was requested by whom and with what rationale, can be difficult. After a “change of guard”, the same will become close to impossible. What’s more, successful science applications may turn out to have a long life. Hence, being able to trace a feature request, its implementation status, and the reasons for accepting/rejecting the request to its origin can be key. Furthermore, having a clear record of who made what request can be critical for re-assessing and re-prioritizing functional requirements, mentioned previously.
BP10- Code Review
Xpert and domain scientists should review each other’s software written. Code review — a second person reading newly written software — is one of the best ways to catch errors and also to improve the code. Despite its effectiveness, it is often ignored, unless strictly mandated, and hence warrants mentioning here. A number of additional reasons make this practice particularly useful for an Xpert-domain scientist relationship:
- The two collaborators have different backgrounds, increasing the chance of detecting an issue that eluded the other.
- The review is effective in verifying that requirement and implementation match.
- It is a good way of transferring the knowledge of what the Xpert did to the domain scientist, who will continue the work after the collaboration completes.
- The discussions happening during code review provide excellent material for documenting the code and project.
BP11- Software Testing
Define test cases that the application and its components must pass before you begin their implementation. Testing is another issue that is covered well in the software engineering literature. The topic has been mentioned often in our Xpert discussions, deserving a place in this list of best practices. Next to raising awareness of the need and the benefit of testing, a key point is that testing must not be an afterthought; one must avoid thinking of test cases only when the software is close to completion. Instead, testing should be considered in the design phase. In the initial communication with the domain researcher, the Xpert needs to ask the question “what test cases should be passed by the tool or application, once it is completed?” Coming up with a thorough set of such cases will not only facilitate the later testing phase; it will also help clarify the specific capabilities that need to be implemented. The implementation of features that do not satisfy any test case, can be postponed or even avoided altogether. Designing test cases can go hand in hand with functional prioritization.
Document your project to ensure long-term success, reproducibility, and obtain proper credit for your work. Software engineering teachings cover extensively the fact that well-documented programs are essential for software maintenance and extensibility. Yet, academic software is notorious for the lack of documentation, as pointed out forcefully by Xpert Network participants. While raising awareness of the benefits is important, one also has to understand that academic processes and reward systems often do not support spending time on documentation. Most academic research funding pays for showing principles and developing prototypes, not production-ready software. What’s more, students are under pressure to graduate, whereby the functionality of the created software is more important than its documentation. If the software becomes successful, the original author often is no longer involved and thus has little incentive to retrofit a proper description. It is often the difficult task of the next generation of students in a team to accomplish these tasks. It helps, however, to understand these relationships. If one can identify the original creator, they may be willing to help, especially when offering them proper credit or co-authorship on a forthcoming publication. Vice versa, properly documenting the original authorship will ensure that such credit can be given. The reproducibility of scientific research is a concern that is currently gaining attention. Proper software documentation can be key to reproducibility.
BP13- Continuous Integration
Integrate new software updates frequently into the application version seen by the end-users in their end environments. Continuous Integration (CI) is generally good software engineering advice. It catches miscommunicated requirements early. CI also allows users to experience new features and provide feedback to the developer early and often. The process may be combined with automated builds and testing, ensuring that new functionality satisfies and continues to satisfy defined test cases.
CI was mentioned as particularly relevant for the work of Xperts, as scientific software tends to be a moving target with frequent changes of requirements. Ensuring that new features are what the end user had in mind, conform to defined test cases, and do not break previous requirements is critical. Many teams find that this approach leads to significantly reduced integration problems and enables the development of cohesive software more rapidly.
Enable reproducibility and transparency by capturing data and software underlying scientific processes, using available software platforms. While the reproducibility of research results is a key requirement in any domain of science, there is a recent, increased focus on this issue in computation-based research. Reproducibility was highlighted as a significant concern in our Xpert Network discussions as well, pointing out that adding supporting software and data to a publication can increase the value significantly. This is of particular importance in large computational studies, where data analysis may play a central role in reaching conclusions. Disclosing the data and software underlying the research methods will add transparency.
Making use of software platforms, such as GitHub, GitLab, and container environments, such as Docker, can dramatically reduce the cost of capturing and describing the computing environment used to produce the scientific results.
Write serial code first, then parallelize. The question of parallel code should be written directly, versus creating serial code first, is an open one. The Xpert Network participants were clear, however, in recommending the latter for creating Computational and Data-Intensive (CDI) applications. Among the arguments were that the benefits of lesser complexity of getting the serial code correct first, combined with better tool support for serial code, outweigh the negatives. The primary negative is that certain serial algorithms are intrinsically hard to parallelize. When keeping this issue in mind and selecting algorithms that are known to parallelize and scale well, this negative can be overcome, however.
Interested in co-authoring the paper?
we are going to publish a paper introducing a set of best practices that have proven effective in many computational and data-intensive (CDI) research settings to RSEs. Please let us know if you are interested in co-authoring that paper by filling out the form below. Submit your name and your email address to us so that we can contact you.
Any comments, opinions, concerns, suggestions, issues, etc.?
These papers mention the practices that if applied to Scientific computing applications can improve the result or can accelerate the research.