Welcome to the HPC ReproHack

class: left, title-slide

# Welcome to the HPC ReproHack<br><img src='https://github.com/reprohack/reprohack-hq/raw/master/hex/reprohack.png' height=200px>
### <br><strong>Please Sign in on the Hackpad:</strong> <br><em>bit.ly/reprohack-hub-launch-hackpad</em>

---

class: inverse

## Welcome to the ReproHack Hub Launch!

### 👋 from the whole ReproHack team

#### _Anna, Daniela, Florencia, Linda, Ricci_

***

### Event Page: [bit.ly/reprohack-hub-launch](https://bit.ly/reprohack-hub-launch)

Contains all event information and links to materials

### <svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;" xmlns="http://www.w3.org/2000/svg">  <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"></path></svg> #ReproHackHub @ReproHack

???

Welcome everyone on behalf of the whole ReproHack team!

So, not all of us could sadly be here today, but you have got myself, as well as Linda Nab, who'm you'll hear a bit more from shortly.

I hope you've signed into the hackpad which has a lot of information, as does our event page. Both links should be in the chat.

Which brings us to why we are all here today.

So yes, we will be ReproHacking, so **working with code and data to try and reproduce** submitted papers.

---
background-image: url("assets/reprohack-hub-screenshot.png")
background-size: contain

???

But what's what really special about today's event is that, for the first time, we'll be using our purpose built ReproHack Hub.

Right from the start of the project, we'd get requests from people that wanted to run their own events and we had tried really hard to create templates and make the infrastructure we used reproducible.

But it was still complicated, a lot of moving parts, held together with a lot duck tape. We always wanted a centralised hub that provided all the required infrastructure so that event organisers (including ourselves) but also authors submitting papers or participants submitting reviews, would find it easy to do what they needed to do and focus on the experience itself.

It took a lot longer than we hoped but we are really so excited to share with you what we've created.

---
class: center, middle

# Thank you N8 CIR

_N8 Centre of Excellence in Computationally Intensive Research_

### <https://n8cir.org.uk/>

???

I want to take the opportunity upfront to thank the N8 CIR,

they've been firm supporters from early on, sponsored a number of previous events and also provided funding for developer time that got the hub off the ground so we really couldn't have  gotten here without their support.

---
class: inverse, center, middle

# Agenda

???

OK I'll do a quick blast through the agenda,

---

### Morning

**10:00 Welcome**
- **Intro to Blackboard Collaborate**
- Ice breaker (split into rooms)

**10:20 📢 Dr. Esther Plomp** ***The Turing Way Community***

**10:45 Introduction & tips for reviewing**

***11:10 Reprohack Round I***
- **Review list, form groups, select papers (break-out rooms)**
- **COFFEE BREAK** 
- Work on papers

???

This morning, after some housekeeping and an Ice breaker, we'll be hearing from Dr. Esther Plomp about the Turing Way Community.

I'll follow on with some tips and tricks for reproducing and then we'll dive straight in and start exploring, selecting and then beginning work on our papers.

---

### Lunchtime

**12:30 Regroup: get someone from each group to tell us a bit about their experiences.**

**12:45 LUNCH BREAK**

***13:45 Reprohack Round II***
- Work on papers (break-out rooms)

???

We'll regroup before lunch to share how we're getting on and then break for an hour at 12:45

After lunch we carry on ReproHacking

---

### Afternoon
***Reprohack Round III***

- Work on papers (break-out rooms)
- 14:45 COFFEE BREAK 🏠 ☕
- Work on papers - **Complete feedback form**

**16:00 📢 Prof. Stephen Eglen -** ***Sharing Reproducible Computational Environments with Binder***

**16:25 Regroup - get someone from each group to tell us a bit about their experiences.**

**16:50 Closing**

???

In te afternoon, remember to take a coffee break and also submit your reviews by the end of the day.

At 4pm, we'll rejoin to hear from Stephen Eglen on the Code Check project.

We'll close by sharing our final thoughts about the experience followed by some closing remarks.

---
class: inverse

# House Keeping:

???

So some quick housekeeping

***

## Tour of Blackboard Collaborate

> ### Linda Nab

???

I'll hand over to Linda who will give you a quick tour of blackboard collaborate.

---

## ReproHack hackpad ➡️   [hackmd.io](https://hackmd.io/s/features) notepad

???

As I've mentioned, we're also using a hackpad, specifically a hackmd hackpad. So this pad understands markdown, which is the text shown on the left and it is rendered on the right. If you are not familiar with markdown, don't worry too much, just use it as a plain text editor or you can have a quick look at hack md documentation, there's a link at the top of the hackpad.

---
class: inverse

# Ice breaker:  Introductions

???

Now we're going to do a quick round of intriductions, and I'll put you in smaller break out groups and there's three questions Id like you to answer.

I'll go first:
---

## Who am I?

> ### Dr Anna Krystalli (@annakrystalli)
> 
> - Research Software Engineer _University of Sheffield_
>
> - 2019 Fellow _Software Sustainability Institute_
>
> - Software Peer Review Editor _rOpenSci_ 
>
> - Core Team Member _ReproHack_

---

## Why am I here?

> I believe there's lots to learn about Reproducibility from working with other people's materials and engaging with real published research code and data.

???

- I feel very strongly that working with these resources is a really powerful working experience. But also that if we don't engage with the materials, we (the authors who have put all this effort in included) won't really know if they are actually reproducible and therefore fit for purpose.
---

## Who is my favorite animated character?

> Stitch!
>
> ![](https://media3.giphy.com/media/95MU6SEzeLnUc/giphy.gif?cid=790b76115d11033236595055776d483b&rid=giphy.gif)

???

I think he's cute and my friends think that's who I turn into after I've had a couple of strong ciders!

---
class: inverse

# Let's hear from the ReproHack Team!

---

# Your turn

### in break-out rooms (5 mins)

> - ### Who are you?

> - ### Why are you here?

> - ### Who is your favorite animated character?

???

OK, so it's your turn now, I'm going to put you into break out groups where you'll have 5 minutes to answer these 3 questions amongst yourself.

### As a group

> ### Name your room! 🏔🏕🏜🎢🏞🏟🏰🗼🌅🏝

???

As I group, for a bit of fun, also decide on a name for the room your group and record it in the hackpad. When we come back Linda will rename them for you.

Please be back in 5 mins sharp for our first talk.

---
class: inverse

## 📢 Dr. Esther Plomp
#### _Data Steward of the Faculty of Applied Sciences at TU Delft, Netherlands. _

<br>

> ### *"The Turing Way Community"*

---
class: inverse

## Welcome back!

***

# Tips for Reproducing & Reviewing

---

## ReproHack Objectives

1. **Practical Experience in Reproducibility**

3. **Feedback to Authors**

5. **Think more broadly about opportunities and challenges**

---

.pull-left[

## Code of Conduct

Event governed by **ReproHack Code of Conduct**

<https://reprohack.org/code-of-conduct>

]

--
.pull-right[
### Additional Considerations

- #### Reproducibility is hard!
- #### Submitting authors are incredibly brave!

### Thank you Authors! 🙌

- #### Without them there would be no ReproHack.
- #### Show gratitude and appreciation for their efforts. 🙏 
- #### Constructive criticism only please!

]
---
class: inverse, center, middle

# 🔍 Reproducing & Reviewing

---

## Selecting Papers

.pull-left[
- **Information submitted by authors:**

- Languages / tools used (tags)
  
  - Why you should attempt the paper.
  
- **No. attempts <svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;" xmlns="http://www.w3.org/2000/svg">  <path d="M184.561 261.903c3.232 13.997-12.123 24.635-24.068 17.168l-40.736-25.455-50.867 81.402C55.606 356.273 70.96 384 96.012 384H148c6.627 0 12 5.373 12 12v40c0 6.627-5.373 12-12 12H96.115c-75.334 0-121.302-83.048-81.408-146.88l50.822-81.388-40.725-25.448c-12.081-7.547-8.966-25.961 4.879-29.158l110.237-25.45c8.611-1.988 17.201 3.381 19.189 11.99l25.452 110.237zm98.561-182.915l41.289 66.076-40.74 25.457c-12.051 7.528-9 25.953 4.879 29.158l110.237 25.45c8.672 1.999 17.215-3.438 19.189-11.99l25.45-110.237c3.197-13.844-11.99-24.719-24.068-17.168l-40.687 25.424-41.263-66.082c-37.521-60.033-125.209-60.171-162.816 0l-17.963 28.766c-3.51 5.62-1.8 13.021 3.82 16.533l33.919 21.195c5.62 3.512 13.024 1.803 16.536-3.817l17.961-28.743c12.712-20.341 41.973-19.676 54.257-.022zM497.288 301.12l-27.515-44.065c-3.511-5.623-10.916-7.334-16.538-3.821l-33.861 21.159c-5.62 3.512-7.33 10.915-3.818 16.536l27.564 44.112c13.257 21.211-2.057 48.96-27.136 48.96H320V336.02c0-14.213-17.242-21.383-27.313-11.313l-80 79.981c-6.249 6.248-6.249 16.379 0 22.627l80 79.989C302.689 517.308 320 510.3 320 495.989V448h95.88c75.274 0 121.335-82.997 81.408-146.88z"></path></svg>** No. times reproduction has been attempted

- **Mean Repro Score <svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;" xmlns="http://www.w3.org/2000/svg">  <path d="M256 8C119.033 8 8 119.033 8 256s111.033 248 248 248 248-111.033 248-248S392.967 8 256 8zm0 48c110.532 0 200 89.451 200 200 0 110.532-89.451 200-200 200-110.532 0-200-89.451-200-200 0-110.532 89.451-200 200-200m140.204 130.267l-22.536-22.718c-4.667-4.705-12.265-4.736-16.97-.068L215.346 303.697l-59.792-60.277c-4.667-4.705-12.265-4.736-16.97-.069l-22.719 22.536c-4.705 4.667-4.736 12.265-.068 16.971l90.781 91.516c4.667 4.705 12.265 4.736 16.97.068l172.589-171.204c4.704-4.668 4.734-12.266.067-16.971z"></path></svg>** Mean reproducibility score (out of 10)

- lower == harder!
    
- Register paper using template in hackpad:
    ```
    ### **Paper:** <Title of the paper reproduced>
    **Reviewers:** Reviewer 1, Reviewer 2 etc.
    ```
]

.pull-right[
<img src="assets/browse-paper-list.png" width="3813" />

### [reprohack.org/paper/](https://reprohack.org/paper/)

]

---
class: inverse, center, middle

# Review as an auditor 📑

#### **tl;dr: Don't be this guy!**

---
class:centers

# 🔍 For FAIR materials

---

.pull-left[

# Access

- How **easy** was it to **gain** access to the materials?

- Did you manage to download all the files you needed?

]

.pull-right[

# Installation

- How **easy / automated** was **installation**?

- Did you have any problems?

- How did you solve them?
]

--- 
---

.pull-left[

# Data

- Were **data clearly separated from code and other items**?

- Were **large data files deposited in a trustworthy data repository** and referred to using a **persistent identifier**?

- Were **data documented** ...somehow...

]

.pull-right[

# Documentation

Was there **adequate documentation** describing:
- how to **install** necessary software including non-standard dependencies?

- how to **use** materials to reproduce the paper?

- how to **cite** the materials, ideally in a form that can be copy and pasted?

]
---

.pull-left[

# Analysis

- **Were you able to fully reproduce** the paper? ✅

- **How automated** was the process of reproducing the paper?

- **How easy was it to link** analysis **code** to:
   - the **plots** it generates
   - **sections in the manuscript** in which it is described and results reported
]
--

.pull-right[

<br>

### If the analysis was not fully reproducible 🚫
 - Were there **missing dependencies?**
 
 - Was the **computational environment not adequately described** / captured?
 
 - Was there **bugs** in the code?
 
 - Did **code run but results (e.g. model outputs, tables, figures) differ** to those published? By **how much?**
]

---
class: inverse, center, middle

# Review as a user 🎮

.pull-left[

### New User

]

.pull-right[

### Invested User

]

---
## Review as a user 🎮

<br>

#### What did you find easy / intuitive?
> Was the file structure and file naming informative / intuitive?
> Was the analysis workflow easy to follow? 
> Was there missing / confusing documentation?

#### What did you find confusing / difficult
> Identify pressure points. Constructive suggestions?

#### What did you enjoy?
> Identify aspects that worked well.

---
class: inverse, center, middle

# Feed back

# 💬

---

## Feedback as a community member

.pull-left[

#### Acknowledge author effort

#### Give feedback in good faith

#### Focus on community benefits and system level solutions

]
.pull-right[

> #### _Help build convention on what a Research Compendium should be and how we should be able to use it_

]
---
# Submit review

1. ### Sign up / Log in

2. ### New Review: [reprohack.org/review/new](https://www.reprohack.org/review/new)
  
  <img src="assets/new-review.png" width="70%" />

---
class: inverse, center, middle

# Participant Guidelines

### [reprohack.org/participant_guidelines](https://www.reprohack.org/participant_guide)lines

---

### Participation guidelines

#### During talks information sessions and regroups
- Please stay on mute until you are invited to speak.
- Add questions for speakers on the hackpad

#### During break out work 
- Try and engage as much as possible.
- Feel free to mute, turn off camera and disconnect if you need to.
- Feel free to move around and go say hello or help out in other rooms.
- We encourage discussions in the break-out rooms but be mindful of others trying to work. The **main room** and **cafe** are always available for hanging out.

### At any time
- Feel free to add to the collaborative notes on the hackpad
- Contact facilitators through the main chat

---
class: inverse

# Let's go! 🏁

## 11:10 - 11:30

### 🔎 Paper List review

+ Have a look at the papers available for reproduction

### 👥 Team formation / project registration

+ Fine to work individually
+ Add your details to the [**hackpad: bit.ly/reprohack-hub-launch-hackpad**](https://bit.ly/reprohack-hub-launch-hackpad).
+ Register your team and paper on the [**hackpad: bit.ly/reprohack-hub-launch-hackpad**](https://bit.ly/reprohack-hub-launch-hackpad)

### 🏠☕Grab a coffee!

---
class: inverse

## 11:30 - 12:30 💻 ReproHack I 
### (break-out rooms)

Work on your papers. Feel free to discuss papers and collaboratively troubleshoot problems. Reach out through the main chat for help outside the group. Feel free to move groups or hang out in the main *room* or *cafe.*

> ### Before Lunch-time Regroup 💭
> #### Summarise group experiences
> - What approaches to reproducibility the papers taken.
> - Anything in particular you like about the approaches so far?
> - Anything you're having difficulty with?

---
class: inverse

## 12:30 - 12:45 💬 Lunch regroup

### Welcome back

***

### Feedback group experiences

---
class: inverse

## 12:45 - 13:45 🥗🌯 LUNCH

Feel free to disconnect or hang out in the main *room* or *cafe.*

---
class: inverse

## 13:45 - 16:00  💻 ReproHack II & III 
### (break-out rooms)

### 14:45 - 15:00 COFFEE BREAK 🏠☕

> ### Before Final-time Regroup 💭
> #### Complete author feedback form ✍️
> - Discuss how you got on with your papers?
> - Summarise final experiences of the group in hackpad

---
class: inverse

## 📢 Prof. Stephen Eglen

_Professor of Computational Neuroscience in the Department of Applied Mathematics and Theoretical Physics, University of Cambridge_

<br>

> ### *"CODECHECK: code review during peer review"*

---
class: inverse

## 16:25 - 16:50 Final regroup 💬

- So, how did the groups get on?

- Final comments.

- On hackpad: **Feedback**
  + One thing you liked
  + One thing that can be improved.

---
class: inverse, center, middle

# Closing Remarks

---
class: inverse, center, middle

# Resources

---

## Resources

- [**The Turing Way**](https://the-turing-way.netlify.com/introduction/introduction): a lightly opinionated guide to reproducible data science.

- [**Statistical Analyses and Reproducible Research**](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.684.9629&rep=rep1&type=pdf): Gentleman and Temple Lang's introduction of the concept of Research Compendia

- [**Packaging data analytical work reproducibly using R (and friends)**](https://peerj.com/preprints/3192/): how researchers can improve the reproducibility of their work using research compendia based on R packages and related tools

- [How to Read a Research Compendium](https://arxiv.org/pdf/1806.09525.pdf): Introduction to existing conventions for research compendia and suggestions on how to utilise their shared properties in a structured reading process.

- [Reproducible Research in R with rrtools](https://annakrystalli.me/rrtools-repro-research/): Workshop: Create a research compendium around materials associated with a published paper (text, data and code) using `rrtools`.  
  - [**Example Compendium**](https://github.com/annakrystalli/rrcompendiumDTB): Demo Research compendium.

---
## Did you enjoy ReproHacking? Get involved!

.pull-left[
# [reprohack.org](https://www.reprohack.org/) 
]
.pull-right[
#### Chat to us:

[![Slack](https://img.shields.io/badge/slack-join%20us-brightgreen)](https://reprohack-autoinvite.herokuapp.com/)
]

- ### Host your own event! 
  train-the-trainer events coming up with the N8 CIR & [AIMOS conference](https://www.aimosconference.com/)
- ### Submit your own papers!

---
## Many ways to ReproHack!

---

# THANK YOU ALL! 🙏

- ### Thank you PARTICIPANTS for coming!
- ### Thank you AUTHORS for submitting!
- ### Thank to the N8 CIR for sponsoring!

# 👋

---

# Acknowledgements

Images throughout the slides watermarked with **Scriberia** were created by [Scriberia](https://www.scriberia.co.uk/) for The Turing Way community and is used under a CC-BY licence 
- _The Turing Way Community, & Scriberia. (2019, July 11). Illustrations from the Turing Way book dashes. Zenodo. http://doi.org/10.5281/zenodo.3332808_