Skip to content

Commit

Permalink
Add internal links
Browse files Browse the repository at this point in the history
  • Loading branch information
woodthom2 committed Aug 20, 2023
1 parent 9eeb4c4 commit 3b1e3b4
Show file tree
Hide file tree
Showing 22 changed files with 116 additions and 111 deletions.
12 changes: 6 additions & 6 deletions content/en/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ blocks:
heading: "Harmonise questionnaire items with **Harmony**."
subheading: Harmony is a tool for retrospective harmonisation of questionnaire items.
copy: |-
Do you need to compare questionnaire data across studies? Do you want to find the best match for a set of items? Are there are different versions of the same questionnaire floating around and you want to make sure how compatible they are? Are the questionnaires written in different languages that you would like to compare?
Do you need to compare questionnaire items across studies? Do you want to find the best match for a set of items? Are there are different versions of the same questionnaire floating around and you want to make sure how compatible they are? Are the questionnaires [written in different languages](/harmony-supports-over-8-languages/) that you would like to compare?
The Harmony project is a data harmonisation project that uses [Natural Language Processing](/guide-natural-language-processing-nlp/) to help researchers make better use of existing data from different studies by supporting them with the harmonisation of various measures and items used in different studies. Harmony is a collaboration project between [Ulster University](https://ulster.ac.uk/), [University College London](https://ucl.ac.uk/), the [Universidade Federal de Santa Maria](https://www.ufsm.br/), and [Fast Data Science](http://fastdatascience.com/).
The Harmony project is a data harmonisation project that uses [Natural Language Processing](https://fastdatascience.com/guide-natural-language-processing-nlp/) to help researchers make better use of existing data from different studies by supporting them with the harmonisation of various measures and items used in different studies. Harmony is a collaboration project between [Ulster University](https://ulster.ac.uk/), [University College London](https://ucl.ac.uk/), the [Universidade Federal de Santa Maria](https://www.ufsm.br/), and [Fast Data Science](http://fastdatascience.com/). Harmony is funded by [Wellcome](https://wellcome.org/) as part of the [Wellcome Data Prize in Mental Health](https://wellcome.org/grant-funding/schemes/wellcome-mental-health-data-prize).
buttons:
- text: Try Harmony Now!
Expand All @@ -29,7 +29,7 @@ blocks:
padding: top
superheading: "**Harmony**"
heading: Introduction to Harmony
subheading: Data harmonisation with natural language processing
subheading: Item harmonisation with natural language processing
youtube: cEZppTBj1NI
image: images/bg-video.jpg

Expand Down Expand Up @@ -95,7 +95,7 @@ blocks:
# padding: top
align: right
heading: "Harmony Docker container"
subheading: We've released Harmony as a Docker container on Dockerhub.
subheading: We've released Harmony as a Docker container on [Dockerhub](https://hub.docker.com/).
links:
- text: Docker
url: https://www.docker.com/
Expand Down Expand Up @@ -145,9 +145,9 @@ blocks:
align: right
image: images/gad-7-scanned-min.webp
copy: |
Our tool, Harmony, allows researchers to upload a set of mental health questionnaires in PDF or Excel format, such as the GAD-7 anxiety questionnaire. It identifies which questions among questionnaires are identical, similar in meaning, or antonyms of each other, and generates a network graph. This allows researchers to harmonise datasets.
Our tool, Harmony, allows researchers to upload a set of mental health questionnaires in PDF or Excel format, such as the [GAD-7 anxiety questionnaire](https://adaa.org/sites/default/files/GAD-7_Anxiety-updated_0.pdf). It identifies which questions among questionnaires are identical, similar in meaning, or antonyms of each other, and generates a network graph. This allows researchers to harmonise datasets.
Uniquely, Harmony relies on Transformer neural network architectures and is not dependent on a dictionary approach or word list. This allows for multilingual support (English and Portuguese are our languages of focus), and Harmony is able to correctly map the GAD-7 used in the UK to the GAD-7 used in Brazil, despite the Brazilian questionnaire being in Brazilian Portuguese.
Uniquely, Harmony relies on [Transformer neural network architectures](https://deepai.org/machine-learning-glossary-and-terms/transformer-neural-network) and is not dependent on a dictionary approach or word list. This allows for multilingual support (English and Portuguese are our languages of focus), and Harmony is able to correctly map the GAD-7 used in the UK to the [GAD-7 used in Brazil](https://pesquisa.bvsalud.org/portal/resource/pt/lil-788637), despite the Brazilian questionnaire being in Brazilian Portuguese.
Using Harmony, our team was able to conduct groundbreaking research into social isolation and anxiety with NLP supplying a quantitative measure of the equivalence of the different mental health datasets.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,13 @@ However, not all constructs can be measured with such simple, categorical questi

How can researchers harmonize such complex measures? One option would be to standardize scores within each data set, thus transforming everyone’s raw score to a rank ordering within their given sample. Although straightforward, this approach has a number of weaknesses. First and foremost, you are assuming that both questionnaires are measuring the same underlying construct, and are measuring it equally well. Second, by standardizing a measure within a cohort, you are removing all information about the mean and standard deviation, making it impossible to compare the average level of a construct across datasets.

An alternative approach is to apply retrospective harmonization at the item-level. Although questionnaires can differ considerably on the number and nature of questions asked, there is often considerable overlap at the semantic/content level. Let’s return to our earlier example of depression. Although there are many different questionnaires that can be used to assess this experience, they often ask the same types of questions. Below is an example of content overlap in two of the most common measures of psychological distress used in children, the Revised Children’s Anxiety and Depression Scale (RCADS), and the Mood and Feelings Questionnaire (MFQ).
An alternative approach is to apply retrospective harmonization at the item-level. Although questionnaires can differ considerably on the number and nature of questions asked, there is often considerable overlap at the [semantic](https://harmonydata.ac.uk/semantic-text-matching-with-deep-learning-transformer-models)/content level. Let’s return to our earlier example of depression. Although there are many different questionnaires that can be used to assess this experience, they often ask the same types of questions. Below is an example of content overlap in two of the most common measures of psychological [distress](https://harmonydata.ac.uk/how-far-can-we-go-with-harmony-testing-on-kufungisisa-a-cultural-concept-of-distress-from-zimbabwe) used in children, the Revised Children’s Anxiety and Depression Scale (RCADS), and the Mood and Feelings Questionnaire (MFQ).

{{< image src="images/blog/blog-pic-1.png" alt="img" >}}

By identifying, recoding, and testing the equivalence of subsets of items from different questionnaires (for guidelines see our previous report), researchers can derive harmonized sub-scales that are directly comparable across studies. Our group has previously used this approach to study trends in mental health across different generations (Gondek et al., 2021), and examine how socio-economic deprivation impacted adolescent mental health across different cohorts (McElroy et al., 2022).

One of the main challenges to retrospectively harmonizing questionnaire data is identifying the specific items that are comparable across the measures. In the above example, we used expert opinion to match candidate items based on their content, and used psychometric tests to determine how plausible it was to assume that matched items were directly comparable. Although our results were promising, this process was time-consuming, and the reliance on expert opinion introduces an element of human bias – i.e. different experts may disagree on which items match. As such, we are currently working on a project supported by Wellcome, in which we aim to develop an online tool, ‘Hamony’, that uses machine learning to help researchers match items from different questionnaires based on their underlying meaning. Our overall aim is to streamline and add consistency and replicability to the harmonization process. We plan to test the utility of this tool by using it to harmonize measures of mental health and social connectedness across two cohort of young people from the UK and and Brazil.
One of the main challenges to retrospectively harmonizing questionnaire data is identifying the specific items that are comparable across the measures. In the above example, we used expert opinion to match candidate items based on their content, and used psychometric tests to determine how plausible it was to assume that matched items were directly comparable. Although our results were promising, this process was time-consuming, and the reliance on expert opinion introduces an element of human [bias](https://fastdatascience.com/how-can-we-eliminate-bias-from-ai-algorithms-the-pen-testing-manifesto) – i.e. different experts may disagree on which items match. As such, we are currently working on a [project](https://fastdatascience.com/starting-a-data-science-project) supported by Wellcome, in which we aim to develop an online tool, ‘Hamony’, that uses machine learning to help researchers match items from different questionnaires based on their underlying meaning. Our overall aim is to streamline and add consistency and replicability to the harmonization process. We plan to test the utility of this tool by using it to harmonize measures of mental health and social connectedness across two cohort of young people from the UK and and Brazil.

Follow this blog for updates on our Harmony project!

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,27 +7,27 @@ image: /images/blog/noah-buscher-x8ZStukS2PM-unsplash-1536x880.jpg

**5 key things Implementation Science has taught us** **to focus on**

Yesterday the [Harmony team](https://harmonydata.ac.uk/team/) received the wonderful news that we are given the chance to continue working on Harmony for another six months, after which we can put ourselves forward for the final round. The [Wellcome Mental Health Data Prize](https://wellcome.org/grant-funding/schemes/wellcome-mental-health-data-prize) has chosen an unusual (for the research world) approach this time, using a “Dragon’s Den” style scheme where research teams can pitch their ideas to win funding for their projects. We started this journey with 10 other teams around six months ago, and last week we all presented our work and pitched our vision for the next 6 months. This nontypical funding scheme challenges some of the traditionally slow university structures, and I am excited about the creativity with which the Wellcome Trust keeps the research world on its toes.
Yesterday the [Harmony team](https://harmonydata.ac.uk/team/) received the wonderful news that we are given the chance to continue working on Harmony for another six months, after which we can put ourselves forward for the final round. The [Wellcome Mental Health Data Prize](https://wellcome.org/grant-funding/schemes/wellcome-mental-health-data-prize) has chosen an unusual (for the research world) approach this time, using a “Dragon’s Den” style scheme where research teams can pitch their ideas to win funding for their [projects](https://harmonydata.ac.uk/projects-partners). We started this journey with 10 other teams around six months ago, and last week we all presented our work and pitched our vision for the next 6 months. This nontypical funding scheme challenges some of the traditionally slow university structures, and I am excited about the creativity with which the Wellcome Trust keeps the research world on its toes.

As our team embarks on the prototyping journey, I am reflecting on how we can maximise the implementation success for our digital [Harmony tool](https://harmonydata.ac.uk/app/?_ga=2.55018287.544219844.1678452210-721610193.1678452210&_gl=1*7y76do*_ga*NzIxNjEwMTkzLjE2Nzg0NTIyMTA.*_ga_5B3RD8TY0P*MTY3ODQ1MjIxMC4xLjEuMTY3ODQ1MjIyNC4wLjAuMA..). I **want** Harmony to have a life post grant-funding and ensure that it has measurable impact on the wider researcher community and global mental health efforts.

The successful implementation and sustainability of digital products developed through research grant funding, has been shockingly low. We have seen this especially in the digital mental health field, where thousands of apps and platforms have been developed and only very few have been implemented and sustained in the wild. From this line of [research](https://www.psychiatrist.com/jcp/psychiatry/implementing-digital-mental-health-interventions/#ref16) and [my own work with colleagues](https://www.jmir.org/2022/11/e40347) I know that innovation and effectiveness alone are not sufficient to secure real-world adoption .
The successful implementation and [sustainability](https://harmonydata.ac.uk/sustainability) of digital products developed through research grant funding, has been shockingly low. We have seen this especially in the digital mental health field, where thousands of apps and platforms have been developed and only very few have been implemented and sustained in the wild. From this line of [research](https://www.psychiatrist.com/jcp/psychiatry/implementing-digital-mental-health-interventions/#ref16) and [my own work with colleagues](https://www.jmir.org/2022/11/e40347) I know that innovation and effectiveness alone are not sufficient to secure real-world adoption .

{{< image src="images/blog/noah-buscher-x8ZStukS2PM-unsplash-1536x880.jpg" alt=" noah buscher" >}}

So how can we maximise the uptake and implementation of Harmony to give it a longer shelf-life? I’ll draw on the field of implementation science[[i\]](https://harmonydata.ac.uk/harmony-going-forward-5-things-implementation-science-has-taught-us-to-focus-on/#_edn1) which provides useful insights and [frameworks](https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-8-139#Abs1) on share some reflections on how this could be done and what we and our fellow teams may want to focus on at this stage.

**1. Think about uptake and implementation early on**

- Researchers (me included) tend to think about implementation and sustainability too late. Don’t wait until your project is approaching the final stretch.
- Researchers (me included) tend to think about implementation and sustainability too late. Don’t wait until your [project](https://fastdatascience.com/starting-a-data-science-project) is approaching the final stretch.
- If that could be you, look up some of the great work published in the [implementation science journal](https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-1-1#additional-information).
- As a start think about: what does successful implementation mean for your team and what could it look like for your tool?

**2.** **Identify and involve stakeholders when you design and develop your product/tool**

- Understand your users’ journey, motivation, context and create some fun user personas.
- Get feedback from others early on, don’t wait until you have an almost finished product. I understand it can be scary putting something out there before it is ready, but it will be worth it.
- Make sure your product is easy to use and doesn’t require much explanation. User-testing workshops can help you identify where your potential user may stop and doesn’t know what to do next with your product.
- Make sure your product is easy to use and doesn’t require much explanation. User-testing [workshops](https://harmonydata.ac.uk/harmony-tidal-workshop) can help you identify where your potential user may stop and doesn’t know what to do next with your product.
- Ask your users whether your tool is actually “useful” to them and is matches their work flow or how they would like to use it.

**3.** **Think of (existing) structures to integrate your product**
Expand Down
Loading

0 comments on commit 3b1e3b4

Please sign in to comment.