-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathqual-notes.qmd
106 lines (70 loc) · 4.52 KB
/
qual-notes.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
---
title: Qualitative Research Approaches in CS
author: Neil Ernst
date: last-modified
format:
revealjs:
theme: solarized
smaller: false
scrollable: true
incremental: true
footer: "©️ Neil Ernst"
---
# Learning Objectives
Q: what haven't we covered yet that you want to cover?
* Learn about the scope of qualitative research in CS
* Summarize different qualitative research methods
* See some exemplars of excellent qualitative CS papers
#
I don't particularly like the qual/quant distinction - I find it more useful to look at research strategies using the ABC or WWH or Circumplex frameworks.
See the lectures from [Epistemology](https://github.com/neilernst/ResearchSkillsUVic/blob/main/resources/epistemology/epistemology.md)
#
Typically qualitative research refers to research strategies that analyze the evidence *qualitatively*, i.e., interpretively. Data sources that fit well in these strategies are personal narratives, observations of users, interview transcripts, texts like books or videos, open-ended survey answers, etc.
Qualitative research is more often associated with constructivist and interpretivist philosophies. See the [sex workers and tech](https://osf.io/preprints/socarxiv/vzehu/) paper.
Good qualitative research is **no less valid** than quantitative research.
## Yin's case study inference diagram
Focus on units of analysis.
## Qualitative Methods (established)
* surveys (both qual/quant)
* interviews
* ethnography/participant observation
* contextual inquiry
## Methods (emerging/lesser known in CS)
* Narrative analysis
* Action Research
* Diary studies (like narrative)
* Focus groups
* ...
## Analysis Approaches
* [grounded theory](https://dl.acm.org/doi/10.1145/2884781.2884833)
* Thematic coding
* Coding from predefined coding categories (deductive)
## Validity and qualitative research
A common critique of these methods is "the plural of anecdote is not data". As Yin's diagram shows, though, qualitative research is rarely just anecdote (and we can of course learn from anecdote! Aesop's fables are one example). However, a good question to ask when reading such work is "how do I know this is a true account and legitimate interpretation" of the data?
- Member checking
- Rich descriptions (thick)
- Links to evidence
- Measure reliability
- Self-critique and journalling
## Coding Data
Qualitative data is increasingly analyzed with tools, e.g., NLP techniques for extracting narrative summaries, micro-narratives that can be quickly grouped into themes. But human interpretation of the data is still really important. It's both incredibly insightful and incredibly boring to do.
One question we can ask is "given the same data set, would another person find the same code". Note the positivist assumptions here.
We could measure the inter-rater agreement or reliability using common techniques such as Cohen's kappa, and then make an argument as to why that justifies validity of the coding approach.
**My view**: the real question is how reliable is the coding guide. The best way to validate this is to conduct robust coding agreement sessions and refinement of the coding guide/taxonomoy.
# Exercise
Let's suppose we have a study planned into *how people update SO questions*. For example, our working theory might be that the edit comments on SO questions tell us something about how knowledge is curated on SO. So we want to begin looking through the data to see how well that works.
Visit the [Google sheet here](https://drive.google.com/file/d/1KvQCYx9TJXgHdU5CURsbiijKWM2zhmcP/view?usp=sharing). Make a copy of it or download it.
# Exercise
* Q: how do people **update** SO questions?
* on your own: spend 5 minutes adding labels to at least 10 rows of the data.
* with a partner: compare the labels you each came up with.
# Reviewing Qualitative Work
The SIGSOFT standards give plenty of guidance, e.g. on [qualitative surveys](https://acmsigsoft.github.io/EmpiricalStandards/docs/?standard=QualitativeSurveys).
# Exemplar Readings
* See e.g. example papers from the SIGSOFT standards on [qualitative surveys](https://acmsigsoft.github.io/EmpiricalStandards/docs/?standard=QualitativeSurveys).
# Optional and Recommended
* Research design: qualitative, quantitative and mixed methods approaches. Creswell, John W. Sage, 2018.
* The [links on Dr Storey's course page.](https://github.com/margaretstorey/EmseUvic2020/blob/master/resources/methods.md)
* [Sex Work and Tech paper](https://osf.io/preprints/socarxiv/vzehu/)