This session is shorter than most and has no exercise set: it is the argument for everything that follows, not a technique to practise. It fits one sitting. Read it before The GLPX-1 Trial, which introduces the data you will work with for the rest of the course.
TipHow to Use the Code in This Session
Every code chunk below is written so you can copy it into your R console (or an R script) and run it yourself, in the order shown. Where you see a “Run It Yourself” box, stop and run the code above it before reading the explanation that follows.
Run R from the project root (open the repository’s .Rproj file, or setwd() to the folder containing _quarto.yml). Every path in this course is written relative to that root, so data/raw/dm_raw.csv means what it says.
2.1 When Do You Use This?
Tip
You have joined a study team and been handed five CSV files and a question: “how many patients had nausea?” Or a reviewer has asked why the subject count in your table is 400 when the demographics file has 401 rows. Or someone says the word “SDTM” in a meeting and you nod.
In each case the underlying problem is the same: the data cannot be read correctly by anyone who does not already know this particular trial. That is the problem standards solve, and this session demonstrates it before naming a single CDISC variable.
This is the first session of the course and assumes no CDISC knowledge. It assumes you can read dplyr.
2.2 Learning Objectives
After completing this session you will be able to:
Explain why clinical data standards exist, in terms of the specific problems they solve rather than the regulations that mandate them
Demonstrate, in R, that raw trial data carries structure and vocabulary that only the study team understands
Distinguish the three things a standard fixes: variable naming, dataset structure, and controlled vocabulary
Explain why “we will standardise it at the end” is a false economy
Describe the shape of the SDTM → ADaM → TLF pipeline the rest of this course builds
2.3 Background: Data That Can Only Be Read by Its Author
Estimated time: ~15 minutes (reading)
Here is the single idea that ties this session together, and arguably the whole course:
ImportantThe one thing to remember
Raw trial data can only be read by someone who already knows the trial. A standard is what lets it be read by someone who doesn’t.
Every dataset carries two kinds of information. There is the data itself: the numbers, the dates, the terms. And there is the knowledge required to interpret it: that colldt means the date the sample was collected, that a blank AEENDT means the event was still ongoing rather than that someone forgot, that subject 101-012 appears twice for a reason.
In a normal research project, that second kind of information lives in someone’s head, and that is survivable. You wrote the data; you can read the data. If a colleague needs it, you explain it to them over coffee.
Clinical trials break this arrangement in three ways at once.
Scale. A regulator receives study data from many sponsors, for many trials, in many therapeutic areas, every year. Nobody at the agency can have coffee with every study team. If each sponsor invents its own layout, then reviewing \(n\) trials requires understanding \(n\) different conventions, and no tool built for one submission works on the next.
Time. A trial’s data outlives the people who made it. Regulators may return to a submission years later; a safety signal may surface long after the study team has moved on. “Ask the programmer who built it” stops being an available answer.
Consequence. If a reviewer misreads your adverse event data, the result is not an awkward correction in a journal. It is a wrong conclusion about whether a drug harms people.
So the industry did the only thing that scales: it agreed, in advance, on what the columns are called, how the datasets are shaped, and which words are allowed in them. That agreement is what CDISC publishes.
2.3.1 A standard fixes three separate things
It is worth separating these now, because learners routinely collapse them into “renaming columns,” and then wonder why the rest of the course is hard.
Example 1 below demonstrates the first two problems in the GLPX-1 raw data. Example 2 demonstrates the third. Neither example needs you to know any CDISC yet, that is the point. You should feel the problem before you are handed the solution.
2.3.2 The shape of what is coming
The rest of this course is one pipeline, and it is worth seeing the whole shape before you meet any of its parts:
Raw data, what the study team receives: an EDC export, a lab vendor’s file, each with its own conventions. This is where you are now.
SDTM: the raw data, reorganised and renamed to the standard, with the original values preserved. It answers: what happened in this trial? (Part 2)
ADaM, analysis-ready datasets derived from SDTM, carrying the variables a statistician needs to run the analysis and a reviewer needs to reproduce it. It answers: what does the trial show? (Part 3)
TLFs: the tables, listings and figures that go in the study report. (Part 4)
Each stage has a different job, and a great deal of confusion in clinical programming comes from asking one stage to do another’s work. SDTM Concepts and ADaM Concepts draw that line properly.
NoteWhere the requirement actually comes from
The argument in this session stands on its own, but the US regulatory chain behind it is short and worth knowing. Four documents do the work:
The statute and the guidance. Under section 745A(a) of the FD&C Act, once FDA specifies an electronic format in final guidance, that format is required. The specifying document is Providing Regulatory Submissions in Electronic Format, Standardized Study Data(U.S. Food and Drug Administration 2021). It applies to NDAs, ANDAs, certain BLAs and certain commercial INDs, and (unusually for FDA guidance) has binding effect. FDA may refuse to file (NDAs, BLAs) or refuse to receive (ANDAs) a submission whose study data does not conform to the required standards in the Catalog.
The Data Standards Catalog(U.S. Food and Drug Administration 2025a) is the list the guidance points to: which standards FDA can process, review and archive, and from when. It lists CDISC SDTM and ADaM (exchanged as XPT files) and Define-XML among the required standards. The requirement applies based on study start date: after 17 December 2016 for NDAs, ANDAs and BLAs; after 17 December 2017 for the applicable INDs.
The Study Data Technical Conformance Guide(U.S. Food and Drug Administration 2025b) is the technical companion: how to organise, name and document the datasets, including the data definition file (define.xml) that describes every submitted dataset’s metadata. You will build one in define.xml.
The Technical Rejection Criteria (TRC) make part of this automatic. The TRC is not a separate document. It lives inside the Conformance Guide (TCG v6.0, §8.1.2.1–8.1.2.3) (U.S. Food and Drug Administration 2025b), and the clinical-drug case this course works in is §8.1.2.1. The mechanism is worth knowing precisely, because it is mechanical: the check operates through the study tagging file and the trial summary dataset, ts.xpt. A fullts.xpt is expected when a study’s type and initiation date meet the Data Standards Catalog’s criteria for requiring SDTM and ADaM; a simplifiedts.xpt (TCG Appendix G) covers studies where those standards are not required.
That closes the chain: the Catalog sets the requirement → the TRC enforces it via the ts.xpt check → non-conformance triggers rejection, before a human reviewer ever opens the submission. You will meet ts.xpt again in The Analysis Data Reviewer’s Guide.
Two cautions. The dates above attach to study start, not submission date. And the Catalog is a living document, versions of SDTM, ADaM and Define-XML move in and out of the required set over time, so the version to consult is always the current one, at the FDA Study Data Standards Resources page.
NoteWhy not just write good documentation instead?
A reasonable objection: if the problem is that knowledge lives in someone’s head, why not write it down? A thorough data dictionary per trial would solve it, wouldn’t it?
It solves the comprehension problem and not the automation one. A reviewer who can read your dictionary still cannot run last year’s program against your data, because your column is called colldt and last year’s was called SAMPDT. Documentation lets a human understand \(n\) conventions one at a time. A standard lets one program read all \(n\). The difference compounds: every validation rule, every review tool, and every analysis macro is written once against the standard rather than once per trial.
This is also why the standard is not negotiable per-study. A standard that each sponsor adapts to taste is a dictionary with extra steps.
2.4 Example 1: Five Files, Five Conventions
Estimated time: ~20 minutes (worked example)
GLPX-1 is a simulated Phase III trial of a GLP-1-class agent in type 2 diabetes. The next session introduces it properly. For now you only need to know that its raw data is what a study team actually receives: an export from the EDC system where sites enter data, plus a separate file from the central laboratory that analyses the blood samples. Two organisations, two sets of habits.
Load all five raw files and look at nothing but their shape:
Run the chunk above. You should see five files: ae_raw.csv (389 rows), dm_raw.csv (401), ex_raw.csv (400), lb_raw.csv (12,389) and vs_raw.csv (12,788).
Before reading on, look only at the columns column and answer one question: if you had to write a single function that returned the subject identifier from any of these five files, what would it look like?
Everything here is read as character deliberately. Left to itself, read_csv() guesses types, and a guess is exactly the kind of trial-specific knowledge we are trying to expose rather than hide. A date that is sometimes 2024-03-12 and sometimes 1969 is not a date column to R; it is a mess, and we want to see the mess.
The answer to the question in the box is that your function needs a lookup table, because the files do not agree:
The lab vendor lower-cases its columns. R is case-sensitive, so dm$SUBJID and lb$subjid are different names for the same concept. Every join you write must know this.
Eight date columns, eight names, no shared pattern. colldt does not even end in the same convention as the others.
ae_raw.csv has 389 rows
Fewer rows than subjects. Adverse event data has one row per event, not per subject, and subjects with no events have no rows at all.
lb_raw.csv has 12,389 rows
One row per subject per visit per test. The same trial, at a completely different granularity.
Notice what the last two rows of that table mean: “one row” means something different in every file. Structure is not a presentational detail you can fix with rename().
The grep() in that chunk is itself the lesson. To find the subject column, we had to search for it with a case-insensitive pattern, because we already knew (from having looked) that it might be upper or lower case. A standard is what makes that search unnecessary.
2.4.1 When two sources disagree about a fact
Both the EDC and the lab file record each subject’s sex. They were entered by different people, at different times, into different systems. Nothing forces them to agree, so let us check whether they do:
# A tibble: 1 × 3
SUBJID edc_sex lab_sex
<chr> <chr> <chr>
1 111-003 F M
TipRun It Yourself
Run the chunk above. Exactly one subject comes back: 111-003, whom the EDC records as F and the lab records as M.
Now sit with the awkward part: which one is right? The data cannot tell you. Nothing in either file is malformed. Both values are perfectly valid. They simply disagree, and you have no way to resolve it from the data alone.
This is not a trick, and it is not rare. It is a defect deliberately built into the GLPX-1 simulation because it is one of the most ordinary things that happens in a real trial. You will resolve this subject properly in The DM Domain, where you will see that the standard’s contribution is not to know the answer, but to force the question to be asked and the decision to be recorded.
Note the join itself: join_by(SUBJID == subjid). Because the two files name the same concept differently, the join cannot be written without trial-specific knowledge. After standardisation it can.
2.4.2 Counting subjects is not nrow()
One more, and it is the one that catches everybody:
dm <- raw[["dm_raw.csv"]]nrow(dm)
[1] 401
n_distinct(dm$SUBJID)
[1] 400
The demographics file has 401 rows and 400 subjects. If you reported nrow(dm) as the number of enrolled subjects, you would be wrong, and nothing in the file would warn you:
dm |>add_count(SUBJID, name ="rows_for_this_subject") |>filter(rows_for_this_subject >1) |>select(SUBJID, SITEID, RANDDT, ARM)
Subject 101-012 has two rows: one at site 101 randomised on 2024-03-06, one at site 108 randomised on 2024-03-27. Read them as a story rather than as a bug. A subject was screened at one site, moved, and was enrolled again at another site under the same subject number. Both rows are things that genuinely happened and both were genuinely recorded.
The file is not corrupt. It simply has no rule that says one row per subject, so nobody enforced one, and a plausible sequence of events produced two. The standard’s contribution here is the rule itself, stated in advance and checkable by a machine. You will apply it in The DM Domain.
This is the whole argument in miniature. Three ordinary questions (what is the subject column, what sex is subject 111-003, how many subjects are there) and each one needed knowledge that is not in the data.
2.5 Example 2: One Question, Three Answers
Estimated time: ~15 minutes (worked example)
Naming and structure are the visible problems. Vocabulary is the one that will actually hurt you, because it produces answers rather than errors.
Here is a question a regulator genuinely asks, and one of the simplest questions available: how many subjects had nausea?
Start by looking at what sites typed into the adverse event term field:
Case again, plus a misspelling. Diarrea is a typo that no case-folding will rescue.
nausea and vomitting (1)
Two clinical events in one field, and a misspelling. This is not a formatting problem; it is a data problem. One AE record is describing two AEs.
25 rows in total
Twenty-five distinct strings for what sites believed were ordinary, common events.
How many distinct things are actually in this column?
n_distinct(ae$AETERM)
[1] 25
n_distinct(toupper(ae$AETERM))
[1] 12
Twenty-five as recorded; twelve once you ignore case. But GLPX-1 only ever produced ten clinical concepts. Upper-casing gets you from 25 to 12, and then stops:
The two extra entries are DIARREA and NAUSEA AND VOMITTING. No string operation will fix them, because fixing them requires knowing that “Diarrea” means diarrhoea and that “nausea and vomitting” is two events. That knowledge is medical, not textual.
2.5.1 Three defensible rules, three different answers
Now answer the regulator’s question. Watch what happens:
Run the chunk above. You should get 61, 65 and 66.
Before reading on: which of those three numbers would you put in a safety table? Notice that you cannot answer that question from the data either.
NoteReading the Output Line by Line
Rule
Subjects
What it does
AETERM == 'Nausea'
61
Exact match. Silently drops the four subjects whose term was typed in a different case. Wrong, and looks fine.
toupper(AETERM) == 'NAUSEA'
65
Case-insensitive exact match. Catches NAUSEA and nausea. Still misses nausea and vomitting.
str_detect(toupper(AETERM), 'NAUSEA')
66
Substring match. Catches all four spellings, but it counts nausea and vomitting as nausea, which is arguably right, while quietly failing to also count it as vomiting, which is definitely wrong.
Three rules, all defensible, all written by a competent R programmer in good faith. Three different numbers in a safety table.
None of these is a bug. Every one of them runs, returns a plausible integer, and produces no warning. That is precisely what makes this the dangerous failure mode: a naming problem throws an error, a vocabulary problem returns a number.
The standard’s answer has two halves, and this course covers both:
Controlled terminology: the column is not free text. It may hold only values from a published list, so Nausea, NAUSEA and nausea cannot coexist, because two of them are not legal values. You will check values against the published lists programmatically rather than from memory in SDTM Concepts.
Coding: a human (or a human-supervised process) maps each verbatim term a site typed to a dictionary term, and both are kept: the original words and the coded concept. Diarrea and nausea and vomitting are resolved by that step, not by code.
Notice that the raw verbatim text is never destroyed. That is a principle worth carrying through the whole course: standardisation adds an interpretation alongside the original. It does not overwrite it.
2.6 What Can Go Wrong
Estimated time: ~10 minutes (reading)
Treating standardisation as renaming
The most common misreading of this session is “so I should call the column USUBJID instead of subjid.” Renaming is the easiest third of the job and the least important. Example 1 showed structure differing (one row per event vs per subject per visit per test) and Example 2 showed vocabulary differing. rename() addresses neither. If your mental model of SDTM is a lookup table of old name → new name, Part 2 will not make sense.
“We’ll standardise at the end”
Standardising is a decision-recording activity, not a formatting one. Look again at subject 111-003: resolving that conflict requires someone to find out which value is right. That is easy while the site is still open and the data manager still remembers the query. It is expensive a year later and impossible after the study closes. Every deferred standardisation decision gets more costly, and some of them become unanswerable. The dates in dm_raw.csv are the same story: someone knew, at the time, why a subject’s birth date was recorded as 1969.
Assuming the vendor’s file is the trustworthy one
The lab file looks more authoritative, machine-generated, consistent formatting, no free text. It is still the file with the wrong sex for subject 111-003 (or the right one; you do not know yet). Tidiness is not accuracy.
Believing a validator’s silence
Conformance tools check that data conforms to the standard. They cannot check that it is true. Every one of the three nausea counts would pass a conformance check on the resulting dataset, because each produces a structurally valid number. Passing validation means you have not made a standards error; it says nothing about whether you have made a scientific one.
WarningCommon misinterpretations
“Standards are bureaucratic overhead imposed by regulators” The regulation exists because the problem does, not the other way round. Everything demonstrated in this session (the disagreeing sex, the duplicated subject, the 25 spellings) would still be a problem for your own team six months from now if no agency existed.
“My data is clean, so this doesn’t apply to me” Nothing in the GLPX-1 raw data is dirty in the ordinary sense. There are no impossible values, no corrupted rows, no encoding errors. Every problem in this session came from clean data with no agreed meaning.
“The CRO/vendor handles CDISC, so I don’t need to know it” Someone still has to decide what sex subject 111-003 is, and that someone needs to understand both the trial and the standard. Delegating the mapping does not delegate the decisions. It just means you approve decisions you did not make.
“Once the data conforms, the analysis is objective” The three nausea counts differ by rule choice, not by data quality. The standard removes the arbitrary differences (naming, case, structure) so that the remaining differences are visible and arguable. It does not remove judgement; it exposes it.
2.7 Exercises
This session has no exercise set. It makes an argument rather than teaching a technique, and the argument is best tested by the Comprehension Check below.
The first exercises arrive with The DM Domain, by which point you will have something to build. If you want to do something now, re-run Example 2’s three nausea rules against Diarrhoea instead, and see how many different answers you can defend.
2.8 Comprehension Check
Estimated time: ~10 minutes
In your own words, why can’t a thorough data dictionary for each trial replace a standard?
dm_raw.csv has 401 rows and 400 subjects. Why is this not a data error, and what does the standard contribute?
Three defensible R rules gave 61, 65 and 66 subjects with nausea. Which is correct, and what does your answer tell you about where the fix has to happen?
A colleague says: “The lab file is machine-generated, so where the EDC and the lab disagree, trust the lab.” What is wrong with this reasoning?
Name the three distinct things a standard fixes, and give an example of each from the GLPX-1 raw data.
NoteAnswers
A dictionary solves comprehension, not automation. A reviewer can read your dictionary and understand your data, but they still cannot run an existing program against it, because your columns have your names. Documentation scales linearly with the number of trials (read each one); a standard is written against once and applies to all of them. Every validation rule and review tool in the industry depends on that second property.
Subject 101-012 was screened at one site and enrolled again at another after moving, producing two genuine, correctly recorded rows. The raw data has no rule saying one row per subject, so nothing prevented it. The standard contributes the rule itself, stated in advance, applying to every trial, and checkable by a machine rather than by someone noticing.
None of them is correct as a code problem, which is the point. 61 is wrong (it drops case variants). 65 and 66 differ only in how they treat nausea and vomitting, and neither handles it properly, because that record describes two events and no filter can split it. The fix cannot happen in the analysis; it has to happen upstream, by constraining the column to a controlled vocabulary and coding the verbatim terms. If your rule choice changes the answer, the answer is not in the data yet.
It confuses consistency with accuracy. The lab file is machine-generated in its formatting, but the sex value in it originated from a human at some point, exactly like the EDC value. Neither file is more likely to be right a priori, and the disagreement is evidence that at least one human-entered value is wrong. Resolving it requires going back to the source, not picking the tidier file. (Note also that the lab file is where the duplicate record and the missing unit live.)
Variable naming: SUBJID in four files and subjid in the lab file: the same concept, two names, so no join can be written generically. Dataset structure: ae_raw.csv has one row per event (389 rows, fewer than the 400 subjects), while lb_raw.csv has one row per subject per visit per test (12,389 rows) - “one row” means something different in each. Controlled vocabulary: AETERM holds 25 distinct strings for 10 clinical concepts, including Diarrea and nausea and vomitting.