The other half of Define-XML. That session built the document by hand to show its structure; this one builds it the way a sponsor does, from a specification workbook, and examines what the production route costs you. Read them in order. This session assumes you know what an ItemGroupDef is.
TipHow to Use the Code in This Session
Run R from the project root, as always. This session uses three packages you have not seen before: defineR, openxlsx and metacore, all pinned in renv.lock.
Everything here runs against artifacts committed in this repository - data/spec/SDTM_METADATA.xlsx and data/define/. You do not need the CDISC Define-XML package for the main path; one comparison in Example 3 uses it, and is marked.
WarningA Note on defineR
defineR is at version 0.0.6. That leading zero is not decoration: it is a pre-1.0 package, and its interface may change. It is used here because it is the most direct open-source path in R from a specification workbook to a define.xml, and because the shape of what it does is what matters, not because it is settled infrastructure you should build a submission pipeline on without further thought.
14.1 When Do You Use This?
Tip
Define-XML built a document with three variables. A real submission has thousands. Then:
Nobody is going to hand-write xml_add_child() calls for 2,000 variables across 30 domains. What actually produces the file?
Your standards group maintains dataset metadata in a spreadsheet, and has done for years, before anyone mentioned XML. How does that become a submission deliverable?
The tool you use was written against an older version of the standard. What exactly does that cost you, and how would you find out?
How is define.xml actually produced, and what does the production route quietly decide on your behalf?
14.2 Learning Objectives
After completing this session you will be able to:
Describe what a specification workbook contains and why it, not the XML, is the sponsor’s source of truth
Generate a Define-XML document from a workbook with defineR
Identify what a Define-XML 2.0 tool cannot express, by comparing its output against the 2.1 enumerations
Read published metadata back into R with metacore and explain why that round-trip is worth doing
Explain why “our tool produces a valid define.xml” is not the same claim as “our define.xml is right”
14.3 Background: The Workbook Is the Deliverable
Estimated time: ~15 minutes (reading)
Here is the thing that surprises people coming to clinical programming from other data work: in most sponsors, nobody writes define.xml.
The document that people write, argue about, review, version and sign off is a specification workbook: an Excel file, one sheet per kind of metadata, maintained by a standards or data-management group. It lists every dataset, every variable, its type, its length, its label, its origin, its codelist and its derivation. It is a controlled document. It exists before any of the datasets do, because it is what programming is written against.
define.xml is generated from it, usually near the end, often automatically, and frequently by someone who did not write a line of the metadata it contains.
That arrangement has a consequence worth stating plainly, because it governs everything else in this session:
If the define.xml is wrong, the workbook is wrong. Fixing the XML fixes one submission; fixing the workbook fixes the study.
This repository mirrors that arrangement deliberately. data/spec/SDTM_METADATA.xlsx is a committed artifact: a binary file that git cannot show you diffs of, which is a genuine cost and also genuinely how this works. R/build_define.Rreads that workbook and never writes it. There is a second script, R/author_spec_workbook.R, which created the workbook once and stands in for the standards group this course does not have.
14.4 Example 1: Reading a Real Specification
Estimated time: ~20 minutes (worked example)
GLPX-1’s workbook covers two domains, DM and LB. Start with its shape:
Nine sheets. Read them as a list of definitions rather than a spreadsheet and the structure of Define-XML is already visible: TOC_METADATA becomes ItemGroupDef, VARIABLE_METADATA becomes ItemDef, CODELISTS becomes CodeList, COMPUTATION_METHOD becomes MethodDef, WHERE_CLAUSES becomes def:WhereClauseDef. The workbook is the same web of definitions, flattened into sheets.
48 variables across two domains, and every one of them carries an origin. Look at the origin counts closely, because Example 3 comes back to them: Derived, Assigned, CRF, eDT.
Two of those four are about to become a problem.
14.4.4 Where the labels came from
One detail matters more than it looks. The labels in that sheet were not typed from memory. R/author_spec_workbook.R takes 36 of them from the CDISC-published example define.xml shipped in the Define-XML v2.1 package, and the remaining 12 from the SDTMIG v3.4 specification tables (§5.2 for DM, §6.3.5.6 for LB). The script stops with an error if any variable is left unlabelled rather than proceeding with a blank.
That is not fussiness. A wrong variable label is one of the easiest things in a submission to get wrong and one of the hardest for anyone downstream to notice, because it looks like content rather than metadata. It is worth a guard.
14.5 Example 2: Generating define.xml From It
Estimated time: ~15 minutes (worked example)
defineR::write_define() reads the workbook and writes the document. That is the whole production step:
Note the path. write_define() writes its output beside the workbook you hand it, so R/build_define.R copies data/spec/SDTM_METADATA.xlsx into data/define/ first, runs the generation, then deletes the copy. That is why the path above points at a file you will not find on disk: it exists only while the build runs. Hand it the workbook where it lives and your define.xml lands in data/spec/ beside it.
R/build_define.R runs exactly that, and the outputs are committed, so you can inspect them without rebuilding:
The HTML is worth dwelling on. A reviewer does not read raw XML. They open define.xml in a browser, where a stylesheet turns it into a navigable document. That rendering is not a courtesy the tool added; it is produced by an XSL stylesheet CDISC publishes, and the reference to it is a line inside the XML itself.
Two datasets and six codelists match the workbook exactly. 52 ItemDefs from 48 variables does not, and the extra four are not a bug.
They are the value-level metadata. VALUELEVEL_METADATA has four rows, one per lab analyte, because LBSTRESN means something different depending on LBTESTCD: a percentage for HbA1c, mmol/L for glucose, U/L for ALT, µmol/L for creatinine. One variable, four meanings, and Define-XML expresses that by defining four additional items and attaching a where-clause to each.
Four rows in a spreadsheet became four ItemDefs, four ItemRefs and four def:WhereClauseDefs. That is the leverage the workbook route buys you.
defineR emits Define-XML 2.0, whose specification carries the production date 2013-03-05 in its own revision history. The current standard is 2.1. It is tempting to file that as a version lag: an old tool that will catch up. Look at what actually differs before accepting that reading.
Note first that Derived is 23 here against 19 in the workbook: the four value-level items carry origins of their own. The two tables reconcile.
Two things are wrong here by 2.1’s account, and only one of them is the obvious one.
The obvious one: there is no Source attribute.Define-XML established that SDTM requires both Type and Source, and that a central lab result is Collected + Vendor. The workbook has a single ORIGIN column. There is nowhere to put “Vendor”.
The less obvious one is worse. Compare those emitted types against what 2.1 actually permits:
CRF and eDT are not values that 2.1 permits. They are not missing a Source. They do not exist. 2.1 did not extend origin; it reorganised it. The old types were re-expressed as pairs:
Define-XML 2.0
Define-XML 2.1
CRF
Collected + Investigator
eDT
Collected + Vendor
That second row is exactly the pairing the previous session derived for GLPX-1’s central lab results from §4.3.2.1. The two sessions meet here.
So the central intuition of this session is not about version numbers:
The tool is not describing the same thing the standard now asks for. A 2.0 document is not a 2.1 document missing an attribute. It is a document whose vocabulary was retired.
14.6.2 Class
The same pattern, in a second place, and the difference here is structural rather than a matter of spelling. Look for def:Class as an element, the way Define-XML built it:
length(xml_find_all(doc, "//def:Class"))
[1] 0
None. It is not missing, in 2.0 the dataset class is an attribute on ItemGroupDef:
So the class moved from an attribute to an element between versions, and the values changed casing too, from Findings to the FINDINGS that 2.1’s ItemGroupClass enumeration requires. The previous session demonstrated deliberately that a lowercased class name fails schema validation with a [facet 'enumeration'] error. Here the same failure arrives from a tool rather than a typo, and nothing in the workbook would have warned you.
The declared extension namespace ends v2.0. This is why the 2.1 validation returned FALSE immediately: the document is not claiming to be a 2.1 document at all. Everything above (the retired origin types, the relocated class) follows from that one declaration.
14.7 Example 4: Reading the Metadata Back
Estimated time: ~10 minutes (worked example)
A define.xml is machine-readable, which means the trip runs both ways. metacore parses a published define.xml into an R object:
mc <-define_to_metacore("data/define/define.sdtm.xml", verbose ="silent")c(datasets =nrow(mc$ds_spec),variables =nrow(mc$ds_vars),codelists =nrow(mc$codelist),derivations =nrow(mc$derivations))
datasets variables codelists derivations
2 48 6 7
Everything the workbook committed comes back out:
mc$ds_spec
# A tibble: 2 × 3
dataset structure label
<chr> <chr> <chr>
1 DM Special Purpose - One record per subject Demogra…
2 LB Findings - One record per lab test per time point per subject Laborat…
And a codelist can be recovered by name, from the XML, without the workbook present at all:
get_control_term(mc, LBNRIND)
# A tibble: 3 × 2
code decode
<chr> <chr>
1 HIGH HIGH
2 LOW LOW
3 NORMAL NORMAL
14.7.1 Why this matters
Consider who can run that code. Not just the sponsor - anyone holding the submission. A reviewer receiving define.xml can reconstruct the sponsor’s entire declared metadata programmatically and check the datasets against it.
That is what “machine-readable” bought, and it cuts both ways. The metadata is not documentation attached to the submission; it is a set of claims the submission can be tested against, by someone who does not trust you and does not have to ask.
14.8 What Can Go Wrong
Estimated time: ~10 minutes (reading)
The tool validates against the wrong schema.defineR’s own check = TRUE produces check.sdtm.pdf, and the document validates cleanly, against 2.0. A green check from a tool tells you the tool is satisfied. R/build_define.R therefore asserts both schemas explicitly, and is written to complain if the 2.1 result ever changes, so that a future defineR gaining 2.1 support is noticed rather than silently assumed.
The workbook has no place for what you know. This is the failure mode with no error message. You know GLPX-1’s lab results come from a central laboratory. The 2.0 ORIGIN column cannot record it. Nothing warns you, nothing fails, and the information is simply absent from the deliverable. A metadata format’s limits become your metadata’s limits, silently.
Fixing the XML instead of the workbook. Hand-editing generated define.xml to correct something is the single most tempting mistake here, and it does not survive the next regeneration. It also makes the workbook and the deliverable disagree, which is worse than either being wrong on its own.
Links to documents that do not exist.EXTERNAL_LINKS in this workbook names blankcrf.pdf and sdrg.pdf. Neither file is part of this course, and the generated define.xml links to both. A real submission ships them alongside; here they are named and absent, which is exactly the kind of dangling reference a conformance check is supposed to catch and a schema will not.
14.8.1 Common misinterpretations
“The define.xml is the source of truth.” The workbook is. The XML is generated output, and treating it as the master is how a study ends up with two disagreeing versions of its own metadata.
“It validated, so the version is fine.” It validated against the version it was written for. That is close to a tautology.
“2.0 is 2.1 minus some attributes.”CRF and eDT say otherwise. The vocabulary changed, not just the field count.
“Value-level metadata is an advanced feature.” Four rows in a spreadsheet. It is ordinary, and any Findings domain with mixed units needs it.
A reviewer finds LBSTRESN’s label is wrong in your submitted define.xml. Where do you fix it, and why not in the XML?
The workbook has 48 variables but the generated define.xml has 52 ItemDefs. Account for the difference.
Why is Type="eDT" a more serious problem than def:Origin simply lacking a Source attribute?
Your defineR output validates and your check report is clean. What have you demonstrated, and what have you not?
What can a regulatory reviewer do with your define.xml that they could not do with a PDF describing the same datasets?
NoteAnswers
In data/spec/SDTM_METADATA.xlsx, then regenerate. Editing the XML fixes one file and leaves the workbook (the source of truth, and the thing the next regeneration reads) still wrong. It also makes the two disagree, so the next person cannot tell which is intended.
Value-level metadata: four extra ItemDefs, one per lab analyte.LBSTRESN has a different meaning and unit depending on LBTESTCD (% for HbA1c, mmol/L for glucose, U/L for ALT, µmol/L for creatinine), so Define-XML defines an item per value and attaches a where-clause to each.
Because a missing Source is an omission, and eDT is a retirement. Adding Source to a 2.0 document would leave eDT still invalid. It is not in 2.1’s OriginType enumeration at all. The conversion is eDT → Collected + Vendor: one value becomes a pair, and you cannot get there by adding an attribute.
You have demonstrated that the document conforms to Define-XML 2.0. You have not demonstrated conformance to the current standard, nor to the 90 Specification-sourced conformance rules that no schema check touches (see Define-XML), nor that the metadata is correct: a confidently wrong label validates perfectly.
Check it programmatically.define_to_metacore() turns the submission’s declared metadata back into data, so a reviewer can test the datasets against the sponsor’s own claims (codelists, types, lengths, derivations) without asking the sponsor anything. A PDF is readable; only this is checkable.