Topic: Data Pipelines for Ingress and Egress#

Session 1, Room 2

  • Chair: Arron Lacey


  • Challenges building data pipelines?

  • Automation / Reproducibility


  • Challenges

    • Data Quality: creating one solution to ingest different datasets / schemas

    • Different date formats

    • Limits of normality in data

    • Need to harmonise data

      • Should this happen before data arrives in TREs?

      • Does it need to happen inside a TRE (ie, can it typically be done at schema level or does it require sight of aggregate data? record-level data?)

    • Instrument level

  • Automation / Reproducibility

    • Introduce common standards i.e. BIDS, NIFTI, DICOM Need to consult with domain experts.

      • Can TRE operators (information governance authorities?) require data providers to send data only in a set number of formats? (like data depositories do)

    • With increased processes through data pipeline you can introduce more error in the final output