Split the data into train and test set, create a unique identifier for each document, then save to a Delta table

First create a unique identifier for each document, then save to a Delta table

Store each chunk as an independent JSON file in Unity Catalog Volume. For each JSON file, the key is the document section name and the value is the array of text chunks for that section