# Splitting Dataset in the dataset loading script

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsplitting-dataset-in-the-dataset-loading-script%2F23207%3E%5C
**Category:** 🤗Datasets\
**Created:** [September 16, 2022, 7:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsplitting-dataset-in-the-dataset-loading-script%2F23207 "2022-09-16T07:18:03Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![astariul](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@astariul](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fastariul)%5C
**Post date:** [September 16, 2022, 7:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsplitting-dataset-in-the-dataset-loading-script%2F23207%2F1 "2022-09-16T07:18:03Z")

</div>

I’m defining my own dataset. To do this I follow the tutorial of the docs, and create a dataset loading script. (see [the docs](https://huggingface.co/docs/datasets/v2.4.0/en/dataset_script))

But I’m facing an issue : my data is located in a single file, and I would like to split this data into `train` and `test` subsets.

As far as I understand, it’s not possible.  
In `_split_generators()` method, since I have a single file, I can assign it only to a single `SplitGenerator`…

* * *

As an alternative, I made a single split in my dataset loading script, and tried to call `train_test_split()` deterministically, but even when fixing the random seed, it gives different results everytime…

PS : I know I could just split my single file, unfortunately I don’t have control over that file…

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [September 16, 2022, 3:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsplitting-dataset-in-the-dataset-loading-script%2F23207%2F2 "2022-09-16T15:11:17Z")

</div>

Hi! You can define two `SplitGenerator` objects, one for `train` and one for `test`, and pass that file to each of them, and implement the splitting in `_generate_examples`.

The code skeleton you can use:

```python
def _split_generators(self, dl_manager):
    ...
    return [
        datasets.SplitGenerator(name="train", gen_kwargs={"data_file": data_file, "split":"train"}),
        atasets.SplitGenerator(name="test", gen_kwargs={"data_file": data_file, "split":"test"})
    ]

def _generate_examples(self, data_file, split):
    # split data based on the `split` value

```