item_list
in your json request payload. Each item can be any type (object, list, string, etc.) and is treated as a single sample. item_list.batch_size
specifies how many items to include in a single batch./cortex/spec/job.json
in the API containers.file_path_lister
in your submission request payload. You can use file_path_lister.s3_paths
to specify a list of files or prefixes, and file_path_lister.includes
and/or file_path_lister.excludes
to remove unwanted files. The S3 file paths will be aggregated into batches of size file_path_lister.batch_size
. To learn more about fine-grained S3 file filtering see filtering files./cortex/spec/job.json
in the API containers.delimited_files
in your request payload to break up the contents of the file into batches of size delimited_files.batch_size
.delimited_files
, your Batch API will iterate through the delimited_files.s3_paths
to generate the set of S3 files to process. You can use delimited_files.includes
and delimited_files.excludes
to filter out unwanted files. Each S3 file will be parsed as a newline delimited JSON file. Each line in the file should be a JSON object, which will be treated as a single sample. The S3 file will be broken down into batches of size delimited_files.batch_size
and submitted to your workers. To learn more about fine-grained S3 file filtering see filtering files./cortex/spec/job.json
in the API containers.<batch_api_endpoint>?jobID=<jobID>
:<batch_api_endpoint>?jobID=<jobID>
:delimited_files
or file_path_lister
, you can use s3_paths
in conjunction with includes
and excludes
to precisely filter files.s3_paths
. If the S3 path is a prefix, it iterates through each file in that prefix. For each file, if includes
is non-empty, it will discard the S3 path if the S3 file doesn't match any of the glob patterns provided in includes
. After passing the includes
filter (if specified), if the excludes
is non-empty, it will discard the S3 path if the S3 files matches any of the glob patterns provided in excludes
.dryRun=true
query parameter in the job submission request to see the target list.