Chunking of custom algorithms for Machine learning and Optimization

Chunking the custom algorithm .zip file is needed because of the 5 MB upload limit. You must break larger files into smaller chunks before uploading. This topic provides details on handling larger files.

For example, when uploading a compressed file (.zip or .tar) of 15 MB, it must be split into smaller chunks of 5 MB each, as each request can handle 5 MB at a time.

Since 15 MB ÷ 5 MB = 3, the file is divided into 3 chunks. The total file size in bytes is:

15MB × 1024 × 1024 = 15,728,640 bytes

Each request should increment the ChunkNumber from 1 to 3 while keeping all other properties the same. After the final chunk is sent, the custom algorithm's status is updated automatically.

All chunks must be 5 MB except for the last one. The last chunk can be smaller if its size is not a multiple of 5 MB.

Keep these key points in mind:

  • The ChunkNumber must be incremented for each request. For example, 1, 2, 3, and so on.
  • TotalSize represents the full file size. This is 15 MB in bytes.
  • The ChunkSize is fixed at 5 MB (5 × 1024 × 1024 = 5242880 bytes).
  • The TotalChunksNumber is 3. This number, indicate the total parts of the file.

Once all chunks are uploaded, the custom algorithm's status updates automatically.

Chunk 1 (First 5 MB)

curl --location 'https://<BaseURL>/<TenantID>/COLEMANAI/optimization/model/v1/customalgorithms/<CustomAlgorithmName>/code' \ 
 --header 'X-Infor-TenantId: <TenantID> \
 --form 'data=@"example.zip"' \
 --form 'totalSize="15728640"' \
 --form 'chunkSize="5242880"' \
 --form 'chunkNumber="1"' \
 --form 'totalChunksNumber="3"' \
 --form 'fileName="example.zip"' \ 
--form 'encoding="identity"'

Chunk 2 (Second 5 MB)

curl --location 'https://<BaseURL>/<TenantID>/COLEMANAI/optimization/model/v1/customalgorithms/<CustomAlgorithmName>/code' \ 
 --header 'X-Infor-TenantId: <TenantID> \ 
 --form 'data=@"example.zip"' \
 --form 'totalSize="15728640"' \
 --form 'chunkSize="5242880"' \
 --form 'chunkNumber="2"' \
 --form 'totalChunksNumber="3"' \
 --form 'fileName="example.zip"' \
 --form 'encoding="identity"'

Chunk 3 (Final 5 MB, completes the upload)

curl --location 'https://<BaseURL>/<TenantID>/COLEMANAI/optimization/model/v1/customalgorithms/<CustomAlgorithmName>/code' \ 
 --header 'X-Infor-TenantId: <TenantID> \ 
 --form 'data=@"example.zip"' \
 --form 'totalSize="15728640"' \
 --form 'chunkSize="5242880"' \
 --form 'chunkNumber="3"' \
 --form 'totalChunksNumber="3"' \
 --form 'fileName="example.zip"' \
 --form 'encoding="identity"'