XMDBTool Import Process (Transaction and Credit Card Imports)

These are the steps for the XMDBTool import process:

  1. The sftp must be setup to ensure that the sftp related parameters are read from the application parameters or the bkg parameters. If a parameter is specified in the application parameter and the bkg parameter, the bkg parameter overrides the application parameter.
  2. The system attempts to establish a connection with the SFTP server. If the sfptHost parameter is not specified, the system does not connect to the SFTP server. The bkg process continues to execute the default process. If the sftpHost parameter is specified, the system checks for the required parameters. If all the parameters are set, the system attempts to connect to the SFTP server. If the SFTP connection fails, the information is logged and the default tasks are processed.
  3. After the connection is established, the system checks for the files that must be imported, in the SFTP folder specified by the sftpImports parameter. The system transfers the files from the SFTP server to the Expense Management Application server. The files are transferred to the 'sftpimport' folder (created by the system) at <importFileLocation>/<tenantId>/import on the Expense Management's Application machine. The folder is a temporary placeholder for all the files that are transferred from the SFTP server. After the files are transferred to the 'sftpimport' folder, the files are deleted from the SFTP server. This is mandatory to ensure that the same files are not imported in the next run. However, if the files must be stored on the SFTP server, you must set the value of the rm bkg parameter to false. The files transferred to the Expense Management Application server are logged in the bkg log file.
  4. After the files are imported to the Expense Management Application server, the files are stored in the sftpImport folder. The imported files can be transactional files or credit card files. Based on the bkg process command (for example, UPLOAD_DATA_PROC/ UPLOAD_CC_IMPORT), the files are transferred from the sftpImport folder to the nonCC/data folder (if the XMDBToolBKG.Import is running) or are transferred from the sftpImport folder to the cc/data folder (if the XMDBToolBKG.CC is running). The system checks for the files in the xmdbtool_mapper table, and based on the data the files are routed.
    Note: Customers must follow the same naming convention as specified in the mapping table. The files that do not have a mapping are not transferred and are retained in the sftpImport folder. Again the files that are transferred are deleted from the sftpimport folder and the files that are not transferred are retained in the folder.

    For example, if the XMDBToolBkg.Import is running, this first connects to the SFTP server and transfers all the files existing in the SFTP server to the Expense Management Application server. All these files are stored in the sftpImport folder. So, tThe files in the sftpImport folder, have CC and nonCC files. Now since the bkg process running is an Import BKG command, all the non CC files from the sftpImport folder are moved to the <importFileLocation>/<tenantID>/import/nonCC/data folder. The cc files, if any, exist in the sftpImport files are not transferred at this point of time. The non CC files that are transferred are deleted from the sftpimport folder.

    Next time, whenever the XMDBToolBkg.CC is run, the same process if followed to connect to the sftp server and download all the files to the sftpImport folder. This time however, since the CC bkg process is already running, all the cc files from the sftpimport folder are transferred to <importFileLocation>/<tenantID>/import/cc/data folder and are deleted from the placeholder folder sftpImport.

    Important: If for some reason, the BKG is not able to connect to the SFTP server, the information is logged that the connection is not possible, but can scan the sftpimport folder and any existing relative file is transferred and proceeds.

  5. After the files are transferred to their respective folders, the BKG proceeds for further processing. Before the processing starts all the files present in the data folder are archived. This archiving process is performed by both cc and nonCC bkg. The number of days that the archive can be kept, is set by the app parameter xmdbtool.archiveDays for Import and xmdbtoo.archiveDaysCC for the cc Import. Once the bkg is done processing, log files that are required to be transferred to the customer SFTP server exist. So after the bkg is done processing this again connects to the SFTP server and transfers the logs to the folder on the SFTP server specified in the sftpHostLog parameter. If the sftpHostLog parameter is not specified then the logs are not sent to the SFTP Server.
  6. Once the logs are sent successfully, the bkg process exits.
  7. XMDBTool 10.2 now supports chunking of large data files, for non cc files, during import through background process. The XMDBToolBKG.Import background process must be able to chunk the import files (non-cc) to small units so that load is much faster.

    Currently this feature is used for loading large files through the XMDBTool client, but now this feature is also enabled for the XMDBTool background process, XMDBTool.Import, for importing large non-cc files.

    To enable this feature, the background parameter chunk must be set to true. When chunk=true is specified in the bkg parameter (of the XMDBTool.Import), import files are chunked into smaller sized files and are processed.

    The default size of the files that are chunked is 100000 lines.

    To change the default value of the size of files to be chunked, the background process parameter chunkSize must be used.

    For example, if the chunkSize = 50000, then the files are chunked into smaller files, each having 50000 lines.

    An example of the XMDBToolBkg.Import bkg parameters with the chunk features enabled: c=UPLOAD_DATA_PROC, chunk=true,chunkSize=50000.