Archive and Delete Transactions

Introduction

An archive is a collection of records from computer files that have been packaged together for backup, for transport to some other location, for saving away from the computer so that additional hard disk storage is made available, or for other purposes.

An archive can include a simple list of files or files organized under a directory or catalog structure.

What is deletion?

Deletion is the permanent removal of computer files from the computer or hard disk.

General background regarding M3 Business Engine and disk space usage

Tables containing transactions tend to increase in number. This is due to the nature of the business and the setup of the M3 Business Engine system. M3 Business Engine uses a database where indexed tables are used. Almost each one of these tables has several indexes connected to it, which also takes up disk space. Sometimes, a single index can use 30-40 percent of the disk space volume of its related table, all depending on how the access path is specified. If a table contains deleted records, its indexes will still contain access information for those deleted records.

To identify which tables use the most disk space, open 'Table Analyzer. Open' (AMS050). (AMS050) lists all tables that have an archiving or mass delete function. To generate the latest database statistics, use option F14='Generate' to update the sub file when necessary. Use standard sorting order 2-'Size, Table' to sort the list on the largest disk space usage or 3-'Archiving progr, Delete program, Table' to sort by archiving function.

Note: Some tables, like OINVOH, have multiple archiving functions. These tables only display one of the archive functions.

Important considerations

Testing
If you are archiving for the first time, we recommend that you test the archiving first in a separate test database (a copy of the production database).
Operational and system requirements
Consult your responsible system operator, since this routine requires knowledge of how database schemas are saved and cleared. If many transactions and records are expected to be archived in a run, the system operator must ensure that the disk usage is not too high before the archiving begins, since it will not be reduced until the archiving schemas are cleared. If the server has enough disk space available, previously performed archives can be restored to the archiving schema before the archiving job is started. Then, new data will be added to the existing tables, and the archive function will work faster since the files do not have to be created.
Permanent changes resulting from archiving
It is important to understand that after you archive, you will not be able to perform certain functions, view certain information, and restore the information using a standard M3 Business Engine function.

These are examples of actions that are not possible:
- Copy invoices for orders that have been archived
- Recreate sales statistics for archived orders
- Recreate balance key information for ledger information that was archived
- View M3 BE Finance Management details from standard inquiry functions. Instead, the archived transactions can now be viewed from 'Information Browser. Open' (CMS100).
New fields in table
New fields may be added to a table that already exists in the archiving schema. If you want to archive more records in this table, you must move the previously archived transactions to another schema and remove the table from the archive schema before you do the new archiving.

Suggested M3 Business Engine archiving and deletion workflow

This section describes the workflow for any of the archiving and deletion functions in M3 Business Engine.

Check functions per table in 'Table. Open' (MNS120).
Define the archiving setup in 'Archiving. Open Toolbox (AMS100).
Create M3 BE Control Reports relevant to the archiving function being processed. They will be used to ensure that the stored table values before and after the archive are consistent.
Run the archive functions and make the selections in the routine's start panel. These functions will submit a batch job. Afterward, check the printed receipt to verify that the selected function has worked as expected.
Check the status of the archiving function in 'Archiving Log. Open' (AMS300) and 'Archiving Log Tables. Open' (AMS310).
Recommended process for housekeeping and storage of the archived transactions is to save the M3 BE database and archiving schema (library) to suitable media. Clear or delete the archiving schema (library) from disk.
Re-run M3 BE Control reports and compare them to the preliminary reports in step 4 above.
If archiving is successful, move the archived records to Data Lake to move the data from M3 BE database storage.

Move archived records to Data Lake

Archived records in the archiving library MVXARCH still use storage of the M3 BE database. In (AMS100), use option 22='Move Archived Records to Data Lake') to move all tables related to the archiving function to Data Lake. You can only move archived records to Data Lake one function at the time. You cannot start an archiving run while a move to Data Lake is in progress. Similarly, a move to Data Lake cannot start when an archiving run is in progress

Archiving toolbox (AMS100)

The function (AMS100) is a program where all archiving functions can be managed.

In this program, you can:

List with all archiving functions, the list is generated with function key F14
View status per function, 00='Function is not running', 20='Function is running'
View next scheduled run date/time, this information is taken from 'Job Schedule Entry. Update' (SHS010)
Use function key F15 to mass update settings on Archiving functions
Use these options to:
- Reset statuses if a job ends abnormally or is paused and should not be restarted (option 7='Reset current archiving job')
- Pause ongoing job or restart paused job for some jobs (option 8='Pause/Restart')
- Run a specific function (option 9='Run archiving function')
- Display a function log (option 11='Display archiving log')
- Display included tables (option 21='Display archiving table')
- Move records from the archiving library to Data Lake (option 22='Move Archived Records to Data Lake')

Archiving policy

The purpose of the archiving policy is to prevent accidental archiving of periods that are recent and those that need to remain in the live environment. The policy is defined as a number of periods back in time, calculated from the current period, during which archiving is not allowed. It is checked against the type of date used by the specific archiving program, such as the invoice date or accounting date. The number of periods is defined in (AMS100).

The policy is to always use Period type 1 in 'System Calendar. Open Period' (CRS910) to calculate back from the archiving date specified in the individual archiving function.

For example: the archiving policy is set to 36 periods (months). If the current date is February 25, 2012, you cannot archive transactions with a date later than January 31, 2009.

Note: To reduce the risk of accidental archiving, using 0 is not allowed in the archiving policy. If you still use 0 period, you must set the archiving policy to -1.

Archiving function log

Every archiving run is logged and given a unique archiving run number. Example of information that is logged:

Start and end date/time
Selection on dates and division
Name of user starting the archiving run.

The archiving function log is displayed in (AMS300).

Archiving record log

For every archiving run, a detail record log can also be created. This information is logged per table:

Number of records in the table at archiving start in production library/schema
Number of records in the table at archiving end in production library/schema
Number of archived records (count is done in every archiving program)
Number of records in the table at archiving start in archiving library/schema
Number of records in the table at archiving end in archiving library/schema

Whether an archiving record log should be created is determined by field 'Archiving log table' in (AMS100). You can use these alternatives:

1 = No
2 = Yes, but only for the master table
3 = Yes, for all archived tables
4 = Yes, but only for the master table (without before and after count)
5 = Yes, for all archived tables (without before and after count)

The archiving record log is displayed in (AMS310).

Archiving error log

Some archiving functions log which records are excluded from the archive run in 'Archiving Error Log. Open' (AMS350). The records' primary keys together with the error message are included in the log. From (AMS350), you can also open the record's browse program using related option 11='Open'. You can open (AMS350) from both (AMS050) and (AMS100) with related options.

Integration to table master

In 'Table. Open' (MNS120), you can see the archiving function per table and open (AMS100) and (AMS310) with related options.

Integration to job scheduler

In (AMS100), you can see if an archiving function is scheduled to be run. The job number, scheduled date, and time are displayed on panel E.

If an attempt is made to run an archiving function that is already scheduled, a warning is displayed. Although, you can override the warning and run the archiving function even if it is scheduled at a later date.

Archiving viewer

The archiving viewer, 'Archive Viewer. Open' (AMS200), is a program that displays data from any M3 BE table and archiving library. The data is only presented in the subfile, no detail panels exist.

Authorization checks are performed on different levels in 'Information Browser. Open' (CMS100):

Program authority: If you are not authorized to run the master program for a specific table, you cannot see the data in (CMS100).
Division authority: You can only see data in the divisions where you are allowed to work.
Facility and warehouse authority: For all tables where facility or warehouse exists in the primary key, normal facility or warehouse authority checks are performed
Object Access Group: All records that have a value in the 'Object Access Group' field are checked against the Users group for access
Accounting authority: The setting for access to 'General Ledger. Display Transactions' (GLS210) is used to check authorization for accounting dimensions 1-7. The access is set in 'Settings - Access Authority Check' (GLS005).

Information browser category

'Information Browser Category. Open' (CMS010) defines which table and archiving library to retrieve the data from. By defining an information browser category and using option 'Activate', the 'Inquiry type', 'Panel version' and 'Field group' are automatically created. One standard information browser category per table is created by using function key F14='Standard' in (AMS100).

Information Browser

By selecting the information browser category defined in (CMS010) in 'Information Browser. Open' (CMS100), records are displayed for the connected table/archiving library.

Archiving viewer

'Archive Viewer. Open' (AMS200) is used to start (CMS100). Use option 1='Select'. By defining a 'Viewer ID', default values for (CMS100) can be set. (AMS200) is started from the menu or from (AMS100), (AMS30) or (AMS310).

One standard 'Viewer ID' per table is created by function key F14='Standard' in (AMS100). Default values that can be set for (CMS100) are Inquiry type, Panel version, Number of filters, and Filter field values.

To be able to use user-defined sorting options in the Archiving viewer (CMS100), they must first be created in the archiving library. That is done from 'Archiving Library. Create Sorting Option' (AMS020).