Icam Regression Utility

The Icam Regression Utility (test.exe) is used to regression-test CAM post-processor output. It runs Icam applications (GENER, CERUN or PSE) on a set of test jobs, uses “filters” to compare the produced output files against stored master files, and reports any differences. Filters are applied before comparison so that irrelevant content, such as dates, version numbers and absolute paths, does not cause false differences.

The utility can be used interactively through its main window, or in batch mode from the command line for automated regression runs.

Overview 

A test suite is a folder containing a vericut_test.json file that describes one or more tests. Each test names an Icam job file (CPJ or RPJ) and the output files it produces. For every output, a master file stored in the suite's .master folder serves as the known-good baseline.

A typical session consists of the following steps:

Select the Root Folder and press Discover Tests. The utility scans the folder recursively and lists every discovered test in the test tree.
Select the Application folder identifying the Icam installation to test against (e.g., C:\Program Files\ICAM V27).
Check the desired tests and press Run checked Tests. Each test application is launched, its outputs are captured and compared against the masters.
Review the results in the test tree and details panel, open the HTML report, or examine individual differences side by side.

A test passes when all of its outputs match their masters after filtering, and the application exited with an acceptable exit code.

Test suites can be written by hand, or created with one of the two built-in wizards: Add Test… builds a suite from existing Icam CPJ/RPJ job files, and New Test Suite… builds a complete suite directly from one or more post databases (see Creating tests).

Main Window 

Root Folder 

This input field selects the folder to scan for test suites. You can type a path or use the browse button located to the right of the field. The utility searches the folder and all of its subfolders for vericut_test.json files and presents each discovered test in the test tree. The last three root folder selections are remembered and can be re-accessed from the drop-down list.

Application folder 

This input field selects the Icam installation used to run the tests. The executable for each test (gener.exe, cerun.exe or pse.exe) is not stored in the test files; it is resolved at run time from this folder, according to the TARGET statement found in the test's job file. The folder itself, and its exec and bin subfolders, are searched for the executable.

Because the executable is resolved at run time, the same test tree can be re-run against any installed Icam version simply by changing this folder. The last three selections are remembered and can be re-accessed from the drop-down list.

Note

The Run checked Tests button remains disabled until this field refers to an existing folder. Diff and master operations do not launch a process and are therefore not affected by this field.

Test tree 

The tree lists every discovered test together with its outputs and, after an operation has completed, its latest result. A check box next to each test selects it for the next operation; Check All and Uncheck All set or clear every check box at once. Icons indicate the result of the last run: passed, failed, warning, or not yet run. Selecting a node displays its details in the details panel.

Right-clicking a node opens a context menu whose entries depend on the node type:

Open File…: Opens the generated output file (output nodes) in its associated application. Enabled only when the file exists on disk.
Open Master File…: Opens the stored master file for the selected output. Enabled only when a master exists.
Open Folder…: Opens the folder containing the selected test or output in Windows Explorer.
Preview Filter…: For an output node, shows how its assigned filter transforms the file (see Preview a filter).

Details panel 

The read-only details panel describes the selected test or output. For a test that has been run, the latest result is shown first, followed by the test configuration. File paths are shown relative to the root folder. For an output with differences, up to five difference lines are shown inline; press Show Diff to see the complete comparison.

Max Parallel 

This spin control sets the number of tests executed simultaneously (default 4). While an operation is running, the status bar reports the live progress, for example: Completed 7 of 12 | 6 passed 1 failed. Pause suspends a running operation and turns into Resume; Stop ends it after the tests already in progress finish.

Operations 

Run checked Tests 

Launches the application of every checked test, waits for it to finish, then compares the captured outputs against the stored masters. Tests run in parallel up to the configured worker count. A test fails when an output differs from its master, an expected output file is missing, the application could not be started or timed out, or the application exited with a non-acceptable exit code.

Diff checked Tests 

Compares the outputs already present on disk against the stored masters without launching any application. Use this to re-examine results, or after changing filter assignments.

Add Masters 

Creates master files for outputs that do not have one yet, by copying the actual output into the suite's .master folder. Use this once for every new test, after verifying that the produced output is correct. Existing masters are not modified.

Update Masters 

Overwrites the masters of outputs whose actual output has changed. Use this to accept an intentional change in post-processor output as the new baseline. This operation cannot be undone; verify the new output before updating.

Open Report 

Opens the HTML summary report generated by the last operation. The summary lists every test and links to a detailed per-output report.

Show Diff 

Opens a side-by-side comparison window for the selected output, showing the master file and the actual output with the differing lines highlighted. The Previous Difference and Next Difference buttons step through the differences.

Preview a filter 

The Preview Filter… context-menu item on an output node shows exactly what the output's assigned filter does, so the assignment can be verified before relying on it. The same side-by-side window is reused, but with the unfiltered text under an Original (raw) column and the filtered text under a Filtered column; the lines the filter removed or rewrote are highlighted.

The preview uses the generated output if one exists, otherwise it falls back to the output's master file (the window title is then annotated with (master)). If the test has not been run and no master exists yet, run the test first. When the output has no filter assigned, the filtered text equals the original and there is nothing to preview.

Creating tests 

There are two methods to create test suites without hand-editing JSON. Add Test… wraps existing Icam job files; New Test Suite… builds a suite directly from Icam databases.

Add Test…

The Make Test File wizard creates a vericut_test.json file from one or more Icam CPJ or RPJ job files, so that test suites do not have to be written by hand. Open it with Add Test… on the main window.

Job files. Use the button to the right of All added job files (cpj/rpj) to select one or more CPJ/RPJ job files. All job files in one wizard session must reside in the same folder; the generated vericut_test.json is placed alongside them, and the job files are copied into the output folder so that the suite is self-contained. The wizard reads each job file and determines the processing mode (GENER, CERUN or PSE) and the produced outputs (listing file and tape file).

Filter files. Use the button to the right of All added filter files to load one or more filter definition files. A preconfigured filter definition file can be found in the filters product installation subdirectory (e.g., C:\Program Files\ICAM V27\filters\Icam_default.filter). It contains individual filters based on file type that are available for assignment to the discovered outputs.

Output list. The output list shows every output discovered from the job files, together with its assigned filter. A filter supporting the output's file extension is selected automatically. Click the Selected filter column to change the assignment, or choose (No filter) to compare the output without any normalization. When no loaded filter supports an output's extension, (No filter) is used. With exactly one job selected, Add Output… attaches an additional output file to that job; the file must reside within the job-files folder.

Save test file. Save test file writes the vericut_test.json file. The generated test stores an empty application path; the executable is resolved at run time from the Icam application folder on the main window. After the file is written, the main window automatically re-discovers the new suite.

New Test Suite from Posts 

This two-step wizard builds a complete test suite directly from Icam databases, generating the CPJ/RPJ job files and the vericut_test.json for you. Open it with New Test Suite… on the main window.

Step 1 — choose the inputs. The first page collects everything the suite is built from:

Post databases (.dbf): Use Add database… to add one or more Icam databases. Each database is scanned for its post objects and reverse-post (CE) objects.
Posts: Every post and CE object found in the added databases is listed with a check box; check the ones to include. The Kit cell is editable — click it to pick the interface kit for that object, or leave it on Automatic.
CL files (for GENER / PSE): Use Add CL file… to add cutter-location files. Each checked post is paired with each CL file to produce a GENER job (and a PSE job when the post has a matching reverse post).
Tape files (for CERUN): Use Add tape… to add NC tapes. Each checked CE object is paired with each tape file to produce a CERUN job.
Filter file (optional): A filter definition file applied to every generated output.
Output folder: Where the generated test tree is written.

Press Preview > to expand the selections into a list of planned jobs and continue to step 2.

New Test Suite — Preview & Edit 

Step 2 — review and adjust. The planned jobs appear in a tree, grouped by subdirectory, each labelled with its target and base name and carrying a check box. Selecting a job fills the edit panel on the right, where its Target (GENER, PSE or CERUN), Post, Revpost (CE), Kit, Input file, and Job base name can be changed; the Subdirectory the job will be written to is shown below. Clear Include this job to skip a job without removing it. Add Job clones the selected job (give it a different input file), and Remove Job deletes one.

Press Create to generate the suite: the output tree is created, a CPJ or RPJ file is written for each included job, and a vericut_test.json is written in every subdirectory. Each target binds a different kind of database object — GENER uses a post, CERUN uses a reverse post (CE), and PSE uses both. As with the Job File Wizard, the generated tests store an empty application path and resolve the executable at run time from the Icam application folder.

Filters 

Filters normalize machine-specific or time-varying content before comparison, so that only real differences in output are reported. A filter definition file is a JSON document containing a set of named filters. A preconfigured filter definition file is provided at C:\Program Files\ICAM V27\filters\Icam_default.filter.

Each filter definition file can contain multiple filters. Each filter specifies the file extensions it applies to, along with an ordered list of rules:

replace: Substitutes the matched text within the line. The replacement string may reference capture groups ($1, $2, …).
change: Rewrites the entire line when the pattern matches.
delete: Removes the line entirely when the pattern matches.

Each rule may carry an optional condition, a secondary pattern that must also match the line for the rule to fire. Rules are applied in the order they are listed.

All patterns use the ECMAScript regular-expression flavor. Supported constructs include \d, \w, \s, lookahead (?=...) and non-capturing groups (?:...); lookbehind is not supported. Patterns are validated when the suite is discovered; an invalid pattern marks the test Invalid config before any process is launched.

The supplied reference filter file, Icam_default.filter, covers date and time stamps, version strings, absolute paths, copyright years and Icam command-line echo lines, and is sufficient for most Icam listing and tape outputs.

Results 

Each output receives one of the following results:

Passed: The output matches its master after filtering.
Differences: The output differs from its master. Use Show Diff to inspect the differences, or Update Masters to accept them.
MissingOutput: The application did not produce the expected output file.
MissingMaster: No master exists for this output. Use Add Masters to create the baseline.
InvalidConfig: The test description or an assigned filter is invalid. The error message identifies the cause.
ProcessError: The application could not be launched, timed out, or exited with a non-acceptable exit code.

Application exit codes 

Icam applications return a severity-based exit code, classified as follows:

Code(s)	Icam band	Accepted
0 – 3	Success	Always
4 – 7	Caution	Only when listed in acceptableExitCodes
8 – 15	Failed	Never
16 – 99	Fatal	Never
100 – 207	Abort	Never

A test that is expected to end with a caution code must declare it in its vericut_test.json, for example:

"application": {
   "path": "",
   "type": "icam",
   "acceptableExitCodes": [4]
}

Without this declaration the result is ProcessError, even when the output matches the master. Each application is allowed 3 hours to complete by default; the limit can be changed per test with the timeoutSeconds field.

Command Line 

When batch qualifiers are present, the utility runs without a window, writes its progress to the console and an optional log file, and ends with an Icam severity-band exit code. The general form is:

$\textbf{test } \; \textbf{/root}\mathtt{=}\mathit{folder} \; \Big[\, \textbf{/run} \; \textbf{/appdir}\mathtt{=}\mathit{folder} \,\Big] \; \Big[\, \textbf{/diff} \,\Big] \; \Big[\, \textbf{/}\mathit{qualifiers} \,\Big]$

$\textbf{/root}\mathtt{=}\mathit{folder}$: Root folder to scan for vericut_test.json suites. Required for all batch operations. Given alone, the discovered suites are listed and the utility exits.
$\textbf{/run}$: Launches each test application and captures its outputs. Requires /APPDIR.
$\textbf{/appdir}\mathtt{=}\mathit{folder}$: Icam application folder used to resolve gener.exe, cerun.exe or pse.exe at run time.
$\textbf{/diff}$: Compares outputs against the stored masters. May be combined with /RUN.
$\textbf{/addmasters}$: Creates master files for outputs that have none yet.
$\textbf{/updatemasters}$: Overwrites masters whose actual output has changed.
$\textbf{/jobs}\mathtt{=}\mathit{n}$: Number of tests executed simultaneously (default 4).
$\textbf{/quiet}$: Reports only failures and the final summary.
$\textbf{/report}\big[\mathtt{=}\mathit{folder}\,\big]$: Writes the HTML reports to the given folder (default: root\report).
$\textbf{/ef}\big[\mathtt{=}\mathit{file}\,\big]$: Mirrors all console output to a log file (default: root\diff.log).

$\begin{array}{l} \rule{0pt}{2.5ex} \textbf{/help} \\ \rule{0pt}{2.5ex} \textbf{/?} \end{array}$

Prints the usage text and exits.

Examples:

test /root=D:\tests /run /appdir="C:\Program Files\ICAM\V27" /diff /quiet /ef
test /root=D:\tests /diff
test /root=D:\tests /run /appdir="C:\Program Files\ICAM\V27" /addmasters

Exit codes 

Code	Meaning
0	All tests passed
8	Differences found or expected files missing
16	Process or fatal error
100	Invalid command line
101	License failure