Date: March 20, 2001
By: Lofton Henderson, for NIST, in partnership with CGM Open
Description of Change
|1.0||2000-02-20||First CGMO-NIST review version.|
|1.1||2000-03-05||Incorporate CGMO & NIST revisions; add TOC and numbered sections (xslt post-process).|
|1.2||2000-03-20||Public release version.|
This report describes the proposed methodology, as well as the proposed content, for a WebCGM Conformance Test Suite.
Although not all issues regarding ultimate test suite content are completely resolved, the specifications in this report are sufficiently complete that production can begin on the WebCGM test suite content.
1.1 Goals & Purpose
1.2Scope of WebCGM Conformance Specifications
1.3Scope of Project
1.4Characteristics of the Suite
1.6Roadmap to this Document
2. Methodology Review
2.3Process of Building Test Suite Content
2.4 Principles Applicable to Test Suite Content
3. Inventory of Existing Materials
3.2Static Conformance Materials
3.2.1ATA 2.4 Grex Certification Suite
3.2.2MetaCheck Test Files
3.2.3Vendor Test Files
3.3Dynamic & Interactive Test Materials
3.3.2WebCGM Tutorial Demo
3.4.2NIST ATA Harness
3.4.3SVG Harness Design
4. WebCGM Conformance Suite Plan
4.3Modularization & Prioritization
4.4.2Static Rendering Components
4.5.2Dynamic Module Components
4.6Packaging, Organization, and Presentation
4.6.1Standalone versus Browser
4.6.3Test Naming Convention
4.7Operator Script Contents
4.8Generating the Reference Images
4.8.1PNG for Static-graphics Test Cases
4.8.2Reference Images for Dynamic Test Cases
5. WebCGM Conformance Suite Specificatons
5.1.2ATA Test Case Inventory & Disposition
220.127.116.11Definition of Dispositions.
5.2.4Categorical TP Prioritization
5.2.5What to Test
5.2.6How to Test It
6. Individual TC Descriptions
6.2Format for TC Descriptions
8.1Basic Effectivity Test (BE)
8.2Demo Test (DM)
8.3Detailed Test (DT)
8.5Error Test (ER)
8.6Intelligent Graphics (IG)
8.7Semantic Requirement (SR)
8.8Test Assertion (TA)
8.10Test Requirement (TR)
8.11Test Purpose (TP)
8.12Test Case (TC)
WebCGM 1.0 has been a W3C Recommendation for two years. Currently, there is no publicly available test suite which is specific to WebCGM, and which is comprehensive enough to at least touch upon the major functional components of WebCGM.
The principal goal of this project is to produce an initial, usable test suite which covers all of the major functionality of WebCGM. It will be as comprehensive and detailed as resources permit. Comprehensiveness at a basic level of detail - i.e., full coverage of WebCGM functionality - has priority over detailed examination of any particular functional area.
The principal intended purposes of the suite are:
It is not a goal of this project to build a certification suite or establish a certification service.
WebCGM 1.0 ([WEBCGM10]) consists of static and dynamic functionalities. Static functionalities include the precise definition of a set of graphical drawing elements, for drawing static images. These are a subset of those descibed in the Model Profile of the ISO CGM standard ([CGM1999]), with a couple of additions (such as Symbol Libraries).
The dynamic functionalities define a set of Intelligent Graphics (IG) functionalities. They are specified by a set a set of Application Structures (APS) and APS attributes, and include:
WebCGM 1.0 hyperlink navigation functionality is defined in the context of HTML 4.0 documents. Standardized WebCGM behaviors assume an environment of current HTML-based Web browsers. Full functional conformance of a WebCGM viewer can only be measured in conjunction with a Web browser - in other words, only browser plugins (or their functional equivalent) can be fully tested for WebCGM 1.0 conformance.
Caveat. Testing WebCGM viewer conformance in a closely integrate browser environment introduces some inherent risk. When an actual result deviates from an expected result, it may be unclear whether it is the fault of the WebCGM viewer or the browser.
This project's scope is limited to the third - a conformance test suite for CGM interpreters and viewers. Taken together with the previous section, this defines the scope and target of this project: conformance of fully-functional WebCGM browser plugins.
There are several kinds of test suites that one could imagine for WebCGM:
#3 most closely matches our principal goals.
A few real-world demo files are anticipated (#1). We don't plan any "goodness" testing. Presently, there are no plans for a WebCGM certification service, therefore there is no need for #4. (However, this work could provide a foundation, should such plans emerge.)
While the formality and rigor of a certification suite might not be needed, the WebCGM conformance suite should embody "traceability" - what normative statement or statements in the standard justify a given test?
Previous analyses (see [HEND01]) suggest that a reasonably comprehensive conformance suite for a standard such as WebCGM is an effort of at least 1.5 engineering-years magnitude, and can easily be twice this much or more. This would be impossible to build "from scratch", given the resources available to this project. Therefore we will leverage existing test suites' content, techniques, and designs whenever possible. Changes will be kept to a minimum, prioritizing generation of needed new materials instead.
Throughout this document, we will nevertheless indicate design choices for an ideal (unconstrained) world. But we will craft the actual solution to maximize our principal goals within resource constraints.
There is a significant body of experience in methodology for tests suites for graphical standards: CALS (CGM), VRML, ATA (CGM), and SVG. The following sub-sections summarize the parts that interest us for WebCGM.
The principle has been established and accepted for some time now, that test suites should be progressive in their organization and structure. Progressive means that the organization and application of the tests progresses from most general and basic, to more detailed and thorough.
The SVG conformance test work [HEND01] coined the term "Basic Effectivity" (BE) for the general, basic level of testing. It is a breadth first testing of the significant functional components across the breadth of the standard. Experience has shown that the largest benefit to implementations, for the effort expended, derives from the BE tests.
BE tests are followed by detailed, drill-down (DT) tests, which methodically probe all testable assertions of the specification. Full confidence in an implementation, and product certification, require a comprehensive body of DT tests. DT tests are typically much more numerous than BE tests. Both CGM (ATA) and SVG conformance test work have acknowledged the value of some "demo" (DM) tests, which provide more natural combinations of functionality from the applicable standard.
These generic test categories are applicable to the WebCGM suite, and are equally applicable to static rendering and dynamic functionalities:
A full set of DT tests is almost certainly beyond the resource constraints of the project. A full set of BE tests may be achievable.
One additional category of tests was prescribed for the SVG test suite development:
Because WebCGM does not specify error response for viewers and interpreters, this category is inapplicable to a WebCGM conformance suite.
Again, there is a significant body of experience in both the theory and practice of building test suite content.
The following basic process is applied for construction of both graphical and non-graphical test suites - CGM, VRML, XML, DOM, SVG, etc. In overview:
An important benefit of following this process is traceability - a test case and its test purpose can be traced back to the supporting requirements in the standard. Ideally, traceability is actually realized in the test case presentation, with buttons or links that connect the test to its supporting requirements from the spec.
While this is the only way to ensure comprehensive coverage and defensibility of a test suite, it is also time consuming and expensive. As we intend to adapt existing materials from elsewhere for this initial WebCGM test suite, and get the most coverage from available resources, we will take short cuts.
Section 4.2 of reference [NIST-VRML] contains a discussion of this methodology - TRs (which it calls SRs) and TCs (the step of generating TPs is implicit in this reference, not explicitly treated as a formal step).
Some basic principles have been learned during the construction of previous test suites (ATA CGM, CALS CGM, and SVG), applicable to both graphics suites and others:
Research ([HEND01]) shows that a comprehensive suite of a reasonably complex standard is an undertaking of at least 1.5 person-years effort. The resources of this project are a fraction of that. Therefore it is critical that we maximize use of existing materials: leverage what we can, and build what we must.
In terms of static graphics functionality, WebCGM and the ATA GRexchange profile have significant overlap. WebCGM 1.0 started with an early revision of the ATA profile, cut out unneeded features and elements, added a few new functionalities, and adjusted a few.
An ATA test suite was written (and is being used for certification service) for ATA revision 2.4. There are 258 test cases in the suite. These probe most (but not all) of the graphical functionality of the ATA GRex 2.4 specification. NIST added an interactive harness, which is based on an XML database description of the test cases.
A document has been written by Cruikshank, et al, at Boeing [CRUIK99], which compares WebCGM 1.0 and ATA GRexchange 2.5.
In the process of developing the MetaCheck conformance test tool, CGM Technology Software (CTS) wrote several score simple test files to probe the non-graphics functionality of WebCGM - the APS and APS Attribute based intelligent content.
The builders and providers of WebCGM viewers likely have significant collections of test files. In most cases, these likely would not align in a clear way with crucial testing principles, but they could be considered for the "demo" (DM) files of the test suite.
Compared to static graphics test materials, there is relatively little suitable existing material for testing the dynamic functionality of WebCGM.
Dieter Weidenbrueck of ITEDO GmbH put together a simple tutorial/demo file for WebCGM. This uses an HTML frameset to test a number of variations on the WebCGM fragment parameters, and to perform basic tests of HTML-to-CGM, CGM-to-CGM, CGM-to-GIF, and CGM-to-HTML navigation.
There are 16 sub-tests, each of which is launched from the same basic HTML page (frameset). Each test has an associated link to a text file, which contains Clear Text CGM code for the essential (V4 APS and APS Attribute) parts of the test. Integrated into a different test harness, these sub-tests are probably suitable as a subset of the needed Basic Effectivity (BE) tests for the dynamic functionality.
A Metacheck-validated version of this demo is publicly available ([WEBCGM-DEMO]).
We are not aware of any other organized body of tests for the dynamic functionality. However, these areas could be productive:
All of the graphical test suites have certain components in common - test file instances, reference images, operator scripts which instruct how to run the test and define the pass/fail criteria. The test suites for interactive environments have added interactive tools to facilitate presentation of these materials in useful configurations - typically using a Web browser - and enable easy navigation through the test cases in the suite. These tools are "test harnesses".
NIST added a test harness to the existing ATA GRex 2.4 test materials, which comprised 258 .CGM instances and .GIF reference images, and approximately 25 HTML files which group the operator scripts for the 258 test cases. The harness is based on a comprehensive XML database in a single file, which has (TestCase) elements for each of the 258 test cases, as well as elements to define the test purpose and provide the operator script.
There is no facility to invoke a CGM viewer. The user is expected to cause CGM viewing to happen somehow, in parallel with using the harness to view the Reference Image and the operator script. In the preceding screenshot, the "View Reference Picture" button has been pushed, which brings up a separate browser window with the GIF image (shown partially overlaid on the principal harness window).
The SVG test suite harnesses are also based on an XML database, which provides operator scripts and navigation links for each test case. Unlike the NIST harness, there is one XML instance per test case, defining navigation paths through the suite, and containing the operator script for the test case. Again unlike the NIST harness, these XML instances are precompiled by XSLT into the SVG navigation harnesses, a set of linked viewing pages.
In fact, there are currently 4 different harnesses available, to accomodate different SVG viewer configurations. These 4 harness sets are each precompiled from the same XML instances using 4 different XSLT stylesheets:
These options accomodate different viewer types - plugins and standalones - as well as different viewer capabilities. #3 is illustrated below:
Neither of these harnesses have traceability (spec and test requirement traceback and display) features built in. Such features can be seen in the NIST DOM test suite, for example.
The following sub-sections describe how we intend to produce test suite. The next major section, WebCGM Conformance Suite Specificatons, presents those detailed content specifications which have been generated by these plans.
An optimal methodology was outlined earlier, which has been developed through the construction of a number of previous test suites.
While such is necessary to ensure comprehensive coverage and defensibility of a test suite, it is also time consuming and expensive. As we intend to adapt existing materials from elsewhere for this initial WebCGM test suite, and get the most coverage from available resources, we will take short cuts:
The existing ATA CGM test materials don't have associated TR materials, nor do they have any sort of traceability or traceback features. These will not be developed retroactively. Rather, the test cases themselves will be subsetted, will be modified appropriate to the global differences between ATA and WebCGM, will be modified for some specific functional differences, and new test cases will be developed where there are serious omissions.
For the new materials that must be developed - mostly for the APS-related WebCGM behavior and navigation functionalities - the optimal methodology will be followed. In particular, traceability information will be generated (whether or not there will be resources to fully incorporate it in the first WebCGM test suite release).
The WebCGM specification divides naturally into semi-independent modules: static & dynamic.
These can be worked on independently and in parallel.
The ideal prioritization is to make an entire breadth-first, basic effectivity (BE) test suite release first, of both the static and dynamic functionality. As remaining resources permit, or in a follow-on project, a drill-down (DT) release should be built.
Notes about BE/DT prioritization:
As indicated previously, we will leverage the existing ATA GRex 2.4 test suite with minimal changes. The changes to the test materials themselves will include:
The precise specifications for each of these 4 classes of changes will be done as follows. Cruikshank et al [CRUIK99] have generated a study of the relationship of the ATA 2.5 profile to WebCGM 1.0, as well as the Model Profile and a draft of CALS Rev B. The differences between ATA 2.5 (of the study) and 2.4 are relatively few and are documented.
This study will be used to derive a differences document, between GRex 2.4 and WebCGM 1.0. This differences document will be applied to the 258 test descriptions of the GRex 2.4 test suite. A list of specific dispositions of each of the GRex 2.4 tests will be generated. The changes will be implemented to produce the static graphics rendering module for the WebCGM suite.
Notes about production methods:
Note about what to test (ref. #4 above):
Issues with this plan:
Each Test Case in the ATA Grex 2.4 test suite contains three principal components:
The WebCGM static rendering module will contain the same three components. However, ideally these changes should be made:
GIF-input and PNG-output is within the functional scope of available raster tools. A tool which could perform conversions in batch mode would make this task trivial.
The OSs of the ATA suite were written appropriately for a certification suite, which would be applied by possibly non-expert operators, and in which the pass/fail rating is paramount. The operator scripts of the SVG conformance suite are more useful and informational - a few sentences which state the test purpose, describe the visible components and sub-tests of the rendered test case, what the results should be, and allowable deviations of the test case rendering from the reference image. The goals and purposes of the WebCGM suite are more similar to the SVG suite, and the OS style should be changed. Effecting such a change would be time consuming.
Note about OS packaging.
For expedience, the initial WebCGM suite will retain the style (and content) of the NIST-ATA XML database. At least one test harness will be included.
Other supporting material should be generated for each test case:
Issues about the static rendering components:
Note about traceability links. One of the difficult issues is being able to link into the WebCGM spec at fine enough level of detail. For example, paragraph, sentence, bullet item may need to be the link targets. The spec does not have such anchors now. However, it would be easy to produce an XSLT stylesheet that would a-tag (anchor) all block-level elements, list items, etc in WebCGM. This would enable the traceability linking to be done. There is still a resource issue with doing it retroactively in the static rendering tests.
The XSLT Conformance TC of OASIS has devised a way to do this, starting with the XML version of the XSLT standard. See [XSLT-SPECLINK].
The dynamic module comprises tests of the Intelligent Graphics (IG) functionality of WebCGM: structuring, linking and navigation, layering, and (optional) search. The IG functionality is defined by APS and APS attributes specifications in WebCGM. The dynamic material must be built from scratch.
Although we don't intend to be overly formal, a TR analysis of intelligent content portions of the WebCGM spec document will be done. We see no other way to get a handle on the scope and upper bound of what can be tested. For the list of TRs, we will derive a collection of test purposes (TP). Those will be prioritized and gathered into a collection of test case definitions.
The WebCGM spec provides a natural to way to group the IG functionality for TR/TP processing:
Our priority order for dynamic test development will be:
Note. Although we will make a more or less comprehensive set of TRs and TPs for the dynamic (IG) functionality, we won't likely have the resources in this project to build it all. A complete BE level, plus some prioritized drill-down (DT) tests should be our goal.
The dynamic module will involve the same components as the static, plus some. There will be:
The reference image is not necessarily a raster image of the WebCGM file instance, in this case. For example, if the test case involves CGM-to-HTML navigation, the reference image might be the HTML target. An ideal solution for the dynamic tests might be a linked set of two or more reference images, indicating initial view, next, ... last. It is to be determined, whether or not we have the resources to pursue this (unlikely).
The dynamic materials will also involve some HTML pages and framesets for some test cases.
It would be ideal if the dynamic test cases could all originate out of the same sort of links in the test harness, as with the static tests. Whether or not this can be achieved will be determined after more detailed design of some of the test cases themselves. The same applies to other details of the components comprising the dynamic portion of the test suite - will be refined as the dynamic functionalities' tests are developed.
As indicated in earlier discussion of WebCGM conformance scope, WebCGM viewers can only be fully-conforming to the WebCGM 1.0 specification if they are browser plugins or controls. There is no conformance subset which excludes the dynamic capabilities, or considers conformance in a non-interactive environment.
Note. For comparison, SVG defines conforming static and conforming dynamic viewers, and further qualifies dynamic functionality requirements with such conditionals as, "...in environments which are interaction-capable". (It is not clear that such hedging would be useful for WebCGM, which is specifically targeted at interactive network documents.)
Nevertheless, we should not arbitrarily require an interactive browser environment and plugin for testing functionality that is not dynamic, i.e., the static graphics tests. The ATA test suite materials are packaged as a set of test .CGM files, .GIF files, and Operator Script files. The ATA-NIST interactive harness incorporates the OS contents directly into the base .XML file which drives the harness. As we do not propose to modify the OS contents initially (but should do so eventually, when more resources are available), then there is no cost now to keeping the 3 static filesets for the WebCGM subset of the ATA test cases.
So for the initial release, at least:
It also has some limitations:
Several alternative harnesses are available - e.g., SVG, CSS, DOM - and some of these resolve some of the limitations. However, attribute #1 is compelling - the NIST-ATA harness could be used immediately, with no modification, for static-rendering viewing of WebCGM test cases (which are in the ATA subset). Therefore we will use the NIST-ATA approach.
There are some modifications that should have high priority:
The details of these are yet to be determined - this will be done in the next project phase.
There are additional desirable modifications, but with lower priority:
Harness issues to resolve:
The test naming convention of the existing tests in the ATA GRex 2.4 test suite are constrained to "8.3" filename limits - filename of at most 8 characters, followed by 3-character extension - and follow a simple file naming convention. The base filename is AAAAAANN, where "A..A" is a 6-character meaningful mnemonic or acronym (e.g., POLYLN, COLTBL, etc) and NN is an ordinal, 01, 02, 03, ...
We will keep this for the static graphics tests because: it exists, is adequate, and spending effort to improve it is not justified.
We will consider a more meaningful naming convention for new dynamic tests. Experience with the SVG test suite showed that a strong naming convention was useful, both for the management of the test suite repository, as well as with to facilitate auto-generation of harnesses.
(For reference, SVG's name convention is: chapter-focus-type-num. 'Chapter' is the SVG spec chapter name, and 'focus' is a functional focus within the chapter. 'Type-num' is a concatenation of the test type - BE, DT, ER, or DM - and its ordinal in the sequence of such tests - 01, 02, ... Examples: path-lines-BE-01, shapes-rect-DT-04, styling-fontProp-DM-02.)
Test naming issue to be resolved:
One of the principles of test suite construction is "atomicity", i.e., each test should consist of a limited number of sub-tests each of which test a simple assertion in isolation. A corallary of this is that the tests should be put into a uniform template. In the case of CGM, this template should give uniform definition to at least these aspects of all test cases:
There can be exceptions for test cases which specifically manipulate and test functionality affecting the standardized template aspects, e.g., addressability tests, font tests, etc.
In the ATA test case CGMs, there are two styles in use the Legend block. The simplest, found in the older tests, features:
Here is a sample Legend block (new style) from the ATA GRex tests:
Missing from the older test cases but present in the newer is:
Missing from both older and newer:
The latter has proved useful (in SVG test suite construction) in judging accuracy of placement of test picture details.
The serial number is a method for ensuring that the raster reference image, the CGM instance, and the CGM rendering are all current, all agree. To be useful, it must unfailingly increment whenever any change is made to the CGM file, in fact whenever the CGM file is saved. A way to automate this is needed. (Note. Such was the case with a former HSI tool called MetaWiz, which defined test cases in a meta-CGM language - CGM with scripting extensions - and generated CGM instances on demand.)
A serial number is important for test suite maintenance and integrity. Its lack in the older (majority) of the tests should be corrected.
The template is not as regular across the existing test cases as it might be. However, with these exceptions missing serial number, it is adequate and resources should probably not be spent to further regularize it. Note, however, that the Legend contents - specifically the profile id and ed, test suite name, release level, and date - must be changed for all test cases instances. New test cases will be built to the newer style of template.
Template (priority) issues yet to address and resolve:
The overall linking structure is decided by the choice of the NIST-ATA harness. Effectively, you get an index of all tests in the pull-down lists of the (left) HTML forms frame. These result in the presentation of one or more TCs in the right frame - test purpose statement, operator script, and reference image.
By comparison, SVG decided a linking structure in which a BE layer is linked throughout the suite (next/previous buttons), DTs drill down from BEs (child button), BEs pop up from DTs (parent button), etc. There is also a Table of Contents (by chapter), but no index - the latter would be a useful enhancement feature, and is equivalent to the NIST-ATA left frame.
There seems to be little purpose in trying to retrofit such a navigation map onto the NIST-ATA materials, especially as there is less underlying structure in the NIST-ATA collection. (There may be some BE/DT classification and structuring in the new dynamic tests). However, navigation buttons for previous and next might be convenient (rather than having to go back to the left frame pull-down menu and selecting.
Linking issues to be investigated and resolved:
The Operator Script will comprise a few sentences, written (per the NIST-ATA test harness) as an element and sub-elements of the XML TestCase element for the test case. The existing NIST-ATA OS content is a terse set of steps and points of evaluation for a non-expert conformance testing operator. We propose to deviate towards the informative and narrative for new tests. OSs for the WebCGM test suite should contain any or all of:
#2 could conceivably be: "CGM rendering should look like the reference image ". However, some specifics could be pointed out, such as (for an accuracy test): "All lines should pass through the cross-hairs", or "Vertexes should be at the locations of the markers". For dynamic tests, #2 will be a narrative description of what should happen.
About #6, optionality. If a test is exploring an optional or recommended feature, that should be clearly indicated right at the beginning of the operator script. Only one WebCGM instance comes to mind: search priority viewers that support text search on the graphics.
#7 refers to a brief description of functionalities, other than the one under test, which are used in the test file instance.
Note. In addition to the other purposes of the Operator Script, a well-written Operator Script can provide an aid to accessibility - an important consideration for W3C standards.
This section will likely develop and evolve further. For now, some variation of this scheme (used initially in the SVG work) is likely to provide a successful fallback to more direct methods:
Note about #2. If there is no CGM viewer available which produces a correct image of the WebCGM instance, then write another instance which produces the desired picture. For example, if no viewer would render the Rectangle element correctly, then draw it as an equivalent Polygon. The latter is the "patch" file.
A better method would be: a direct method, WebCGM to PNG rasterizer, such as MetaRip. We will look into this further in the next project phase.
We also must address the conversion of existing GIF images to PNG (although a direct CGM rasterizer, with a batch mode, could remove the need for it). ImageReady could also be used to convert existing GIFs to PNG, but it is a manual process: Open GIF, SaveAs PNG. Raster format converters with command line interfaces (usable in batch) are available.
We have noticed that there is little relationship between the existing GIF image dimensions (e.g., 491x417) and the canonical WebCGM address space (1000x1000). This is not inherently bad, however if there were a more direct relationship (1-to-1, or 1-to-2), then raster analytic tools such as the pointer-coordinate tool of ImageReady could be conveniently used to diagnose accuracy questions - the reference image would be more useful.
Issues to address for reference images:
The dynamic tests will usually have multiple reference images - at least two, an initial state and a final state. Ideally, the harnesses would incorporate features to deal with multiple images. It will probably suffice to have a single image (the final state, after a dynamic action such as link navigation or layer selection is triggered), if the Operator Script is sufficiently informative.
In addition, the the initial state be either CGM or HTML, and the final state could either be CGM or HTML. CGM reference images can be captured as for static graphics - either direct CGM-to-PNG production, or screen capture and save. HTML states (reference images) could be done by raster screen capture, or perhaps more efficiently by saving an HTML page itself. There may be issues with reproducability of exact screen issue in the latter.
These latter issues will be addressed and resolved in the next project phase.
Issues to address for dynamic reference images:
This is a maintenance issue. It need not be resolved now, but we will mark it and put a placeholder for future consideration. How is the WebCGM suite to be kept and maintained after initial release? For comparison, SVG uses CVS to keep all of the materials (and all working group members have r/w access to the repository.)
The reference document [CRUIK99] was analyzed and edited so that only the differences between ATA 2.5 and WebCGM 1.0 remained. The differences between ATA 2.5 and ATA 2.4 were applied, resulting in:
(Production note. [CRUIK99] was a MS Word .DOC file. It was edited down to only the differences, and exported as HTML. The large and verbose MS HTML was processed through the W3C 'Tidy' utility. The resulting small and clean HTML table had some undesirable formatting artifacts. An XSLT stylesheet was written to process the HTML table and rewrite it with these artifacts corrected.)
An XSLT stylesheet was written to process the NIST-ATA XML database, and produce a table of Test Case name, Element(s) (an informational tag in the XML database), test purpose (ditto), and a final column for disposition (default value, "Keep"). The information from the differences table was applied, and any dispositions other than "Keep" (remove, modify, or qualified keep) were edited into the table. Result:
Note. The anomolies in the NIST-ATA XML database can be seen in the table. The "Elements" classification is sometimes missing, and is inconsistent in content (detail). The "CGM Purpose" information is not uniformly on target. As stated earlier, these aspects of the XML database should be cleaned up (but at lower priority than needed new tests, traceability improvements, etc).
The preceding table contains these dispositions for the ATA static rendering tests:
"Segregate" in item #3 indicates that the file would be grouped and linked with the "DM" files, if the suite were structured that way.
35 of the tests relate to elements which are disallowed in WebCGM: segments, bundled attributes, some advanced geometry, etc.
|Test Cases to Remove from WebCGM Suite|
(Production note. This table was extracted from ATA-test-case table via a simple XSLT stylesheet.)
The following common global modifications must be applied uniformly the routine, template-related parts of the tests:
Notes about #4 & #5. We do not yet know the extent of these changes. In this section, we are dealing with uniform changes to routine template aspects. It is possible that there are none of these usages in the template parts of the test files. Usages in the substantive body of the test will be dealt with in the "Modifications" section below. Second: a good candidate for the process is to CGMconvert to Clear Text (in batch), and then 'grep' in batch for the usages.
Notes about #1, #2, and #3. We are still considering labor-saving ways to accomplish these global changes. One possibility is via batch conversion to Clear Text, and then a combination of automatic (e.g., 'sed') and manual (text editor) procedures. A better possibility is recovery and restoration of the tool MetaWizard of the former HSI, which enables to convert and maintain the test case instances in a "meta-CGM" script language, that features macros, inclusion, and looping control structures. MetaWizard also has an automated serialization (serial number) feature.
This is a preliminary list of modifications to existing ATA tests for WebCGM inclusion:
To do/resolve in next project phase:
There are a number of candidates for addition to the static-graphics tests of the WebCGM, based on:
Here is a preliminary list of candidates for new tests. Initial recommendations are in brackets.
Note about #1. although it is under discussion to deprecate multi-picture metafiles, a significant amount of WebCGM fragment syntax is devoted to multiple pictures. It should be tested.
Note about #7 and related escapes. This is a major extension functionality of WebCGM, which also exists in ATA. It should be tested, even though it is (not-yet) in known use.
Note about #8. These tests should not problem exotic character sets, but rather probe simple, common sets in the given encodings. Initial tests could be:
Note about #10 and ESCAPE 45. Precise color models and transparency effects are very peripheral to the central purpose and constituency of WebCGM.
Because we are starting from scratch on the dynamic test cases - no existing test materials, per se - we will follow the ideal methodology for test suite construction:
We will present the first two of these in this report, and the other two will be part of the next project phase.
TR extraction has been done for WebCGM 1.0 (1st Release, WebCGM-REC-20000121), resulting in the tables:
Notes about the tables:
Potential improvements to the table and methodology:
Issues to resolve about TR extraction:
The above-referenced TR Tabulation was analyzed to produce a superset of Test Purposes:
We call this a "superset", because some TPs are redundant, and some very general TPs are made redundant by more specific ones. These are indicated by comments in the table (for example see the first TR). Other cases of apparent redundancy (e.g., very general TPs) are not commented - these may resolve into a single BE test, for example, the general TR statement of permissible structural variations of the fragment specification.
There are some issues yet to resolve related to the TP synthesis:
In terms of general functional category, we propose the following priority order:
By level of detail, we will prioritize:
Note that BE tests can (we believe) lump together a number of TPs without being too careful about atomicity. So no further TP factorization or refinement is required. It is an issue, how much this still pertains for DT tests.
There is an issue that should be addressed in the context of anticipated future functional editions (ProfileEd greater than 1.0) of WebCGM: testing of picture-selection keywords in multi-picture tests. This is an issue because there is current consideration of deprecating and eventually prohibiting multi-picture metafiles. We propose that these should be tested at the BE level, but postpone spending any resources on DT tests until the disposition of the WebCGM spec issue is resolved.
Specific TPs and their Test Case generation will be prioritized in the next project phase. This is largely a pragmatic choice - how to get the most useful test suite with available resources.
The design and specification of a test case for each selected TP will be addressed in the next project phase.
We will incorporate the existing XML file for static-graphics component. It's XML grammar should ideally be expanded XML for the new dynamic tests - add spec references, etc. The XSLT Conformance TC has been pursuing a test description grammar - first early reference can be found at [XSLT-XMLINSTANCE], for example (the grammar has grown considerably in most the most recent proposal, [XSLT-TINMAN]).
The actual individual Test Case (TC) descriptions will be generated in the next project phase. Whether or not we will actually generate formal TC descriptions is undecided. The answer will be pragmatic - does it accelerate test case production. If we do so, the following format will be used.
The material in this section has been adapted from the SVG conformance test project (see [HEND01]).
A formal TC description is most interesting when labor is being divided for specification versus production tasks - different people doing each. That will not likely be the case during this initial WebCGM project.
Nevertheless, the task of writing the description in at least "sketch" detail forces one to actually design the test in sufficient detail that it is then easy to write the WebCGM (and HTML) content to implement it.
It is the Conceptual Description of the test (see below) which is most useful.
In previous work, the a format similar to the following has been used:
The Associated Test Requirements (#5) and Document References (#6) are the crux of traceability. It has never been generated for the existing static-graphics tests (derived from ATA suite). It has been generated during the TR-TP phases of the dynamic tests.
Most of these tools are from the former HSI CGM technology base. We intend to solicit the CGM community for contributions of additional tools, especially viewers, graphical editors, and rasterizers.
Issue to address about tools:
A test which lightly exercises one of the basic functionalities of the SVG specification. Collectively, the BE tests of an SVG functional area (chapter) give a complete but lightweight examination of the major functional aspects of the chapter, without dwelling on fine detail or probing exhaustively. BE tests are intended to simply establish that a viewer has implemented the functional capability.
A test which is intended to show the capabilities of the SVG specification, or a functional area thereof, but is otherwise not necessarily tied to any particular set of testable assertions from the SVG specification.
Also called drill-down tests. DT tests probe for exact, complete, and correct conformance to the most detailed specifications and requirements of SVG. Collectively, the set of DT tests is equivalent to the set of testable assertions about the SVG specification.
See Detailed Test.
An Error Test probes the error response of viewers, especially for those cases where the SVG specification describes particular error conditions and prescribes viewer error behavior.
Graphics which contain non-graphical metadata, which associate non-graphical application-specific semantics or intelligence with the graphical elements.
See Test Requirement.
See Test Requirement.
The program, process, or context assiciated with test suite materials, which organize the materials and test results for presentation to the user, and facilitate execution of and navigation through the test suite.
A testable assertion which is extracted from a standard specification. Also called Semantic Requirement (SR) or Test Assertion (TA) in some literature. Example. "Non-positive radius is an error condition."
A reformulation of a Test Requirement (or, one or more TRs) as a testing directive. Example. "Verify that radius is positive" would be a Test Purpose for validating SVG file instances, and "Verify that interpreter treats non-positive radius as an error condition" would be a TP for interpreter or viewer testing.
As used in this project, an executable unit of the material in the test suite which implements one or more Test Purposes (hence verifies one or more Test Requirements). Example. An SVG test file which contains an elliptical arc element with a negative radius. In practice (and abstractly), the relationship of TRs to TCs is many-to-many.
The ability, in a test suite, to trace a Test Case back to the applicable Test Requirement(s) in the standard specification, and ultimately to the appropriate section of the standard.