Job scheduler – dependency graph

One of the primary goal of globtech scheduler is to provide maximum flexibility to organization of jobs. We have chosen simple but powerful kind of organization – dependency graph. A dependency graph is composed of objects like job, lock or connector and dependencies among them. Two jobs can be synchronized using predecessor-successor dependency. If this kind of dependency is not sufficient lock can be used. Lock is typically used as an abstraction for table, file etc. to assure restricted access of jobs to a resource. Access can be fully exclusive or shared with limited number of actual users. Connector is another type of object, which is used by jobs to access target environments like databases or file systems. As for locks, connectors can be restricted for concurrent access as well.

Dependency graph has many advantages over other organizations like tree organization or workflow. Above all they are:

• Maximally clear arrangement of jobs – any two jobs are equal and does not define hierarchy. It is then easy to look up for the real direct or indirect dependencies. On contrary, trees or workflows add unnecessary dependencies because of their structure. Jobs are often grouped on similarity or modularity rules which induces more dependencies than is needed for correct run.
• Easy administration – changing job dependencies means simply changing the dependencies. No tree reorganization, no workflow/subworkflow redesign is necessary not mentioning that reorganization can be very tricky and error prone.
• Maximum autonomy of jobs – all you need to define is the dependencies necessary for correct execution. It helps scheduler to effectively order jobs for execution. Operator can easily and effectively restart or skip sets of selected jobs.

Job scheduling

Job processing based on dependency graph allows maximum parallelism, which can be used for various methods of scheduling. The simplest one that is used in scheduler is scheduling by priorities. But more important method for processing of big graphs of jobs is scheduling based on statistics of previous job executions. Goal of this scheduling method is to prefer jobs from critical paths and thus complete the whole execution in the least possible time, which is crucial requirement for etl processing.
Another criterion of job scheduling, which is orthogonal to the previous ones, is load-balancing. Load-balancing constrains the job selection by keeping maximum acceptable level of target environment load not exceeded. For better estimate of expected load during job execution collecting of statistics of previous executions are helpful. Load-balancing is important in effective job scheduler because even small exceeding of server resources (typically it is memory) can significantly prolong the execution time of the job processing. Shortage of free memory causes swapping of memory pages. Similarly overloaded data cache in database server causes lots of page misses. Both examples increase number of i/o read and write disk operations, which are very time expensive.

Software architecture

Scheduler has the following subparts: repository database, runtime database, runtime server and user console.

Schema

• Repository database contains metadata like graph definition. Definitions can be parameterized, which is useful for example for definition of extract file set i.e. files containing the same kind of data and differ only in their names containing date of export. Repository can be accessed using db api (views and stored procedures). DB api can be used for advanced metadata generation using other types of metadata.
• Runtime database contains actual state of objects and other operational and history data like event log, statistics, etc.
• Runtime server is a service (demon) process that is responsible for job selection (load-balancing), execution, monitoring and collection of statistic. One service is able to execute multiple jobs in parallel (maximum number is dynamically changeable). It is also possible to spawn multiple services in cooperating mode or backup mode.
• Web console is intranet user interface for metadata administration, controlling a monitoring of jobs, resources and other objects. console is described in detail below.

List of functionality

Repository and runtime database

• Scheduler is strongly oriented on database applications, that is why we support db api. Database always contains actual data.
• Database variables are similar to system environment variables and can be used for metadata parameterization e.g. jobs can be created based on business data or environment or active statistics, etc.

Connector

• Uniform access to target environment – abstraction from concrete connection, e.g. connector with the same name can be mapped to different database in test environment and in production. Metadata remains the same. It is possible to map more than one connector to one target environment.
• Use of credentials in uniform way (see server).
• Ability to manage load balancing using max number of parallel access.
• Ability to restrict access to connector for exclusive access.
• Temporary blocking connector by operator or programmatically using db api.

Lock

• Used for additional synchronization among jobs. Lock can be used as an abstraction of file, table, database, processor or any kind of resource whose parallel access needs to be synchronized.
• Lock can be used as a trigger or a latch (lock that is locked by default) and can be released on demand by operator or programmatically using db api. This kind of lock can be used to launch processing of subgraph of jobs dependent on external system. The subgraph processing can then be triggered programmatically using db api or by an operator.

Job

• System jobs – ability to launch any executable file in the operating system where the runtime server runs.
• Enabled/disabled – ability to block/unblock job execution by operator or programmatically using db api.
• Abort – ability to abort job.
• Skip – ability to skip job, which means scheduling the job for skipping. The real skipping is done when the job would be ready to run and would have resolved all dependencies thus all successor jobs are waiting since skipping job predecessors are resolved and the skipping job is skipped and resolved.
• Timeout – ability to set timeout for job execution. the timeout event can be handled by an action (e.g. abort) or by sending an alert.
• Deadline – ability to set an expected time of completion on a job. This attribute is used for preferential job scheduling based on critical paths in graph.
• Priority – ability to set a priority on a job. Typically priority is less significant for job scheduling than effective priority (computed from critical paths) and so is not important except the immediate priority that causes job to be executed immediately (still dependencies have to be satisfied).
• Statistics – ability for statistics collecting. Multiple statistics can be defined depending on time period like daily statistics, month ultimo statistics etc.
• Restart count, restart delay – ability to set automatic restart of job in case of a failure with max count of repeat and a delay between two executions.
• Event handling – ability to handle job execution events with an action. events are: on start, on finish, on end, on success, on error, on abort, on expire, on skip.
• Recovery plan – complex way of treatment of job finish or timeout.
• Job history – every job has recorded history of its execution. History contains basic job info and all output attributes like number of inserted/updated/deleted rows. Set of output attributes can be extended and is configurable in job type metadata.

Runtime server

• Can run as native service or as part of application server.
• Fail-over – timestamp of last activity, handled lost connection to database, in case of lost connection to database critical operations are repeated with some period until they succeeds, backup runtime servers.
• Ability to set server to idle state that causes suspension of new job processing.
• Ability to set maximum number of currently running jobs on the server or time period for looking up for new jobs.
• System logs – configurable format and file name of log files, configurable maximum size and splitting to multiple log files. Level of logging is configurable for different parts of server or modules independently. Log records are easily identifiable with server, worker, and job. Ability to log into windows system log or other 3rd party logging software supporting log4j library.
• Event log – important events that occurred in scheduler are logged into runtime. Ability to log user defined events from service job, event handler or from database using db api.
• Alerts – events can be matched with rules for alerts and in case of match an alert occurs and an email is sent.
• Credentials – saved in runtime database (managed by operator) or in server configuration (managed by system administrator). Integration with java keystores including system keystore in windows.
• Mailer – simple service for email delivery that is used for email notification. Mailer service can be used in service job or event handler.
• Statistics collecting – automatic collecting of statistics of job executions. Collecting can be disabled that is useful in cases when statistics are generated by external procedures.
• Environment variables – server propagates environment variables of operating system to jobs, moreover, server propagates job properties of java runtime environment and variables of runtime database and runtime server (saved in runtime database and managed by operator).
• Extendibility – server can be easily integrated with any java library and functions can be exported to scheduler expression language. Libraries that require lifecycle management can be integrated as internal services.

Target environments

• System jobs – ability to launch any executable file in the operating system and record standard and error outputs and archive them.
• RDBMS database – access through jdbc api to any jdbc enabled database. Ability to execute any kind of command supported by the jdbc driver. Support of input and output parameters (output parameters can be archived in job history). Database command can be composed dynamically using expression language.
• Informatica integration services – support of native api that allows to launch and control workflows in integration server. After workflow completion logs and statistics can be downloaded, saved and accessed from the console.
• File transfer – scheduler supports file transfer from/to local file system and servers supporting http(s), webdav, ftp(s), sftp, scp, samba protocols. For authentication user/password or client/server certificate can be used (concrete support of authentication method depends on whether connector library supports it or not).
• Servis job – allows to execute a command in the runtime server. Explicitly exported functions can be executed only, which includes functions of internal services or imported functions from integrated libraries.
• Basic types of jobs (listed above) can be used to derive user defined job types. User defined job types serves as templates for reusing identical parts of job definitions, e.g. ssis job can be derived from system job and defined as a template for dtexec command line utility that is used to execute ssis package.

User defined job types

One of the powerful features of scheduler is extendible type system. Existing types of jobs (or other objects, e.g. lock) can be reused to create derived user defined types, which can predefine common attributes or override general attributes like command with an expression composing the value from more user friendly attributes. For example a user want to use a command line utility which has non-transparent options that are hardly to remember. Furthermore, many options remain unchanged for most of the jobs. In scheduler user can create new job type for this command line utility derived from system job. This new type will introduce new attributes one attribute for for each command line option that is expected to be used. Then the type will override attribute command with predefined path to the executable file of the utility and will also override attribute arguments with an expression that will compose command line arguments from the new options attributes. Jobs derived from this new type define only the necessary attributes and leave all the command line utility specialties on the derived job type.

Calendar, execution plan

Calendar defines list of holidays and time zone. Execution plan defines sequence of timestamps that are based on calendar and execution plan parameters. Execution plan can be defined as „execute daily at 2:00“, then the execution plan defines sequence of timestamps (time zone is not stated) …, 1.1.2010 2:00, 2.1.2010 2:00, 3.1.2010 2:00, … execution plan has parameters to constrain timestamp sequence for concrete or periodic amount of time unit (e.g. on second day of every third month at 1 hour am every second minute) time unit can be one of quarter, month, day/workday in month, day in week, last day/workday in month, hour, minute. Execution plan can be constrained by a start and/or an end date (e.g. plan is valid only until the end of the year) or by a daily window (e.g. execute only from 8:00 to 18:00). Job can be scheduled using more than one execution plan.

Web console

Scheduler user interface allows access in three different roles: superuser, administrator, operator and common user. Administrator is responsible for metadata administration, i.e. objects definitions. Runtime data is not allowed to be edited for administrator. On contrary, operator is responsible for runtime state of jobs, resources and runtime servers. That is why operator is allowed to modify (using predefined operation actions) runtime data and thus react on incidents. For example he is allowed to solve unpredicted job failures by restart or resolve, he can block temporarily set of jobs, target environments or stop runtime servers. But operator is not able to modify metadata, which are accessible to operator for reading only. Role user is for the other users of scheduler that are allowed only to see data but are not allowed to modify them however. Role superuser merges administrator and operator roles. These roles are also applied on database level i.e. for db api access.

Effective browsing, navigation, editing, operations

Console is designed and optimized for browsing and editing of dependency graphs containing big number of objects and dependencies. That is why we generally use tables for browsing data. In object details it is possible to find all direct dependencies and quickly navigate to them using hyperlinks. For another way to browse hierarchy we use tree. Objects in tables can be filtered and sorted.

Browsing and editing objects in console

For bulk operations on jobs (e.g. restart, spool) there are specially design dialogs that helps to effectively select set of jobs with some properties. Upper table is used for browsing jobs in hierarchy towards predecessors of selected root job or towards its successors. Alternatively jobs can be listed without hierarchy. Jobs in the table can be filtered, sorted and then selected. Actual selection of jobs is shown in the lower table. When all required jobs are selected user can set up additional parameters of operation and confirm the operation. All modifications are processed in one database transaction.

Bulk operations on jobs

Monitoring

Overview of the current state of execution is shown and periodically updated in the left side panel. This gets a user general information about current state of processing. More detailed information about processing is available in monitoring view. This view contains several tables with important operational data about jobs (running, waiting, finished and waiting for resolve), event log and current state of the runtime server(s).

File system

Jobs and runtime server(s) generate lots of files e.g. system log file, standard output of system jobs or informatica log files. Console provides remote access to this files. User is able to browse them, sort, filter them or download them to local computer.

Remote file system explorer

User interface customization

UI interface can be customized on three levels:

• Environment styling – helps to differentiate different environments (e.g. development, test and production), this kind of styling changes color and title of the console header.
• Branding styling – is for changing style to assimilate ui with company gui standards.
• Detail customization – using css styling and templates with forms and tables layout.

The post Job schedulers – GTL Scheduler appeared first on GTL Tools.

Job scheduler – grouping tasks into groups – jobs, steps

• Job – group of logically related tasks.
• Step – task – one job step.
• Job contains several steps, sequence of steps in job is fixed.
• Dependencies between jobs – order of processing not only in the job, but also among them.

Job configuration screenshot

Parallelism and prioritization

• Launch more tasks at the same time
• Possibility to define, which tasks can and which cannot run in parallel.
• Possibility to influence the order of processing tasks with priority.

Advanced planning of the jobs and steps

• Comparable opportunities with schedules v sql agent server, launch of tasks daily,weekly, monthly, just in working days, etc.
• Possibility to combine more of execution plans.

Jobs planning screenshot

Fault resistance

• Separation launching tasks from launcher administration – fall of task does not mean fall of launcher.
• Each of the task runs as independent process (sql server agent job).

E-mail notification

• Sending messages generated while processing – successful end of job, task errors, etc.

Intelligent handling of errors

• Mechanism of recovery – automatic reaction to the outcome of the jobs run (email sending, task restart, launch of corrective task).
• Critical and not critical errors: critical tasks – when an error occurs in the task to suspend further processing non critical tasks – the processing will not be stopper by the failure of the task.
• Errors logging in launcher running also in task running, browsing the error log directly from the application.

Editing environment

• Launcher complete administration from user interface, minimum direct intervention into database(special maintenance).
• Possibility to create jobs, steps, tasks, executive plans.
• Monitoring of processing, possibility to actively intervene in the run of processing (suspend, restart, task canceling).

Jobs processing screenshot

Customization

• Color tuning application.
• Types of launching tasks.

The post Job schedulers – GTL Launcher appeared first on GTL Tools.

• Source and target objects for transformed data (Informatica sources and targets).
• Mapping with transformations.
• Sessions and workflows for generated mappings.
• Auxiliary files for mapping – parametrical files and definition plans for external components.
• SQL code (BTEQ script, saved procedures) based on model input.

Generation results in Informatica are PowerCenter objects saved in XML which may be imported directly to the Informatica PowerCenter repository.

Tool architecture

Informatica code generator consists of three basic components:

• Application GTL Generator (Java) installed on the workstation.
• Mapping Architect for Visio (part of the installation IPC).
• Metadata repository in Oracle or PostgreSQL database.

Metadata for SW GTL Generator tool can be stored on any existing database server, there is no need to have dedicated database server.
GTL Application Generator runs locally on the PC, where must be installed client tools IPC and MS Visio. The application requires access to the repository via IPC client tools (command line tool pmrep.exe).

Main functions

• IN data- data management in the input layer. Allows reading, viewing and deleting data structures within the data model version.
• DS data – manage data structures in inner layer of the application. Performs within the version of the data model again.
• WRK data – manage working data structures. It allows preparing a full version of the data model or the difference between two data model versions for next objects generation.
• Source and Target – generate IPC sources and targets for the specific model version.
• Transformations – templates of IPC objects management, import parameter values and object generation.
• Text Templates – text templates management and texts/scripts generation.

IN data

Loading data to the interface layer contains two steps:

• Load of data structures of the specific model version.
• Load of history rules of the specific model version.

Data structure means the table definitions – list of columns, constraints, keys, indexes and so on. All imported definitions support generation of sources and destinations. Definitions are generally related to the data model version, it is possible to perform version comparison and update sources and destinations in case of model changes.
Data load is done from a text file in the format of csv with semicolon as a separator.Character set is optional.
Data entities containing history rules are used during load into core of the data warehouse – target. Rules import creates a metadata layer, which is used in mapping templates filling target.
Data load is done from a text file in the format of csv with semicolon as a separator. Character set is optional.

Screenshot with loaded tables in chosen model version.

DS data

Input layer contains data for each model version. For application full functionality internal data processing has to be made in DS data function.

Working (wrk) data

WRK – working data are processed internally by generator and supports:

• Database objects created in the user’s schema – schema with object definition rigths for user. Database objects are then the source for the template parameter values.
• Sources and targets generating.

Working data are in fact data structures and other metadata that depends on the selected operating mode. There are two types of working mode:

• FULL- working data contain all metadata for a specific model version.
• DIFF- working data contain only difference between two model versions. The advantage is that the generated sources, targets and parameter values possibly are performed only for new or modified objects.

Screenshot in operating mode FULL with a list of new or changed tables.

Sources and targets

Sources and targets are directly generated IPC objects. Generator on the selected operating mode base:

• Generate them using to xml file on the path selected.
• Import them into the repository IPC.

Transformations

Transformation is a logically grouped set of templates and includes templates for workflow, mapping, session or parameter file. Template is created from object exported from IPC (mapping, session, workflow). Template type matches type of the exported object. Template definition it is possible to define as following sequence of steps:

• Export the object from IPC (xml files).
• Import object to the generator, in case of mapping to Mapping Architect for Visio.
• Setting the variable attributes – object parameters (in case of mapping in Mapping Architect for Visio).
• Completion of the template definition.

Screenshot with selection and definition of parameterized attributes in template.

Parameter values – import and export

A prerequisite for object generating is Import of values for the parameters in the template. There are two ways to import metadata:

• From text file in csv format.
• From database view in the user’s schema.

Example of the text file with parameter values.

Export is an operation that can be used in case of once imported file correction or can help to create a file for a different template, but with similar parameters.

IPC objects generation and import

Generating objects is the final operation and uses two options:

• Generate them using to xml file.
• Import them into the repository IPC.

Text templates

Text templates use Apache Velocity technology integrated in the application. Templates can be used for example in SQL queries in ETL transformations. There are usually the same SQL queries /scripts with differences only in the names of tables, columns list, joining conditions etc. Such SQL queries can be parameterized and then generated.

Text templates parameters in text/script.

The post Informatica code generator – GTL Generator appeared first on GTL Tools.

Test tool – key features

• Platform independence – gui master is a platform-independent solution. it supports web applications, reporting systems, b2b, erp, sap, infor, navision, oracle, siebel, etc., as well as terminal applications (e.g. as400).
• Simplicity – one solution for testing all applications. no extra software costs or “applications- specific” plugins are required.
• Flexibility – the flexibility of gui master far exceeds that of other “boxed products”. Available functionality can be extended at once in accordance with project needs. Test scenarios are portable among various pc’s as well as between windows operating systems.
• Fast test scenario configuration – the test scenario design in gui master is fast and efficient. Users do not write a “test script”; rather they build a test scenario using the mouse. Any part of the test scenario can be re-used in another test scenario with different parameters.
• Programming skills not required – GUI Master does not require programming/scripting skills. The test scenarios are built/maintained in a graphical environment with good “bug tracking” support.
• Data-driven testing support – in GUI Master, the data is separated from the test scenario “flow” and kept in a (xls/txt) file or sql database. Various test scenarios are created by adding an extra set of data into the file/database.
• Automated evaluation of test results – the scenarios are automatically evaluated according to the given guidelines. The results achieved can be compared to the data taken from an error message or displayed report and compared to a value stored in a file or database table.
• Detailed reporting – the execution of the test scenario (including screenshots) and its results are reported. A number of formats are supported, including the possibility to generate a video sequence from the test.
• Easy integration with test management systems – GUI Master fully integrates with major alm applications, such as spiratest, hp qc, or ibm rational, to name just a few.
• Test scenarios resistant to changes in the tested application – gui master supports automated testing even in a dynamic environment, where tested applications change frequently.

How does test tool GUI Master work

GUI Master uses image recognition to run the test scenario trough tested application. Typical test scenario defines, which transaction should be started, the parameters to used and where these parameters should be entered. GUI Master seeks those objects (button, radio button, combo box, drop down list, etc.) on tested screen and subsequently process required action (i.e. enter value, select an item from the drop down menu, etc). Exact position of these objects is irrelevant. An object can be found anywhere on a tested screen, including areas not currently shown. Additionally, the bitmap sample can be dynamically adapted to changed conditions (e.g. different colours, screen resolution, or new font) of the application, which makes the test scenario quite stable and minimize maintenance cost despite changes implemented into tested application. The “visual recognition of objects” gives gui master the ability to automate the testing of virtually any application, including those accessible only via citrix or other remote desktop. In fact the gui master approach is very similar to what a human tester would do when executing the same test scenario: the tester starts the test, looks for the graphical object specified in the documentation, takes the action required (pushes a button, enters data, etc.) and (sometimes) writes a report. GUI Master is also able to react to the status of the tested screen and adapt to the next course of action according to the situation. Because of the similarity of the approach to the “human way of testing”, the test scenario design is easy to learn and quite intuitive. The difference (with a human tester) is that gui master is cheaper, runs faster, evaluates each test scenario in line with given rules, and keeps detailed records of actions taken for further auditing reference. Apart from the “screen-based actions”, GUI Master also actively uses the command line to execute other operations (such as ftp, telnet, various utilities and system services) to further enhance its ability.

GUI Master editor

The GUI Master editor is a component for building and editing test scenarios. In GUI Master, the test scenario is not built in the form of script (vb,java). Instead, tool keeps the test scenario in a visual “tree” structure, where the test designer inserts (by mouse) specialized procedures seeking visual objects and determines what the test scenario should do. Each procedure was designed to execute a specific event, such as “find a bitmap”, “enter a string in a combo box”, “push a button” or “collect data for validation”, etc. the gui master library contains approximately 40 procedures and covers all typical functionality needed to build automated test scenarios. In order to offer maximum flexibility, there is a built-in sdk, which allows advanced users to build new procedures in order to accommodate any functionality not covered by standard procedures. Most standard procedures are supported by the wizard to simplify and speed up the setup of procedure-specific parameters. Such procedure is then virtually set with a few “clicks of the mouse”. This innovative approach makes a significant impact on productivity, as work in the editor is faster and more efficient, compared to the conventional method of typing a script into GUI Master also supports modularity. A set of procedures (e.g. the sequence for login into the system) can be saved in a block (macro) and used furthermore as a procedure with different parameters.

GUI Master launcher

The launcher is a component that provides the following operations:

• Structures and organizes test scenarios
• Prepares test data.
• Launches test scenarios.
• Collects data.
• Evaluates results.
• Creates test reports.

Test data preparation

In GUI Master, the data can be separated from the test scenario and kept in a text file or an xls file. The data can also be stored in a database table. GUI Master accesses the data source via an odbc connector/driver while processing the test scenario. An alternative test scenario is then created simply by adding an additional set of data to the data source. GUI Master also supports the automatic generation of test data. for example, in order to validate an “entry form” it is necessary to enter “allowed” in each combo box, as well as “disallowed” values. GUI Master has a component that makes it possible, with simple statements, to describe these values, and GUI Master then subsequently generates the required values (please see qacegen for more details).

Launching test scenarios

In GUI Master, test scenarios can either be launched manually from a launcher or from a batch file while processing more than one test scenario at the same time. A batch file can run straight from a command line and it is also possible to launch the file from external test management applications (including spiratest, hp qc and ibm rational). While executing the batch file, gui master carefully manages the test cycle. Should a scenario fail or the tested application crash, gui master always attempts to return the next test scenario to a defined “starting point”. That way, a failure of one scenario does not affect the execution of the remaining scenario.

Data collection

Each test scenario is automatically evaluated according to the given guidelines. Should data from the test scenario be saved on a database table, gui master would generate and run an sql statement to validate the results achieved. In some instances, it is necessary to validate the results against the data displayed only on the screen (reports, error messages, etc.). In that case, GUI Master downloads the required data directly from the screen and – if necessary – applies built-in ocr functionality to convert the image. The collected data is stored in internal variables and subsequently used during result verification.

Results verification/validation

GUI Master also provides very strong support for results verification. There is a number of ways to prepare verification statements, which can be written in any programming language or in the inbuilt “results verification editor” using simple comparisons. Probably the simplest way is to store the expected results together with the test data in a text or xls file. GUI Master then only compares achieved results with the data recorded in the file.

Reporting

Presenting achieved results is an integral part of the testing process. gui master offers fully automated reporting functionality. Each scenario is carefully documented during its execution and subsequent verification, including screen snapshots. once the scenario is completed, gui master compiles a report. The level of details and graphical template can be edited according to specific reporting requirements. Alternatively, GUI Master can also store the results in a database.

GUI Master adaptability

The “bitmap samples” sought by gui master on the tested screen should not be understood as static objects. Colors may be changed, other fonts used, or a new graphical object added. GUI Master can easily adapt to these changes without modifications in test scenarios. For example, the typical “bitmap sample” gui master frequently uses is a description of a combo box. This description can be stored as a text value. prior to the execution of the test, GUI Master detects what font and color are used within the tested operation, and based on this information it dynamically creates the “bitmap sample”, which is subsequently used during the test.

Integration with other applications

We focus entirely on test automation. We realize there are many existing solutions for application lifecycle management (alm). In 2012 gartner compared approximately 20 alm applications and we have no ambition to compete there. Therefore, instead of merely building another alm, we gave gui master the ability to integrate its superior functionality into existing alm applications or virtually any reporting system in that area. The test scenarios built in gui master can be launched from an alm application. After completion, gui master returns the required information back to the alm. It is also possible to integrate gui master into existing test automation scripts. GUI Master also supports automatic test data generation. For example, each “entry form” has defined specific data types to be entered into each box, eventually there are filters and constrains applied. GUI Master can automatically generate data to validate entry of “allowed” data types into a box, eventually generate data for “forbidden” data types.

The post Test tools – GTL GUI Master appeared first on GTL Tools.

• Up to 50% faster test data preparation than conventional methods.
• Covers 100% of tested business logic.
• Increased transparency of the testing process.
• Complex tests (logical and performance tests).
• Improved security of production data.

Test tool concept

Functionality of QAceGen is driven by the “focus to the point” concept:

• Easy access to all necessary information.
• Elimination of non-value added tasks.
• Focus on the problem solving.

“Focus to the point” means that work with test tool QAceGen is very efficient. All necessary information about data types, primary and foreign keys, constraints and relations impacting the test project is always easily available within the tool. Tester does not have to look for relevant information in documentation, databases or in other information sources. QAceGen also eliminates vast majority of the “non-value added” activities: keys, constraints, filters and records in all referential tables are automatically generated by qacegen. All data fields, although not directly related to the tested logic, but required due to the data integrity, are generated automatically. This significantly shortens the time to prepare test data. This innovative concept introduced by qacegen offers substantial user benefits. QAceGen saves up to 50% of the time needed for test data preparation when compared with conventional methods. When preparing test data manually, a tester has to often deal with very complex data relations, which is very time-consuming and often leads to grossly incorrect results. QAceGen however allows a tester to focus entirely on the tested business logic. Any “non-value added” activities (i.e. not directly related to tested business logic) are processed by the system.
Note: “business logic” specification defines rules which a software application follows. this is used for both development and application testing. QAaceGen has been designed as a “business logic driven data generator”. Tt means that the system generates test data based on the rules defined in the business specification. In QAceGen, the business logic is divided into logically indivisible elementary units called test scenarios. Each test scenario is then expressed in the dgl language (data generation language) of QAceGen. These definitions are then used by QAceGen to generate set of records for each scenario. Resultant set of test data covers every logical branch of a given business logic. Another useful feature of qacegen is self-documentation. Each test project has a defined logical structure, driven by the split of a tested logic into individual test scenarios. Project documentation is integrated directly into each test scenarios. These comments can be easily exported into a separate document (while preserving its original logical structure) and subsequently distributed to all involved parties. This feature prevents project documentation to lag behind the actual status of a project. The “self-documentation” feature minimizes administrative effort and costs and keeps the documentation always up to date. QAcegen generates test data in two modes:

• Data for a logical test.
• Performance test data.

Logical test data is primarily used to test the business logic. Tool generates only small volume of data (of a few thousands of records) to cover every logical branch of a tested business logic. Once successfully completed, the system (based on all test scenarios) prepares data for a performance test. In this mode, the volume of records is significantly higher (millions of records) to simulate a real-life situation and/or a “peak events” to verify that tested application meets speed requirements.
Note: qacegen always generates complex data, which covers entire business logic. No matter how insignificant change request is implemented, tool always generates data for a complex test. Only this way can be identified possible collateral effects of the change request, which may not show during an incremental test.
QAceGen offers an easy transportation functionality to deliver test data into all source files and/or tables. Metadata of each table/file are stored in a “source profile”, which defines location, data format and access rights. The “data transportation” icon transports data into required location automatically. When test scenarios include the “verification statements”, qacegen validates the results of each test scenario and creates detailed protocol. First, these results are displayed in a graphical mode in qacegen. Subsequently, the results can be exported into a text report while preserving the logical structure of the project. The level of detail can vary from the simple results overview to a detailed information about each transaction used during the test. QAceGen’s security features are system’s another advantage. Data used by the system contain only artificial strings and numbers unrelated to the real production data. No sensitive data from production system is ever used during test.

Main applications

QAceGen system has been primarily designed for following type of type of activities:

• Data preparation – (e.g. development, training).
• Applications Testing.
• ITO applications.
• Complex systems.
• Data quality management (dqm).

Note: ito = Input / transformation / output (i.e. an application processing data)
ITO application – takes data from input tables/files, transforms data according to the business logic and stores results into output tables/files. An etl process is an example of an ito application, however similar processes/applications can be found among data warehouse application or transaction systems. Complex systems – specialized solutions (e.g. erp, b2b), which can be seen as a “black box” by users. Data processing and storage of data inside database is not exactly known but these systems communicate with users or other application via pre-defined interface according to given business processes. Data preparation – according to the business logic defined in the specification. As mentioned above, the business logic is split into scenarios and qacegen generates set of records covering each scenario. Such data can be used for applications development or loading data into test/training environments, where the use of production data is either not possible or not recommended due to security issues. Testing – designed scenarios in addition to the “data preparation” include “verification statements” which validate results of a tested application. QAceGen also supports data generation/testing in a complex systems environment. Many organizations run many specialized systems from various vendors and on various platforms. It is often necessary to test how these systems exchange data and cooperate among themselves. These systems communicate via a pre-defined interface. This implies that the input data structures as well as the loaders feeding those systems have a defined structure. Test scenarios are defined by a given business process, which defines both list of required input data (e.g. client, order, transaction) and a sequence of input data loads. QAceGen tool generates data according to the input tables/files structures and then it runs a relevant api or a script to transport input data. In addition, data input using screen forms can be automated. If required, the results verification can be addressed similarly. QAceGen can run an api/script with the relevant business process, record and verify the result. Data quality management (dqm) feature controls data quality. The feature includes:

• Duplicity monitoring.
• Cardinality monitoring.
• SCD2 (slowly changing dimensions type2 – e.g. historization, incidences, …).

Using the simple dqm commands, it is possible to set checks above tables or joins of tables. These commands then generate a set of control sql statements and after execution are automatically verified by QAceGen. The outcome is recorded into a standard report. The main advantage of this solution is fast control of virtually hundreds of tables across different databases. Also, dqm functionality is an “open solution” where new dqm commands can be designed if required.

Benefits

• Lower project costs than traditional solutions.
• Time efficient.
• Enhanced transparency of a test project.
• Improved data security.

The post Test tools – GTL QAceGen appeared first on GTL Tools.

Informatica custom transformation input groups

• Group working contains data, which needs to be consolidate with historical data, which means data records for a certain period. One group can contain only data changes or entire time record.
• Group target contains historical data, that means the history of each record/object. Data are consolidated with data from input working.

Informatica custom transformation output groups

• Group output – new records, which needs to be added to historical data, so than can collect consolidated changes.
• Group original – on this output are send target records for to reduce validity, because there are working record with changed attributes or records on the input. These changed records or attributes does not exist on the new record and its validity should be terminated.

Input and output groups of component

Consolidation plan – port type values