Name Parsing and Gender Determination

Centerprise integrated name component accepts free-form names and then automatically splits them into standard components: Prefix, First, Middle, Last and Suffix no matter what the original format. Next, gender is determined using a built-in name dictionary.

The component can process all styles of names including inverse, natural order, hyphenated and multi-part last names.

Address Parsing and Cleansing

The address parser component parses free-form United States addresses into individual components. USPS compliant address lookup component can be used to correct, complete, and normalize incoming addresses. This component performs all the operations locally and does not need connection to a service. This architecture delivers high performance and is capable of handling large data volumes. There is no limit for data volumes or per unit charge.

Insert an action after another action

You can add a new action after an existing action by dragging the item from the toolbox and dropping them to the right of the top node or the top output node. The action is inserted after the specified node and all the outgoing maps are rerouted. This way, if you need to add a new transformation between two existing actions, you do not have to reroute maps manually. You can insert any set transformations such as filter, sort, join, union, route, etc.

Insert an action before another action

Add certain actions before another action by dragging from the toolbox and dropping them to the input port of the top node or the top input node. The action is inserted before the specified node and all the incoming maps are rerouted.

Insert an action before or after an element

Insert an action before and after an element by dragging it from the toolbox and dropping it on to the input or output port. This enables you to insert single transformations such as expressions, lookups, functions, etc. before and after elements. If the element is already mapped, Centerprise reroutes the maps where appropriate.

Delete an action and reroute maps

This feature can be used to remove an action and automatically reroute the incoming and outgoing maps. This action is allowed in certain conditions:

• Input and output layout is the same (such as filter, sort, etc.)
• Both input and output layouts contain a single element. Examples are list lookup, expression with one inbound element, and functions with single input and output element.

Cascade Element Delete

This feature enables you to remove an element from an action and any destination elements. Elements are deleted only if the action supports add/delete elements.

Reading Data

You can use a number of built-in features to optimize database reading. These include partitioning, change data capture, multi-table query, and others.

Read Only What is Necessary. When reading from a database, use where clause to filter out the records you do not need. Using filter transformation after reading the data reads unneeded records. If subsequent transformations expect data to be sorted, use ORDER BY clause to sort records in the database.

Do Not Map All Fields. For database table source, Centerprise reads only the fields that are used in maps. This can substantially improve database read performance. Therefore, avoid mapping the entire database table and map only the fields that you need.

Use Multi-Table Query. If you access data from multiple tables in the same database and then use Centerprise Join Transformation to join them, consider using Centerprise Multi-table Query source. This component features a sophisticated query engine to build queries that span across tables.

Use SQL Query Source. If you have an existing stored procedure or would like to write a query that is optimized for the specific scenario, use SQL query source.

Partition Data. You can use data partitioning feature of Database Table Source. This feature parallelizes database reading by creating multiple readers.

Use Change Data Capture. Centerprise includes multiple change data capture strategies to minimize data reads. These include Audit Field, Incremental Read as well as database writing strategies. If you transfer data at regular intervals and only a small fraction of data changes between transfers, CDC can help speed up your dataflows.

Writing Data

Centerprise supports native bulk insert for popular databases such as SQL Server, Oracle, DB2, Sybase, MySQL, Access, Netezza, Teradata, and more. This enables you to efficiently process high data volumes. Where applicable, use bulk insert option while writing to database. Centerprise also batches updates and deletes to optimize writes. Centerprise includes several write strategies that can be employed to speed up database writes. These include data-driven strategy, Slowly Changing dimensions, and Diff Processor.

Transformations

It is important to understand performance and memory characteristics of transformations. Transformations are blocking and non-blocking. Blocking transformations hold data until all the records have been received and processed. Non-blocking transformations continuously process and release records. Blocking transformations often impede parallel processing because succeeding transformations must wait for the entire dataset to be received and processed. Sort is an example of a blocking transformation. Join, distinct, aggregate, denormalize, and other transformations are non-blocking if data is sorted and blocking if data is not sorted. Avoid repeated sorting for large datasets. If applicable, use ORDER BY to sort data in the database. Remember, checking a ‘Sort Incoming Data’ has the same effect as inserting Sort Transformation.

Distinct and Aggregate Transformations: Distinct and Aggregate Transformations use little memory when the incoming data is sorted on key or group by. If, however, the incoming data is not sorted, these transformations store values in memory. This could consume enormous amount of memory for large data sets and will slow down the rest of the server.

Field Profile: Field Profile Transformation can consume significant memory if the number of distinct items is very large. Use this transformation with caution on large datasets.

Lookups: Lookups can slow down your dataflows considerably. If a lookup table is large, ensure that it is indexed on lookup fields. You can use appropriate caching options depending on the size of lookup table and the number of values you are looking up. Where appropriate, consider using Join Transformation instead of lookup.

Excel Files

Excel files are memory hogs. While reading, he entire file is loaded in memory. This slows down the dataflow and consumes substantial memory resources. Excel Destination keeps the entire file in memory and writes them to disk at the end of the dataflow. Avoid using Excel workbooks for large data volumes.

Electronic Data Interchange is a predominant means of data exchange between businesses. Integrating EDI transactions is a challenging undertaking. Businesses often need to map incoming EDI transactions with their in-house applications. These applications are usually based on relational databases necessitating conversion from hierarchical to relational model. Building outgoing EDI transactions is even more challenging and often entails custom programming or complicated scripting.

Centerprise now delivers support for common EDI standards and transactions in a drag-and-drop environment designed for business users. You can quickly parse and create complex EDI transactions using the visual dataflow designer. Key features include:

• Support for key EDI standards including X12, HL7, HIPAA, EDIFACT, CARGO, and IATA
• Parsing of incoming EDI transactions
• Creation of outgoing EDI transactions from disparate data sources
• Multi-table query and tree join features enable visual building of complex EDI transactions
• High performance parallel processing delivers throughput for processing large data volumes

Centerprise provides built-in ability to move binary data between applications, databases, and files via simple drag and drop that involves no programming or scripting. With Centerprise, you can:

• Move picture and video files to database tables
• Move BLOB, CLOB, and other large data between databases
• Exchange files with web applications using SOAP, or HTTP/REST
• Move binary data between local database and cloud applications such as Salesforce.com and Microsoft Dynamics CRM

Additionally, Centerprise’s high-performance parallel processing engine, combined with native bulk load support for popular databases, delivers the throughput to handle the most demanding integration tasks.

• Support for all REST methods
• For GET, common paging patterns are built-in. These include next page and item range.
• Common authentication mechanisms including basic and OAuth are supported
• Serialization and deserialization support for XML, JSON, and HTTP name-value formats
• Use sample XML, JSON, or name-value text to automatically build input or output layout
• Auto-determination of output layout for GET requests

Quickly Integrate with Popular Web Services

With Centerprise REST connector, you can quickly integrate your internal applications with any REST web service. These services include:

• Google, Yahoo, and Bing REST services
• Amazon Web Services
• Facebook, LinkedIn, Twitter, and other social media sites
• Salesforce.com, QuickBase, Zoho, and other online databases
• ZenDesk, Bugzilla, and many others

Integrate your In-house systems with cloud web services

Centerprise connects to an array of databases, file formats, and applications. With Centerprise REST connector, you can connect your in-house applications with web based platforms using a point-and-click interface requiring zero programming.

Centerprise now includes the capability to connect to any OLE DB data source. This feature expands Centerprise connectivity to include any OLE DB drivers. With this capability, you can query Microsoft Active Directory, Microsoft FoxPro and Jet Data Sources, and numerous third parties data sources that provide OLE DB connectivity.

A partial list of Microsoft OLE DB providers can be found in http://msdn.microsoft.com/en-us/library/ms187072.aspx.

Here is another useful link: http://www.carlprothman.net/Default.aspx?tabid=87

Centerprise Data Integrator represents a new generation of integration products that provide comprehensive data integration functionality in a single, unified platform. Centerprise’s codeless, visual environment enables you to integrate data from any data source including databases, files, services, and unstructured data sources.

Astera customers range from Fortune 500 enterprises to small and medium sized businesses including Wells Fargo, Raymond James, Alcatel-Lucent, AutoTrader, Montana-Dakota Utilities, and El Paso Gas.

Centerprise is designed for nontechnical users while providing all the power and functionality required for complex, high-volume data integration scenarios. Using Centerprise’s acclaimed user interface, technical and nontechnical users are able to easily create data flows to transform, profile, and cleanse data.

Today, we are launching a free Community Edition of Centerprise Data Integration Platform. The Community Edition provides the functionality to enable development and execution of complex dataflows and deliver the performance to efficiently process large data volumes.

Centerprise Community Edition provides connectivity to a wide array of data sources including popular databases, file formats, printed reports, and on-premise and cloud applications such as Salesforce.com, Microsoft Dynamics CRM, and QuickBooks.

The Community Editions is free and is available at www.astera.com/centerprise/download

Astera is also making available detailed Centerprise documentation and training videos online. Centerprise documentation and training videos are available  here.

Many require a series of workarounds or multi-step processes requiring staging tables and custom programming. As a result, companies frequently rely on custom, ad-hoc programs to manage their complex data conversions. These ad hoc programs are difficult to create, expensive to maintain, often create performance issues and do not scale well.

If you are responsible for validating, normalizing, and boarding data from multiple sources, perform complex data mappings and conversions, and routinely deal with format inconsistencies and data quality issues, you know how challenging it is to find a solutions that provides the functionality required to manage those processes. Astera developed Centerprise specifically to enable the processing of complex data conversion tasks by business analysts.

The Centerprise user interface was developed and refined in collaboration with Astera’s customers to provide enable a high level of productivity and support enormously complex data conversions. As a result of these efforts, Centerprise delivers a superior solution for complex data conversion tasks.

Key Features which make complex data mapping easy:

• Drag and drop visual flow designer enables creation and visualization of even the most complex dataflows
• A single-click WYSIWYG data preview provides an instant view of data at any step in a dataflow
• Workflow designer enables orchestration of jobs and dependencies
• Integrated environment featuring database browser, and query editor; provides detailed visibility into your data sources and targets
• Full featured scheduler, event-driven job triggering, and other capabilities assist with visually design and automate your data conversion workflow

Customers with complex data conversion needs select Centerprise because it makes these data conversions easy, boosts their productivity, provides constructs to handle layout discrepancies and data quality issues, and delivers the performance and scalability to efficiently process high data volumes.

Learn more about complex data mappings and conversions.