Category Archives: Astera Technical Blog

An Automated Approach to Modeling Your Slowly Changing Dimensions

Business data is inherently susceptible to change with the passage of time and impacts the business in different ways. In data warehouses, the effect of time on our dimensions and facts requires careful study for the repository to meet the business intelligence objective of delivering up-to-date information to decision makers.

Question is, how best to handle these changes?

Developing a dimensional model that captures the different states of your data with respect to time is a key objective of an Enterprise Data Warehouse. For measures in our fact tables, we can use date dimensions and link them using foreign keys. For dimensions, the complexity of handling changes increases greatly. Each step of the Slowly Changing Dimension (SCD) flow must be hand-coded using multiple, complex SQL statements. The implementation is lengthy and complex, and affects the business’ ability to maintain its data quickly and reliably – which is always a critical consideration.

Slowly Changing Dimensions in Centerprise

Compared to the traditional hand-coded approach to the slowly changing dimension flow, Astera offers an automated implementation using a completely drag-and-drop interface. Source data is mapped to an SCD object in Centerprise, which pushes system-generated SQL statements directly to the target data warehouse (Read: Pushdown Optimization Mode in Centerprise) based on the field layouts defined by the user. Each column in the user’s table can be designated as Surrogate Key, Business Key, SCD1, SCD2, etc. (see below) within the component’s properties in Centerprise. The platform handles the update strategy, performance considerations, routing, and complex joins automatically on the backend, as long as the SCD Field Types in below screen are defined correctly.

Field Layout - Slowly Changing Dimensions component

SCD Object Properties in Centerprise

Automating Type 1 & 2 Slowly Changing Dimension Implementation

Centerprise supports both Type 1 and Type 2 SCD to update records with and without maintaining history.

SCD Type 1

This type deals with updates in the dimensional table, for cases when preserving history is not a consideration and you need to replace the old values in your table with recent ones.

To use SCD Type 1 in Centerprise, you can mark your column as ‘SCD1 – Update’ in the Layout Fields menu of the SCD object in Centerprise, as seen in above screenshot for the ‘Contact Title’ column.

SCD Type 2

This type deals with changes in your dimension that need to be tracked. A new record is inserted with each change, and the existing record is marked as expired, by date, version, or status.

To use SCD Type 2 in Centerprise, mark your chosen column as ‘SCD2 – Update and Insert’, as seen in above screenshot for ‘ContactName’ column.

Push-Down Optimization

Once the layout is defined and flow executed, the Astera SCD transformation generates the SQL code necessary to compare, join, route, and insert data in your target dimension and pushes the transformation logic down to the database for processing.

Using this approach, the maintenance of large dimensions is significantly faster because all the processing is done by the database rather than the Centerprise server performing the operations and going back and forth between the database to read, compare, and write the data.

To learn more about the automated Slowly Changing Dimensions component in Centerprise and how to use it to manage your dimensions, download the white paper: How to Manage Slowly Changing Dimensions Using Centerprise.

EMR and HL7 within EDIConnect

In October, we went over two healthcare EDI transactions: EDI 834 and 837. This week, we’ll take a look at two more healthcare facets of EDIConnect: the ability to handle EMRs and compliance with HL7.

Electronic Medical Records (EMRs)

Electronic medical records (EMRs) are electronic records generated and maintained by hospitals and healthcare organizations. They are a vast improvement over paper records. They allow more than one person to use a patient’s chart, are better organized, eliminate illegible handwriting, and allow storage of more information.

The demand for healthcare systems with Electronic Medical Records (EMR) interfaces is rising. This increase is due to many factors, including the growing adoption of EMR systems and emerging clinical healthcare data standards such as HL7.

 

Health Level-7

Health Level-7(HL7) refers to a set of international standards for software applications used by healthcare providers for clinical and administrative transaction data. The HL7 standards are produced by the Health Level Seven International, an international standards organization, and are adopted by other standards issuing bodies such as American National Standards Institute and International Organization for Standardization.

HL7 International specifies a number of flexible standards, rules, and procedures for healthcare systems used by hospitals and other healthcare provider organizations to communicate with each other.  This helps information to be shared and processed in a uniform and consistent manner and allow hospitals and healthcare organizations to easily share clinical information.

Electronic data interchange (EDI) format also plays a very important role with EMR files and records. It helps to share clinical and other administrative data and medical records across different healthcare systems as well as individuals. Patients can also access their medical records through internet, allowing them to stay well-informed about their health status and ongoing medical treatments.

 

EDIConnect

Astera Software provides a complete solution for handing documents such as EMR and many others such EDI documents. EDIConnect offers a user friendly and intuitive user interface to accurately and efficiently handle bi-directional EDI data integration. It is scalable and powerful enough to fulfill entire EDI transaction processes.

 

EDIConnect Benefits:

  • Handles electronic medical records and HL7 documents.
  • Powerful and scalable
  • Fast, comprehensive and accurate data exchange
  • Easy to use graphical interface for both technical and business users.

Multi-Column Support

Many documents have newspaper-style formatting, which contains more than one column with a repeating pattern of records. As a result, the layout is more complex than a single-column document, and extracting useful information can pose a challenge.

ReportMiner 7 now provides a Multi-Column layout option to handle documents with multiple columns.  In the past, if documents had more than one column with a repeating pattern, it would be very difficult to extract information from all the columns in a clean and efficient manner.  This was due to the way in which the software looks for information: it scans the data in horizontal sweeps. With the latest version of ReportMiner, you can now process your multi-column documents within minutes for perfectly editable and searchable data.

Here’s how to use ReportMiner 7’s Multi-Column feature: 

First, load your multi-column document in ReportMiner.

Add a Data Region to create matching patterns.

In this case, we’ll create a matching pattern for Names and Phone Numbers in the document.

Next, add Data Fields for Names and Phone Numbers in the document.

When previewed, the data is displayed accurately in a list format.

As seen in the screenshot below, a blank bar appears as soon as you check off the Multi-Column option in the Data Region. Click on the bar and a black dotted vertical line will appear indicating a column boundary. If a line is placed incorrectly, click on it within the bar to remove it and try again. Make sure that the line is flush with the left side of the first column of characters in your document.

Since there are three columns in the sample document, another column boundary is added just before the start of the second column. All records in both columns have now been successfully identified.

Preview your data and export it to a destination file type of your choice with easy access to the extracted information.

From one column to multiple columns, Astera can extract information with ease. Thanks to ReportMiner7, your data is more accessible than ever before. 

Top EDIConnect Features: EDIConnect EDI Transaction Process

Many businesses use Electronic Data Interchange (EDI) to send and receive messages. Last week we covered EDI source files and parsing. This week, we reach our number one feature for Centerprise’s EDIConnect connector: the EDIConnect Transaction Process.

  1. EDI Builder and Destination File

To build an EDI document, use the EDI Builder feature. There are two steps in this process. First, build the EDI message using EDIConnect Visual Designer. Next, validate it against Partner Standards and send it out.

EDI Builder

Select File > New > Dataflow.

To create an EDI file, select the EDI Message Serializer from the Toolbox on the left as shown in the screenshot below. Drag and drop the EDI Message Serializer to the Dataflow.

1

Setting EDI Builder Properties

Right-click on the EDIBuilder message and select Properties. A dialog box opens as seen below. First, point this file to a Trade Partner Profile: this automatically selects the transaction type for the outgoing EDI message.

2

EDI Builder Structure

The EDI builder now has a structure for the selected transaction type in the previous step. Any problems and errors that occur during a preview are shown in the Errors section.

3

EDI Destination file

EDI destination files can also be selected from the toolbox and dragged and dropped into the Dataflow. Use the file properties to specify the location of the output file and the Trade Partner Profile data.

4

Mapping

Finally, map the EDI Builder file to the EDI Destination file as seen in the screenshot below. This Destination file will contain the final EDI message.

 5

EDIConnect can be used to handle entire EDI transaction process. It supports all EDI message standards and customizes transaction documents: EDI transactions have never been easier.

That concludes our top 4 EDIConnect features! We hope that you’ve enjoyed this series and that you’ll check back next week for more exciting new information.

Top EDIConnect Features: EDI Source Files and Parsing

Businesses across the globe send and receive messages via Electronic Data Interchange (EDI). Last week we covered our Centerprise connector, EDIConnect’s, Custom Repository feature. Our focus this week are our number two features for EDIConnect: EDI Source Files and Parsing.

Once an EDI file has been received, validated, and once the acknowledge process is complete, it goes to a dataflow to be parsed and translated. After that, it can be transformed, mapped, and written to destination.

EDI Source File

This reads the incoming EDI messages.  First, create a new Dataflow by selecting File > New > Dataflow. Once a new dataflow opens up, drag and drop the EDI Source File option from the toolbox on the left to the dataflow.

Now, specify source file location information and Trade Partner Profile settings. These contain all information about processing incoming data using EDI Source file properties.

2

To preview EDI source file data, right-click on the EDI Source File box in the dataflow and select Preview Output.

3

The output is in hierarchical format as seen below.

4

EDI Parser

This generates the transaction structure that can be mapped to any destination. The Parser also includes the Trade Partner Profile file: all information required to parse a transaction is in one place. From here, users can specify all transaction sets expected from a partner.

The EDI Parser can be dragged and dropped in to the dataflow using the Toolbox on the left. To set its properties, right-click on it and select Properties. The screenshots below demonstrate the EDI Parser process.

5

6

Mapping

To map the EDI Parser to the EDI Source file, drag the Transaction Node of the EDI Source file onto the EDI Parser file.

7

Destination file

Once the EDI message has been parsed, it’s ready to be mapped to a destination file, such as an XML file as shown in the example below.  Drag and drop the XML/JSON File Destination option from the Toolbox on the left under the Destinations category to the dataflow.

8

Make sure you specify the output file location and provide an XML Schema file that matches the EDI Parser transaction structure to the XML Destination File Properties.

9

Now, map your EDI Parser to your XML Destination File. Your output file is ready to view.

Want to know more about EDIConnect? We’ve been covering our top four EDIConnect features over the past several weeks. Be sure to check back for the final installment in this series!

Top EDIConnect Features: EDI Custom Repository

Many businesses use Electronic Data Interchange (EDI) to send and receive messages. Last week we covered our Centerprise connector, EDIConnect’s, Trade Partner Management features. Our focus this week is our number three feature for EDIConnect: the EDI Custom Repository.

EDI Custom Repository

With EDIConnect, users can customize transaction sets, segments, composite elements, and elements, and store them in a custom repository.

Create a new Custom Repository

To create a new Custom Repository, Select File > New > Edi Custom Repository.

This will create an empty custom repo that contains all the standard definitions for transaction sets, segments, elements and composite elements.

1

Customize a Transaction Set

In the screenshot below, all the standard EDI Transaction sets are listed on the right and each transaction set’s details can be seen and modified once clicked.

2

Custom segments can also be added or deleted.

3 4

Customize Segments/Composite Elements/Elements

As with the transaction sets, every segment, composite element, and element can also be customized: EDIConnect is robust and powerful enough to handle various EDI transaction documents with ease. The screenshots below demonstrate the numerous possibilities to customize EDI documents.

5 6 7

All artifacts that come with standard EDI messages can be modified using EDIConnect and can be stored in an EDI Custom Repository.

Want to know more about EDIConnect? We’re covering our top four EDIConnect features over the next several weeks. Be sure to check back for the third installment!

Top EDIConnect Features: Trade Partner Management

Many businesses use Electronic Data Interchange (EDI) to send and receive messages. Last week we covered what EDI is and two specific use cases. This week, we take a closer look at how to get started with Astera’s  Centerprise connector, EDIConnect. Below is our number four feature.

Trade Partner Management

The Trade Partner Management feature in EDIConnect allows users to set up trading partners, manage all aspects of trading partner relationships (such as inbound and outbound transaction maps definition and  separators for segments,) and has ways to customize EDI message standards and use them within trade partner definition.

Create new Trade Partner Profile

To create a new Trade partner profile, select File>New> Edi Trade Partner Profile. From there, specify the Trading partner profile name and settings, Inbound/Outbound Maps, and Sequences.

X12 Settings lets the user define acknowledgement information, functional groups, Validation settings, etc. The screenshot below shows the options available.

Specify Inbound Maps and Outbound Maps

Use this option to detail how transactions from the partner will be treated and the number of transactions expected from said partner. Choose from standard or customized versions of EDI messages, as well as standard or customized transaction processes.

Outbound maps are where the user can establish and customize which maps will be sent out to this partner. 

Sequences

All sequence generations for a particular EDI partner are controlled by the properties specified in the Sequences tab (see screenshot below.) Options such as Database Information, Functional Group Control numbers, etc., can be set via the Sequences tab.

4

Want to know more about EDIConnect? We’re covering our top four EDIConnect features over the next several weeks. Be sure to check back for the second installment!

Optical Character Recognition Support

Welcome to The Highlight Reel, Astera Software’s blog series on ReportMiner 7’s newest features.

ReportMiner 7 now offers built-in Optical Character Recognition (OCR). Combined with our sophisticated pattern based text extraction functionality, ReportMiner can be used to unlock data trapped in scanned documents seamlessly.

How does it work in ReportMiner?

ReportMiner uses OCR as a preprocessing step to get the text equivalent of the image found in the scanned pdf documents. Once the equivalent text is available, rest of the process is exactly same as other text based documents. Let’s review the OCR process for PDF documents containing textual information as images:

  1. Once we select File > New > Report Model, we can go ahead and set the path to the PDF document containing textual information that we would like to run OCR on.

Make sure that the “Run OCR” option is checked, so that ReportMiner will run OCR on the document.

An important thing to be noted here is the option of zoom level and its default value being set to 100%.

Selecting an appropriate zoom level results in both speed and accuracy. If the image containing text is very small, increasing the image size can result in better text recognition with improved accuracy. Hence, you can adjust this zoom level until you get the desired results.

  1. Below is a screenshot of the PDF document we are trying to read using ReportMiner.

  2. As soon as you select “Ok”, ReportMiner will start running OCR on the document.

 

 

  1. As shown below, Report Miner grabs the textual information from the PDF document and displays it on the screen.

Now that you have your document digitized, it can be processed by ReportMiner. It can be used to create report models to create data regions and identify matching patterns, grab data and then export it to your desired destination.

Be sure to check back every Thursday for more highlights. If you’d like to see the current list of featured features, click here.

Microsoft Word and Rich Text Format Support

Welcome to The Highlight Reel, Astera Software’s blog series on ReportMiner 7’s newest features.

ReportMiner 7 now includes support for Microsoft Word (Doc and Docx) and RTF formats: enjoy efficient and easy information extraction from more source files than ever before. Now you can process invoices, purchase orders, receipts, forms and other Word/RTF-formatted files with ReportMiner 7.

The screenshots below illustrate .docx extension support in ReportMiner.

Select File > New > Report Model and choose your source document.

As seen in the screenshot below, the .docx file opens and is ready for processing.

You can now continue to create your report model.

We at Astera Software are proud to take every step towards making our products the best that they can be. Be sure to check back every Thursday for more highlights.

Trade Partner Management in EDIConnect

EDIConnect is a complete solution for accurate and efficient bi-directional EDI data integration. It offers EDI data integration capability in an intuitive user interface with visual tools to build bi-directional integration. This blog will discuss how to set up a trade partner relationship in EDIConnect, including how to define incoming and outbound condition maps, how to define separators for the segment elements, and how to customize standards and use them inside the trade partner definition.

First, let’s review the EDIConnect process for receiving and sending messages.

1_EDI1

You define your trade partner and specify the settings to receive incoming messages.

Once you receive the message, it goes into the system, where you can validate it and generate acknowledgements such as X12, 997, and 999 or Tier One.

Now let’s take a closer look at the process for defining a trade partner and specifying the settings to receive incoming messages.

Begin by creating a new trade partner. Go to file/new and click on EDI Trade Partner Profile.

2_EDI1

In the screen, you can specify the partner name and also specify your settings, inbound maps, outbound maps, and sequences.

3_EDI1

Let’s look at these options one by one.

First, select the dialect for the partner, in this case X12.

4_EDI1

Within the X12 settings, you can specify information for the interchange headers, functional groups, validation settings, and acknowledgement.

5_EDI1

Within the Functional Group you can specify GS properties as shown below.

6_EDI1

Similarly, for Interchange Header you can specify all ISA properties, as well as separators for your data elements, as shown below.

7_EDI1

For the validation settings, you can specify what should be validated when a message is received for the partner.

8_EDI1

You can specify acknowledgements such as what kind of acknowledgement you want to convey when the messages arrive, such as FunctionalAckTransactionSet, GenerateAk2ForAcceptedTransactionSet, or SendTechnicalAcknowledgement.

9_EDI1

Now we’ll move on to how to specify inbound maps. This is where you specify how specific transactions from the partner are treated and how many you are expecting from that partner. For example, from this partner I am expecting only 850 transactions.

10_EDI1

So I’m going to add one 850 as an incoming transaction set and specify how I want it to be processed in the system. You can have a standard 850 or you can have a customized version of it. In the case of a customized version, you will pick your customized 850 version as a mapping so that when the messages arrive the customized information is used to process that message.

11_EDI1

Similarly, for outbound messages you specify what maps you are going to send out to this partner and if there is any customization, in which case you map the standard message to the customized message in the customization window as shown below.

12_EDI1

Now we’ll move on to sequences. All the sequence generation for this EDI partner is controlled by the properties specified in this tab.

13_EDI1

Here you specify where your sequence object resides, your database information, and your functional group control numbers, interchange control numbers, and conditions of control numbers.

Now that you know how to set up a trade partner relationship, our next blog will show you how to use the EDI builder and EDI destination file in EDIConnect.