The Dilemma of Build vs Buy – How it applies to Enterprise Software

If Shakespeare was an IT manager, the famous question ‘To be, or not to be’ would have been ‘To build or to Buy’. In fact, the phenomena of DIY-ing something or buying a commercial product is not only limited to enterprise software. IKEA is running a whole business out of providing utility to ‘build proponents’ and DIY enthusiasts. While building furniture can be fun, building an enterprise level software – not so much.

Build vs Buy

Like any other business decision, the decision to either build a software or to buy a commercial product is significantly influenced by total cost of the approach and return on investments. If you’re facing a similar dilemma, the table below summarizes the prospects and consequences of both the approaches.

         Metrics and KPIs                     Build Approach                        Buy Approach
Cost of deployment Hiring a team of developers, designers, programmers to build the solution License fee of the product and deployment costs
Time to market – Time to develop the product

– Time for performing QA analysis

– Time to fix any patches or bugs found

– Time to deploy the solution

– Product development, QA analysis and patch fixes are already taken care of by the solution provider. Therefore, solution can directly be deployed.

– Time to configure and install the product.

Ongoing maintenance and support costs A dedicated team of IT professionals should be on-board to help with ongoing product support and maintenance Updates, maintenance and customer support are handled by the solution provider. However, the solution provider might charge a fee for providing these services
Learning curve A steep learning curve is usually associated with the developed product Commercial products are developed to be used by a wide range of audience with varying levels of technical skills therefore, in most cases these solutions are designed to be more intuitive and user-friendly

 

When is ‘Building’ the right approach?

Building a software is going to be beneficial for your business if:

  • The software is going to give you sustainable competitive advantage
  • No other available solution can meet your business needs
  • The end-points from where your business collects data are not volatile or prone to frequent changes
  • You have substantial resources to cover the costs associated with building and maintaining the software

When is ‘Buying’ the right approach?

You should opt for buying a commercial software if:

  • Building a software is not the core of your business and is not going to yield you any competitive advantage
  • You have limited resources and you would rather invest them in improving your core business activities
  • There are solutions available that address the challenges your business is facing
  • You are looking for a quick solution that can be immediately deployed

IT manager at Brickell Bank, formerly known as Espirito Santo Bank, faced challenges in migrating broker data from MS Access database to IBM mainframe data warehouse. Learn more about the approach he opted for and other factors that influence build vs buy decision by downloading the free white paper.

An Automated Approach to Modeling Your Slowly Changing Dimensions

Business data is inherently susceptible to change with the passage of time and impacts the business in different ways. In data warehouses, the effect of time on our dimensions and facts requires careful study for the repository to meet the business intelligence objective of delivering up-to-date information to decision makers.

Question is, how best to handle these changes?

Developing a dimensional model that captures the different states of your data with respect to time is a key objective of an Enterprise Data Warehouse. For measures in our fact tables, we can use date dimensions and link them using foreign keys. For dimensions, the complexity of handling changes increases greatly. Each step of the Slowly Changing Dimension (SCD) flow must be hand-coded using multiple, complex SQL statements. The implementation is lengthy and complex, and affects the business’ ability to maintain its data quickly and reliably – which is always a critical consideration.

Slowly Changing Dimensions in Centerprise

Compared to the traditional hand-coded approach to the slowly changing dimension flow, Astera offers an automated implementation using a completely drag-and-drop interface. Source data is mapped to an SCD object in Centerprise, which pushes system-generated SQL statements directly to the target data warehouse (Read: Pushdown Optimization Mode in Centerprise) based on the field layouts defined by the user. Each column in the user’s table can be designated as Surrogate Key, Business Key, SCD1, SCD2, etc. (see below) within the component’s properties in Centerprise. The platform handles the update strategy, performance considerations, routing, and complex joins automatically on the backend, as long as the SCD Field Types in below screen are defined correctly.

Field Layout - Slowly Changing Dimensions component

SCD Object Properties in Centerprise

Automating Type 1 & 2 Slowly Changing Dimension Implementation

Centerprise supports both Type 1 and Type 2 SCD to update records with and without maintaining history.

SCD Type 1

This type deals with updates in the dimensional table, for cases when preserving history is not a consideration and you need to replace the old values in your table with recent ones.

To use SCD Type 1 in Centerprise, you can mark your column as ‘SCD1 – Update’ in the Layout Fields menu of the SCD object in Centerprise, as seen in above screenshot for the ‘Contact Title’ column.

SCD Type 2

This type deals with changes in your dimension that need to be tracked. A new record is inserted with each change, and the existing record is marked as expired, by date, version, or status.

To use SCD Type 2 in Centerprise, mark your chosen column as ‘SCD2 – Update and Insert’, as seen in above screenshot for ‘ContactName’ column.

Push-Down Optimization

Once the layout is defined and flow executed, the Astera SCD transformation generates the SQL code necessary to compare, join, route, and insert data in your target dimension and pushes the transformation logic down to the database for processing.

Using this approach, the maintenance of large dimensions is significantly faster because all the processing is done by the database rather than the Centerprise server performing the operations and going back and forth between the database to read, compare, and write the data.

To learn more about the automated Slowly Changing Dimensions component in Centerprise and how to use it to manage your dimensions, download the white paper: How to Manage Slowly Changing Dimensions Using Centerprise.

Pushdown Optimization Mode in Centerprise Data Integrator

How does Pushdown Optimization mode work in Centerprise?

Moving data, containing millions of records, between source, ETL server, and target database can be a time-consuming process. When source and target database reside on the same server, unnecessary data movements and delays can be prevented by applying transformations to data in pushdown optimization mode.

Pushdown optimization mode pushes down the transformation logic to the source or target database. Centerprise integration server translates the applied transformation logic into automatically generated SQL queries. This eliminates the need for extracting data from the source, migrating it to staging tables on an ETL server for applying transformations, and then loading the transformed data on the target database. As a result, performance is significantly improved and data is readily made available to the end-users.

ELT, ETL, Pushdown optimization mode

Types of Pushdown Mode

There are two types of pushdown optimization modes:

  1. Full pushdown optimization mode
  2. Partial pushdown optimization mode

In full pushdown optimization mode, the Centerprise integration server executes the job completely in the pushdown mode. And in partial pushdown mode, the transformation logic is either pushed down to the target database or the source database, depending on the transformation logic and database provider.

Database Providers supported in Pushdown Mode by Centerprise

Centerprise supports following database providers:

  1. MySQL
  2. SQL
  3. Oracle
  4. Postgres
  5. MSSQL

Verify Pushdown Mode

Certain transformation logic cannot be executed in a pushdown mode. ‘Verify Pushdown Mode’ feature in Centerprise identifies the transformation logic that can be pushed down to the source or destination database.

To learn more about Pushdown Optimization mode in Centerprise and its use cases, download the white paper Centerprise Automated Pushdown Optimization.

Optimizing Business Capabilities with a Data Integration Software

Businesses are increasingly adopting a data-driven culture. The significant surge in the volume of the exchanged data indicates that the trend is creating a paradigm shift – a shift from manufacturing to an information economy. To put this in perspective, Google processes petabyte of information by the hour and The Economist recently declared data as the most valuable resource, even more than the oil.

Data integration with Centerprise

“The world’s most valuable resource is no longer oil, but data.”

-The Economist

But the true utility of any resource comes from its consumption or the value it delivers to the consumers. The same principle applies to data. To gain maximum utility out of data, businesses must be able to (quickly and reliably) integrate incoming data from disparate sources and make that information available to the relevant stakeholders, both internally and externally. Your business needs a data integration tool to perform this task efficiently.

A data integration tool can help you in following ways to optimize your current business capabilities:

By extracting data from structured and unstructured sources

Incoming data can be structured, semi-structured, poly-structured or unstructured. For instance, text-based PDF files, PDF forms and scanned PDF images are used as a medium for exchanging information by many organizations. But the data contained in PDF files is unstructured and is required to be extracted for crucial business decisions. A data integration tool can automate the data extraction process and integrate the extracted data with the internal systems for further processing and analysis.

By integrating data from hierarchical files

Integrating data from flat files is comparatively easier but business users face challenges when they try to extract, parse and integrate information from hierarchical data files such as XML, JSON, EDI and COBOL. To perform hierarchical data integration, business users rely on IT, which increases the burden on them. A data integration tool can effectively bridge this gap between business executives and IT.

Learn how Centerprise Data Integrator enables business users to work with hierarchical data, without the need for custom coding and programming, by downloading the whitepaper Hierarchical Data Integration for Business Users.

By making data readily available to business users

A data integration tool with a user-friendly interface and a comprehensive library of built-in functions can help limit the reliance on IT. It readily makes the data available to business users who can then work with the available information and get business insights without delay. Additionally, data integration tools can automate the ETL process, which eliminates the need for manual integration and significantly reduces the chances of errors.

The performance of a business is optimized when the executives are more focused on making critical business decisions rather than collecting and integrating the data.

By checking for data-quality

A data integration tool cleanses, validates and ensures the trustworthiness of the incoming data. Poor quality data can adversely affect business insights that can prove to be expensive for the business.

Overall, a data integration tool that simplifies the ETL process for the users is an investment that organizations should make to stay relevant in the current data driven business environment. It can prove to be beneficial for the business in more than one ways. By bridging the gap between IT and business executives, it helps in efficient division of workload. It empowers business users to drive insights from the data by giving them prompt access to it. And when executives delegate the task of data integration and extraction to software, they can focus on more critical aspects of the business. The result is faster and more accurate business decisions, minimized costs and increased revenue.

Astera’s Centerprise Data Integrator is a complete data integration solution that provides the mentioned benefits to its users, and more. The user-friendly interface and visual drag-and-drop environment eliminates the need for manual scripting and enables business users to work with data without relying on IT. Contact Astera’s sales and support to get more information.

Streamlining Data Extraction

Most of the crucial business data is stored in unstructured formats, while machines require structured data for processing. Businesses need data extraction tools to bridge this gap.

Unstructured Data | Data Extraction

Data extraction has evolved with technology, from manual extraction to complete automation. Constant innovation and developments in this field are making data extraction easier, flexible, and scalable for users.

Automation of Data Extraction

Previously, organizations were heavily dependent on manual extraction of data. In some cases, the IT department was responsible for writing custom scripts to extract data points, and in other, employees manually read through every document to extract data. In both cases, the data required further massaging based on the needs of end users, delaying business decisions.

Today, the key goal of a data extraction tool is to automate the entire process for its users. Template-based data extraction is a popular route to automation, giving greater control to users. It involves converting incoming documents using extraction templates which can be re-used for documents with similar layouts. Moreover, modern tools provide a Graphical User Interface (GUI) for the creation of these extraction templates, enabling business users to extract documents on their own, without the need to script or code.

Other than this, technologies like Natural Language Processing (NLP) enable computers to understand free-form text and make it analyzable through speech tagging, deep learning, text analytics, and other methods. Tools that leverage Machine Learning (ML) use algorithms to understand text structures and word morphology.

Automating data extraction process accompanies several benefits for businesses. Some of them are listed below:

  • Saves Time and Effort

Reusability of extraction templates for similar documents saves time and effort.

  • Faster Decision Making

Data can be processed in real time. This makes meaningful data readily available for business analysis, ensuring faster decision-making.

  • Streamlined Document Processing

Data patterns are used for recognizing documents and can allow for automatic classification of documents.

Conclusion

Automation has reshaped the business landscape. In today’s dynamic environment, it is important for businesses to focus on the quality and accessibility of data to stay ahead of their competitors. Accurate data can be made available in real-time through the automation of data extraction process.

For more information about how the concept of data extraction has evolved to meet modern business needs download the whitepaper ‘State of the Art of Data Extraction’.

Mainframe Modernization

Common Challenges of COBOL Data Extraction and How Centerprise Addresses Them

Although technologies like Ruby, Hadoop, and Cloud Computing continue to dominate headlines, there are still a large number of businesses that rely on legacy technologies. Many businesses, particularly those operating in the banking and insurance sector, use solutions that are COBOL-based.

According to Reuters, over 220 billion lines of COBOL code are in use today. As a result, a tremendous amount of data remains tied up in legacy systems. For any legacy modernization and BI initiative to be successful, it is important that this data must be integrated, transformed, and offloaded onto an analytics platform.

While extracting data from COBOL-based legacy applications is essential for improved decision-making, it remains a challenge for most businesses due to two primary reasons:

  • Shortage of COBOL Skills

There is a growing gap between the number of skilled COBOL programmers and organizations relying on the programming language. The average age of COBOL programmers is 55 years, and 70 percent of universities are favoring fancy languages like Java, C++, Linux, and UNIX over COBOL.

  • Need for Custom Programming

Analyzing data by directly querying the mainframe is a complex process. It requires custom development and therefore can be time-consuming and costly, with billing based on MIPS.

To addresses these two challenges, businesses need a solution that can fuel their data integration efforts, while ensuring data quality and reducing the need for hand-coding the processes.

How Centerprise Facilitates COBOL Data Extraction

Centerprise is a complete data integration solution that allows users to import data from a variety of sources, including legacy systems, transform it and write it to a destination of their choice. With its user-friendly, drag-and-drop interface and unparalleled data mapping capabilities, Centerprise makes the process of extracting data from COBOL-based systems simple, quick, and cost-effective.

cobol data extraction, legacy modernization

Centerprise offers complete support for COBOL data extraction with the functionality to:

  • Read a COBOL File — Centerprise features a high-speed COBOL file reader that can efficiently process large COBOL files.
  • Parse a Copybook — The built-in copybook parser reads a COBOL copybook and automatically builds the layout. When a copybook is not available, users can import a COBOL data file as a fixed length file and manually define field markers, data types, and numeric formats.
  • Identify USAGE, REDEFINES, and OCCURS — Centerprise offers support for different clauses used in a COBOL data file, including REDEFINES, OCCURS, and USAGE, such as COMP, COMP-3, COMP-5.

Once a COBOL data file has been imported, users can leverage the code-free, drag-and-drop environment of Centerprise to transform and write data to a destination of their choice.

Download our whitepaper to learn how Centerprise can help you combine legacy COBOL data with modern data streams and get a unified view of your information assets.

Automate Partner Data Exchange and Integration through Astera’s Customer Portal

Effective business intelligence demands proper data collection and integration processes. Modern businesses receive data in different formats from a variety of communication protocols, when efficient collaboration and tracking of data exchanges are more important than ever before.

Partner Data Exchange and Integration Automation

Partner Data Exchange – Centerprise Data Integration Software

As data increases, the maintenance and management of multiple files from disparate sources takes its toll on IT teams. File based partner data exchanges need to be fast, reliable, and secure for transfers, especially on large-scale operations. Manual data entry and file exchange over channels like email aren’t sufficient anymore. Businesses need a centralized platform where customers, partners, suppliers, and employees can upload data files, ensure compliance, and track file status.

Enter the new Astera Cloud Customer Portal.

Key features include:

Centralized Uploading

Partners upload data files directly on the portal URL using their unique login credentials. Validation and cleansing take place automatically on the backend Centerprise server using custom integration flows. A response file is generated at the end for both the admin and users.

File Tracking

The Customer Portal comes with inherent file tracking: all involved remain up-to-date on file statuses in real-time via the Dashboard. For unsuccessful uploads, the response file contains error codes and explanations. Status emails can also be configured for all types of accounts. This ensures all necessary files are uploaded correctly after data validation, allowing the final workflow to run as scheduled.

Security

Secure Sockets Layer (SSL) and file encryption are supported on the Customer Portal. Uploaded encrypted files are automatically decrypted, processed, and then re-encrypted by Centerprise to be sent back.

Scalability

The Customer Portal is built to handle unlimited data volume and borrows on Centerprise’s powerful scalability. Cloud servers can also be added for reliable load balancing.

White Label Capability

Astera Cloud’s Customer Portal is customizable, with the ability to use your company logo and domain name. Get all the functionality of Centerprise while maintaining brand cohesion.

For a live demo and information on pricing for our partner data exchange and integration solution, call us at +1 888-772-7837 or email us at sales@astera.com

Business Intelligence Tools and Centerprise

Astera Software’s Centerprise Data Integrator is an easy-to-use and robust data integration tool which can be used for ETL tasks, data quality and profiling. Centerprise seamlessly integrates with various data visualization and reporting tools to create a full-stack Business Intelligence process.

data-warehouseing-img

What is BI Software?

BI software is typically designed to analyze, transform and prepare data for reporting. BI tools use data which has been previously stored and which may or may not be stored in a data warehouse/data mart. BI tools enable users to transform raw data into legible information that enables businesses to make better decisions, which in turn allows for growth and increased revenue.

Centerprise is a complete data integration solution which can be used as a standalone product or can be integrated with other BI tools such as prediction analysis and data visualization tools to achieve a complete data mining and analytics process.

Centerprise provides a scalable, powerful, robust platform which helps in many processes, including: data extraction, data scrubbing, data warehousing, and data migration.

Some of the benefits of Centerprise include:

  • Support for complex hierarchical data
  • Easy to use graphical interface
  • Well-suited for business as well as technical users
  • High-performance and scalable
  • Built-in extensive set of data transformation features
  • Data profiling

    If you’re interested in seeing Astera Software products in action, check out our monthly webinar series! You can find out more on the Astera Software Events page.

Build vs. Buy: How it applies to Enterprise Software

Nowadays, enterprise software is used almost everywhere and at different levels within organizations. It is the lifeblood of most IT departments. One dilemma that tech company leaders face is whether to build a business solution (i.e. custom software) from scratch or to buy commercial, off-the-shelf (COTS) products and mold them according to business requirements.

Some key points when deciding whether to buy or build software are:

  1. The core business requirements, the scope of the problem in question, and the how complex the solution should be in order to fit the business needs and size
  2. Available resources and skills in terms of people, software, hardware, tools, etc. capable of building, maintaining, and supporting the business solution
  3. The amount of time required to develop the solution in-house

Companies that decide to build their own solution tend to overlook issues and expenses, such as:

  1. Replacing existing technology, adding features and functionality to a legacy system is extremely tough and a complete reengineering and rebuilding of business solutions may be needed
  2. A learning curve always comes with building a new software or recruiting additional people with specific skills to build in-house solutions
  3. Higher costs and potentially longer implementation compared to integrating COTS products

One of the biggest software building failure stories is the Bank of America’s MasterNet case study. According to “The Incremental Commitment Spiral Model: Principles and Practices for Successful Systems and Software” by Barry Boehm, Jo Ann Lane, Supannika Koolmanojwong, and Richard Turner:

“In the 1950s and 1960s, Bank of America (BofA) was the leading pioneer in banking automation with its electronic check processing capability. Subsequent BofA leaders had other interests, allowing BofA’s banking automation capabilities to degrade over time. In 1981, BofA’s new president, Sam Armacost, had an agenda to regain its automation leadership by “leapfrogging into the 1990s.” After an in-house effort that spent $6 million and failed to develop a workable trust management system, Armacost appointed a new executive vice president of the trust management department, Clyde Claus, with the charge of either modernizing the department or discontinuing it.

“…[However,] system problems continued…[and] clients began dropping off, with BofA’s base dwindling from 800 to 700 accounts and from $38 billion to $34 billion in institutional assets.… Eventually, in May 1988, BofA transferred its whole trust business to other banks, after an overall expenditure of $80 million and more than four years of project effort. The previous president, Tom Clausen, replaced Armacost in late 1986, and Claus resigned in October 1987.”

Advantages abound in buying COTS products and then customizing them. Some of the pros of buying software are:

  • Lower up-front costs
  • Clear and definite processes available for customization
  • Ready-made solutions available immediately
  • Less time has to be spent on customization and getting a product into the market
  • Readily available customer support

Although buying software may cost more than building a solution at the outset, it does provide better ROI over the long term.

Astera Software provides powerful, commercial, off-the shelf data extraction, integration and processing software including ETL tools which can be easily integrated and adapted by enterprises and companies to reduce time and effort required to perform data extracting and processing tasks and in turn improve revenue.

Some of Astera’s available software include:

centerpriselogo_hd_transparent

  • Centerprise Data Integrator delivers a powerful, scalable, high-performance, and affordable integration platform that is easy to use and cost friendly. It is robust enough to overcome even the biggest and most complex data integration challenges. A complete data integration solution, Centerprise includes data integration, data transformation, data quality, and data profiling in a flexible environment that enables users to choose from multiple integration scenarios. It comes with job scheduling and orchestration for automatic scheduling, file drop events, and API calls.

RMTextLogoTransparent

  • ReportMiner enables you to extract business data from PDF, TXT, XLXS, XLS, etc. formats. Data can be integrated into the main database system and used in electronic applications for business operations and business intelligence. ReportMiner provides an easy to use interface and helps the user to identify desired data, build the data extraction logic, and save the extracted data in a number of destinations. The best part about this product is that it can be used efficiently by business users with no technical background, but is robust enough for IT professionals.

edi-logo-trans

  • EDIConnect is a complete solution for handing EDI documents such as EDI 834 and 837. EDIConnect offers a user friendly and intuitive user interface to accurately and efficiently handle bi-directional EDI data integration. It is scalable and powerful enough to fulfill entire EDI transaction processes.

Astera Software also provides training and support. Customers can learn more about the products, view the demo, and reduce the learning curve. In conclusion, Astera Software provides easy to use, robust and manageable off-the shelf products to support companies and provide them with powerful data mapping, transformation and processing tools that are not only budget-friendly but also have a sizable advantage over building in-house business solutions.

EMR and HL7 within EDIConnect

In October, we went over two healthcare EDI transactions: EDI 834 and 837. This week, we’ll take a look at two more healthcare facets of EDIConnect: the ability to handle EMRs and compliance with HL7.

Electronic Medical Records (EMRs)

Electronic medical records (EMRs) are electronic records generated and maintained by hospitals and healthcare organizations. They are a vast improvement over paper records. They allow more than one person to use a patient’s chart, are better organized, eliminate illegible handwriting, and allow storage of more information.

The demand for healthcare systems with Electronic Medical Records (EMR) interfaces is rising. This increase is due to many factors, including the growing adoption of EMR systems and emerging clinical healthcare data standards such as HL7.

 

Health Level-7

Health Level-7(HL7) refers to a set of international standards for software applications used by healthcare providers for clinical and administrative transaction data. The HL7 standards are produced by the Health Level Seven International, an international standards organization, and are adopted by other standards issuing bodies such as American National Standards Institute and International Organization for Standardization.

HL7 International specifies a number of flexible standards, rules, and procedures for healthcare systems used by hospitals and other healthcare provider organizations to communicate with each other.  This helps information to be shared and processed in a uniform and consistent manner and allow hospitals and healthcare organizations to easily share clinical information.

Electronic data interchange (EDI) format also plays a very important role with EMR files and records. It helps to share clinical and other administrative data and medical records across different healthcare systems as well as individuals. Patients can also access their medical records through internet, allowing them to stay well-informed about their health status and ongoing medical treatments.

 

EDIConnect

Astera Software provides a complete solution for handing documents such as EMR and many others such EDI documents. EDIConnect offers a user friendly and intuitive user interface to accurately and efficiently handle bi-directional EDI data integration. It is scalable and powerful enough to fulfill entire EDI transaction processes.

 

EDIConnect Benefits:

  • Handles electronic medical records and HL7 documents.
  • Powerful and scalable
  • Fast, comprehensive and accurate data exchange
  • Easy to use graphical interface for both technical and business users.