Data Migration Framework

Framework & ETL Tools

Prorigo’s DMF (Data Migration Framework) can migrate data and files from any source system to target system. Framework uses ETL based tools to extract, transform, and load data and files. ETL tools have been built for most of the famous PLM systems, by just configuring the tools, data migration can be done with minimal effort. Data extractors can be created for any CRM, ERP, Teamwork Cloud etc. with minimal effort. Framework supports adding new systems by just developing the plug-ins. Framework and tools have been tested with many customers and with massive amounts of data. Framework and tools have been used by US Defense Organizations and have been scanned for software vulnerability assessment.

Prorigo Data Migration Framework
DMF - DATAFLOW
DMF Data Flow

Benefits

  • Framework based on ETL methodology, with a collection of fast, scalable and configurable tools.​
  • Configuration driven, with ready to use tools for various established PLM applications.​
  • Supports all type of files and metadata, including history and revisions.​
  • Designed with unique techniques such as direct database access for high performance, and use of multiple parallel threads for scaling up the process.​
  • Can migrate datasets with over 100 million records in 16 hours.​
  • End-to-end data migration solution, with support for extraction and loading of full or incremental (specific date and time) datasets.​
  • Detailed and real-time diagnostics information is provided to review progress of various activities.​
  • Data migration can be scheduled as an automated process by chaining the tools for delta migration at regular intervals.​
  • Tried and tested with many complex and large volume data migration projects.​
DMF – DETAILED DATAFLOW
DMF Detailed Data Flow

Data Extractor

  • Extracts metadata and files with a single command for a pre-defined process area (many types).
  • Extraction is driven by config file that defines all types, relationships, formats (files) for extraction.
  • Supports extraction of object data, relationship data, files, and history.
  • Supports full or incremental extraction (date range).
  • Supports parallel extraction (reduces overall extraction time).
Data Extractor
Data Extractor
  • Extraction based on filters like vault, policy, state, date range etc.
  • Connects directly to the database to optimise extraction process instead of using API.
  • Provides diagnostic information to give precise idea about extraction status and count.
  • Can extract up to 100 million records in a span of 8 hours.
  • Sample Extractor Command.
    ProrigoEnoviaOracleDataExtractor --fromDate "2022-05-08 14:40:52" --types “ManufacturingPart, Part, Document”

Data Transformer

  • Command-line tool, that reads flat files (extracted source data) and transform to a form, suitable for loading the data to the target system.
  • Datatype validation and massaging for object attribute values.
  • Supported Transformations for CAD models include generation of Thumbnails, Derived Outputs, Viewables.
  • SubComponent Rollup for CAD Models.
  • Classification transformation.
Data Transformer
Data Transformer
  • Splitting items from source system to multiple items and relationships.
  • Merging items and relationships from source system.
  • Supports parallel transformation (reduces overall extraction time).
  • Transformation is driven by config file that defines transformer sequence, source/output folders, parameters etc.
  • Provides diagnostic information to give precise idea about transformation status and count.
  • Sample Transformer Command.
    ProrigoEnoviaToArasTransformer --transformType “CadMajorMinor, CadRollup”​

Data Loader

  • Loads data from transformed files to target database with a single command.
  • Tool is driven by config file that defines all types, relationships, formats (files).
  • Supports loading of object data, relationship data, files, revisions, and history.
  • For loading each type data, it uses SSIS (SQL Server Integration Services) package. SSIS is essentially a part of the Microsoft SQL Server database program that is used to carry out large-scale data movement.
  • Tool can use prebuilt SSIS package or given the configuration it can build the package on the fly.
  • Handles load dependencies among types and relationships by sequencing their loads.
  • Copies the files directly to InnovatorVault server by creating their vault folder structure.
Aras Data Loader
Aras Data Loader
  • SSIS package has logic to resolve any dependencies current type data has with other types data. For example, while loading specific part we may have to lookup in the user table to update creator property on the part.
  • Using SSIS package approach provides seamless and high speed data connectivity with Microsoft SQL server.
  • Supports incremental updates over already migrated data.
  • Supports parallel data load if multiple types are configured.
  • Can load up to 100 million objects within a span of 8 hours.
  • Sample Loader Command.
    ProrigoSsisDataLoader --types “ManufacturingPart, Part, Document”​