Document Archiving

Archive paper documents with ABBYY FineReader Engine: preprocess scans, run parallel recognition, and export to searchable PDF, PDF/A, or MRC-compressed PDF for long-term storage.

This scenario is used for processing paper documents to save them into a digital archive, especially when creating an archive of contracts, project documentation, invoices, certificates, etc.

In this processing scenario, paper documents are converted into non-editable digital copies containing all the document information in a searchable format. As a result of such processing, digital copies of documents may be easily found in an electronic archive using full-text search, document text segments may be copied, and documents may be sent by e-mail or printed out.

To create a digital copy, the document first needs to go through several processing stages, each of which has its own peculiarities:

1. Preprocessing of scanned images

   Scanned images may require some preprocessing prior to recognition, for example, if scanned documents contain background noise, skewed text, inverted colors, black margins, wrong orientation, or resolution.

2. Simultaneous recognition of a large volume of documents

   To extract text data from a document, it must be recognized. When processing a large volume of documents, simultaneous document processing may come in useful. In this case, analysis and recognition workload can be spread over the processor cores, which makes it possible to speed up processing.

3. Export to an archive format

   The recognized document is saved to a suitable storage format. The most convenient formats for storing documents are PDF, PDF/A, PDF, and PDF/A with MRC. When saving to these formats, one may use a mode, under which the text is placed underneath the document image — this enables full preservation of the document formatting and provides a full-text search. The MRC settings allow a significant reduction of file size without loss of visual quality. Also, when saving to the PDF format, one may customize the security settings of the document protecting it from unauthorized viewing and printing.

Scenario implementation

The code samples provided in this topic are Windows -specific.

Below is the detailed description of the recommended method of using ABBYY FineReader Engine 12 for creating digital copies of the documents for archiving. The proposed method uses processing settings that are most suitable for this purpose. In this implementation the document scanning phase is omitted. Please see Additional optimization for specific tasks below for the tips on implementing scanning.

Step 1. Loading ABBYY FineReader Engine

To start your work with ABBYY FineReader Engine, you need to create the Engine object. The Engine object is the top object in the hierarchy of the ABBYY FineReader Engine objects and provides various global settings, some processing methods, and methods for creating the other objects.

To create the Engine object, you can use the InitializeEngine function. See also other ways to load Engine object (Win).

C#

public class EngineLoader : IDisposable
{
    public EngineLoader()
    {
        // Initialize these variables with the full path to FREngine.dll, your Customer Project ID,
        // and, if applicable, the path to your Online License token file and the Online License password
        string enginePath = "";
        string customerProjectId = "";
        string licensePath = "";
        string licensePassword = "";
        // Load the FREngine.dll library
        dllHandle = LoadLibraryEx(enginePath, IntPtr.Zero, LOAD_WITH_ALTERED_SEARCH_PATH);
           
        try
        {
            if (dllHandle == IntPtr.Zero)
            {
                throw new Exception("Can't load " + enginePath);
            }
            IntPtr initializeEnginePtr = GetProcAddress(dllHandle, "InitializeEngine");
            if (initializeEnginePtr == IntPtr.Zero)
            {
                throw new Exception("Can't find InitializeEngine function");
            }
            IntPtr deinitializeEnginePtr = GetProcAddress(dllHandle, "DeinitializeEngine");
            if (deinitializeEnginePtr == IntPtr.Zero)
            {
                throw new Exception("Can't find DeinitializeEngine function");
            }
            IntPtr dllCanUnloadNowPtr = GetProcAddress(dllHandle, "DllCanUnloadNow");
            if (dllCanUnloadNowPtr == IntPtr.Zero)
            {
                throw new Exception("Can't find DllCanUnloadNow function");
            }
            // Convert pointers to delegates
            initializeEngine = (InitializeEngine)Marshal.GetDelegateForFunctionPointer(
                initializeEnginePtr, typeof(InitializeEngine));
            deinitializeEngine = (DeinitializeEngine)Marshal.GetDelegateForFunctionPointer(
                deinitializeEnginePtr, typeof(DeinitializeEngine));
            dllCanUnloadNow = (DllCanUnloadNow)Marshal.GetDelegateForFunctionPointer(
                dllCanUnloadNowPtr, typeof(DllCanUnloadNow));
            // Call the InitializeEngine function 
            // passing the path to the Online License file and the Online License password
            int hresult = initializeEngine(customerProjectId, licensePath, licensePassword, 
                "", "", false, ref engine);
            Marshal.ThrowExceptionForHR(hresult);
        }
        catch (Exception)
        {
            // Free the FREngine.dll library
            engine = null;
            // Deleting all objects before FreeLibrary call
            GC.Collect();
            GC.WaitForPendingFinalizers();
            GC.Collect();
            FreeLibrary(dllHandle);
            dllHandle = IntPtr.Zero;
            initializeEngine = null;
            deinitializeEngine = null;
            dllCanUnloadNow = null;
            throw;
        }
    }
    // Kernel32.dll functions
    [DllImport("kernel32.dll")]
    private static extern IntPtr LoadLibraryEx(string dllToLoad, IntPtr reserved, uint flags);
    private const uint LOAD_WITH_ALTERED_SEARCH_PATH = 0x00000008;
    [DllImport("kernel32.dll")]
    private static extern IntPtr GetProcAddress(IntPtr hModule, string procedureName);
    [DllImport("kernel32.dll")]
    private static extern bool FreeLibrary(IntPtr hModule);
    // FREngine.dll functions
    [UnmanagedFunctionPointer(CallingConvention.StdCall, CharSet = CharSet.Unicode)]
    private delegate int InitializeEngine(string customerProjectId, string licensePath, 
        string licensePassword, string tempFolder, string dataFolder, bool isSharedCPUCoresMode, 
        ref FREngine.IEngine engine);
    [UnmanagedFunctionPointer(CallingConvention.StdCall)]
    private delegate int DeinitializeEngine();
    [UnmanagedFunctionPointer(CallingConvention.StdCall)]
    private delegate int DllCanUnloadNow();
    // private variables
    private FREngine.IEngine engine = null;
    // Handle to FREngine.dll
    private IntPtr dllHandle = IntPtr.Zero;
    private InitializeEngine initializeEngine = null;
    private DeinitializeEngine deinitializeEngine = null;
    private DllCanUnloadNow dllCanUnloadNow = null;
}

C++ (COM)

// Initialize these variables with the path to FREngine.dll, your FineReader Engine customer project ID,
// and, if applicable, the path to the Online License token and Online License password
wchar_t* FreDllPath;
wchar_t* CustomerProjectId;
wchar_t* LicensePath;  // if you don't use an Online License, assign empty strings to these variables
wchar_t* LicensePassword;
// HANDLE to FREngine.dll
static HMODULE libraryHandle = 0;
// Global FineReader Engine object
FREngine::IEnginePtr Engine;
void LoadFREngine()
{
    if( Engine != 0 ) {
    // Already loaded
    return;
    }
    // First step: load FREngine.dll
    if( libraryHandle == 0 ) {
        libraryHandle = LoadLibraryEx( FreDllPath, 0, LOAD_WITH_ALTERED_SEARCH_PATH );
        if( libraryHandle == 0 ) {
            throw L"Error while loading ABBYY FineReader Engine";
        }
    }
    // Second step: obtain the Engine object
    typedef HRESULT ( STDAPICALLTYPE* InitializeEngineFunc )( BSTR, BSTR, BSTR, BSTR, 
        BSTR, VARIANT_BOOL, FREngine::IEngine** );
    InitializeEngineFunc pInitializeEngine =
    ( InitializeEngineFunc )GetProcAddress( libraryHandle, "InitializeEngine" );
    if( pInitializeEngine == 0 || pInitializeEngine( CustomerProjectId, LicensePath, 
        LicensePassword, L"", L"", VARIANT_FALSE, &Engine ) != S_OK ) {
    UnloadFREngine();
    throw L"Error while loading ABBYY FineReader Engine";
    }
}

Step 2. Loading settings for the scenario

ABBYY FineReader Engine enables loading of all processing settings that are most suitable for this scenario using the LoadPredefinedProfile method of the Engine object. This method receives the profile name as an input parameter. Please see Working with Profiles for more information.

ABBYY FineReader Engine supports 2 variants of settings for this scenario:

Profile name	Description
DocumentArchiving_Accuracy	The settings have been optimized for accuracy: Enables detection of maximum text on an image, including text embedded into the image. Full synthesis of the logical structure of a document is not performed. The profile is not intended for converting a document into RTF, DOCX, or text-only PDF. Use the document conversion profiles for such purpose.
DocumentArchiving_Speed	The settings have been optimized for processing speed: Enables detection of maximum text on an image, including text embedded into the image. Skew correction is not performed. Full synthesis of the logical structure of a document is not performed. The processes of document analysis and recognition are speeded up. The profile is not intended for converting a document into RTF, DOCX, or text-only PDF. Use the document conversion profiles for such purpose.

C#

// Load a predefined profile
engine.LoadPredefinedProfile("DocumentArchiving_Accuracy");

C++ (COM)

// Load a predefined profile
Engine->LoadPredefinedProfile( L"DocumentArchiving_Accuracy" );

If you wish to change processing settings, use appropriate parameter objects. Please see Additional optimization for specific tasks for further information.

Step 3. Loading and preprocessing the images

ABBYY FineReader Engine provides the FRDocument object which allows processing multi-page documents. Using of this object allows you to preserve the logical organization of the document.

To load images of a single document and preprocess them, you should create the FRDocument object and add images to it. You may do one of the following:

• Create the FRDocument object using the CreateFRDocumentFromImage method of the Engine object. This method creates the FRDocument object and loads images from the specified file.
• Create the FRDocument object with the help of the CreateFRDocument method of the Engine object, then add images to the created FRDocument object from file (use the AddImageFile, AddImageFileWithPassword, or AddImageFileWithPasswordCallback method of the FRDocument object).

C#

// Create the FRDocument object from an image file
FREngine.IFRDocument frDocument = engine.CreateFRDocumentFromImage( "C:\\MyImage.tif", null );

C++ (COM)

// Create the FRDocument object from an image file
FREngine::IFRDocumentPtr frDocument = Engine->CreateFRDocumentFromImage( L"C:\\MyImage.tif", 0 );

Step 4. Document recognition

To recognize a document, we suggest that the methods of the FRDocument object analysis and recognition be used. This object provides a whole array of methods for document analysis, recognition, and synthesis. The most convenient method allowing document analysis, recognition, and synthesis using just one method is the Process method. It also uses simultaneous processing features of multiprocessor and multicore systems in the most efficient manner. However, you can also perform consecutive preprocessing, analysis, recognition, and synthesis using Preprocess, Analyze, Recognize, and Synthesize methods.

C#

// Analyze, recognize, and synthesize the document
// There is no need for additional parameters because they are set up by processing profile
frDocument.Process( null );

C++ (COM)

// Analyze, recognize, and synthesize the document
// There is no need for additional parameters because they are set up by processing profile
frDocument->Process( 0 );

Step 5. Document export

To save a recognized document, you can use the Export method of the FRDocument object by assigning the FileExportFormatEnum constant as one of the parameters. In this scenario, you can save the document, for example, to the PDF format using MRC in the export mode PEM_ImageOnText (property TextExportMode of the PDFExportParams object). You can change the default parameters of export using the corresponding export object. Please see Additional optimization for specific tasks below for further information.

After you have finished your work with the FRDocument object, release all the resources that were used by this object. Use the IFRDocument::Close method.

C#

// Save a recognized document to an archive format (for example, PDF)
// Create a PDFExportParams object
FREngine.PDFExportParams exportParams = engine.CreatePDFExportParams();
// Set the necessary parameters
exportParams.MRCMode = FREngine.PDFMRCModeEnum.MRC_Auto;
exportParams.TextExportMode = FREngine.PDFExportModeEnum.PEM_ImageOnText;
// Use the parameters during export
frDocument.Export( "C:\\MyText.pdf", FREngine.FileExportFormatEnum.FEF_PDF, exportParams );
// Release the FRDocument object
frDocument.Close();

C++ (COM)

// Save a recognized document to an archive format (e.g., PDF)
// Create a PDFExportParams object
FREngine::IPDFExportParamsPtr params = Engine->CreatePDFExportParams();
// Set the necessary parameters
params->MRCMode = FREngine::MRC_Auto;
params->TextExportMode = FREngine::PEM_ImageOnText;
// Use the parameters during export
frDocument->Export(L"C:\\MyText.pdf", FREngine::FEF_PDF, params);
// Release the FRDocument object
frDocument->Close();

Step 6. Unloading ABBYY FineReader Engine

After finishing your work with ABBYY FineReader Engine, you need to unload the Engine object. To do this, use the DeinitializeEngine exported function.

C#

public class EngineLoader : IDisposable
{
    // Unload FineReader Engine
    public void Dispose()
    {
        if (engine == null)
        {
            // Engine was not loaded
            return;
        }
        engine = null;
        // Deleting all objects before FreeLibrary call
        GC.Collect();
        GC.WaitForPendingFinalizers();
        GC.Collect();
        int hresult = deinitializeEngine();
 
        hresult = dllCanUnloadNow();
        if (hresult == 0)
        {
            FreeLibrary(dllHandle);
        }
        dllHandle = IntPtr.Zero;
        initializeEngine = null;
        deinitializeEngine = null;
        dllCanUnloadNow = null;
        // throwing exception after cleaning up
        Marshal.ThrowExceptionForHR(hresult);
    }
    // Kernel32.dll functions
    [DllImport("kernel32.dll")]
    private static extern IntPtr LoadLibraryEx(string dllToLoad, IntPtr reserved, uint flags);
    private const uint LOAD_WITH_ALTERED_SEARCH_PATH = 0x00000008;
    [DllImport("kernel32.dll")]
    private static extern IntPtr GetProcAddress(IntPtr hModule, string procedureName);
    [DllImport("kernel32.dll")]
    private static extern bool FreeLibrary(IntPtr hModule);
    // FREngine.dll functions
    [UnmanagedFunctionPointer(CallingConvention.StdCall, CharSet = CharSet.Unicode)]
    private delegate int InitializeEngine( string customerProjectId, string LicensePath, string LicensePassword, , , , ref FREngine.IEngine engine);
    [UnmanagedFunctionPointer(CallingConvention.StdCall)]
    private delegate int DeinitializeEngine();
    [UnmanagedFunctionPointer(CallingConvention.StdCall)]
    private delegate int DllCanUnloadNow();
    // private variables
    private FREngine.IEngine engine = null;
    // Handle to FREngine.dll
    private IntPtr dllHandle = IntPtr.Zero;
    private InitializeEngine initializeEngine = null;
    private DeinitializeEngine deinitializeEngine = null;
    private DllCanUnloadNow dllCanUnloadNow = null;
}

C++ (COM)

void UnloadFREngine()
{
 if( libraryHandle == 0 ) {
  return;
 }
 // Release Engine object
 Engine = 0;
 // Deinitialize FineReader Engine
 typedef HRESULT ( STDAPICALLTYPE* DeinitializeEngineFunc )();
 DeinitializeEngineFunc pDeinitializeEngine =
  ( DeinitializeEngineFunc )GetProcAddress( libraryHandle, "DeinitializeEngine" );
 if( pDeinitializeEngine == 0 || pDeinitializeEngine() != S_OK ) {
  throw L"Error while unloading ABBYY FineReader Engine";
 }
 // Now it's safe to free the FREngine.dll library
 FreeLibrary( libraryHandle );
 libraryHandle = 0;
}

Required resources

You can use the FREngineDistribution.csv file to automatically create a list of files required for your application to function. For processing with this scenario, select in the column 5 (RequiredByModule) the following values:

Core

Core.Resources

Opening

Opening, Processing

Processing

Processing.OCR

Processing.OCR, Processing.ICR

Processing.OCR.NaturalLanguages

Processing.OCR.NaturalLanguages, Processing.ICR.NaturalLanguages

Export

Export, Processing

Export.Pdf

Export.Pdf, Opening.Pdf

If you modify the standard scenario, change the required modules accordingly. You also need to specify the interface languages, recognition languages and any additional features which your application uses (such as, e.g., Opening.PDF if you need to open PDF files, or Processing.OCR.CJK if you need to recognize texts in CJK languages). See Working with the FREngineDistribution.csv File for further details.

Additional optimization for specific tasks

Below is the overview of the Help topics containing additional information regarding customization of settings at different stages of document processing:

• Scanning - Windows Only

  • Scanning
    Description of the ABBYY FineReader Engine scenario for document scanning.

• Recognition

  • Tuning Parameters of Preprocessing, Analysis, Recognition, and Synthesis
    Customization of document processing using objects of analysis, recognition and synthesis parameters.

• Recognize handwriting
  The DocumentArchiving_*** profiles do not include handwritten or handprinted text recognition. If you need to recognize handwriting, set the DetectHandwritten property of the PageAnalysisParams object to TRUE.

• PageProcessingParams Object
  This object enables customization of analysis and recognition parameters. Using this object, you can indicate which image and text characteristics must be detected (inverted image, orientation, bar codes, recognition language, recognition error margin).

• SynthesisParamsForPage Object
  This object includes parameters responsible for restoration of a page formatting during synthesis.

• SynthesisParamsForDocument Object
  This object enables customization of the document synthesis: restoration of its structure and formatting.

• MultiProcessingParams Object - Implemented for Linux and Windows
  Simultaneous processing may be useful when processing a large number of images. In this case, the processing load will be spread over the processor cores during image opening and preprocessing, layout analysis, recognition, and export, which makes it possible to speed up processing.
  Reading modes (simultaneous or consecutive) are set using the MultiProcessingMode property, and the RecognitionProcessesCount property controls the number of processes which may be started.

• Export

  • Tuning Export Parameters
    Customization of document export using objects of export parameters.
  • PDFExportParams Object
    This object allows you to tune PDF (PDF/A) export with only several parameters.
  • To customize the PDF (PDF/A) format export mode, use the TextExportMode property of the PDFExportParams object, and to customize MRC settings, use the MRCMode property.
  • In addition, you can customize image export settings to ensure faster processing, additional reduction of file size, etc. For example, you can save a colored image as a grayscale, or black and white image, if this fits your scenario (use the Colority property of the PDFExportParams object).
  • You can change the image resolution in such a way that the resulting electronic copy may subsequently be printed out on a printer, viewed on a computer screen, or you can select low resolution allowing only for the reading of a text and providing very poor quality of graphics (use the Resolution and ResolutionType property of the PDFExportParams object).

• Separation into documents

  • Under this scenario, the batch of images may have to be separated into documents. ABBYY FineReader Engine 12 does not support automatic document separation. However, you can use ABBYY FlexiCapture Engine to implement automatic separation. The documents may be separated, for instance, based on the number of pages in a document or based on pages having separating barcodes. When implementing barcode separation, you can use the scenario for extraction of barcode values only from the document.

See also

Basic Usage Scenarios Implementation