Machine-Readable Zone Capture

Capture the machine-readable zone (MRZ) on passports and ID documents with ABBYY FineReader Engine — extract per-field data conforming to ICAO Document 9303.

The official travel or identity documents of many countries contain a machine-readable zone (MRZ) that ensures more accurate processing of the document data. The MRZ includes 2 or 3 lines with the OCR-B font text written in accordance with ICAO Document 9303 (see the specifications on the ICAO website).

This scenario is used for extracting data from a machine-readable zone on ID documents during customer onboarding or verification processes. The system recognizes MRZ on the document image and extracts the data from it. The extracted data contains several fields with the personal information about the document and its holder (document’s type and expiry date, the first and the last names of the document holder, etc.). You may search through the fields, verify the data and save it to an external file for further processing.

To extract the data from MRZ, image files obtained by scanning or saved in the electronic format typically go through several processing stages, each of which has its own peculiarities:

1. Preprocessing of scanned images or photos

   You either scan or take a photo of an ID document’s identity page with MRZ. Photos made with digital cameras of mobile devices may have low resolution and quality. Also, images may require some preprocessing prior to recognition.

2. Extracting data from MRZ

   No more than one MRZ may be captured from each image. The text of each of the 2 or 3 lines will be recognized and parsed to extract the data fields. Some of the fields and the MRZ as a whole have checksums, which will help you to verify the data.

3. Export to an external file

   You may also save the extracted data in an external format: XML and JSON are supported.

The procedure described below is implemented in the MRZExtraction code sample for Linux and Windows.

Scenario implementation

The code samples provided in this topic are Windows -specific.

Below is the detailed description of the recommended method of using ABBYY FineReader Engine 12 in this scenario. The proposed method uses processing settings that are most suitable for this scenario.

Step 1. Loading ABBYY FineReader Engine

To start your work with ABBYY FineReader Engine, you need to create the Engine object. The Engine object is the top object in the hierarchy of the ABBYY FineReader Engine objects and provides various global settings, some processing methods, and methods for creating the other objects.

To create the Engine object, you can use the InitializeEngine function. See also other ways to load Engine object (Win).

C#

public class EngineLoader : IDisposable
{
    public EngineLoader()
    {
        // Initialize these variables with the full path to FREngine.dll, your Customer Project ID,
        // and, if applicable, the path to your Online License token file and the Online License password
        string enginePath = "";
        string customerProjectId = "";
        string licensePath = "";
        string licensePassword = "";
        // Load the FREngine.dll library
        dllHandle = LoadLibraryEx(enginePath, IntPtr.Zero, LOAD_WITH_ALTERED_SEARCH_PATH);
           
        try
        {
            if (dllHandle == IntPtr.Zero)
            {
                throw new Exception("Can't load " + enginePath);
            }
            IntPtr initializeEnginePtr = GetProcAddress(dllHandle, "InitializeEngine");
            if (initializeEnginePtr == IntPtr.Zero)
            {
                throw new Exception("Can't find InitializeEngine function");
            }
            IntPtr deinitializeEnginePtr = GetProcAddress(dllHandle, "DeinitializeEngine");
            if (deinitializeEnginePtr == IntPtr.Zero)
            {
                throw new Exception("Can't find DeinitializeEngine function");
            }
            IntPtr dllCanUnloadNowPtr = GetProcAddress(dllHandle, "DllCanUnloadNow");
            if (dllCanUnloadNowPtr == IntPtr.Zero)
            {
                throw new Exception("Can't find DllCanUnloadNow function");
            }
            // Convert pointers to delegates
            initializeEngine = (InitializeEngine)Marshal.GetDelegateForFunctionPointer(
                initializeEnginePtr, typeof(InitializeEngine));
            deinitializeEngine = (DeinitializeEngine)Marshal.GetDelegateForFunctionPointer(
                deinitializeEnginePtr, typeof(DeinitializeEngine));
            dllCanUnloadNow = (DllCanUnloadNow)Marshal.GetDelegateForFunctionPointer(
                dllCanUnloadNowPtr, typeof(DllCanUnloadNow));
            // Call the InitializeEngine function 
            // passing the path to the Online License file and the Online License password
            int hresult = initializeEngine(customerProjectId, licensePath, licensePassword, 
                "", "", false, ref engine);
            Marshal.ThrowExceptionForHR(hresult);
        }
        catch (Exception)
        {
            // Free the FREngine.dll library
            engine = null;
            // Deleting all objects before FreeLibrary call
            GC.Collect();
            GC.WaitForPendingFinalizers();
            GC.Collect();
            FreeLibrary(dllHandle);
            dllHandle = IntPtr.Zero;
            initializeEngine = null;
            deinitializeEngine = null;
            dllCanUnloadNow = null;
            throw;
        }
    }
    // Kernel32.dll functions
    [DllImport("kernel32.dll")]
    private static extern IntPtr LoadLibraryEx(string dllToLoad, IntPtr reserved, uint flags);
    private const uint LOAD_WITH_ALTERED_SEARCH_PATH = 0x00000008;
    [DllImport("kernel32.dll")]
    private static extern IntPtr GetProcAddress(IntPtr hModule, string procedureName);
    [DllImport("kernel32.dll")]
    private static extern bool FreeLibrary(IntPtr hModule);
    // FREngine.dll functions
    [UnmanagedFunctionPointer(CallingConvention.StdCall, CharSet = CharSet.Unicode)]
    private delegate int InitializeEngine(string customerProjectId, string licensePath, 
        string licensePassword, string tempFolder, string dataFolder, bool isSharedCPUCoresMode, 
        ref FREngine.IEngine engine);
    [UnmanagedFunctionPointer(CallingConvention.StdCall)]
    private delegate int DeinitializeEngine();
    [UnmanagedFunctionPointer(CallingConvention.StdCall)]
    private delegate int DllCanUnloadNow();
    // private variables
    private FREngine.IEngine engine = null;
    // Handle to FREngine.dll
    private IntPtr dllHandle = IntPtr.Zero;
    private InitializeEngine initializeEngine = null;
    private DeinitializeEngine deinitializeEngine = null;
    private DllCanUnloadNow dllCanUnloadNow = null;
}

Step 2. Loading settings for the scenario

You can load the processing settings suitable for this scenario using the LoadPredefinedProfile method of the Engine object. This method uses the name of a settings profile as an input parameter. Please see Working with Profiles for more information.

The settings for this scenario are available in the MachineReadableZone predefined profile:

• Enables detection and extraction of all text on an image (pictures, vector graphics and tables are not detected).
• Resolution and geometry correction are performed automatically.

C#

// Load the predefined profile
engine.LoadPredefinedProfile("MachineReadableZone");

If you wish to change processing settings, use appropriate parameter objects. Please see Additional optimization for specific tasks below for further information.

MRZ capture is only possible if your ABBYY FineReader Engine license supports the MRZCapture module.

Step 3. Loading and preprocessing the document images

To load images to FineReader Engine, you can use the methods of these objects:

• FRDocument
• BatchProcessor for Linux and Windows

Linux and Windows users can learn about the advantages and disadvantages of each approach in Parallel Processing with ABBYY FineReader Engine. The current topic focuses on FRDocument.

To load images to the FRDocument object, do one of the following:

• When creating the FRDocument object, use the CreateFRDocumentFromImage method of the Engine object.
• Add images to the created FRDocument object from file (use the AddImageFile, AddImageFileWithPassword, or AddImageFileWithPasswordCallback method).

All these methods take as an input parameter a PrepareImageMode object which allows you to specify different parameters of image preprocessing. Create this object by calling the IEngine::CreatePrepareImageMode method, then change its properties as necessary and pass it to the image opening method.

C#

// Create document
FREngine.IFRDocument frDoc = engine.CreateFRDocument();
// Add image file to the document
document.AddImageFile( imagePath, null, null );

Step 4. Extracting data from MRZ

To extract data from MRZ:

1. [optional] Create the MrzProcessingParams object with the help of the CreateMrzProcessingParams method of the Engine object. Set up its properties to the values you need.
2. Call the ExtractMrz method of the FRPage object, passing the MrzProcessingParams object you configured in the previous step as the input parameter; for default MRZ capture settings, simply pass NULL. You will receive an MrzData object containing the parsed information from the captured MRZ.

C#

// Extract MRZ
FREngine.IFRPage page = document.Pages.Item(0);
FREngine.IMrzData mrzData = page.ExtractMrz( null );

Step 5. Working with the extracted data

The MrzData object contains all the data extracted from the MRZ. You may access the lines of machine-readable text using the GetLine method and iterate through fields using the GetField and GetFieldByType methods. The fields of the following types are extracted:

• Document type
• Document subtype
• Issuing country
• Last name
• Given name
• Document number
• Nationality
• Birth date
• Sex
• Expiry date
• Personal number
• Optional data line 1
• Optional data line 2

The MrzField object provides the complete information about the extracted field. Use its Text property to get the field value and the Region property to get the field location on the image. To verify the data, use the checksums available through the Checksum, HasChecksum and IsChecksumVerified properties of the MrzData and MrzField objects. Not all the field types support checksums; the HasChecksum property is provided so that you can check its value before trying to retrieve the checksum.

You may edit the recognized text of the fields manually using the Insert and Remove methods of the MrzField object.

C#

// Verify the checksum of the document number
FREngine.IMrzField documentNumberField = mrzData.GetFieldByType(FREngine.MrzFieldTypeEnum.MFT_DocumentNumber);
bool isNumberVerified = documentNumberField.IsChecksumVerified;

Step 6. Exporting the extracted data

The extracted data may be saved to an XML or JSON file. To export the data with the default parameters, call the ExportToFile method of the MrzData object and pass the path to the file as an input parameter. To export the data with the user-defined parameters, call the ExportToFileEx method of the MrzData object and pass the pointer to the MrzJsonExportParams object or the MrzXmlExportParams object as an input parameter.

C#

// Save to XML format
mrzData.ExportToFile("C:\\ExtractedData.xml", FREngine.MrzExportFormatEnum.MEF_Xml);

Step 7. Unloading ABBYY FineReader Engine

After finishing your work with ABBYY FineReader Engine, you need to unload the Engine object. To do this, use the DeinitializeEngine exported function.

C#

public class EngineLoader : IDisposable
{
    // Unload FineReader Engine
    public void Dispose()
    {
        if (engine == null)
        {
            // Engine was not loaded
            return;
        }
        engine = null;
        // Deleting all objects before FreeLibrary call
        GC.Collect();
        GC.WaitForPendingFinalizers();
        GC.Collect();
        int hresult = deinitializeEngine();
 
        hresult = dllCanUnloadNow();
        if (hresult == 0)
        {
            FreeLibrary(dllHandle);
        }
        dllHandle = IntPtr.Zero;
        initializeEngine = null;
        deinitializeEngine = null;
        dllCanUnloadNow = null;
        // throwing exception after cleaning up
        Marshal.ThrowExceptionForHR(hresult);
    }
    // Kernel32.dll functions
    [DllImport("kernel32.dll")]
    private static extern IntPtr LoadLibraryEx(string dllToLoad, IntPtr reserved, uint flags);
    private const uint LOAD_WITH_ALTERED_SEARCH_PATH = 0x00000008;
    [DllImport("kernel32.dll")]
    private static extern IntPtr GetProcAddress(IntPtr hModule, string procedureName);
    [DllImport("kernel32.dll")]
    private static extern bool FreeLibrary(IntPtr hModule);
    // FREngine.dll functions
    [UnmanagedFunctionPointer(CallingConvention.StdCall, CharSet = CharSet.Unicode)]
    private delegate int InitializeEngine( string customerProjectId, string LicensePath, string LicensePassword, , , , ref FREngine.IEngine engine);
    [UnmanagedFunctionPointer(CallingConvention.StdCall)]
    private delegate int DeinitializeEngine();
    [UnmanagedFunctionPointer(CallingConvention.StdCall)]
    private delegate int DllCanUnloadNow();
    // private variables
    private FREngine.IEngine engine = null;
    // Handle to FREngine.dll
    private IntPtr dllHandle = IntPtr.Zero;
    private InitializeEngine initializeEngine = null;
    private DeinitializeEngine deinitializeEngine = null;
    private DllCanUnloadNow dllCanUnloadNow = null;
}

Required resources

You can use the FREngineDistribution.csv file to automatically create a list of files required for your application to function. For processing with this scenario, select in the column 5 (RequiredByModule) the following values:

Core

Core.Resources

Opening

Opening, Processing

Processing

Processing.OCR

Processing.OCR, Processing.ICR

Processing.OCR.NaturalLanguages

Processing.OCR.NaturalLanguages, Processing.ICR.NaturalLanguages

Export

Export, Processing

If you modify the standard scenario, change the required modules accordingly. You also need to specify the interface languages, recognition languages and any additional features which your application uses (such as, e.g., Opening.PDF if you need to open PDF files). See Working with the FREngineDistribution.csv File for further details.

Additional optimization

These are the sections of the Help file where you can find additional information about setting up the parameters for the various processing stages:

• Loading Engine
  • Different Ways to Load the Engine Object for Windows
    Describes the ways of loading Engine object in detail.
  • Using ABBYY FineReader Engine in Multi-Threaded Server Applications for Windows
    Discusses the specifics of using FineReader Engine in server applications.
  • Working with Profiles
    Provides detailed description of predefined and user profiles.
• Recognition
  • Tuning Parameters of Preprocessing, Analysis, Recognition, and Synthesis
    Customization of document processing using objects of preprocessing, analysis, recognition, and synthesis parameters.
• Working with the extracted data
  • Machine-Readable Zone Fields
    The list of fields, which may be extracted from a machine-readable zone by means of ABBYY FineReader Engine 12, and their brief descriptions.
  • Working with Text
    Working with the recognized text, paragraphs, words, and characters.
  • Using Voting API
    Working with words and character recognition alternatives.

See also

Basic Usage Scenarios Implementation