Import flat files

You can import documents into the application using flat files, or text-only load files. You can use the flat file import process with the following flat file types:

Metadata flat file: DAT, CSV, or TXT files only.

Image flat file: OPT or LFP files only. The image flat file links the metadata flat file to associated TIFF images.

When you import flat files, you can copy all associated full-text OCR and native source files to the application. When you add a flat file job, the application automatically saves all job settings and field mappings for later use.

Flat file import guidelines

Use the following guidelines to load non-native data into the application using the flat file import feature.

Data load file (.DAT file): The first line in the load file contains the field names. Each subsequent line in the load file represents the metadata for a document. Standard delimiters include the following options:

Field delimiter: ( ¶ ) character 020

Text qualifier: ( þ) character 254

New line character: ( ® ) character 174

Multi Value delimiter: ( ; ) semicolon

The application also accepts load files with a .TXT or .CSV extension.

Date/time values: Always use separate fields for date and time values. For example, “Date Modified” and “Time Modified” must be separate fields, and always contain only the date and time, respectively.

Image reference file (.OPT or .LFP): The application requires a standard format .OPT or .LFP file for loading of images. The image reference file contains path and document break information for images. The number of lines in the OPT file must match the number of images provided. Always ensure the number associated with the first page of each document matches the value in a field for the corresponding row in the DAT file. We recommend that you name the first page of the document with Document ID and Control Number.

Document text: Provide document text as separate text files. Use one text file per document and name each file using the Document ID and Control Number. A field in the DAT file (or other data load file) must reference the relative path to the text file. The application also accepts document text in a single field in the DAT file.

Images: Document images must be copied to the exact path in the OPT file. Always verify that the number of lines in the OPT file matches the number of images.

Native files: If native files are provided, the DAT file (or other data load file) must have a field with the relative path to the native file. Always name native files using the Document ID and Control Number of the document.

Flat file import process

To create a flat file import:

On the Case Home page, under Manage Documents, click Imports.

Click Add.

Note: As you complete each page in the Import window, the application saves the details. To close the window, you can click Cancel or submit the job. To retain the in-progress settings for your the job, in the Import window, on the Schedule page, select Place on hold, and then click Save as draft.

Complete the Import Details page

in the Import window, on the Import Details page, use the following procedure to complete the information required.

In the Job type list, select Flat file.

Type a Name for the flat file import job.

Optionally, type a Description.

To update the All Custodians or related fields for new documents in the import job, select Update group coding fields. Selecting this option also updates the fields for existing and future family duplicate documents in the application.

To run an indexing and enrichment job immediately after the import job completes, select Run indexing and enrichment. For more information about indexing and enrichment, see Index and analyze documents.

In the Date format list, select the default date format for the imported files.

Note: The application supports all date formats. The application uses the default date format only when a date is unclear, for example, 06/05/15.

Click Next.

Complete the Metadata File page

In the Import window, on the Metadata File page, use the following procedure to complete the information required. 

Select the metadata file (.DAT, .CSV, .TXT) for import:

In the File repository list, select the file repository to import the data to. This list contains the file transfer repositories configured for your case.

In the Folder name list, select a folder.

In the File name list on the right, select the metadata file that you want to import.

Click Next.

Complete the Image File page

On the Image File page, you can select an image file (.OPT or .LFP) for import.

In the Import window, on the Image File page, use the following procedure to complete the information required.

Select Import image file.

In the Folder name list, select a folder, and in the File name list on the right, select the image file that you want to import.

Click Next.

Complete the Source Files page

In the Import window, on the Source Files page, use the following procedure to include source files with the import.

Select Copy source files to the server.

To overwrite matching source files, select Overwrite matching source and/or content files already on the server.

In the Folder name list, select a folder.

Click Next.

Complete the Delimiters page

On the Delimiters page, you can set the delimiter character that separates field values in the flat file. You can also set the text qualifier that encloses the field value. After setting the delimiters, use this page to search for fields in the flat files. You can proceed to the next page only after the field search.

In the Import window, on the Delimiters page, use the following procedure to complete the information required.

In the Column - fields list, select the delimiter character for columns and fields. For example, if the flat file is delimited by commas, select , (44).

In the Text qualifier list, select the text qualifier character, if required.

In the Row - documents list, do not change the default <CR>/<LF> value.

To search the flat file for available fields, click Get fields.

Note: Due to RPF server demands, this process can take a while. If you select an incorrect delimiter, an error message appears. If you change the import details or delimiters, click Get fields again to run the search again.

Click Next.

Complete the Map Files page

In the Import window, on the Map Files page, use the following procedure to configure the location from which the application copies the associated native or full-text files.

If you are copying source files, in the Native path list, select the field to access native files. This option appears only when you select Copy source files to the server on the Source Files page.

Note: The selected field must contain the relative path to the source files that you set on the Source Files page.

Under Full text, select one of the following options:

None: Do not import full-text files.

Metadata field: If you are importing text content in the flat file, select the field that contains this content. The application uses the selected field values to create associated text files for searching.

File path: If you are importing full-text or OCR files, in the File path list, select the field that will reference the path to these files.

Click Next.

Complete the Existing Documents page

In the Import window, on the Existing Documents page, use the following procedure to determine how the application handles the import of existing documents.

Note: This page is similar to the Import documents feature. For more information, see Import documents.

Select one of the following Import action options:

Abort import, if it contains an existing document ID

Import new documents only: Data for documents that already exist in the case is ignored.

Update existing documents only

Import new documents and update existing documents

If you select Update existing documents only or Import new documents and update existing documents, additional options appear. See the following table for descriptions of the options.

Options	Descriptions
For existing documents, update	To update the listed fields, select Document Date, Description, Title, Type, or Source and attachments. To use null values to represent missing data when you update the selected fields, select Allow NULL updates (for fields above). For example, if a load file contains a document with a null value as its title, then the existing document is updated or overwritten with the null title. To update levels, select Levels.
Fields data	Do not update fields: Does not update data for existing documents. Only new documents contain import fields. Delete all field values. Do not update fields: Deletes all field values for documents in the import data set. When selecting this option, none of the updated documents will have import field values. Note: This option only clears field values for documents in the import data set. For example, if a case contains existing documents A, B, and C, and the import data set contains documents C, D, and E, selecting Delete all field values does not affect documents A and B. It clears the metadata only for document C. Update only mapped fields: Deletes field values for documents in the case, and then adds those fields from the import data set to the case. This applies only to mapped fields. All other fields remain the same. Append only mapped fields: Adds new field values for one-to-many fields for documents in the import data set. Existing values in one-to-one fields do not change. This only applies to mapped fields. All other fields remain the same.
Parties data	Do not update any parties: Does not update parties data in existing documents. Delete all party values. Do not update parties: Deletes all parties data values for documents in the import data set. When selecting this option, none of the updated documents will have parties data values. Update all mapped parties: Deletes the parties data values for documents in the import data set, and then adds new parties data values to the case. This option applies only to mapped parties data in the target data set. Append all mapped parties: Adds new mapped parties data values to documents and removes all parties information for updated documents.
Pages data	Do not update pages: Does not update pages data in documents. Delete all page values. Does not update pages: Deletes all pages data values for documents in the import data set. When selecting this option, none of the updated documents have pages data values. Update all pages: Deletes the pages data values for documents in the import data set, and then adds new pages data values to the case. This applies to both mapped pages data and all other pages data values in the target data set. Append all pages: Either overwrites new pages data fields in documents or overwrites fields for documents in the import data set.

Click Next.

Complete the Levels page

On the Levels page, you can set the levels into which the application imports documents. By default, the maximum number of documents is 1,000 per level, but this is configurable. When the number of imported documents exceeds the set value, the application creates additional levels at the same depth, and appends each level with a number, for example, Level_0001, Level_0002. The application truncates level names that are longer than 20 characters.

In the Import window, on the Levels page, use the following procedure to set the levels.

To set the maximum number of documents per level, in the Max documents per level list, select a number.

Do one of the following:

To select an existing level, in the Level list, select the root and sublevels. To create a new level for the imported documents, the application uses the name of the selected level and adds the suffix "_00001."

Note: You must select at least one level.

Tip: If there are many levels, filter the list by entering the first few letters of the level name.

To create a new level, in  the  Level list, select Add new level, and then type the name of the level. The new level inherits the parent or root-level security settings.

Note: After you create a level, always synchronize the security groups. For information about syncing security groups, see Sync security.

If the load file has a field with level name values, such as "Custodian," in the Level list, select Use data from load file, and then select the field.

Click Next.

Complete the Document ID page

On the Document ID page, you can set the Document ID fields that the application uses to identify imported files.

Use the value of a field in the load file as the Document ID

To use the value of a field in the load file as the Document ID:

On the Document ID page, select Existing Document ID. 

See the following table for descriptions of the options that appear on this page.

Option	Description	Additional options
Document ID	Selects the metadata field with the document ID.
Image Document ID	Selects the metadata field that contains the value that links to the image load file. Usually, this field is the Document ID.
Source and Attachment relationship	Selects the method that will be used to define the relationships between source and attachment documents. Available options are: None LFP: Use the child designation from the LFP load file to create the source and attachment relationships. Parent Doc ID: If this option is selected, the application expects attachment documents to contain the Document ID of the parent document in the selected field. Attachment Doc IDs: If this option is selected, the application expects parent documents to contain the Document IDs of the attachment documents. If there are multiple attachments for one parent document, separate the Document IDs by a comma (,) or a semi-colon (;). Family Range: If this option is selected, the application expects all documents to contain a range of Document IDs, for example, DOC_0000001 - DOC_0000015. The first Document ID is the parent, and all other Document IDs between the first and last, including the last, are attachments.	If you select Parent Doc ID, Attachment Doc IDs, or Family Range, the Field defining Source and Attachment relationship list appears.  The options in this list are based on the column names of the load file. The selected option determines the field in the load file that references the other documents in the family.

Click Next.

Use levels as the Document ID

To use the levels selected on the Levels page as the Document ID:

On the Document ID page, select Get Next - Levels Based. 

Note: The application assigns the level names and a five-character suffix as the ID number for the first document you upload. This number increments as you upload subsequent documents.
For example, If you select Docs_1, Docs_2, and Docs_3 as levels, the application generates the document ID as Docs_1.Docs_2.Docs_3.00001, Docs_1.Docs_2.Docs_3.00002, and Docs_1.Docs_2.Docs_3.00003.

See the following table for descriptions of the options that appear on this page.

Option	Description	Additional options
Level separator	Separates the levels in the Document ID. This option uses a period by default, but you can replace the period with a different single character.
Page length	Sets the number of padding zeroes to use for the page number portion of the document ID of the imported documents
Image Document ID	Selects the metadata field that contains the value that links to the image load file. Usually, this field is the Document ID.
Source and Attachment relationship	Selects the method that will be used to define the relationships between source and attachment documents. Available options are: None LFP: Use the child designation from the LFP load file to create the source and attachment relationships. Parent Doc ID: If this option is selected, the application expects attachment documents to contain the Document ID of the parent document in the selected field. Attachment Doc IDs: If this option is selected, the application expects parent documents to contain the Document IDs of the attachment documents. If there are multiple attachments for one parent document, separate the Document IDs by a comma or a semi-colon. Family Range: If this option is selected, the application expects all documents to contain a range of Document IDs, for example, DOC_0000001 - DOC_0000015. The first Document ID is the parent, and all other Document IDs between the first and last, including the last, are attachments.	If you select Parent Doc ID, Attachment Doc IDs, or Family Range, the following lists appear: Field defining Source and Attachment relationship: The options in this list are based on the column names of the load file. The selected option determines the field in the load file that references the other documents in the family. Source and Attachment Document ID: The options in this list are based on the column names of the load file. The selected option determines which field to use as the initial document ID when determining source and attachment relationships.

Click Next.

Use a sequential number as the Document ID

To use a sequential number as the Document ID:

On the Document ID page, select Get Next - Sequential. 

The application generates the document ID based on existing documents in the level.

See the following table for descriptions of the options that appear on this page.

Option	Description	Additional options
Document ID prefix	Allows you to create a custom document ID prefix. The application assigns a sequential number to the prefix. For example, if the prefix is AA, the application generates the document ID as AA000000001, AA000000002, and AA000000003.
Image Document ID	Selects the metadata field that contains the value that links to the image load file. Usually, this field is Document ID.
Source and Attachment relationship	Selects the method that will be used to define the relationships between source and attachment documents. Available options are: None LFP: Use the child designation from the LFP load file to create the source and attachment relationships. Parent Doc ID: If this option is selected, the application expects attachment documents to contain the Document ID of the parent document in the selected field. Attachment Doc IDs: If this option is selected, the application expects parent documents to contain the Document IDs of the attachment documents. If there are multiple attachments for one parent document, separate the Document IDs  by a comma or a semi-colon. Family Range: If this option is selected, the application expects all documents to contain a range of Document IDs, for example, DOC_0000001 - DOC_0000015. The first Document ID is the parent, and all other Document IDs between the first and last, including the last, are attachments.	If you select Parent Doc ID, Attachment Doc IDs, or Family Range, the following lists appear: Field defining Source and Attachment relationship: The options in this list are based on the column names of the load file. The selected option determines the field in the load file that references the other documents in the family. Source and Attachment Document ID: The options in this list are based on the column names of the load file. The selected option determines which field to use as the initial document ID when determining source and attachment relationships.

Click Next.

Complete the Field Map page

On the Field Map page, use the following procedure to map the import fields to case fields, binders, or issues.

Note: If you are mapping one-to-many fields, you can select delimiters for the fields. Delimiters are not required for one-to-one fields.

To ignore imported fields that you do not want to map or match to existing import fields, click Apply at the top of the page. The application will change all Action column values from New to Ignore if no Case field is selected.

For each Import field, do the following:

To change the mapping of an import field, click a value in the Action column, and then select one of the following options:

New: Creates a new field in the case database on import.

Ignore: Excludes the field from the import.

Import to: Allows you to select the case field to map to.

In the Case field column, select a binder, issue, or field value that corresponds to the action selected for the import field.

To define a character to use as a delimiter, click the field in the Delimiter column and select a character. The default value is ; (59).

To set a default value for a field in an imported document, click the field in the Default value column and then type a value.

Click Next.

Complete the Field Trumping Order page

On the Field Trumping Order page, you can determine the order in which the import fields will populate the application case field. Case fields affected by many-to-one mapping for the import job appear in the Case field pane on the left, if the case field is a one-to-one field.

Note: If the case field is a one-to-many field, values from all mapped import fields will be added to the case field.

In the Import window, on the Field Trumping Order page, use the following procedure to determine the order in which the import fields will populate the application case field.

To view and reorder the import fields, click the name of a case field. The order in which the import fields are used to populate the case field appears in the Rank column.

To change ranking order, click on the handle of an import field row and drag it up or down.

Click Next.

Complete the Schedule page and finish the import job request

In the Import window, on the Schedule page, use the following procedure to set the start time for the import job, place the job on hold to save as a draft job for later use, and finish the import job request.

To schedule a time, in the Start time box, select a date. In the second box, select a time.

To submit the job for processing immediately, leave the default Now setting.

To place a job on hold for later, select Place on hold, and then click Save as draft. You can return to a draft job at any time from the Imports page.

To schedule the job to run as a test, select Schedule as a test run only.

Click Finish.

Note: If you schedule the job to run as a test, when you click Finish, the application adds the word TEST at the beginning of the description of the import on the Imports page.

The application returns to the Imports page and indicates that your job is scheduled. The application saves your job definitions for processing. To view the import job details, click the name of the import job. The application displays the details in the Properties page.

View flat file import results

When the flat file import job completes, it appears on the Imports page. Note that jobs saved as draft jobs also appear on the Imports page, but will not have run.

To view the flat file job details, click the Name of the import job. The application displays the details on the Properties page.

If the flat file import job completes with errors, view the errors on the Errors page.

After the import job finishes, you can view the documents on the Documents page, in the List pane.