Document structure
A document is a file that has been indexed by the document search engine. Every time a file is uploaded, updated or deleted in the Files API, it will also be scheduled for processing by the document search engine. After some processing, it will be possible to search for the file in the document search API. The document search engine is able to extract content from a variety of document types, and perform classification, contextualization and other operations on the file. This extracted and derived information is made available in the form of aDocument object.
The document structure consists of a selection of derived fields, such as the title, author and language of the document, plus some of the original fields from the raw file. The fields from the raw file can be found in the sourceFile structure. The derived fields are described in more detail below.
Derived fields
title
title
Some document types (such as PDFs) contain additional metadata fields. If the document contains its title as part of this metadata, this field will be populated with that title.
We do not currently extract the title from the document content itself. If there is a need for this, we may consider adding such functionality in the future.
producer
producer
The
producer field also exists in the document metadata. It contains information about the software or the system that was used to create the document.createdTime
createdTime
The
createdTime we assign to the document is not exactly the same as the one found in the Files API. We first try to extract the created time from the document metadata. If the document does not contain such a timestamp, we fall back to the time set in the Files API.mimeType
mimeType
If there is a mime type set on the file in the Files API, this field will be set to the same mime type. If there is no mime type set on the file, we will try to auto-detect it.
extension
extension
This field contains the extension of the file, derived from the file name. For instance, if the file name is
My Document.docx, the extension field will contain docx.pageCount
pageCount
Contains the number of pages in the document, if possible to determine.
type
type
The
type field contains a high-level file type, derived from the mime type. Mime types are not that pleasant to look at, and not always easy to understand. That is why we map the mime types into more user-friendly types. Below is the list of types currently returned, but be aware that this list may be extended in the future.geoLocation
geoLocation
If there is a geolocation set on the file in the Files API, then this field will contain the same geolocation. If there is no explicitly assigned geolocation, the document processing system will try to detect a location using two different techniques:
- Embedded GPS locations: Extract locations from files that contain embedded GPS locations. Photos and videos often have this kind of metadata.
- Related assets: Look at related assets that have locations, and assign the same location(s) to the document.
File type support
We create a document for each uploaded file, but only derive data from certain files. The following file types are eligible for further data extraction and enrichment:- PDF files
- Spreadsheets, documents, and presentations from the Microsoft, Libre Office and macOS office suites
- Plain text files
- Images
Automatic extraction
CDF automatically extracts text and metadata from uploaded files. Once ingested, documents are indexed for search. You can query by standard metadata fields or by semantic meaning using passage search.Organizing documents
Documents are associated with files in CDF. You can filter by source type, metadata, and content. Use aggregations to understand document distributions and properties across your corpus.Document content extraction happens asynchronously. Large files may take time to process before they appear in search results.
List and content
POST /documents/list pages through every document in the project. Unlike search, the cursor is not capped at 1000 documents. Filters follow document search.
The pipeline stores up to 1 MiB of extracted text per document. Search and list responses truncate that text. POST /documents/content returns the full text. Set the Accept header to text/plain. Any other value returns HTTP 406.
Document elements
POST /documents/elements returns the parsed layout of a PDF: titles, paragraphs, and tables, then lines and words. Each word has a bounding box with its page and position. Paragraph and line boxes are not returned. Compute them from the word boxes.
This operation is alpha-only. Send the cdf-version header with an alpha calendar version. Only PDF files are supported.
Aggregates
Whenaggregate is omitted, the endpoint counts the documents that match the filter or search. Supported aggregate types are described in Aggregation types. You can filter metadata keys or values by prefix before the documents are counted.
Key capabilities
- Search by metadata or content
- Semantic passage search for meaning-based retrieval
- Aggregate properties for analytics
- Retrieve document content for display or processing
Index status
Extraction runs asynchronously after a file is uploaded.POST /documents/status reports the index status for a set of documents so you can tell when those files are ready for search.