Including Diagrams in Federated Search

We are excited to announce our latest improvement to our federated manual search – an AI computer vision integration which can understand manuals and include them in the materials it searches through and returns to you, the user.

‍
Including Diagrams in Federated Search

‍
As a crucial component of manual’s content, diagrams are invaluable to include in the search process and present as part of the answer to the user. We implement this with a GPT-based image-to-text model. Here’s how it works:
Processing a new manual:

1. Convert the diagram image into a detailed description of what is going on in the diagram. Include the name of the diagram in question to be retrieved by the federated search engine.
2. Insert the text in place of the diagram, mark the text as describing a diagram and include information on where to find the original image.

Processing a new query:

1. Run the federated search algorithm.
2. If a diagram is included in relevant results, it will be converted back to image form, and a coherent string of text and images will be presented back to the user for the answer response.

We value versatility, and, like most of our implementations, our approach is model-agnostic: whatever the leading image-to-text model out there is can be plugged into our codebase, out of the box. Likewise, it does not matter what the format of the image is in the manual. Partners also have full control over how detailed the model should be in its conversions and whether it should pay attention to anything in particular during new manual processing.

‍
We look forward to continuing to develop the ability of our framework to master the complexity of manuals and offer an effective copilot in repair jobs.

‍