Batch Image Dataset Upload to Qdrant via Multimodal Embeddings - n8n Workflow

• Data scientists and ML engineers managing large image datasets.


• Automation specialists building RAG or search systems using vector databases like Qdrant.


• Users looking for advanced n8n templates integrating cloud storage, third-party AI APIs (Voyage AI), and vector databases.


• Developers needing a robust n8n node pipeline for data preprocessing and ingestion.

Managing large datasets for vector search or anomaly detection requires efficient batch processing. This specific n8n workflow addresses the challenge of ingesting image data from Google Cloud Storage into a modern vector database, Qdrant. By leveraging the power of multimodal embeddings from Voyage AI, this n8n automation extracts meaningful vectors from images (agricultural crops) in batches. The primary value of this n8n template is providing a resilient and structured process that handles collection creation conditionally, generates necessary UUIDs for Qdrant points, manages batch API limits, and ensures data integrity by attaching extracted metadata (like crop names) to the vector payloads. This robust n8n workflow is the first step in building applications like k-Nearest Neighbors (KNN) image classification or anomaly detection.

The n8n workflow begins with the 'When clicking ‘Test workflow’ n8n trigger.

1. Configuration: The workflow initializes global variables (Qdrant URL, collection name, embedding dimensions, and batch size) using a Set n8n node.


2. Qdrant Check: An HTTP Request n8n node checks if the target Qdrant collection already exists.


3. Conditional Setup: An If n8n node checks the existence result. If the collection is missing, an HTTP Request n8n node creates it, defining the voyage vector configuration (1024 dimensions, Cosine distance).


4. Indexing: Regardless of collection creation, a subsequent HTTP Request n8n node creates a payload index on the crop_name metadata field within Qdrant, optimizing future filtering operations.


5. Data Retrieval: The Google Cloud Storage n8n node fetches image metadata from the specified bucket (n8n-qdrant-demo) using the prefix agricultural-crops.


6. Metadata Processing: A Set n8n node generates the publicly accessible image link and extracts the cropName from the file path structure.


7. Data Filtering: A Filter n8n node explicitly excludes 'tomato' images to prepare the dataset for anomaly detection testing, ensuring the integrity of the subsequent steps in this n8n workflow.


8. Batching and UUID Generation: A Code n8n node (Python) dynamically splits the filtered items into manageable batches (size defined by the initial variables) and assigns a unique, immutable UUID for every point, essential for Qdrant batch upserts.


9. API Formatting: A Set n8n node restructures the batch data to conform strictly to the input requirements of both the Voyage AI embedding API and the Qdrant batch upload API.


10. Embedding: An HTTP Request n8n node calls the Voyage AI multimodal embeddings service, converting the batch of image URLs into 1024-dimensional vectors.


11. Batch Upload: The final HTTP Request n8n node sends the comprehensive batch request to Qdrant, combining the UUIDs, the generated vectors, and the associated metadata payloads, completing the high-volume data ingestion process facilitated by this n8n template.

To deploy this n8n workflow and run it successfully:

1. Import: Copy the provided JSON and import it into your n8n instance via the 'Workflows' screen using the 'New' -> 'Import from JSON' option.


2. Credentials: Set up the required credentials:

1. Variables Update: Modify the 'Qdrant cluster variables' n8n node to match your specific configuration (Cloud URL, desired Collection Name, and Google Cloud Storage prefix).


2. Activation: Ensure the n8n workflow is activated and run the 'When clicking ‘Test workflow’ n8n trigger to initiate the data pipeline.

This advanced n8n workflow utilizes several specialized n8n node types:

Manual Trigger n8n trigger: Initiates the n8n workflow on demand for testing or manual execution.
Set (Qdrant cluster variables) n8n node: Central hub for configuring global variables like qdrantCloudURL, collectionName, and batchSize.
HTTP Request n8n node (Check Collection Existence, Create Collection, Payload Index, Embed crop image, Batch Upload): These nodes handle all communication with external APIs (Qdrant and Voyage AI). Crucial configurations include defining the vector distance ('Cosine') during collection creation and configuring specific JSON bodies for batch embedding and batch upsert operations in Qdrant.
Google Cloud Storage n8n node: Fetches the list of image objects. Key configuration uses the listFilters parameter to specify the prefix of the dataset being ingested.
Filter (Filtering out tomato) n8n node: Uses a condition check (notEquals) on the extracted cropName to manage the dataset contents as required for the intended machine learning application.
Code (Split in batches, generate uuids) n8n node: Executes a Python script to handle complex data transformation, chunking the dataset into batches and generating UUIDs dynamically—a critical step that maximizes throughput for the vector database ingestion by leveraging the efficiency of the Qdrant batch upload API. This is a common pattern in effective n8n templates.

• Set (Batches in the API's format) n8n node: Prepares the final data structures using complex JSON expressions (=batchVoyage, =batchPayloadQdrant) necessary for the subsequent Voyage and Qdrant n8n node requests.