Computer Vision Image Classification with Nifi, CDSW, Kudu, Solr, and HDFS

Computer vision applications often involve processing large amounts of image and video data, and require efficient storage, retrieval, and analysis of this data.

Image Classifier takes an input image and uses algorithms to identify and classify the image.

Architecture

Services Used

• Nifi is used to collect and manage data from various sources. In this use case, Nifi is used to collect images from various sources, including external sources and HDFS, and then process and transfer the data to the next service in the pipeline.

• HDFS is used for large-scale data storage and processing. In this use case, HDFS is used to store the images and the results of the image classification process for later retrieval and analysis.

• Kudu, a fast and flexible database, is used for fast and efficient data storage and retrieval. In this use case, it is used to store the processed images and the results of the image classification process.

• CDSW is used for data science workbench. In this use case, it is utilized for the development and execution of machine learning models. The data collected from Nifi is passed to CDSW for processing and model creation.

• Solr is used for full-text search and indexing. In this use case, Solr is used to index the image data and results, making it easily accessible and searchable.

Artifacts

Create a Table in Kudu

SQL to create a table in Hue.

create table computer_vision.image_predictions(
request_id string,
requested_timestamp timestamp,
hdfs_location string,
image_size string,
image_extension string,
image_dimension string,
requestor_ip string,
prediction_value string,
prediction_timestamp timestamp,
prediction_duration_in_seconds INTEGER,
content_type string,
image_name string,
PRIMARY KEY(request_id, requested_timestamp)
)  
STORED AS KUDU
TBLPROPERTIES ('kudu.num_tablet_replicas' = '1');

Drop Table

drop table computer_vision.image_predictions