Solving the Apple Pathology problem using Artificial Intelligence

Photo by Sara Cervera on Unsplash

Apple orchard productivity and quality are threatened by a silent foe: leaf diseases. These diseases can wreak havoc on trees, and traditional methods of diagnosis rely on manual scouting by human experts. This method, however, is both time-consuming and costly, posing a significant challenge for apple farmers. With the entire orchard’s survival at stake, there is an urgent need for a solution that can accurately diagnose apple leaf diseases in a timely and cost-effective manner.

Through time, countless jobs, hours, and production skills have been utilized for this cause of simply scanning apple tree leaves; in a world of growing demand, this certainly would not be the ideal way to thrive. With the population set to rise by another billion in the next few decades, we humans cannot afford to waste resourceful talent in agriculture. To provide for a thriving, healthy planet, we need to look at more efficient and swifter solutions.

Photo by Olav Ahrens Røtne on Unsplash

Current methods of solutions

As of now, there is much discussion about finding effective solutions to manage leaf diseases in apple orchards, with proposed strategies including genetic resistance through plant breeding, biological control through the introduction of natural predators or beneficial microorganisms, and remote sensing technologies such as drones or satellite imagery to detect early signs of disease. Humans worldwide have started to realize the value of the time wasted completing these farm-scanning tasks, which could undoubtedly be replaced, but the only question is, what would be the best alternative?

Here is an in-depth analysis of these prospect solutions:

Genetic Resistance: Plant breeders are working to create apple varieties that are genetically resistant to common leaf diseases. This can reduce the need for pesticides and other environmentally hazardous treatments. Furthermore, genetic resistance in apple varieties can improve the sustainability and resilience of apple orchards by reducing reliance on chemical treatments and improving tree health and productivity. The limit in the use of pesticides can also be an overall benefit for humans consuming the apples and for the surrounding environment and food webs, which may also thrive off of the plant.

Biological Control: Some farmers use biological control methods to manage leaf diseases, such as introducing natural predators or beneficial microorganisms. This can be a more environmentally friendly and long-term approach to disease management. Furthermore, biological control can also promote the biodiversity and ecological balance of the orchard ecosystem, which can positively impact soil health and nutrient cycling, as well as other ecosystem services provided by the orchard.

Remote Sensing: Remote sensing technologies, such as drones or satellite imagery, can be used in addition to machine learning algorithms to detect early signs of leaf disease. Farmers can then take action before the disease spreads and becomes more difficult to control. Farmers can have a more efficient and effective tool for managing leaf diseases in apple orchards by using remote sensing technologies, which can lead to higher crop yields, less pesticide use, and overall improved sustainability of the orchard ecosystem.

But could there be possibly more effective and Cost-efficient Solutions?

Photo by Diego PH on Unsplash

One particular solution not being explored is the influence artificial intelligence and machine learning can have on this prospect. Artificial intelligence and machine learning can potentially revolutionize how leaf diseases are managed in apple orchards by enabling faster and more accurate spotting of particular diseases and predicting the possibility of a future outbreak based on those results. Integrating artificial intelligence and machine learning into apple orchard leaf disease management can potentially benefit apple farmers. Machine learning models can detect early signs of diseases that are not visible to the human eye by analyzing images and data collected from orchards, allowing farmers to take immediate action to prevent the disease from spreading. Machine Learning can also optimize other aspects of apple production, such as yield forecasting, harvest timing, and fruit grading. By analyzing data from sensors and other sources, machine learning algorithms can provide farmers with insights into maximizing productivity while minimizing waste. However, there are some potential drawbacks to using artificial intelligence and machine learning in orchard management, such as the need for reliable data collection and skilled professionals to develop and implement the models. Nonetheless, the advantages of this technology make it a promising research and development area for the Apple industry. We have previously seen machine learning revolutionize manufacturing by implementing various techniques such as predictive maintenance, quality control, and supply chain optimization. Motivating the auto industry leaders to completely transform the car manufacturing process from the top down. Maybe it's time for Agriculture to have its own AI takeover.

My mini Project focuses on an Aspect of this Problem:

When coming across the perspective of individuals who have to go through this process of scouting apples daily, I thought there had to be a way to ease this process. Coming from an era of automation and technology, it seemed inevitable that this could be a possible alternative solution.

Keeping this thought in mind, I created a machine learning-based model that can accurately classify a given leaf image from the test dataset to a specific disease category and identify an individual disease from multiple disease symptoms on a single leaf image. This eliminates this problem of manual scouting, making work a lot easier for those scouting Apple Trees! This project can start a revolution in this field, making the production of goods in agriculture more efficient, effective, and sustainable.

I also envisioned using machine learning algorithms to help predict the likelihood of future disease outbreaks based on leaf types and other factors, enabling farmers to take preventive measures before the disease spreads and becomes more difficult to control.

Here’s how I got started: (Importing Libraries)

First, as usual, I got started by importing the necessary Libraries which I planned on using throughout the process of this code; here is what I got started with:

import numpy as np
import pandas as pd
%matplotlib inline
import seaborn as sns
import matplotlib.pyplot as plt
import cv2
import os
import warnings
warnings.filterwarnings('ignore')
import tensorflow as tf
import random
import albumentations as A
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.layers import Dense,Activation,Flatten, Conv2D, MaxPooling2D
from tensorflow.keras.models import Sequential
from tensorflow.keras.callbacks import ModelCheckpoint,EarlyStopping

These libraries all have different significations within the code

Overall, libraries are used in programming for data analysis and data visualization, but here is a brief:

• numpy: It is a Python library for numerical computations. It includes powerful tools for working with arrays and matrices, frequently used in scientific computing and data analysis.
• pandas: It is a data manipulation and analysis library. It includes tools for reading and writing data to and from various file formats, dealing with missing data, and performing data analysis tasks such as filtering, grouping, and merging.
• %matplotlib inline: It is a command that allows plots to be displayed within Jupyter notebooks.
• seaborn: It is a data visualization library that offers a high-level interface for creating informative and appealing statistical graphics.
• matplotlib.pyplot: It’s a Python library for creating static, animated, and interactive visualizations.
• cv2: It is a library for performing computer vision tasks such as image and video processing.
• os: It is a library that allows you to interact with the operating system by providing functions for accessing files and directories.
• warnings: It is a library that handles the program’s warnings.
• tensorflow: It is a deep learning model building and training library.
• random: It’s a library that can generate random numbers and sequences.
• albumentations: It is an image augmentation library that generates new training samples by applying random transformations to existing ones.
• ImageDataGenerator: It is a class in Keras, a high-level neural networks API, for generating batches of augmented image data.
• Dense, Activation, Flatten, Conv2D, MaxPooling2D: These are layer classes in Keras that are used for building convolutional neural networks.
• Sequential: It is a class in Keras for building sequential models, which are models where each layer is connected to the next one linearly.
• ModelCheckpoint, EarlyStopping: These are callback functions in Keras for saving the best model during training and stopping training early if the model's performance does not improve.

After this, I also imported some datasets I would be using; in this case, I imported training and testing images. The testing set consisted of three different types of images correlating to each disease type, whereas the training images were a set of 18000+ images of the disease scenarios. The images are classified as the following to signify what type of image it is:

• Scab
• Healthy
• Fog Eye Leaf Spot
• Rust
• Complex
• Powdery Mildew
• Scab Frogeye Leafspot
• Scab Frogeye Leafspot Complex
• Frogeye Leafspot Complex
• Rust Frogeye Leafspot
• Rust Complex
• Powdery Mildew Complex

After importing the required datasets using the following code:

train_image_path = '../input/plant-pathology-2021-fgvc8/train_images'
test_image_path = '../input/plant-pathology-2021-fgvc8/test_images'
train_df_path = '../input/plant-pathology-2021-fgvc8/train.csv'
test_df_path = '../input/plant-pathology-2021-fgvc8/sample_submission.csv'

Batch Visualization

I visualized the images using batch visualization, which utilizes my data frame, batch size, and image path. This segment aims to call the batch visualization function, which takes my df_train, batch size, and train image path. Here is a visualization:

def batch_visualize(df,batch_size,path):
    sample_df = df_train.sample(9)
    image_names = sample_df["image"].values
    labels = sample_df["labels"].values
    plt.figure(figsize=(16, 12))
    
    for image_ind, (image_name, label) in enumerate(zip(image_names, labels)):
        plt.subplot(3, 3, image_ind + 1)
        image = cv2.imread(os.path.join(path, image_name))
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        plt.imshow(image)
        plt.title(f"{label}", fontsize=12)
        plt.axis("off")
    plt.show()
    
batch_visualize(df_train,9,train_image_path)

This Python function definition has three arguments: df, batch size, and path.

The function takes a random sample of 9 images from the input df (a pandas data frame containing information about images and their labels). It uses matplotlib to display these images and their corresponding labels.

Here’s an explanation of what the code does:

1. sample_df = df_train.sample(9): The sample df variable is assigned a random sample of 9 rows (i.e., images) from the input DataFrame df train.
2. image_names = sample_df["image"].values: The values from the “image” column of the sample df DataFrame are extracted and assigned to the image names variable.
3. labels = sample_df["labels"].values: The values from the sample df DataFrame’s “labels” column are extracted and assigned to the labels variable.
4. plt.figure(figsize=(16, 12)): This function generates a new matplotlib figure with the dimensions 16x12 inches.
5. The following loop iterates over the 9 images and their corresponding labels, and for each image:

— — — plt.subplot(3, 3, image_ind + 1): Specifies the subplot’s position in the figure. The first argument 3 indicates the number of rows in the subplot grid, the second argument 3 indicates the number of columns in the subplot grid, and the third argument image ind + 1 indicates the index of the current subplot.

— — — image = cv2.imread(os.path.join(path, image_name)): OpenCV (cv2) reads the image file from the specified path and assigns it to the image variable.

— — — image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB): Converts the color space of the image from BGR to RGB.

— — — plt.imshow(image): Displays the image on the current subplot.

— — — plt.title(f"{label}", fontsize=12): Adds a title with the corresponding label to the current subplot.

— — — plt.axis("off"): Turns off the current subplot’s axis labels and ticks.

6. plt.show(): Displays the matplotlib figure with the images and labels from the 9 subplots.

The function is then called with three arguments: df train, 9, and train image path.

The first argument is presumably a pandas DataFrame containing image and label information, the second argument is the batch size (which is not used in this function), and the third argument is the path to the image directory.

Training and Validation

To simplify this even more, the data frame train and test were predefined as such:

df_train = pd.read_csv(train_df_path)
df_test=pd.read_csv(test_df_path)

The function pd.read csv(train df path) reads a CSV file from the path specified by the train df path and creates a pandas DataFrame from its contents.

Similarly, pd.read csv(test df path) reads another CSV file from the path specified by the test df path, creating a pandas DataFrame from its contents.

The DataFrames df train and df test that result can be used to manipulate and analyze the data in the CSV files.

The structure and content of the CSV files will determine the operations that can be performed on the DataFrames.

As a cause of this code, we could also visualize how each leaf looked and if they were accurately defined.

I also figured that the visualization process could be made a lot easier when incorporating a bar plot; thus, I utilized the Seaborn library, based on the data in the df_train DataFrame, to do this, here is what I came up with.

plt.figure(figsize=(15,12))
labels = sns.barplot(df_train.labels.value_counts().index,df_train.labels.value_counts())
for item in labels.get_xticklabels():
    item.set_rotation(45)

I used such methods to efficiently visualize all types of leaves within the model, leading to the slightly daunting experience of TensorFlow Data Generation. For this scenario, we will be putting into use the Image Data Generator into. This function's main utility is to generate batches of tensor image data with real-time data augmentation. To start off, I labeled some parameters, ranging from Height, Width, batch size, and Seed. Within the function, one of my motives is to also rescale the image from 0–1, while currently, the pixel values range from 0–255. I create my train data-gen by setting particular variables such as validation_split and shear_range, which will be used during my train_dataset phase. Similarly, I also go on to do this for the validation stage.

HEIGHT = 128
WIDTH=128
SEED = 45
BATCH_SIZE= 64

train_datagen = ImageDataGenerator(rescale = 1/255.,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    validation_split = 0.2,
    zoom_range = 0.2,
    shear_range = 0.2,
    vertical_flip = False)

train_dataset = train_datagen.flow_from_dataframe(
    df_train,
    directory = train_image_path,
    x_col = "image",
    y_col = "labels",
    target_size = (HEIGHT,WIDTH),
    class_mode='categorical',
    batch_size = BATCH_SIZE,
    subset = "training",
    shuffle = True,
    seed = SEED,
    validate_filenames = False
)

validation_dataset = train_datagen.flow_from_dataframe(
    df_train,
    directory = train_image_path,
    x_col = "image",
    y_col = "labels",
    target_size = (HEIGHT,WIDTH),
    class_mode='categorical',
    batch_size = BATCH_SIZE,
    subset = "validation",
    shuffle = True,
    seed = SEED,
    validate_filenames = False
)

test_datagen = ImageDataGenerator(
    rescale = 1./255
)
INPUT_SIZE = (HEIGHT,WIDTH,3)
test_dataset=test_datagen.flow_from_dataframe(
    df_test,
    directory=test_image_path,
    x_col='image',
    y_col=None,
    class_mode=None,
    target_size=INPUT_SIZE[:2]
)

This code prepares image datasets for deep learning model training and testing. It employs the Keras ImageDataGenerator class to augment images on the training dataset and rescale pixel values on the testing dataset.

The variables HEIGHT, WIDTH, SEED, and BATCH SIZE are constants for the image height and width, the seed value for random number generation, and the batch size for training.

To add variety to the training dataset, the ImageDataGenerator object is created with various data augmentation parameters such as rotation range, width shift range, height shift range, horizontal flip, zoom range, and shear range. It also specifies a validation split of 0.2, which means that 20% of the training data will be used for validation during training.

The flow_from_dataframe() method creates training and validation datasets from the df_train DataFrame. It determines the directory where the images are located, the columns for the images and labels, the target image size, the class mode as categorical, the batch size, and the subset as “training” or “validation,” depending on the use case, and whether to shuffle the data and validate the filenames. A separate test_datagen object is created for the testing dataset, which only rescales the pixel values of the images.

The flow from dataframe() method is also employed in generating the testing dataset from the df test DataFrame. It specifies the image directory, the image columns, the target size, and the class mode as None because we are not dealing with labels.

Finally, the variable INPUT SIZE is defined as a tuple of (HEIGHT, WIDTH, 3), where 3 denotes the number of color channels in the images (RGB).

Building the CNN

At this stage, the sequential model would come into play, wherein I added the required models for building my Convolutional Neural Network.

model=Sequential()
model.add(Conv2D(32,(3,3),activation='relu',padding='same',input_shape=(HEIGHT,WIDTH,3)))
model.add(MaxPooling2D(2,2))
model.add(Conv2D(64,(3,3),activation='relu',padding='same'))
model.add(MaxPooling2D(2,2))
model.add(Conv2D(64,(3,3),activation='relu',padding='same'))
model.add(MaxPooling2D(2,2))
model.add(Conv2D(128,(3,3),activation='relu',padding='same'))
model.add(MaxPooling2D(2,2))
model.add(Flatten())
model.add(Dense(12,activation='softmax'))

To preserve the spatial dimensions of the input, the Conv2D function adds a convolutional layer with 32 filters, each with a size of 3x3, a ReLU activation function, and the same padding. (HEIGHT, WIDTH, 3) is the input shape, with 3 representing the number of color channels in the image (RGB).

To downsample the feature maps, the MaxPooling2D function adds a pooling layer with a pooling window of size 2x2.

This is repeated for three more convolutional layers, each with 64, 64, and 128 filters, followed by another MaxPooling2D layer after each convolutional layer.

To prepare for the fully connected layer, the Flatten function reshapes the output of the last pooling layer into a 1D array.

The Dense function adds a fully connected layer with 12 neurons (equal to the number of classes in the dataset) and a softmax activation function to generate class probabilities.

Overall, the model extracts features from the input image using convolution and pooling operations, followed by a fully connected layer that classifies the image into one of the 12 classes.

Compilations

After completing that, I compiled the overall code for the model using the Adam optimizer alongside the loss function categorical_crossentropy.

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss='categorical_crossentropy',
    metrics=['accuracy'])
model.summary()

model.compile Configures the model for training by specifying the optimizer, loss function, and evaluation metrics.

The Adam optimizer is used in this case, with a learning rate of 0.001. Because this is a multi-class classification problem, the categorical cross-entropy loss function is used.

The accuracy metric is used to evaluate the model.

model.summary() Prints a summary of the model architecture, including the number of trainable parameters in each layer and the total number of trainable parameters in the model. This is useful for understanding the model's complexity and debugging any issues with the model's structure.

Checkpoints

To be safe, I also implemented model checkpoints on the neural network. Model checkpoints are important in neural networks because they allow the model to be saved at various points during training. This is useful in several ways:

1. Resuming training: Model checkpoints can be used to resume training from where it was left off rather than starting from scratch if training is interrupted due to a power outage or other similar reasons.
2. Saving the best model: During training, the model may overfit or underfit the data. The best model can be saved based on validation accuracy or loss by using model checkpoints. This ensures that the best possible model is used for prediction.
3. Experimentation: Model checkpoints enable you to experiment with various hyperparameters and architectures. Different models can be trained with different hyperparameters, and model checkpoints can be used to compare each model’s performance.
4. Deployment: Model checkpoints can be used to deploy the trained model for inference, where new data is fed into the model to generate predictions.

Here is how I implemented the checkpoint to the code:

checkpoint_path = "training_1/cp.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)

# Create a callback that saves the model's weights
cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,
                                                 save_weights_only=True,
                                                 verbose=1)

The callback saves the model’s weights at certain intervals during training so that the weights can be loaded later for evaluation or further training.

Specifically, the code creates a callback called a sample of the tf.keras.callbacks.ModelCheckpoint class. Soon after, the filepath the parameter determines the path where the checkpoint will be saved. In this example, the checkpoint will be reserved at the path "training_1/cp.ckpt". The save_weights_only parameter is set to true, meaning only the model weights will be saved, not the entire model architecture. The verbose parameter is set to 1, meaning the callback will print a message when a checkpoint is saved.

Later, during training, this callback can be handed to the fit model method as a list, and the model will preserve the weights at specified intervals during training, allowing the training process to be resumed from where it left off if needed.

Fitting:

Finally, using model.fit generator, I can fit my model employing the parameters such as train dataset, validation data, validation dataset, and more, I can train the model.

model_history=model.fit_generator(train_dataset,
                                  validation_data=validation_dataset,
                                  epochs=5,
                                  steps_per_epoch=train_dataset.samples//128,
                                 validation_steps=validation_dataset.samples//128,
                                 callbacks=[cp_callback]
                                 )

The fit_generator function trains the model on batches of data generated by the train_dataset and validation_dataset generators.

The train_dataset and validation_dataset generators were created earlier using the ImageDataGenerator.flow_from_dataframe method and have been predefined. These generators generate batches of image data and their corresponding labels.

The epochs parameter specifies the number of times to iterate over the entire training dataset. In this case, the model is trained for five epochs. However, this function can be interchangeable based on preference.

The steps_per_epoch parameter specifies the number of batches of data to be processed in each epoch. In this case, the train_dataset.samples parameter specifies the total number of images in the training dataset, which is divided by the BATCH_SIZE of 64 to get the number of batches per epoch. This, too, can be interchangeable based on your liking.

The validation_steps parameter defines the number of batches of data to be processed for verification in each epoch. In this case, validation_dataset.samples parameter specifies the total number of images in the validation dataset, which is divided by the BATCH_SIZE of 64 to get the number of batches for validation per epoch.

The callbacks parameter specifies a list of callbacks to be used during training. In this case, the list includes only the cp_callback which is the model checkpoint callback described earlier. This will save the model weights at the end of each epoch to be loaded later if needed.

The model_history object returned by the fit_generator method contains the training and validation loss and accuracy values for each epoch. These can be used to evaluate the model's performance during training and diagnose any issues.

Prediction

After creating the network, I used the system to predict the state of particular using the press function and chose the argmax value of the output predictions.

train_dataset.class_indices.items()

preds = model.predict(test_dataset)
print(preds)

preds_disease_ind=np.argmax(preds, axis=-1)

preds_disease_ind

train_dataset.class_indices.items() Returns a dictionary-like object containing the class labels and their corresponding numerical index values generated during the creation of train_dataset. This is useful for mapping the numerical output of the model back to their respective class labels. These numeral values are the ones used when a prediction is being made.

model.predict(test_dataset) Generates predictions for the test dataset using the trained model. The output is an array of predicted probabilities for each image in the test dataset. Each array row corresponds to an image, and each column corresponds to a class label. The probabilities generated utilize the numerical values.

np.argmax(preds, axis=-1) Returns the indices of the maximum value along the last axis of the preds array. This means finding the class label index with the highest predicted probability for each image in the test dataset. This is mapping out the possible likely outcomes and narrowing down your results.

preds_disease_ind Is an array of predicted class labels, where each element corresponds to an image in the test dataset. These predicted labels are generated by taking the argmax of the predicted probabilities for each image.

After this process, you will get the result of your predicted leaf types! Demonstrating whether or not the leaves are safe.

Takeaways:

Photo by Unseen Studio on Unsplash

After constructing this project, it was evident that such a model could do miracles in the real world. However, this was only a test prototype; the impact that such a model could have on predicting apple tree leaf types could save plenty of valuable time and energy for working farmers, and the implementation of such a model could possibly come into play by utilizing image capturing technology, which could possibly capture the features of the apple tree leaves which make up a particular area. This information can then possibly be fed into a neural network like such, which can scan the area and select the healthy trees for you, allowing the farmers to emphasize their time and effort for enhancing the condition of the trees which are not healthy instead of scanning and differentiating the trees.

Here are some skills and insights which I developed through the coding procedure:

Throughout the process, I learned how to use new libraries like Keras, TensorFlow, and Argumentation to preprocess data and build a neural network.

In addition, I improved my debugging skills by identifying and resolving several errors during the coding process.

One aspect of the coding process that I found particularly difficult was ensuring the data was properly preprocessed. This is a critical step in developing a precise and effective neural network. It allowed me to discover the significance of resizing images and employing data augmentation techniques; it was surprising to see how this was like a hack to improve the model's overall accuracy.

Self Reflection

When looking back at this project, it is not very challenging to readily put to use across countless farms worldwide. Artificial Intelligence has provided us with a medium to thrive, and it would be best if we utilize this medium for the right causes. When looking at the evolution of AI through time, it is fascinating to see how far we have come and how this concept can undoubtedly be used to our advantage, such as in this situation; we just need to carefully consider how AI is employed to ensure it is used in ways that are beneficial to society.

If innovations like such were to be implemented in the real world, it would revolutionize traditional farming and irrigation methods. Farmers can gain greater insights into soil health, crop growth, and water usage by leveraging such technologies. Ultimately leading to more efficient and sustainable farming ready to provide for a growing human population. Finally, a sustainable future is the only future.