Cover image

Multimodal Data Fusion For
Precision Agriculture

Lucas Waltz, Sushma Katari, Chaeun Hong, Adit Anup, Julian Colbert, Anirudh Potlapally, Taylor Dill, Canaan Porter, John Engle, Christopher Stewart, Hari Subramoni, Raghu Machiraju, Osler Ortez, Laura Lindsey, Arnab Nandi, and Sami Khanal.
Cover image

Our goal is to produce AI-powered insights from the fusion of multimodal data in agriculture to benefit farmers.

Cover image

We believe that the most impactful AI use cases in agriculture are not the ones that try to tell people what to do, but the ones that provide explainable insights to enable better decisions that maximize profit and manage risk for each unique situation.

Publications

Lucas Waltz, Sushma Katari, Chaeun Hong, Adit Anup, Julian Colbert, Anirudh Potlapally, Taylor Dill, Canaan Porter, John Engle, Christopher Stewart, Hari Subramoni, Raghu Machiraju, Osler Ortez, Laura Lindsey, Arnab Nandi, and Sami Khanal. 2024. Cyberinfrastructure for Machine Learning Applications in Agriculture: Experiences, Analysis, and Vision. In Proceedings of the 16th International Conference on Precision Agriculture (unpaginated, online), Monticello, IL. International Society of Precision Agriculture.

Lucas Waltz, Sushma Katari, Taylor Dill, Canaan Porter, Osler Ortez, Laura Lindsey, Arnab Nandi, and Sami Khanal. 2024. A Growth Stage Centric Approach to Field Scale Yield Estimation for Corn Leveraging Machine Learning Methods From Multimodal Data. In Proceedings of the 16th International Conference on Precision Agriculture (unpaginated, online), Monticello, IL. International Society of Precision Agriculture.

Multimodal Data

We routinely source multimodal data from three Ohio State agricultural research stations geographically dispersed across Ohio. Each site includes 80 plots for corn and 80 plots for soybeans. The data is spatiotemporal in nature, it is indexed by both plot and time. The following data was collected via sensors:

  • Aerial RGB Imagery (via UAV)
  • Aerial Multispectral Imagery (via UAV)
  • Timelapse Imagery (downward facing to detect uniformity of emergence)
  • Smartphone Images/Videos
  • Soil Moisture (via in-situ sensors)
  • Matric Potential (via in-situ sensors)
  • Canopy Microclimate (via in-situ sensors)
  • Photosynthetic Active Radiation (via in-situ sensors)
  • Meteorological Data (via on-site weather stations)
  • Soil Available Nitrogen and Soil Respiration levels (via routine soil tests)

Additionally, the following data was collected manually:

  • Growth Stage Identification
  • Weed-growth Evaluation
  • Disease Incidence and Severity Analysis

A generic square placeholder image with rounded corners in a figure.
UAV taking flight.
A generic square placeholder image with rounded corners in a figure.
UAV image with plot orthomosaic.
A generic square placeholder image with rounded corners in a figure.
In-situ soil sensor control box.

Machine Learning

The multimodal data is used to power machine learning. Currently, three accurate models have been created for one of the research sites. Details are provided below. We are currently working on extending this to the other research sites.

  • Growth Stage Classification using Vision Transformer (ViT)
  • Soil Moisture Prediction using Recurrent Neural Network (RNN)
  • Corn Kernel Weight Prediction using Multi-layer Perceptron (MLP)

Figures showing results are depicted below.

A generic square placeholder image with rounded corners in a figure.
ViT Results.
A generic square placeholder image with rounded corners in a figure.
RNN Results.
A generic square placeholder image with rounded corners in a figure.
MLP Results.

Data Visualization

We envision an accessible platform for farmers to receive AI-powered insights and interact with all of the multimodal data. To do this, we are developing a web-based data visualization that will enable farmers to engage with the information accross spatial and temporal dimensions.

A concept dashboard will be completed and shared here in Summer 2024. Some concepts and wireframes are provided below.

A generic square placeholder image with rounded corners in a figure.
Wireframe concept.
A generic square placeholder image with rounded corners in a figure.
Web-based growth stage visualization.

Videos

Supporters

Nationwide Logo
Ohio State Logo
Ohio Farm Bureau Logo

Funding for this project was provided by the Nationwide AgTech Innovation Hub and leveraged the Battle for the Belt research plots.

Contributors

Luke Waltz Luke Waltz is a PhD student whose research interests are centered around using technology and artificial intelligence to benefit farmers, with a specific focus on nitrogen use efficiency. Prior to entering the PhD program, he spent more than two decades leading product and engineering teams for an industrial equipment manufacturer, with responsibility for a broad range of product and technology platforms and product lines.
Dr. Arnab Nandi Dr. Nandi’s expertise is in the area of data infrastructure, specifically in use cases involving humans-in-the-loop. Most recently, Prof. Nandi spun out his research on spatiotemporal analytics into a connected vehicle data infrastructure startup, Mobikit. The startup focused on interactive analysis of vehicle risk through the fusion of a variety of data sources and statistical models, such as vehicle sensor data, GPS trajectories, weather data, and location-specific risk profiles.
Dr. Sami Khanal Dr. Khanal is an assistant professor in the Department of Food, Agricultural, and Biological Engineering at OSU. The focus of her teaching and research includes the application of remote sensing (satellite and UAV) and GIS techniques/tools, ecosystem modeling, and data analytics in agricultural production systems. Some of her works relevant to this project include monitoring the in-season nitrogen status of corn plants, soybean defoliation, assessment of soil compaction due to tractor traffic during planting, estimation of soil properties, and crop yield and biomass prediction, all based on imagery acquired by manned aircraft and UAV.
Dr. Laura Lindsey Dr. Lindsey is an associate professor in the Department of Horticulture and Crop Science. She is a Soybean and Small Grain Extension Specialist. Dr. Lindsey works closely with farmers and other stakeholders to generate best management practices for soybean to maximize yield and economic return while minimizing environmental impact.
Dr. Osler Ortez Dr. Ortez is an assistant professor in the Department of Horticulture and Crop Science. He is the statewide Corn and Emerging Specialist at Ohio State. He has worked in applied agronomic research for about ten years across various crops (corn, soybeans, wheat, sorghum, sunflower, and coffee) and geographies (U.S. Midwest, Central America, and Argentina). He manages state and multi-state projects. Dr. Ortez works with extension teams, farmers, and other stakeholders to generate research and recommendations for crop management practices that maximize productivity and profitability while reducing environmental impacts. He will guide ground-truth data collection of corn stands, growth stages, yield components, and crop yield data.
Canaan Porter Canaan Porter is a 4th-year Computer Science and Engineering student at The Ohio State University who specializes in Data Engineering, set to graduate in May. Over the past year, he has collaborated with PhD student Luke Waltz to compile and aggregate agricultural data for machine learning. Post-graduation, Canaan has accepted a position at Exact Sciences to apply his skills in data engineering to advance early cancer detection research.
Sushma Katari Sushma Katari is a PhD student in the Food, Agricultural, and Biological Engineering department, focusing primarily on leveraging remote sensing and artificial intelligence (AI) techniques to develop automated tools for precision agriculture. She holds a bachelor's degree in Computer Science and Engineering, as well as a master's degree in Geoinformatics. Additionally, she has four years of industry experience working in an AgTech company named CropIn before joining Ohio State.
Chaeun Hong Chaeun Hong is a master's student and graduate research assistant in Computer Science and Engineering at The Ohio State University. She specializes in Computer Vision and image data processing, focusing on developing an advanced UAV image processing pipeline. She holds a bachelor's degree in Computer Science and Engineering with a specialization in Artificial Intelligence and has completed an internship project focused on high-definition map updates using image data processing.
Julian Colbert Julian Colbert is a BS/MS student at The Ohio State University, double majoring in Computer Science and Sociology with a minor in Statistics. He specializes in data visualization and software development, with experience creating interactive dashboards and applications. He is also currently employed as a Software Engineer Co-Op for Battelle. Julian contributed to the CropFusion project by cleaning data, creating visualizations, and developing a Dash application to combine and interactively display the data.
Adit Anup Adit Anup is an undergraduate Computer Science and Engineering student at The Ohio State working in the ixLab research group led by Prof. Arnab Nandi. He has held internship positions with Deutsche Bank, Nationwide Insurance, and American Electric Power. Outside of work and school, Adit has been recognized as a TEDx speaker and serves on the executive board of the OHI/O hackathon program on campus.