Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

Control Board Microcontroller IMU ESCs Speed commands Jetson Docker ROS2 Daemon ZED node Processed data Rust program Processed data Movement commands Sensor data Manipulation Systems Board Dropper Grabber Torpedoes Power to magents/actuators High level action commands Bottom Camera Raw camera stream ZED Camera Raw depth camera data

What does the aquapack software team do?

  • The aquapack software team handles the decision making of the robot through the control board and Jetson.
  • The competition aquapack competes in is RoboSub and the challenges for this competition are all automated, as such there are no inputs sent to the robot during its run. Writing functional code for the missions is very important.
  • In Summary The code makes the robot do things.

What languages and tools are used?

  • The Rust programming language for implementing high-level logic
  • Computer vision
    • YOLO (You only look once) for model based object detection
    • OpenCV for classical computer vision
  • ROS2 (Robot Operating System) to communicate with peripherals

How can I get started?

Continue to the next chapter, or jump straight to the Rust installation instructions

Repository Overview

The repository overview is a breakdown of all of the repositories that make up the software stack, what they do, and how they fit together.

Current Repositories

SW9S

SW9S is the main codebase for SeaWolf 9. At a high level, it receives sensor data, uses that data to make decisions, then acts on them by sending commands to the locomotion and manupulation systems.

AUVControlBoard

This repository contains the PCB designs, firmware, Python library, and Rust libraries for the AUVControlBoard project. SW9S uses the Rust library to send commands to SeaWolf’s control board to move the robot and recieve depth and orientation data.

SW9-MSB

This repository contains the PCB designs, firmware, and Rust library for SeaWolf 9’s manipulation systems board (MSB). SW9S uses the Rust library to send commands to the MSB to operate peripherals like the torpedo launchers, marker dropper, and grabber.

SW9S-IsaacSim

The Issac Sim simulator for SeaWolf 9. It is currently a work in progress.

SW9S-GazeboSim

The Gazebo Sim simulator for SeaWolf 9. It is currently a work in progress.

Legacy Repositories

SW8S-Rust

The core codebase for SeaWolf 8

SW8S-UnitySimulator

The simulator used for SeaWolf 8

Hardware

  • NVIDIA Jetson Orin Nano
  • Stereolabs ZED X Mini

Computer Vision

SeaWolf runs model based and classical computer vision algorithms using footage from it’s two cameras.

Communications

SW9S communicates with the various peripherals on SeaWolf in two main ways:

  1. Serial: This is used for communication with out electrical boards, including the control board, main electronics board, and manipulation systems board
  2. ROS2: We use ROS2 to collect data from several sensors, including our front depth camera and sonar unit.

Installation

Linux and MacOS

Follow the Linux and MacOS section of the Rust book’s installation instructions.

Windows

You have two options:

  1. The native Windows installer. For this option, follow the Windows section of the Rust book’s installation instructions.
  2. Install Rust inside Windows Subsystem for Linux (WSL)
  3. Follow the VS Code WSL setup guide
  4. Open a new terminal from the top menu within VS Code and run these commands one by one to download Rust: 1. sudo apt update 2. sudo apt upgrade 3. sudo apt install build-essential rustup 4. rustup default stable
  5. Go to the VS Code ‘Extensions’ tab and install the rust-analyzer extension

Learning Resources

  • The Rust book: This is the best primary resource for learning Rust in my opinion. It walks you through almost the entire language and has good tutorials.
  • Rust by Example: This covers a lot of the same material as the Rust book, but presents it as example code that is explained line by line.
  • Rustlings: This has some more approachable tutorials
  • Rust Cookbook: This is a good supplementary resource that has examples for many common programming tasks.

Active Learning Pipeline

A work in progress active learning pipeline is being worked on here.

Documentation Draft

This pipeline integrates YOLOv8 (Ultralytics) for object detection, DagsHub for metadata and dataset versioning, and Label Studio for manual annotation creation and adjustment. Source code and configuration files are on GitHub while the raw images, annotations, and model weights are left on DagsHub.

Step 0: Setup You’ll need to create an account with DagsHub. It shares some similarities with GitHub and is tailored for working with computer vision projects. Once you have an account, create a repository for this project. There are two things you’ll put in this repo initially: the images for your dataset and the ALL.ipynb code file. Create a folder called data and create another folder inside that called images. Put all of your raw image files into that folder. You’ll need to install JupyterLab or Jupyter Notebook to view and run the code.

Step 1: Installing Required Libraries If you do not have the dagshub and ultralytics libraries installed in your Jupyter Notebook environment, run the first cell containing the pip commands. You only need to do this once. The second cell contains import statements for libraries used in most, if not all, of the code cells. You’ll need to run this whenever you close and reopen the project.

Step 2: Connect to DagsHub In the third cell, change the REPO_OWNER and REPO_NAME to your DagsHub username and repository name respectively. This code will create or connect to the repository’s Datasource, which is essentially what DagsHub uses to handle data. You’ll also need to run this cell whenever you close and reopen the project.

Step 3: Create Training Images To start, choose 50-100 images to manually annotate in your annotation software of choice (such as makesense.ai). Make sure that each object you’ll label is represented in a good number of images. If it gives you a choice between “Object Detection” and “Image Recognition,” choose “Object Detection.” Create your labels and start annotating–remember the order your labels are in, since the YOLO format won’t preserve the names. It’s recommended to create them by alphabetical order. Make sure that the bounding box is just large enough to fit the entire object. Once you’re done annotating, export your annotations, making sure they are in YOLO format. In the project files, create another folder in data called labels and put the annotation files in that folder (unzip the files if needed).

Step 4: Upload Training Data to DagsHub Remember your label order? In the fourth cell–in the yaml_content block–go down to the ‘names’ block and edit the label names to match your label order. Add or delete entries as needed, just update the ‘nc’ line to your number of classes and copy the format the other entries are written in. Once you run this code cell, a file called data.yaml will be created in your project folder that DagsHub will use to know where the image files are and which labels are what. All the images in DagsHub will be tagged with metadata, with their split value being ‘train’ for those with annotations and ‘unlabeled’ for those without. If you want to check if your labels were uploaded correctly, run the fifth code cell to create a Label Studio project in DagsHub. Make sure to select the “Load annotations from datasource field” option and point it at the “data_labels” field. Once all the data is imported, click the BLUE “Settings” button in the top right and select “Labeling Interface” in the menu. In “Add label names,” write your labels in the order you created them in the annotation software. Alternatively, switch to “Code” mode and copy paste the text block between the quotes in the code cell (remember to edit the names and format to match your actual label format). You should now be able to click through your training images and verify their annotations are correct.

Step 5: Training a Model Run the sixth cell. This will create a file called train_list.txt holding the filepaths of every training image. It then creates another file called yolo_train.yaml that will tell the training process where the training images are. Edit the ‘nc’ and ‘names’ lines to match what’s in your data.yaml file. Now go to the seventh cell. Since you’ll be training a base YOLO model, uncomment out the first code block and comment out the second code block. If you want to, change the model = YOLO() line to your desired YOLO model; by default, it is set to YOLOv8 Nano. Run the code to start training. The code is set up to run the process on your CPU; running it on your dedicated GPU (if you have one) is much faster, but it is not implemented currently. If you have to stop in the middle of training, it’s simple to continue: comment out the first code block, uncomment the second code block, verify that the filepath in the code matches where the model weight files are stored, and run the cell again.

Step 6: Create and Correct Predictions Now that the model has undergone its initial training run, run the eighth cell to have the model draw its own annotations on the unlabeled images. Make sure the filepath assigned to ‘model’ is correct, and update ‘pred_yaml’ to match your previous two YAML contents. This is where the active learning comes in; the program takes all the images where the model is uncertain about even one annotation (by default, a confidence between 30% and 70%) and tags them as the ‘al_preds’ group. Once the program is done, it should open DagsHub’s Label Studio and prompt you to create a new project. Like before, make sure to select “Load annotations from datasource field.” This time, the predicted annotations are stored in “yolo_preds”. Once again, go to Settings and edit “Labeling Interface” so the annotations can be seen. You, the human, will have to go through all of the model’s predicted annotations and edit any that are drawn or labeled incorrectly. Once you’re done with review, click “Export”, select “YOLO” (just regular YOLO), and click “Export” again. This downloads the annotations as a .ZIP file of text files.

Step 7: Correct Label Format Unzip the text files into your project directory but keep them in a folder separate from data/labels. You can rename this folder to something simpler if you’d like. In the ninth code cell, edit ‘ls_classes’ to your labels in alphabetical order and edit ‘model_classes’ to the order your labels were originally in–if your created them in alphabetical order, it’s fine for them to be the same. Change ‘extract_dir’ to whatever the folder with your reviewed annotation text files is called and run the code cell. Since Label Studio exports labels in alphabetical order, this script changes them to match the original order so the model doesn’t get confused. Label Studio also prepends an 8-character hash to each annotation file, so this script also removes that. It also moves the text files to data/labels for you, so once it’s done you can delete the folder you created and exported to.

Step 8: Restarting the Active Learning Cycle Run the tenth code cell to put the ‘al_batch’ annotations under the ‘train’ group. Now, you can go back to Step 5 and repeat the process. Remember for the sixth cell to change the ‘model’ line to the path of your ‘best.pt’ file, the same that you put in ‘model’ for the seventh cell. You should see that the model is trained better and its predicted annotations are better. Do this cycle as many times as you think is appropriate.