Security Cam 101: DeepStack training

My test set

My current test set can be downloaded from here.

Hardware

This workstation is set up for video editing so it is pretty fast. It was in it:

3.50 gigahertz AMD Ryzen 9 3950X 16-Core
64 GB of RAM
Samsung SSD 970 EVO Plus 1TB [Hard drive]
NVIDIA GeForce RTX 3080

Dataset prep

If you are looking to prep your own dataset you might want to look at the utils I have here.

You may want to look at Why Image Preprocessing and Augmentation Matter as well.

Method Compare

I used the original 91 pics with 194 labeled objects of 4 types of my test set:

Colab 2.973 hours
Jupyter hybrid method 18 minutes
local 0.299 hours (just under 18 minutes)

The models created appear to be the same. The latest model with the latest data set can be downloaded from here.

Local

Start here. Took me a while to sort what I was doing wrong as the instructions can be a bit misleading and python being at least 20 something on the list of languages I use the most. I finally figured it out while trying to debug the "local Colab" / jupyter method.

Be sure to install CUDA and CUDNN not just one. Remember the versions.

PyTorch

Instead of the suggested something like

pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio===0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

Instead try

pip3 install torch torchvision==0.10.1+cu111 torchaudio torchtext -f https://download.pytorch.org/whl/torch_stable.html

The +cu111 bit seems to align to Cuda 11.1. Leaving off the other == bits lets it sort versions it needs itself.

Download training data to deepstack-trainer../deepstack/train

So assuming your workspace is C:\DeepStackWS

deepstack-trainer would cloned in C:\DeepStackWS\deepstack-trainer

and your training data would be in C:\DeepStackWS\deepstack\train

Then cd deepstack-trainer and run

python3 train.py --dataset-path --exist-ok "../deepstack"

Running this way not only avoids the notebook framework making automation easier but puts more progress info to the screen to see how things are going.

Options from train.py (with my notes)

--model, default='yolov5m', help='yolo model to use' (best option most cases)
--classes,default='', help='model.yaml path'
--dataset-path, required=True, help='path to dataset folder' (Should contain at least a train folder. May also contain test and validate folders. If folders exist they should contain some image files and copy of classes.txt or programs will crash.)
--hyp, default='data/hyp.scratch.yaml', help='hyperparameters path'
--epochs, default=300
--batch-size, default=16, help='total batch size for all GPUs' (>16 did not work for me)
--img-size, default=[640, 640], help='[train, test] image sizes'
--rect, help='rectangular training'
--resume, default=False, help='resume most recent training'
--nosave, help='only save final checkpoint'
--notest, help='only test final epoch'
--noautoanchor, help='disable autoanchor check'
--evolve, help='evolve hyperparameters'
--bucket, default='', help='gsutil bucket'
--cache-images, help='cache images for faster training'
--image-weights, help='use weighted image selection for training'
--device, default='', help='cuda device, i.e. 0 or 0,1,2,3 or cpu'
--multi-scale, help='vary img-size +/- 50%%'
--single-cls, help='train as single-class dataset'
--adam, help='use torch.optim.Adam() optimizer'
--sync-bn, help='use SyncBatchNorm, only available in DDP mode'
--local_rank, default=-1, help='DDP parameter, do not modify'
--log-imgs, default=16, help='number of images for W&B logging, max 100'
--workers, default=8, help='maximum number of data loader workers' (>8 did not work for me)
--project, default='name folder dataset folders are in', help='save to project/name'
--name, default='exp', help='save to project/name'
--exist-ok, help='overwrite if not first run instead of making new folder' (runTrain.bat assumes this to copy model to DeepStack instance)

Note increasing --batch-size or --workers caused and out of resources exception for me despite having plenty of unused RAM, GPU, CPU and HD. Though there maybe some missed option as Python only appears to be using 1 GB of RAM.

Gotchas

C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: block: [0,0,0], thread: [95,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.

This is caused by a bad mapping file with a class ID larger than the number of classes in the train\classes.txt file.

Note there is a similar problem using the LabelImg program. It will crash if it reads a map file with an ID larger than the number of classes in labelImg\data\predefined_classes.txt

Add debug

If you are seeing another kind of crash or odd behaviour with train.py try changing the set_logging method in deepstack-trainer/utils/general.py to look like this

def set_logging(rank=-1):

# logging.basicConfig(

# format="%(message)s",

# level=logging.INFO if rank in [-1, 0] else logging.WARN)

logging.basicConfig(

format="%(message)s",

level=logging.DEBUG)

Colab Method

The DeepStack docs mention doing this but they kind of assume you know what you are doing with Colab.

Open My edited copy of their notebook for cloud training. Notebook in this instance means project file not notes on how to do something which might be your first guess looking at it.

Click on copy to drive to get a copy you can edit.

I want to have my data and output persist between runs so I'm mounting my Google Drive for storage.

Go to / click in the mount drive section and press Shift-Enter to run it

Go to / click in the clone trainer section and press Shift-Enter to run it

Section 1

This takes a while. (Mine said 4 minutes.)

Note it loads old versions of packages so I was left with this error/result:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. torchtext 0.11.0 requires torch==1.10.0, but you have torch 1.7.0+cu110 which is incompatible. Successfully installed dataclasses-0.6 torch-1.7.0+cu110 torchaudio-0.7.0 torchvision-0.8.1+cu110

While this running you can work on preparing your data. See Preparing Your Dataset. If you just want to try it before creating your own data set you can find mine here. It is the one I'll building on to detect the wildlife types I have here.

Finally run the Add Your Dataset section which will take a fair bit of time. My set of 91 images took ~30 seconds per run and the default is 300 runs.

Note if you do not scroll the output often enough you will get a popup asking if you are still there. At which point I think it pauses your processing till you respond. So do not leave it to run overnight and expect to be done in the morning.

I finally got this result

Epoch gpu_mem box obj cls total targets img_size

299/299 9.26G 0.01827 0.0114 0.001359 0.03103 59 640: 100% 6/6 [00:30<00:00, 5.07s/it]

Class Images Targets P R mAP@.5 mAP@.5:.95: 100% 1/1 [00:00<00:00, 2.16it/s]

all 11 0 0 0 0 0

/usr/local/lib/python3.7/dist-packages/seaborn/matrix.py:198: RuntimeWarning: All-NaN slice encountered

vmin = np.nanmin(calc_data)

/usr/local/lib/python3.7/dist-packages/seaborn/matrix.py:203: RuntimeWarning: All-NaN slice encountered

vmax = np.nanmax(calc_data)

Exception in thread Thread-10:

Traceback (most recent call last):

File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner

self.run()

File "/usr/lib/python3.7/threading.py", line 870, in run

self._target(*self._args, **self._kwargs)

File "/content/deepstack-trainer/utils/plots.py", line 122, in plot_images

colors = color_list() # list of colors

File "/content/deepstack-trainer/utils/plots.py", line 32, in color_list

return [hex2rgb(h) for h in plt.rcParams['axes.prop_cycle'].by_key()['color']]

File "/content/deepstack-trainer/utils/plots.py", line 32, in <listcomp>

return [hex2rgb(h) for h in plt.rcParams['axes.prop_cycle'].by_key()['color']]

File "/content/deepstack-trainer/utils/plots.py", line 30, in hex2rgb

return tuple(int(h[1 + i:1 + i + 2], 16) for i in (0, 2, 4))

File "/content/deepstack-trainer/utils/plots.py", line 30, in <genexpr>

return tuple(int(h[1 + i:1 + i + 2], 16) for i in (0, 2, 4))

TypeError: int() can't convert non-string with explicit base

Exception in thread Thread-11: