Wednesday, October 6, 2021

Quick Blue Iris with DeepStack debug

Note this write refers to Blue Iris version 5.5.0.13 which is a beta release so there may be some differences to the version you are using.

First things first

Make sure you only have one or no custom model in the folder Blue Iris is looking for custom models in. The doc indicate multiples are supported but this appears to be wrong. If there is more than one it appears none of them work and in some versions even the non custom models stop working.

Also if you are running DeepStack with -e MODE=High try -e MODE=Medium instead and see if that fixes things. I've was seeing random 500s in High mode though the system did not appear to be maxed out in any way. It was taking a lot longer to respond too. In Medium getting around 300 ms times while High was taking seconds. High mode will detect more objects though.

Turn on DeepStack save details.

To do real debug you need to turn on Save DeepStack analysis details for the camera. Open camera properties, go to the Trigger tab and click on Artificial Intelligence.
AI options dialog

This will create a ".dat" file in the Alerts folder for each alert with info about the frames DeepStack processed.
Alerts folder listing

Find the alert pic.

If you are here you are probably have this but if you are just not seeing an alert clip for something you know should be there start here. Open the cancelled alerts folder and find the alert picture you think should have triggered and alert. 

Note if there is no alert you have a motion detection issue. DeepStack analysis below sends frames ignoring motion and object to detect settings for the camera.

If you have an alert that you think DeepStack should have triggered an notification for here are some simple things to check:

Does the motion alert have a return 100 or -1 for the memo?

If so DeepStack has crashed or hung. Either way it needs restarted.

Still getting a return 100 or -1 for the memo after restart?

Ran into this the other day on one of my servers. After a restart DeepStack would work for a few minutes and the stop. Looked in the logs (docker logs for Docker installs) and found the return codes of  200 changed to a 400 then 500s. I tried several things. I backed out the last change, restarted and watched to see if I could sort an event causing the crash but it just kept working. So hard to say what to do other than keep restarting DeepStack till it stays running.

Does the motion alert have a nothing found for the memo?

Double click on the alert picture you want to debug and turn on analyze by right clicking in the playing clip and selecting.

Analyze options select 

I find it helpful to turn on overlays as well.
Overlay options selection

Next open the DeepStack console via the status icon.


You will want to restart playing the clip at this point and play it backwards. Note the Alert picture appears to be the last frame of the clip just double clicking on that alert plays forward from that position.

Stop when you get to the place where the thing was not detected that you think should have been.

IDed frame


Now you can compare the saved (.dat file) info against the IDed frame. In the above instance it IDed a "cat" at a few places but checking the 2 frames sent to DeepStack we see the alert clip does not necessarily match. 
First DeepStack frame

Second DeepStack frame

Notice the above is the alert picture and that they are in reverse order in the listing.


If I switch to Analyze with motion detector and play forward from a little before the first sent frame I can then see what motion triggered the alert. It appears to be about half a second after the first sent frame. It appears this is related to the make time and the first frame sent to DeepStack is from the start of the make time and the rectangle below is the end of the make time.

So at least one issue here is that the IDed frame from analyze came between the frames sent. So in some cases (but not this one) playing with the make and break times might help. 

In this case the camera is set to send a picture to DeepStack every 750ms for up to 30 images but BI appears to have only sent 2 frames 2.061 seconds apart. Though looking at the timestamps on the images shows about 4 seconds actually passed between them. This would seem to indicate something it wrong in Blue Iris.

Then there is also the issue with DeepStack not seeing anything in the second sent frame. That would seem to indicate a model issue. Short of creating your own model there is little to be done about that.

It IDed something else in the picture but not the thing I wanted.

Again running analyze on the clip may help. For example in this image below from DeepStack analyze we see this.

And from motion analyze we see this.

You will note there are 6 raccoons in this shot. The lone one is mainly IDed from its tail. Two motion blocks are on a second raccoon by BI's motion detection and the reset are unseen. DeepStack seems to connect the lone raccoon to its tail to ID it as a cat but takes the 4 in a group, along with the dog house as one objects and IDs it as a dog. Note too the confidence was higher in the second clip but the results returned were from the first clip. I could see how this might been interpreted as a bug though whether to return the min, max, average, confirmed, first or last could be a matter of taste or best fit for a given target. This also might give some insight into tweaks in motion detection settings that might help you get better notifications.

Keeping an eye on DeepStack

If you are running DeepStack in Docker the log output goes to the console so the easiest way to monitor DeepStack is to start it from the command line and leave the window open on the screen in a reduced state so it looks something like this.


DeepStack on Docker

Stop instance named DSfaces
docker stop DSfaces

List local instances and their status
docker ps -a

Start instance named DSfaces. (note runs in background with no logs to screen)
docker start DSfaces

Tails log if started with docker start DSfaces
docker logs -f DSfaces

Remove all non running Docker images
docker system prune --all --volumes --force

Run DeepStack pulling new image if needed.
docker run -e VISION-FACE=True -e VISION-DETECTION=True -e VISION-SCENE=True -v localstorage:/datastore -e MODE=Medium -v /deepstack/mymodels:/modelstore/detection -v /deepstack/myfaces:/DeepStack/Faces --name DSfaces -p 82:5000 deepquestai/deepstack
Options are:
-e VISION-FACE=True 

-e VISION-DETECTION=True 

Do scene detection Not very useful for a static camera.
-e VISION-SCENE=True 

Mode to run in. Same as Blue Iris options
-e MODE=Medium 

Where custom models are. Note maps local path outside of Docker /deepstack/mymodels to container path /modelstore/detection  
-v /deepstack/mymodels:/modelstore/detection 

Where known faces images are. Note maps local path outside of Docker
-v /deepstack/myfaces:/DeepStack/Faces 

This specifies the local volume where DeepStack will store all data. Note maps local path outside of Docker
-v /deepstack/datastore:/datastore 

Name to use to interact with image. Random name is assigned if not given. Note will fail it image with that name already exists on your system.
--name DSfaces 

External port:internal one. Internal is always 5000 and 82 is assumed external port by BI.
-p 82:5000 

Name of cloud image to use.
deepquestai/deepstack

If using the gpu version you will want to use.
--gpus all deepquestai/deepstack:gpu

Check free space on image is available
docker exec DSobjects df -k

# docker exec DSobjects df -k
Filesystem               1K-blocks      Used  Available Use% Mounted on
overlay                 3835070876 992674812 2842396064  26% /
tmpfs                        65536         0      65536   0% /dev
tmpfs                     16298888         0   16298888   0% /sys/fs/cgroup
shm                          65536         0      65536   0% /dev/shm
/dev/mapper/centos-root   52403200  20850180   31553020  40% /DeepStack/MyModels
/dev/mapper/centos-home 3835070876 992674812 2842396064  26% /etc/hosts
tmpfs                     16298888         0   16298888   0% /proc/asound
tmpfs                     16298888         0   16298888   0% /proc/acpi
tmpfs                     16298888         0   16298888   0% /proc/scsi
tmpfs                     16298888         0   16298888   0% /sys/firmware

Log onto the container you look at files and processes. Note there is not much to see.
docker exec -it DSfaces /bin/bash

At the container's bash prompt try running this to see what files are recent and therefore maybe info worth lookin at.
find / -newer /sys/hypervisor -ls |  grep -v /proc

Sample shell script to check for image and if missing download auto download and start. Otherwise just restart current image and monitor log.
if [ -z "`docker ps -a | grep DSfaces`" ]
then
        docker run -e VISION-FACE=True -e VISION-DETECTION=True -e VISION-SCENE=True -v localstorage:/datastore -e MODE=Medium -v /deepstack/mymodels:/modelstore/detection -v /deepstack/myfaces:/DeepStack/Faces --name DSfaces -p 82:5000 deepquestai/deepstack
else
        docker start DSfaces
        docker logs -f DSfaces
fi

Sample shell script to auto download and start. Then if crashes, blows the local image, gets the image from the cloud and then runs it.
while [ 1 -eq 1 ]
do
        sleep 10
        docker system prune --all --volumes --force
        docker run -e VISION-FACE=True -e VISION-DETECTION=True -e VISION-SCENE=True -v localstorage:/datastore -e MODE=Medium -v /deepstack/mymodels:/modelstore/detection -v /deepstack/myfaces:/DeepStack/Faces --name DSfaces -p 82:5000 deepquestai/deepstack
done

To do a quick or extensive check of your DeepStack setup run fullTest from my DeepStack utils repo which include some other sample tools.