CodeProject.AI Discussions

Re: CPU docker with coral TPU; coral module crashes after a bit and has to be manually restarted -- Open

10-Mar-24 7:34

I'm getting somehow the same issue that renders BlueIris to generate an AI Error 500.
Using Coral USB TPU on a Proxmox LXC CPAI Docker.
The only difference in my case is that the process restart itself eventually so I don't have to do it.

13:02:23:objectdetection_coral_adapter.py: WARNING:root:No work in 120.0 seconds, watchdog shutting down TPU runners!
13:02:23:objectdetection_coral_adapter.py: I20240310 17:02:23.944063 252967 pipelined_model_runner.cc:172] Thread: 124925159392832 receives empty request
13:02:23:objectdetection_coral_adapter.py: I20240310 17:02:23.944077 252967 pipelined_model_runner.cc:245] Thread: 124925159392832 is shutting down the pipeline...
13:02:23:objectdetection_coral_adapter.py: I20240310 17:02:23.944214 252967 pipelined_model_runner.cc:255] Thread: 124925159392832 Pipeline is off.
13:02:23:objectdetection_coral_adapter.py: I20240310 17:02:23.944253 266018 pipelined_model_runner.cc:207] Queue is empty and `StopWaiters()` is called.
13:02:23:objectdetection_coral_adapter.py: I20240310 17:02:23.944337 252967 pipelined_model_runner.cc:172] Thread: 124925159392832 receives empty request
13:02:23:objectdetection_coral_adapter.py: E20240310 17:02:23.944345 252967 pipelined_model_runner.cc:240] Thread: 124925159392832 Pipeline was turned off before.
13:02:23:objectdetection_coral_adapter.py: I20240310 17:02:23.944375 252967 pipelined_model_runner.cc:207] Queue is empty and `StopWaiters()` is called.
13:02:23:objectdetection_coral_adapter.py: E20240310 17:02:23.944383 252967 pipelined_model_runner.cc:240] Thread: 124925159392832 Pipeline was turned off before.
13:02:23:objectdetection_coral_adapter.py: E20240310 17:02:23.944387 252967 pipelined_model_runner.cc:147] Failed to shutdown status: INTERNAL: Pipeline was turned off before.

<pre>Module 'Object Detection (Coral)' 2.1.4 (ID: ObjectDetectionCoral)
Valid:         True
Module Path:   <root>/modules/ObjectDetectionCoral
AutoStart:     True
Queue:         objectdetection_queue
Runtime:       python3.9
Runtime Loc:   Local
FilePath:      objectdetection_coral_adapter.py
Pre installed: False
Start pause:   1 sec
Parallelism:   1
LogVerbosity:  
Platforms:     all
GPU Libraries: installed if available
GPU Enabled:   enabled
Accelerator:   
Half Precis.:  enable
Environment Variables
   CPAI_CORAL_MODEL_NAME = YOLOv8
   CPAI_CORAL_MULTI_TPU  = true
   MODELS_DIR            = <root>/modules/ObjectDetectionCoral/assets
   MODEL_SIZE            = small
Status Data:  {
  "inferenceDevice": "Multi-TPU",
  "inferenceLibrary": "TF-Lite",
  "canUseGPU": "false",
  "successfulInferences": 53,
  "failedInferences": 490,
  "numInferences": 543,
  "averageInferenceMs": 23.132075471698112
}
Started:      10 Mar 2024 2:44:04 PM Coordinated Universal Time
LastSeen:     10 Mar 2024 5:32:49 PM Coordinated Universal Time
Status:       Started

modified 14-Mar-24 12:17pm.

Anthony m Jul202311-Mar-24 2:34

Anthony m Jul2023

11-Mar-24 2:34

I get the same error with multi tpu enabled. If I turn it off, it goes away.

modified 14-Mar-24 12:17pm.

Re: CPU docker with coral TPU; coral module crashes after a bit and has to be manually restarted -- Open

Member 1496256211-Mar-24 2:47

Re: CPU docker with coral TPU; coral module crashes after a bit and has to be manually restarted -- Open

11-Mar-24 2:47

Good,
I'll give it a try.
Thanks Smile | :)

modified 14-Mar-24 12:17pm.

Member 1496256211-Mar-24 9:19

Re: CPU docker with coral TPU; coral module crashes after a bit and has to be manually restarted -- Open

11-Mar-24 9:19

After disabling Multi-Tpu support I'm now in the same boat. After a while AI stop responding and the only way is to restart the module.

modified 14-Mar-24 12:17pm.

Sean Ewington12-Mar-24 10:07

Sean Ewington

12-Mar-24 10:07

Can you please try uninstalling and reinstalling the Coral module? We've applied a fix (we hope).

Thanks,
Sean Ewington
CodeProject

modified 14-Mar-24 12:17pm.

Re: CPU docker with coral TPU; coral module crashes after a bit and has to be manually restarted -- Open

seashellz5915-Mar-24 8:39

seashellz59

15-Mar-24 8:39

Have the exact same problem. Goes down multiple times a day. The Docker container should be catching this and restarting itself, but it's not.

Re: CPU docker with coral TPU; coral module crashes after a bit and has to be manually restarted -- Open

Member 1496256220-Mar-24 2:20

Re: CPU docker with coral TPU; coral module crashes after a bit and has to be manually restarted -- Open

20-Mar-24 2:20

So far the latest update fixed the issue for me Smile | :)

seashellz5920-Mar-24 10:32

seashellz59

20-Mar-24 10:32

Hasn't for me, but I've heard they're working on an update to auto-restart when it goes down, so that's cool

LPR 3.0.2 No Predictions Returned -- Resolved

Marc Oliver 20245-Mar-24 20:41

5-Mar-24 20:41

Successfully installed LPR 3.0.2 with no errors however no plates are detected when queries are passed to CPAI from Agent.
Loading CPAI Explorer and processing images of license plates returns no predictions.
I have face detection also running on this VM with no issues.

Server version:   2.5.6
System:           Windows
Operating System: Windows (Microsoft Windows 10.0.19045)
CPUs:             AMD FX(tm)-6300 Six-Core Processor              (AMD)
                  1 CPU x 2 cores. 2 logical processors (x64)
GPU (Primary):    VirtualBox Graphics Adapter (WDDM) (Oracle Corporation) 
                  Driver: 6.0.18.0
System RAM:       4 GiB
Platform:         Windows
BuildConfig:      Release
Execution Env:    Native
Runtime Env:      Production
Runtimes installed:
  .NET runtime:     7.0.10
  .NET SDK:         Not found
  Default Python:   Not found
  Go:               Not found
  NodeJS:           Not found
Video adapter info:
  VirtualBox Graphics Adapter (WDDM):
    Driver Version     6.0.18.0
    Video Processor    VirtualBox VESA BIOS
System GPU info:
  GPU 3D Usage       0%
  GPU RAM Usage      0
Global Environment variables:
  CPAI_APPROOTPATH = <root>
  CPAI_PORT        = 32168

19:54:11:Module ALPR installed successfully.
19:54:11:
19:54:11:Module 'License Plate Reader' 3.0.2 (ID: ALPR)
19:54:11:Valid:         True
19:54:11:Module Path:   <root>\modules\ALPR
19:54:11:AutoStart:     True
19:54:11:Queue:         alpr_queue
19:54:11:Runtime:       python3.9
19:54:11:Runtime Loc:   Local
19:54:11:FilePath:      ALPR_adapter.py
19:54:11:Pre installed: False
19:54:11:Start pause:   3 sec
19:54:11:Parallelism:   0
19:54:11:LogVerbosity:
19:54:11:Platforms:     all
19:54:11:GPU Libraries: installed if available
19:54:11:GPU Enabled:   enabled
19:54:11:Accelerator:
19:54:11:Half Precis.:  enable
19:54:11:Environment Variables
19:54:11:AUTO_PLATE_ROTATE            = True
19:54:11:MIN_COMPUTE_CAPABILITY       = 6
19:54:11:MIN_CUDNN_VERSION            = 7
19:54:11:OCR_OPTIMAL_CHARACTER_HEIGHT = 60
19:54:11:OCR_OPTIMAL_CHARACTER_WIDTH  = 30
19:54:11:OCR_OPTIMIZATION             = True
19:54:11:PLATE_CONFIDENCE             = 0.7
19:54:11:PLATE_RESCALE_FACTOR         = 2
19:54:11:PLATE_ROTATE_DEG             = 0
19:54:11:
19:54:11:Started License Plate Reader module
19:54:11:Installer exited with code 0
19:54:15:Module ALPR started successfully.

modified 7-Mar-24 10:23am.

Mike Lud6-Mar-24 1:42

6-Mar-24 1:42

Does it work if you run a test image like the below

Marc Oliver 20246-Mar-24 14:42

6-Mar-24 14:42

No it doesn't detect anything

Mike Lud6-Mar-24 16:09

6-Mar-24 16:09

Are you seeing any errors in the log? Also what Object Detection module are you using.

Marc Oliver 20246-Mar-24 16:37

6-Mar-24 16:37

When I upload an image and select read license plates the log returns a single line of:

16:33:59:Response rec'd from License Plate Reader command 'alpr' (...d722a0)

Mike Lud6-Mar-24 16:56

6-Mar-24 16:56

Are you running Object Detection (YOLOv5 6.2) or Object Detection (YOLOv5 .NET) module? Because the LPR module needs one of these modules to work.

Marc Oliver 20246-Mar-24 19:33

6-Mar-24 19:33

Ah, now that would make sense! I did not have any object detection running, thought I was fine as I had already had face detection successfully running on the same host.

Have started the .NET module and now license plate detection is working. Apologies for bothering you.

Mike Lud6-Mar-24 20:13

6-Mar-24 20:13

Anytime

StefanLU6-Mar-24 22:36

StefanLU

6-Mar-24 22:36

I have the Yolo V5 6.2 running and it works perfect with BI.

But LPR doesn't work. I tested several times now. Always "No Predictions Returned". Yolo V5 6.2 returns "DayPlate found" and marks it correctly if I run it with the license-plate model.

During installation I found an error because of CUDNN. CUDA 12.2 is installed and Yolo V5 6.2 is running in GPU mode.

Is it mandatory for LPR to install CUDNN? I skipped it because "Face Processing" and "Yolo V5 6.2" didn't need it.

Mike Lud7-Mar-24 1:40

7-Mar-24 1:40

It is mandatory to have cuDNN installed also depending on the age of your GPU CUDA 12.2 will not work and needs CUDA 11.8

modified 7-Mar-24 9:08am.