|
Hey Sean, I updated to 2.5.6+0001 with 2.1.5 Coral Module and it crashed overnight and only came alive again when I manually restarted shortly after 8am. Hope this helps.
Thanks,
James
01:43:42:Response rec'd from Object Detection (Coral) command 'detect' (...cd5d1a) [''] took 399ms
01:43:43:Response rec'd from Object Detection (Coral) command 'detect' (...bd98fb) [''] took 402ms
01:43:43:Response rec'd from Object Detection (Coral) command 'detect' (...914b57) [''] took 403ms
01:43:43:Response rec'd from Object Detection (Coral) command 'detect' (...e1c14b) [''] took 400ms
01:43:45:Response rec'd from Object Detection (Coral) command 'detect' (...9d101b) [''] took 398ms
01:43:54:Response rec'd from Object Detection (Coral) command 'detect' (...e95380) [''] took 405ms
01:45:00:Object Detection (Coral): [Empty] : Traceback (most recent call last):
File "/app/preinstalled-modules/ObjectDetectionCoral/objectdetection_coral_adapter.py", line 167, in _do_detection
result = do_detect(opts, img, score_threshold)
File "/app/preinstalled-modules/ObjectDetectionCoral/objectdetection_coral_multitpu.py", line 86, in do_detect
predictions, inferenceMs = _tpu_runner.process_image(options, image, score_threshold)
File "/app/preinstalled-modules/ObjectDetectionCoral/tpu_runner.py", line 613, in process_image
result = rs_queue.get(timeout=self.max_idle_secs_before_recycle)
File "/usr/lib/python3.9/queue.py", line 179, in get
raise Empty
_queue.Empty
01:45:56:objectdetection_coral_adapter.py: WARNING:root:No work in 120.0 seconds, watchdog shutting down TPU runners!
01:45:56:objectdetection_coral_adapter.py: I20240313 01:45:56.127516 259 pipelined_model_runner.cc:172] Thread: 281472822014240 receives empty request
01:45:56:objectdetection_coral_adapter.py: I20240313 01:45:56.127547 259 pipelined_model_runner.cc:245] Thread: 281472822014240 is shutting down the pipeline...
01:46:00:Object Detection (Coral): [Empty] : Traceback (most recent call last):
File "/app/preinstalled-modules/ObjectDetectionCoral/objectdetection_coral_adapter.py", line 167, in _do_detection
result = do_detect(opts, img, score_threshold)
File "/app/preinstalled-modules/ObjectDetectionCoral/objectdetection_coral_multitpu.py", line 86, in do_detect
predictions, inferenceMs = _tpu_runner.process_image(options, image, score_threshold)
File "/app/preinstalled-modules/ObjectDetectionCoral/tpu_runner.py", line 613, in process_image
result = rs_queue.get(timeout=self.max_idle_secs_before_recycle)
File "/usr/lib/python3.9/queue.py", line 179, in get
raise Empty
_queue.Empty
08:03:54:Update ObjectDetectionCoral. Setting Restart=now
08:03:54:Sending shutdown request to python3.9/ObjectDetectionCoral
08:04:27:Forcing shutdown of python3.9/ObjectDetectionCoral
|
|
|
|
|
Hey Sean
I changed from pi4 to pi5 with Ubuntu 23.10 and after a few teething issues the USB Coral was working and stable for a few days within docker on 2.5.6.1 Tonight I connected my Pineberry HatAI! with the PI5, install PCIe drivers and ran the 2.5.6.1 docker image with /dev/apex_0 instead of usb and it all appears to be working (took a few restarts of docker to stop erroring and falling back to CPU due to device being busy)
|
|
|
|
|
Thanks for letting me know. You might the be first user to use CodeProject.AI Server with Pineberry HatAI, so good to know!
Thanks,
Sean Ewington
CodeProject
|
|
|
|
|
So very cool.
cheers
Chris Maunder
|
|
|
|
|
I'm getting this error, and the only fix so far seems to be a manual stop/start of the coral module. Anyone ever run into this one before?
14:12:50:objectdetection_coral_adapter.py: WARNING:root:No work in 120.0 seconds, watchdog shutting down TPU runners!
14:12:50:objectdetection_coral_adapter.py: I20240306 14:12:50.575723 2034 pipelined_model_runner.cc:172] Thread: 22785103492672 receives empty request
14:12:50:objectdetection_coral_adapter.py: I20240306 14:12:50.575732 2034 pipelined_model_runner.cc:245] Thread: 22785103492672 is shutting down the pipeline...
14:12:50:objectdetection_coral_adapter.py: I20240306 14:12:50.575780 2034 pipelined_model_runner.cc:255] Thread: 22785103492672 Pipeline is off.
14:12:50:objectdetection_coral_adapter.py: I20240306 14:12:50.575785 2034 pipelined_model_runner.cc:172] Thread: 22785103492672 receives empty request
14:12:50:objectdetection_coral_adapter.py: I20240306 14:12:50.575786 2034 pipelined_model_runner.cc:245] Thread: 22785103492672 is shutting down the pipeline...
14:12:50:objectdetection_coral_adapter.py: I20240306 14:12:50.575788 10083 pipelined_model_runner.cc:207] Queue is empty and `StopWaiters()` is called.
14:12:50:objectdetection_coral_adapter.py: I20240306 14:12:50.575806 2034 pipelined_model_runner.cc:255] Thread: 22785103492672 Pipeline is off.
14:12:50:objectdetection_coral_adapter.py: I20240306 14:12:50.575893 10084 pipelined_model_runner.cc:207] Queue is empty and `StopWaiters()` is called.
14:12:50:objectdetection_coral_adapter.py: I20240306 14:12:50.575924 2034 pipelined_model_runner.cc:172] Thread: 22785103492672 receives empty request
14:12:50:objectdetection_coral_adapter.py: E20240306 14:12:50.575925 2034 pipelined_model_runner.cc:240] Thread: 22785103492672 Pipeline was turned off before.
14:12:50:objectdetection_coral_adapter.py: I20240306 14:12:50.575949 2034 pipelined_model_runner.cc:207] Queue is empty and `StopWaiters()` is called.
14:12:50:objectdetection_coral_adapter.py: E20240306 14:12:50.575953 2034 pipelined_model_runner.cc:240] Thread: 22785103492672 Pipeline was turned off before.
14:12:50:objectdetection_coral_adapter.py: E20240306 14:12:50.575954 2034 pipelined_model_runner.cc:147] Failed to shutdown status: INTERNAL: Pipeline was turned off before.
14:12:50:objectdetection_coral_adapter.py: I20240306 14:12:50.575985 2034 pipelined_model_runner.cc:172] Thread: 22785103492672 receives empty request
14:12:50:objectdetection_coral_adapter.py: E20240306 14:12:50.575986 2034 pipelined_model_runner.cc:240] Thread: 22785103492672 Pipeline was turned off before.
14:12:50:objectdetection_coral_adapter.py: I20240306 14:12:50.575991 2034 pipelined_model_runner.cc:207] Queue is empty and `StopWaiters()` is called.
14:12:50:objectdetection_coral_adapter.py: E20240306 14:12:50.575992 2034 pipelined_model_runner.cc:240] Thread: 22785103492672 Pipeline was turned off before.
14:12:50:objectdetection_coral_adapter.py: E20240306 14:12:50.575994 2034 pipelined_model_runner.cc:147] Failed to shutdown status: INTERNAL: Pipeline was turned off before.
modified 20-Mar-24 11:12am.
|
|
|
|
|
Thanks very much for your report. Could you please share your System Info tab from your CodeProject.AI Server dashboard?
Thanks,
Sean Ewington
CodeProject
modified 14-Mar-24 12:16pm.
|
|
|
|
|
Sure thing, here it is
Server version: 2.5.6
System: Docker
Operating System: Linux (Ubuntu 22.04)
CPUs: 12th Gen Intel(R) Core(TM) i9-12900K (Intel)
1 CPU x 16 cores. 24 logical processors (x64)
System RAM: 126 GiB
Platform: Linux
BuildConfig: Release
Execution Env: Docker
Runtime Env: Production
Runtimes installed:
.NET runtime: 7.0.16
.NET SDK: Not found
Default Python: Not found
Go: Not found
NodeJS: Not found
Video adapter info:
System GPU info:
GPU 3D Usage 0%
GPU RAM Usage 0
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168
Module 'Object Detection (Coral)' 2.1.4 (ID: ObjectDetectionCoral)
Valid: True
Module Path: <root>/modules/ObjectDetectionCoral
AutoStart: True
Queue: objectdetection_queue
Runtime: python3.9
Runtime Loc: Local
FilePath: objectdetection_coral_adapter.py
Pre installed: False
Start pause: 1 sec
Parallelism: 1
LogVerbosity:
Platforms: all
GPU Libraries: installed if available
GPU Enabled: enabled
Accelerator:
Half Precis.: enable
Environment Variables
CPAI_CORAL_MODEL_NAME = MobileNet SSD
CPAI_CORAL_MULTI_TPU = True
MODELS_DIR = <root>/modules/ObjectDetectionCoral/assets
MODEL_SIZE = Small
Status Data: {
"inferenceDevice": "Multi-TPU",
"inferenceLibrary": "TF-Lite",
"canUseGPU": "false",
"successfulInferences": 10,
"failedInferences": 59,
"numInferences": 69,
"averageInferenceMs": 9.6
}
modified 14-Mar-24 12:16pm.
|
|
|
|
|
I'm getting somehow the same issue that renders BlueIris to generate an AI Error 500.
Using Coral USB TPU on a Proxmox LXC CPAI Docker.
The only difference in my case is that the process restart itself eventually so I don't have to do it.
13:02:23:objectdetection_coral_adapter.py: WARNING:root:No work in 120.0 seconds, watchdog shutting down TPU runners!
13:02:23:objectdetection_coral_adapter.py: I20240310 17:02:23.944063 252967 pipelined_model_runner.cc:172] Thread: 124925159392832 receives empty request
13:02:23:objectdetection_coral_adapter.py: I20240310 17:02:23.944077 252967 pipelined_model_runner.cc:245] Thread: 124925159392832 is shutting down the pipeline...
13:02:23:objectdetection_coral_adapter.py: I20240310 17:02:23.944214 252967 pipelined_model_runner.cc:255] Thread: 124925159392832 Pipeline is off.
13:02:23:objectdetection_coral_adapter.py: I20240310 17:02:23.944253 266018 pipelined_model_runner.cc:207] Queue is empty and `StopWaiters()` is called.
13:02:23:objectdetection_coral_adapter.py: I20240310 17:02:23.944337 252967 pipelined_model_runner.cc:172] Thread: 124925159392832 receives empty request
13:02:23:objectdetection_coral_adapter.py: E20240310 17:02:23.944345 252967 pipelined_model_runner.cc:240] Thread: 124925159392832 Pipeline was turned off before.
13:02:23:objectdetection_coral_adapter.py: I20240310 17:02:23.944375 252967 pipelined_model_runner.cc:207] Queue is empty and `StopWaiters()` is called.
13:02:23:objectdetection_coral_adapter.py: E20240310 17:02:23.944383 252967 pipelined_model_runner.cc:240] Thread: 124925159392832 Pipeline was turned off before.
13:02:23:objectdetection_coral_adapter.py: E20240310 17:02:23.944387 252967 pipelined_model_runner.cc:147] Failed to shutdown status: INTERNAL: Pipeline was turned off before.
<pre>Module 'Object Detection (Coral)' 2.1.4 (ID: ObjectDetectionCoral)
Valid: True
Module Path: <root>/modules/ObjectDetectionCoral
AutoStart: True
Queue: objectdetection_queue
Runtime: python3.9
Runtime Loc: Local
FilePath: objectdetection_coral_adapter.py
Pre installed: False
Start pause: 1 sec
Parallelism: 1
LogVerbosity:
Platforms: all
GPU Libraries: installed if available
GPU Enabled: enabled
Accelerator:
Half Precis.: enable
Environment Variables
CPAI_CORAL_MODEL_NAME = YOLOv8
CPAI_CORAL_MULTI_TPU = true
MODELS_DIR = <root>/modules/ObjectDetectionCoral/assets
MODEL_SIZE = small
Status Data: {
"inferenceDevice": "Multi-TPU",
"inferenceLibrary": "TF-Lite",
"canUseGPU": "false",
"successfulInferences": 53,
"failedInferences": 490,
"numInferences": 543,
"averageInferenceMs": 23.132075471698112
}
Started: 10 Mar 2024 2:44:04 PM Coordinated Universal Time
LastSeen: 10 Mar 2024 5:32:49 PM Coordinated Universal Time
Status: Started
modified 14-Mar-24 12:17pm.
|
|
|
|
|
I get the same error with multi tpu enabled. If I turn it off, it goes away.
modified 14-Mar-24 12:17pm.
|
|
|
|
|
Good,
I'll give it a try.
Thanks
modified 14-Mar-24 12:17pm.
|
|
|
|
|
After disabling Multi-Tpu support I'm now in the same boat. After a while AI stop responding and the only way is to restart the module.
modified 14-Mar-24 12:17pm.
|
|
|
|
|
Can you please try uninstalling and reinstalling the Coral module? We've applied a fix (we hope).
Thanks,
Sean Ewington
CodeProject
modified 14-Mar-24 12:17pm.
|
|
|
|
|
Have the exact same problem. Goes down multiple times a day. The Docker container should be catching this and restarting itself, but it's not.
|
|
|
|
|
So far the latest update fixed the issue for me
|
|
|
|
|
Hasn't for me, but I've heard they're working on an update to auto-restart when it goes down, so that's cool
|
|
|
|
|
Successfully installed LPR 3.0.2 with no errors however no plates are detected when queries are passed to CPAI from Agent.
Loading CPAI Explorer and processing images of license plates returns no predictions.
I have face detection also running on this VM with no issues.
Server version: 2.5.6
System: Windows
Operating System: Windows (Microsoft Windows 10.0.19045)
CPUs: AMD FX(tm)-6300 Six-Core Processor (AMD)
1 CPU x 2 cores. 2 logical processors (x64)
GPU (Primary): VirtualBox Graphics Adapter (WDDM) (Oracle Corporation)
Driver: 6.0.18.0
System RAM: 4 GiB
Platform: Windows
BuildConfig: Release
Execution Env: Native
Runtime Env: Production
Runtimes installed:
.NET runtime: 7.0.10
.NET SDK: Not found
Default Python: Not found
Go: Not found
NodeJS: Not found
Video adapter info:
VirtualBox Graphics Adapter (WDDM):
Driver Version 6.0.18.0
Video Processor VirtualBox VESA BIOS
System GPU info:
GPU 3D Usage 0%
GPU RAM Usage 0
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168
19:54:11:Module ALPR installed successfully.
19:54:11:
19:54:11:Module 'License Plate Reader' 3.0.2 (ID: ALPR)
19:54:11:Valid: True
19:54:11:Module Path: <root>\modules\ALPR
19:54:11:AutoStart: True
19:54:11:Queue: alpr_queue
19:54:11:Runtime: python3.9
19:54:11:Runtime Loc: Local
19:54:11:FilePath: ALPR_adapter.py
19:54:11:Pre installed: False
19:54:11:Start pause: 3 sec
19:54:11:Parallelism: 0
19:54:11:LogVerbosity:
19:54:11:Platforms: all
19:54:11:GPU Libraries: installed if available
19:54:11:GPU Enabled: enabled
19:54:11:Accelerator:
19:54:11:Half Precis.: enable
19:54:11:Environment Variables
19:54:11:AUTO_PLATE_ROTATE = True
19:54:11:MIN_COMPUTE_CAPABILITY = 6
19:54:11:MIN_CUDNN_VERSION = 7
19:54:11:OCR_OPTIMAL_CHARACTER_HEIGHT = 60
19:54:11:OCR_OPTIMAL_CHARACTER_WIDTH = 30
19:54:11:OCR_OPTIMIZATION = True
19:54:11:PLATE_CONFIDENCE = 0.7
19:54:11:PLATE_RESCALE_FACTOR = 2
19:54:11:PLATE_ROTATE_DEG = 0
19:54:11:
19:54:11:Started License Plate Reader module
19:54:11:Installer exited with code 0
19:54:15:Module ALPR started successfully.
modified 7-Mar-24 10:23am.
|
|
|
|
|
Does it work if you run a test image like the below
|
|
|
|
|
No it doesn't detect anything
|
|
|
|
|
Are you seeing any errors in the log? Also what Object Detection module are you using.
|
|
|
|
|
When I upload an image and select read license plates the log returns a single line of:
16:33:59:Response rec'd from License Plate Reader command 'alpr' (...d722a0)
|
|
|
|
|
Are you running Object Detection (YOLOv5 6.2) or Object Detection (YOLOv5 .NET) module? Because the LPR module needs one of these modules to work.
|
|
|
|
|
Ah, now that would make sense! I did not have any object detection running, thought I was fine as I had already had face detection successfully running on the same host.
Have started the .NET module and now license plate detection is working. Apologies for bothering you.
|
|
|
|
|
|
I have the Yolo V5 6.2 running and it works perfect with BI.
But LPR doesn't work. I tested several times now. Always "No Predictions Returned". Yolo V5 6.2 returns "DayPlate found" and marks it correctly if I run it with the license-plate model.
During installation I found an error because of CUDNN. CUDA 12.2 is installed and Yolo V5 6.2 is running in GPU mode.
Is it mandatory for LPR to install CUDNN? I skipped it because "Face Processing" and "Yolo V5 6.2" didn't need it.
|
|
|
|
|
It is mandatory to have cuDNN installed also depending on the age of your GPU CUDA 12.2 will not work and needs CUDA 11.8
modified 7-Mar-24 9:08am.
|
|
|
|