|
Hello, I can't seem to get Project AI Object Detection to enable my newly installed Tesla P4. Any idea what I'm doing wrong? YOLOv5 Doesn't have the option to enable the GPU and only the CPU shows.
18:46:20:System: Windows
18:46:20:Operating System: Windows (Microsoft Windows 10.0.19045)
18:46:20:CPUs: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz (Intel)
18:46:20: 1 CPU x 4 cores. 4 logical processors (x64)
18:46:20:GPU (Primary): Tesla P4 (8 GiB) (NVIDIA)
18:46:20: Driver: 551.61, CUDA: 12.4 (up to: 12.4), Compute: 6.1, cuDNN: 8.9
18:46:20:System RAM: 16 GiB
18:46:20:Platform: Windows
18:46:20:BuildConfig: Release
18:46:20:Execution Env: Native
18:46:20:Runtime Env: Production
18:46:20:Runtimes installed:
18:46:20: .NET runtime: 7.0.10
18:46:20: .NET SDK: Not found
18:46:20: Default Python: Not found
18:46:20: Go: Not found
18:46:20: NodeJS: Not found
18:46:20:App DataDir: C:\ProgramData\CodeProject\AI
18:46:20:Video adapter info:
18:46:20: Intel(R) HD Graphics:
18:46:20: Driver Version 10.18.10.4425
18:46:20: Video Processor Intel(R) HD Graphics Family
18:46:20: NVIDIA Tesla P4:
18:46:20: Driver Version 31.0.15.5161
18:46:20: Video Processor
18:46:20:STARTING CODEPROJECT.AI SERVER
18:46:20:RUNTIMES_PATH = C:\Program Files\CodeProject\AI\runtimes
18:46:20:PREINSTALLED_MODULES_PATH = C:\Program Files\CodeProject\AI\preinstalled-modules
18:46:20:MODULES_PATH = C:\Program Files\CodeProject\AI\modules
18:46:20:PYTHON_PATH = \bin\windows\%PYTHON_NAME%\venv\Scripts\python
18:46:20:Data Dir = C:\ProgramData\CodeProject\AI
18:46:20:Server version: 2.5.6
18:46:23:
18:46:23:Module 'Object Detection (YOLOv5 3.1)' 1.9.1 (ID: ObjectDetectionYOLOv5-3.1)
18:46:23:Valid: True
18:46:23:Module Path: <root>\modules\ObjectDetectionYOLOv5-3.1
18:46:23:AutoStart: True
18:46:23:Queue: objectdetection_queue
18:46:23:Runtime: python3.7
18:46:23:Runtime Loc: Local
18:46:23:FilePath: detect_adapter.py
18:46:23:Pre installed: False
18:46:23:Start pause: 1 sec
18:46:23:Parallelism: 0
18:46:23:LogVerbosity:
18:46:23:Platforms: all,!macos-arm64
18:46:23:GPU Libraries: installed if available
18:46:23:GPU Enabled: enabled
18:46:23:Accelerator:
18:46:23:Half Precis.: enable
18:46:23:Environment Variables
18:46:23:APPDIR = <root>\modules\ObjectDetectionYOLOv5-3.1
18:46:23:CPAI_MODULE_ENABLE_GPU = True
18:46:23:DATA_DIR = C:\ProgramData\CodeProject\AI
18:46:23:MODE = MEDIUM
18:46:23:MODELS_DIR = <root>\modules\ObjectDetectionYOLOv5-3.1\assets
18:46:23:PROFILE = desktop_gpu
18:46:23:TEMP_PATH = <root>\modules\ObjectDetectionYOLOv5-3.1\tempstore
18:46:23:USE_CUDA = True
18:46:23:YOLOv5_VERBOSE = false
18:46:23:
18:46:23:Started Object Detection (YOLOv5 3.1) module
18:46:26:Server: This is the latest version
18:49:12:Sending shutdown request to python/ObjectDetectionYOLOv5-3.1
18:49:12:detect_adapter.py: Object Detection (YOLOv5 3.1) started.
18:49:13:Module ObjectDetectionYOLOv5-3.1 has shutdown
18:49:13:detect_adapter.py: has exited
18:49:45:ObjectDetectionYOLOv5-3.1 went quietly
18:49:45:
18:49:45:Module 'Object Detection (YOLOv5 3.1)' 1.9.1 (ID: ObjectDetectionYOLOv5-3.1)
18:49:45:Valid: True
18:49:45:Module Path: <root>\modules\ObjectDetectionYOLOv5-3.1
18:49:45:AutoStart: True
18:49:45:Queue: objectdetection_queue
18:49:45:Runtime: python3.7
18:49:45:Runtime Loc: Local
18:49:45:FilePath: detect_adapter.py
18:49:45:Pre installed: False
18:49:45:Start pause: 1 sec
18:49:45:Parallelism: 0
18:49:45:LogVerbosity:
18:49:45:Platforms: all,!macos-arm64
18:49:45:GPU Libraries: installed if available
18:49:45:GPU Enabled: enabled
18:49:45:Accelerator:
18:49:45:Half Precis.: enable
18:49:45:Environment Variables
18:49:45:APPDIR = <root>\modules\ObjectDetectionYOLOv5-3.1
18:49:45:CPAI_MODULE_ENABLE_GPU = True
18:49:45:DATA_DIR = C:\ProgramData\CodeProject\AI
18:49:45:MODE = MEDIUM
18:49:45:MODELS_DIR = <root>\modules\ObjectDetectionYOLOv5-3.1\assets
18:49:45:PROFILE = desktop_gpu
18:49:45:TEMP_PATH = <root>\modules\ObjectDetectionYOLOv5-3.1\tempstore
18:49:45:USE_CUDA = True
18:49:45:YOLOv5_VERBOSE = false
18:49:45:
18:49:45:Started Object Detection (YOLOv5 3.1) module
modified 29-Mar-24 11:06am.
|
|
|
|
|
|
Thank you! That did help keep YOLOv5 from crashing but YOLOv5 still only uses the CPU and no option to enable GPU. Whats odd is that BlueIris is responding 50x faster as before I installed the Tesla P4. Here is the system info:
Server version: 2.5.6
System: Windows
Operating System: Windows (Microsoft Windows 10.0.19045)
CPUs: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz (Intel)
1 CPU x 4 cores. 4 logical processors (x64)
GPU (Primary): Intel(R) HD Graphics (2 GiB) (Intel Corporation)
Driver: 10.18.10.4425
System RAM: 16 GiB
Platform: Windows
BuildConfig: Release
Execution Env: Native
Runtime Env: Production
Runtimes installed:
.NET runtime: 7.0.10
.NET SDK: Not found
Default Python: Not found
Go: Not found
NodeJS: Not found
Video adapter info:
Intel(R) HD Graphics:
Driver Version 10.18.10.4425
Video Processor Intel(R) HD Graphics Family
NVIDIA Tesla P4:
Driver Version 30.0.14.7168
Video Processor Tesla P4
System GPU info:
GPU 3D Usage 0%
GPU RAM Usage 4 MiB
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168
|
|
|
|
|
Could you please run the nvidia-smi command and then the nvcc --version command and let me know what it shows?
Thanks,
Sean Ewington
CodeProject
|
|
|
|
|
[nvidia-smi.exe]
Unable to determine the device handle for GPU 0000:01:00.0: GPU is lost. Reboot the system to recover this GPU
[nvcc --version]
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:41:10_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
[nvidia-smi.exe] after reboot
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 471.68 Driver Version: 471.68 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla P4 WDDM | 00000000:01:00.0 Off | 0 |
| N/A 34C P8 7W / 75W | 146MiB / 7680MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 852 C+G Insufficient Permissions N/A |
+-----------------------------------------------------------------------------+
modified 12-Mar-24 13:27pm.
|
|
|
|
|
Okay, just a random thought: Do you know if your YOLOv5 was installed with the dependencies necessary to run on GPU? Perhaps it was 'not detected' previously, so those required dependencies were not downloaded?
Maybe, worth to try to reinstall YoloV5?
|
|
|
|
|
Just reinstalled YOLOv5. GPU still not showing available to enable. Thank you for the suggestion though.
|
|
|
|
|
it may not be relevant but I am having similar issues with YOLO not working on 2 CodePorject installs. Both systems where I am having problems are using a Tesla P4 with the Nvidia Grid drivers, CUDA 11.8 and CUDNN 8.9.
CodeProject.AI Server: AI the easy way.[^]
|
|
|
|
|
Sounds very similar. Does your object detection work as if there is no issues... as if the Tesla is processing images?
|
|
|
|
|
Actually, no. I took a picture from one of my BlueIris alerts and tried Object detection in explorer. For both systems where I am having issues object detection immediately reports back "No predictions returned" I took the same picture and ran it through object detection on a temp machine I threw an OS and CodeProject on just to see what would happen and it works perfectly, it correctly detected a person, a car, even a backpack. I installed CodeProject exactly the same way on all 3 systems and the OS is the same, the only difference is the hardware. I can't imagine why the Tesla GPU would cause this but it's starting to feel like that's part of my issue.
|
|
|
|
|
I just noticed that you are using YOLOv5 3.1. And based on your first post, your Tesla P4 is having CUDA 12.4:
18:46:20:GPU (Primary): Tesla P4 (8 GiB) (NVIDIA)
18:46:20: Driver: 551.61, CUDA: 12.4 (up to: 12.4), Compute: 6.1, cuDNN: 8.9
Based on the description of YOLOv5 3.1, it is for CUDA 10 or 11 for older GPUs
Object Detection (YOLOv5 3.1)
2024-02-08
GPL-3.0Provides Object Detection using YOLOv5 3.1 targeting CUDA 10 or 11 for older GPUs.
Project by Chris Maunder, Matthew Dennis, based on Deepstack. Uses Python, PyTorch, YOLO. Your CUDA version is above the 3.1's description. Maybe that's why it is not using GPU?
Now, if you look at the YOLOv5 6.2's description, it is for 11.5+ which matches with your Tesla P4's setup:
Object Detection (YOLOv5 6.2)
2024-02-08
GPL-3.0Provides Object Detection using YOLOv5 6.2 targeting CUDA 11.5+, PyTorch < 2.0 for newer GPUs.
Project by Matthew Dennis, based on Ultralytics YOLOv5.
My computer that has Tesla P4 is using YOLOv5 6.2 and it is using the GPU.
I hope this solves it.
|
|
|
|
|
Thank you! Based on your suggestion I did a full tear-down and used the GRID driver, CUDA 11.8, cuDNN-8.9.4 and YOLOv5 6.2. Everything worked for a good 10 minutes and then it crashed. Do you recommend a specific CUDA version?
|
|
|
|
|
This is what I have for my GPU:
GPU (Primary): Tesla P4 (8 GiB) (NVIDIA)
Driver: 538.15, CUDA: 12.2 (up to: 12.2), Compute: 6.1, cuDNN: 8.9
I'm using the driver version of 538.15 from the following link: Drivers for NVIDIA RTX Virtual Workstation (vWS) | Compute Engine Documentation | Google Cloud[^].
Note: Left side of 538.15 says the CUDA version is 16.3, I don't know why it is saying that because it seems to be incorrect.
If possible, please post the log so we know what errors your CodeProject AI ran into. It should give some idea what caused the crash. And if it is something deeper, it might be helpful for Sean and others.
|
|
|
|
|
OMG! Thank you! Driver: 538.15 updated and CUDA has been going strong for 12 hours! I believe this has now been resolved; I can now do other AI projects.
[Issue Resolved]
|
|
|
|
|
That is great news! I hope it just keeps going non-stop!
|
|
|
|
|
I performed a fresh install on Ubuntu server 24.04/CUDA/CUDNN/CodeProject.AI and encountered the following issues.
- Installation of CodeProject.AI and Dependencies Failure: The installation of CodeProject.AI itself completes without any issues. However, during the "FINAL REQUIREMENTS" portion of the installation of additional required components for Face Processing and Object Detection (YOLOv5 6.2), I encountered problems.
pushd "/usr/bin/codeproject.ai-server-2.5.4/" && bash setup.sh && popd
- Running the CodeProject.AI-Service Instability: Although I could enable and start the CodeProject.AI-Service, I noticed that it was not stable—shutting down and restarting every 5 to 30 seconds.
Server version: 2.5.4
System: Linux
Operating System: Linux (Ubuntu 22.04)
CPUs: Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (Intel)
1 CPU x 4 cores. 4 logical processors (x64)
GPU (Primary): NVIDIA RTX A2000 12GB (12 GiB) (NVIDIA)
Driver: 550.54.14, CUDA: 12.4 (up to: 12.4), Compute: 8.6, cuDNN: 9.0.0
System RAM: 16 GiB
Platform: Linux
BuildConfig: Release
Execution Env: Native (SSH)
Runtime Env: Production
.NET framework: .NET 7.0.16
Default Python: 3.10
Go Version:
Video adapter info:
Device 1234:
Driver Version
Video Processor
GA106 [RTX A2000 12GB] (rev a1):
Driver Version
Video Processor
System GPU info:
GPU 3D Usage 0%
GPU RAM Usage 884 MiB
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168
modified 19-Apr-24 11:51am.
|
|
|
|
|
There is a known bug in 2.5.4 that prevented the server from starting.
Please upgrade to 2.5.6.
We have pull the 2.5.4 release.
"Mistakes are prevented by Experience. Experience is gained by making mistakes."
|
|
|
|
|
Thank you @matthew-dennis! Please excuse my ignorance, but where do I download 2.5.6, I'm only seeing 2.5.4?
|
|
|
|
|
I'm facing same problem... 2.5.6 seems only available for windows and not for native Linux...
|
|
|
|
|
At least I have company now. I thought I was just really daft being unable to find the 2.5.6 linux version. In the meantime, I just keep a tmux session open running:
sudo bash /usr/bin/codeproject.ai-server-2.5.4/start.sh
|
|
|
|
|
Hi Matthew, I have since upgraded to 2.6.2 and experience nearly identical behaviour. Following the same steps above gets me up and running with the missing libraries and unfortunately utilizing the service continues to start and stop.
|
|
|
|
|
Are you running the installer under sudo?
cheers
Chris Maunder
|
|
|
|
|
Hi Chris, yes, I ran the installer as sudo. I looked in my history and the exact command was:
sudo dpkg -i codeproject.ai-server_2.6.2_Ubuntu_x64.deb
Please let me know if there is anything you would like me to try.
|
|
|
|
|
We've just updated the Ubuntu version to 2.6.4. Curious to know if 2.6.4 works better for you.
Thanks,
Sean Ewington
CodeProject
|
|
|
|
|
Thanks Sean! Okay, this looks MUCH better. Everything seemed to go without a hitch. I didn't have to manually install any libs, the service fired up, no shutdowns and appears stable after a reboot.
I did observe the following logs errors, but it otherwise all appears to be working:
Pillow reported an initial failed check, then subsequently reported already installed.
10:12:01:FaceProcessing: Python packages will be specified by requirements.linux.cuda11_5.txt
10:12:09:FaceProcessing: - Installing Pandas, a data analysis / data manipulation tool... (✅ checked) done
10:12:19:FaceProcessing: - Installing CoreMLTools, for working with .mlmodel format models... (✅ checked) done
10:12:25:FaceProcessing: - Installing OpenCV, the Open source Computer Vision library... (✅ checked) done
10:12:26:FaceProcessing: - Installing Pillow, a Python Image Library... (❌ failed check) done
10:12:33:FaceProcessing: - Installing SciPy, a library for mathematics, science, and engineering... (✅ checked) done
10:12:33:FaceProcessing: - Installing PyYAML, a library for reading configuration files...Already installed
10:13:23:FaceProcessing: - Installing PyTorch, an open source machine learning framework... (✅ checked) done
10:14:23:FaceProcessing: - Installing TorchVision, for working with computer vision models... (✅ checked) done
10:14:34:FaceProcessing: - Installing Seaborn, a data visualization library based on matplotlib... (✅ checked) done
10:14:34:FaceProcessing: Installing Python packages for the CodeProject.AI Server SDK
10:14:35:FaceProcessing: Searching for python3-pip...All good.
10:14:36:FaceProcessing: Ensuring PIP compatibility... done
10:14:36:FaceProcessing: Python packages will be specified by requirements.txt
10:14:37:FaceProcessing: - Installing Pillow, a Python Image Library...Already installed
10:14:37:FaceProcessing: - Installing Charset normalizer...Already installed
10:14:41:FaceProcessing: - Installing aiohttp, the Async IO HTTP library... (✅ checked) done
10:14:43:FaceProcessing: - Installing aiofiles, the Async IO Files library... (✅ checked) done
10:14:45:FaceProcessing: - Installing py-cpuinfo to allow us to query CPU info... (✅ checked) done
10:14:45:FaceProcessing: - Installing Requests, the HTTP library...Already installed
I don't ultimately use the following, but see a "self-test failed" from the auto install on launch.
10:14:57:Module FaceProcessing started successfully.
10:15:15:ObjectDetectionYOLOv5Net: Downloading ObjectDetectionYOLOv5Net-CUDA-1.10.1.zip...Expanding... done.
10:15:15:ObjectDetectionYOLOv5Net: Moving contents of ObjectDetectionYOLOv5Net-CUDA-1.10.1.zip to bin...done.
10:15:25:ObjectDetectionYOLOv5Net: Downloading YOLO ONNX models...Expanding... done.
10:15:25:ObjectDetectionYOLOv5Net: Moving contents of yolonet-models.zip to assets...done.
10:15:31:ObjectDetectionYOLOv5Net: Downloading Custom YOLO ONNX models...Expanding... done.
10:15:31:ObjectDetectionYOLOv5Net: Moving contents of yolonet-custom-models.zip to custom-models...done.
10:15:31:ObjectDetectionYOLOv5Net: Scanning modulesettings for downloadable models...No models specified
10:15:32:ObjectDetectionYOLOv5Net: Self test: Self-test failed
10:15:32:ObjectDetectionYOLOv5Net: Module setup time 00:00:37
10:15:32:ObjectDetectionYOLOv5Net: Setup complete<br />
10:15:32:ObjectDetectionYOLOv5Net: Total setup time 00:00:38
10:15:32:Module ObjectDetectionYOLOv5Net installed successfully.
Thanks again for this update!
|
|
|
|