HUGE Rendering Problem GPU

OK, after 2 sleepless nights :smiley: i finally solved it. Reinstalled OS X, Yosemite, CUDA, Nvidai Web driver… and nothing… took from time machine 4 TB data back, over long hours selectively to have it as clean as it gets… and then still nothing… :stuck_out_tongue:
Installed latest official and did a lot of animation tests… over 12 hurs long… still the same problem below :stuck_out_tongue: soon i nearly gave up…
Booted in Windows… worked perfect… lol…

THE PROBLEM: It is not cuda, or the drivers, it were .py script addons that manipulates the geometry at render time. Somehow Cycles loses sync between the GPUs and fills the whole GPU memory. Some weird .py interaction is happening.

THE SOLUTION:

I had to delete all addons in the blender directory and delete any preferences and re-set blender settings fresh. Now i have no 3rd party rig system, but it will be ok, due to the simple animation i have to do.
The only things i kept were the clearly not harmful themes and shaders.

At least i have the production system fresh and TM restored with Yosemite. Now it works flawless. :slight_smile:

The good side: I did with this occasion a lot of cleanup :smiley: and ordering in my file system. :wink: Always look on the bright side of life. :stuck_out_tongue:

Gosh was i desperate… :slight_smile: Now i can sleep. LOL


Hello,

i have a huge rendering problem now with 2.72a and b with rendering out ANY animation.
The frames are rendered, but after the 3rd or 4th frame, the cards simply jump over the tiles, not rendering anything, and if i let it so for 10 secs and do not abort, it will crash blender completely and stuck at building BHV!

Viewport rendering is also affected in another style: it won’t crash, but after switching Rendering On/Off a few times, CX CUDA Memory error or Sync Error will happen although not crashing blender in this case (this although will happen with memory intensive stuff, yet still smaller than about half of the GPU free memory!).

In BOTH cases, you can’t even render after that even a still, i need to close and open blender again. Then it works again, but with the same issues. A restart won’t fix it either.

The cards are 780 6GB Models and the scene is really small 200mb ? OSX Mavericks 10.9.5, Official Nvidia Webdriver, latest CUDA.

This even happens if i choose a single card only. No matter the scene, heavy or light or any shaders. Also this happens both on experimental AND supported GPU.

Everything else works perfectly, it can’t be CUDA. Octane Render, LuxRender, Adobe Mercury Engine, DaVinci, etc… flawless.

i7 3770K - 16GB - 3x780 6GB - OSX

Something is not ok i guess with Cycles and CUDA.

I need to render out on 2.72, AND i need ALL the GPUs. Please help!

UPDATE: This also happens on another IDENTICAL non-render machine with Yosemite but only two 780 6GB cards.

UPDATE 2: Latest buildbot blender-2.72-88fe896 same issues.

PS: I have the feeling that somewhere between 2.70 and 2.72 the sync of the gpus got messed up for animation ?

Hi enilnacs, I had no problem to render animation on my Cuda cards lately, use own build a few days ago (specs in signature).
If there where problems with Cuda animations since 2.71 we had a lot of bug reports in the bug tracker.
I is may a OSX only problem, btw. how do you get 3 GTX 780 to work on a MacPro?
The power supply is way to small for it.
May nobody use Cuda on OSX, most OSX user have very good CPU`s.
You can post a bug report to the tracker with sys specs and test file.
If it is possible to reproduce on other systems the developer can fix it.

https://developer.blender.org/

Cheers, mib

SOLVED: Look at the first post for the solution.


Yes, it is not a Mac Pro but a Hackintosh that runs out of the box. It is production proven since 2 years, and since Blender on Mountain Lion using official Nvidia Drivers.
The Hardware is not the issue, any CUDA program works in really heavy EXR production flawless. Ok, i will revert to 2.70, and maybe it will be fixed till 2.73 but before that i will test an little older CUDA version, maybe there are some changes… and open an ticket like you said.
Still many thanks for the reply.

UPDATE:
I made some CUDA memory diagnostics and i found out that as it renders it FILLS all the GPU memory till its full, and then not even quitting with an error but “rendering” transparent tiles till it crashes, one card after another. The primary (less memory due to displays) being the first to get into transparent “rendering”.

Conclusion: it doesn’t flush or only update BVH or frame buffer into the memory before rendering next frame, but launches another BVH render scene, one after another till the GPU RAM is full. Scene is only 97mb, a test sphere. If i stop the rendering before crashing, i have only 124 MB free from 6GB GPU ram.
Tried all combinations under Performance panel and turned off Motion Blur, etc.
Need to get into this deeper and fill a ticket.

I’ve been having the same problem rendering with Cycles and not updating BVH data. My scene renders on 2.69 perfectly. it has hair particles NO animation. But it has NOT successfully rendered on ANY builds since 2.69. I even increased memory within Blender thinking that was the problem. It crashed 2.71 and 2.72.

Im using a Mac Pro OX 10.9.3 (trash can) with 32 Gb. I tried deleting addons screwed the entire way the file displayed. Going to add the back one by one. I read some other place that the developers think that running out of memory IS NOT A BUG. How can a regressive step not be a bug. SHRUGS