this is what happens when mechatronics goes bad

After watching a long print job that started well, I went to bed. This is what I found in the morning:


Not shown is a burnt out 12V wire, no fan sounds, no comm with webcontrol, and the hot-end assembly ripped out of the effector platform. Oh my.

The last day or two I was getting random going-crazy-jumping-and-jamming-into-the-bed at some point while doing auto-calibration patterns. (its a delta). But I kept repeating until sucess then ran print jobs without a problem.
I had the bed routine working pretty hard and thought a stepper was skipping so I mellowed the moves and got no more out-of-control behavior.

I suspect it went crazy long into the print job as opposed to catching on a lifted part or something. The clue was a pattern of heavy melted craters clustered on the print itself where the nozzle would have banged into the bed really hard while jumping around.

Apparently this went on for some time before self-destructing.
So far I have fixed the problems and there is no apparent damage (yet) to the hot-end or remote-direct-drive (Zesty). But although there are lights on the Duet Ethernet, I can't talk to it.

-I ordered a replacement Duet but I'm concerned:
-What was the source of the original erratic motions? If the Duet, why? If not, I may damage the new Duet as well. There isn't much else however to blame except wires, motors, LEDS, and switches, none of which can explain this behavior.

If I recall correctly, my accels were at 5000, jerks at 400, and my print speed sliced for 80 mm/min. I rarely run this fast but it's a wide thin part with lots of straight moves and at .2 layer x .4 width it's well below the maximum flowrate for the e3d hotend. In fact, up until it crashed I was getting a *flawless* finish with this eSun PLA+. (sob) (and I mean flawless as in no visible nozzle stroke lines)

I'm am now converging on the speed thing/accel thing as the culprit. I've been running the aggressive bed probing for a while now without problems but I've never run 80mm/min on a job. Maybe it was hurting the board, or stressed a weak part on the pcb?

Thoughts?

Edited 1 time(s). Last edit at 11/26/2018 06:08PM by shadowphile.

Drivers getting hot for the steppers at that speed with those accelerations? What current values are you running on each axis? What's the MCU temp at idle, and then what's the MCU temp when you start running at those speeds? Does your Duet have cooling fans underneath the drivers or any sort of air flow across the board?

It could be one f your axis drivers are getting too hot and eventually starts to fail. This is a complete guess, but might explain why it works while probing at low speeds but fails at high speeds.

Thanks for the reply PDBeal.

Unfortunately I don't have some of that info. I can't verify the drivers were getting hot and at a glance nothing looked like it lost its smoke. I thought the drive chips had thermal protections.
I haven't messed with the drive current limits, so maybe 1 or 1.5 amps? The motors are NEMA 17s but my delta effector is rather heavy (unfortunately, for now).

I was relying on the passive cooling but did plan on adding some kind of active cooling but only as a performance tweak. Sounds like not so optional!

Do you have a suggestion about how I can test my limits? Indirectly monitor the driver temps via the MCU, but what else?
thanks again

I'm sorry to hear of your misfortune.

From the picture, it looks like your magnetic joints detached. This may have been caused by the nozzle hitting a curl up or blob at high speed. Using a small amount of Z lift for travel moves can help avoid that.

The burned out 12V wire may be a clue. What wire was it? If it's the main VIN wire, then check the terminal block too. If that's burned, then the screw worked loose (perhaps because the wire was incorrectly terminated), which would have led to the motors cutting out when the voltage at the Duet got too low.

Acceleration of 4000 is high, and may be beyond what your machine can achieve reliably.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

all good suggestions.
Im pretty sure its all collateral damage though, to whatever caused the motor going chaotic. No sign of a peel up to snag on. Motor went crazy but job dont know that and beats things around until something dies (usually messily). It must have shorted a 12V line that feeds some LEDS. Still havent looked closely at the Duet but it probably didnt like its power rail violently switched to ground:
To Do: add cooling to the duet and c:arefully evaluate my performance limits. Ive printed max flowrate before but not at highspeeds. 4000 has always been a good accel for me, but perhaps because I was always printing at lower speeds.

I have some small copper heat sinks just right for the driver chips and I figured I might as well put them to use since I will be improving the cooling.
Should I put them on top of the chip or underneath? I can think of arguments for either one. I like underneath because easier place to mount the fan and heatsinks.
Same question about fan placement.
I just got my new board today so unless somebody stole if off of my front porch I will be facing these choices in a few hours.
thanks

Quote
shadowphile
I have some small copper heat sinks just right for the driver chips and I figured I might as well put them to use since I will be improving the cooling.
Should I put them on top of the chip or underneath? I can think of arguments for either one. I like underneath because easier place to mount the fan and heatsinks.
Same question about fan placement.

None of the above! See [duet3d.dozuki.com]. The fan should be at the edge of the Duet between the VIN terminal block and the nearer corner, so that it blows cool air along the row of driver chips on both sides of the PCB.

It's probably better not to use heatsinks at all. Places on the back, they risk creating a short. Placed on top of the driver chips, they might improve cooling of the chip closest to the fan a little, but they will reduce the airflow to the chips further downstream.

Btw the TMC2660 driver chips produce an over-temperature warning (which is reported by the firmware) well before they go into shutdown mode and miss steps. So if you have never seen any over-temperature warnings, it's most unlikely that they went into over-temperature shutdown.

Edited 4 time(s). Last edit at 11/29/2018 04:40AM by dc42.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

crap. that puts me back in head-scratching territory. i want to add a 'misbehaving' safety mechanism in but its not easy defining what is expected or not

Some people who use magnetic joints on deltas use wires as a backup to prevent the joints detaching if the effector its a bump. I've not found this necessary using Haydn Huntley's magnetic joints because they are very strong. However, I don't generally leave my printers running overnight.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

I've always wonder about deltas that use the magnetic joints. Does that introduce a limit to acceleration/jerk that can be used, or are the magnets normally strong enough to keep the effector attached regardless of motion settings? Are some joints better than others in that regard?

---

Ultra MegaMax Dominator 3D printer: [drmrehorst.blogspot.com]

I have seen several designs of magnetic joints, some very good (providing lots of attachment force all joint angles), and some much poorer ones. Somebody said magnetic joints were no good on these forums a few weeks ago, and in response I tested my delta with a lot of additional weight suspended from it. See [reprap.org]. Maybe I should publish a video to prove the point.

[Sorry, I had to censor the link because the forum software considers it an offensive word.]

Edit: maybe it will let me add the link in this edit, here goes: no that didn't work.

Edit 2: its the post AFTER this one: no, it seems that link is offensive too.

Edit 3: here's the link, but you need to remove the embedded spaces I added: [reprap.org]

*** EDIT *** - fixed the broken link ...

Edited 5 time(s). Last edit at 11/29/2018 05:18PM by VDX.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

... I made my first delta demonstrator not with joints, made form single magnets and a steelball, but with a sort of "self-adjusting" joints with two spheric neodymium magnets - the holding forces was more than enough and they "self-assembled", when torn apart by force and released again.

The main "problem" with them were the more complex calculation -- this weren't singular joints, but a combined/synchronized pair of joints separated by the spheres diameters.

Here is a pretty old (11 years!) video, showing the kinematics:

[vimeo.com]

---

Viktor
--------
Aufruf zum Projekt "Müll-freie Meere" - [reprap.org] -- Deutsche Facebook-Gruppe - [www.facebook.com]

Call for the project "garbage-free seas" - [reprap.org]

At first I didn't trust magnetic ball-joints because of their inherent 'brittleness' so I went with the Trick Laser ball-cap rod ends. They work as well as the magball joints but probably not as popular because harder to print your own accurately and not as 'clean', and more parts. I didn't like the offset springs, which interfered with my particular effector, and the spring loops can break off and I ended up fixing with cable ties.

Nobody seemed to have an issue with the magballs though, so I switched over when designing the next upgrade of my printer. NOW I start reading about issues people have with them! The only issue I saw with nozzle impacts popping joints was when I tried to travel really fast. I dropped it down to reasonable speeds like 150 and now the nozzle has time to thermally mow through any ridges (although really thick extrusions such as from the Volcano might not mow down so well)
They are very clean and work well and I LOVE being able to knock them off in a heart-beat to invert and inspect the underside of the stage. But I don't like the fact that manhandling the stage, such as struggling with a filament, can pop joints without warning. So now I'm back to preferring the ball-cap design, but I would design spring forces that are inline with the two balls and isolated to each ball/ball-cap pair alone, not spanning the set.

The think the most important question to ask yourself though is when do you want the system to fail in order to minimize damage? Formula cars look spectacular now with all the flying pieces that suck energy out of the system to protect the driver. Sometimes you WANT lesser destruction to protect a greater asset (like a human being). Now I can't watch the old formula car races with the cigar-shaped bodies without wincing in awe at the shear brutality on the driver.

Edited 1 time(s). Last edit at 11/29/2018 08:32PM by shadowphile.

I would like to hear more details about the instructions for best cooling. Cross-flow is likely to be laminar and it's standard knowledge that:
(copying from Wikipedia)
"When fluid is pumped through these channels, it can develop two basic characteristics: laminar or turbulent flow. Characterized by turbulence, where fluid does not flow in smooth layers but is agitated. Turbulent flow, due to the agitation factor, develops no insulating blanket and heat is transferred very rapidly"

I'm not saying the duet instructions are wrong or that there aren't mitigating details, but at first appraisal, forced air straight onto a hot surface will remove heat faster.
(straight into the hot surface drills air through any insulating air layer, plus making the air turn 90 degrees creates a lot of turbulence).

I have my new Duet and (finally) a free evening to really study the wreckage. I will let you know if I find anything interesting.

BTW, how do you do really long runs if you don't print while asleep? Stay up for 20 hours? I do have to work weekdays but I can get a 40+ hour run in during the weekend.

When you mount and unmount the effector on magnetic joints, is there any problem with the magnets chipping?

---

Ultra MegaMax Dominator 3D printer: [drmrehorst.blogspot.com]

magnets are recessed inside the plastic forms that contact the ball. all the friction is on the plastic.

Quote
shadowphile
I'm not saying the duet instructions are wrong or that there aren't mitigating details, but at first appraisal, forced air straight onto a hot surface will remove heat faster.
(straight into the hot surface drills air through any insulating air layer, plus making the air turn 90 degrees creates a lot of turbulence).

And you get a dead zone in the centre where there is very little airflow or cooling.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Quote
shadowphile
BTW, how do you do really long runs if you don't print while asleep? Stay up for 20 hours? I do have to work weekdays but I can get a 40+ hour run in during the weekend.

The only really long prints I have done for a while were on my Scara printer, which doesn't have a heated bed. At the end of the day I paused the print and powered the printer down. Then used the resurrect facility the next day.

If I wanted to do a really long print on my delta, I'd probably pause the print and turn the hot end off overnight, leaving the bed heater running although perhaps at a lower temperature than usual. The motors will go down to idle current after 30 seconds. Then in the morning, heat the hot end, re-prime the nozzle, and resume the print. All this with a smoke alarm in the room with the printer.

Edited 2 time(s). Last edit at 11/30/2018 02:20AM by dc42.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

can't argue with that dead center part, if all the flow path is mostly smooth.

BTW, how does the Duet provide the steppers-getting-too-warm warning? When my system went down I lost comm with the browser so anything logged there was lost, or maybe got lost when I poked around it before realizing I might want to see what the last messages were. I will start logging to a file. Does the logging command overwrite the log file each time if the file is always the same name? Appending might be better, or even better, a time-stamped filename.

Your precautions are more than I usually consider, especially resuming jobs after a long pause, but then I have not been confident about positional stability until this current build. As for safety, I think I've still been on my honeymoon. I guess I have to consider this event a wake-up call. :/ Considering that my bed is AC powered, I *DO* have a thermal fuse on the silicone pad.

Now that I was forced to buy a new Duet, I have that extra serial port again. My auxiliary Arduino was for macro buttons, hard knobs, and special conveniences but it will also be useful as an independent backup to watchdog the Duet or anything it reports that might be suspicious.

I'm still pondering the most global way to monitor when the physical behaviour has exceeded some kind of performance or positional boundary. Strain sensors on the rods or perhaps contact-sensing of some kind of the rod joints? Excessive force on the nozzle when not expected ? (via the z-probe sensor). Over-current to the steppers? All of these? (sigh) edit: I just discovered stall detection although it sounds like it could be difficult to use without false positives.

Edited 1 time(s). Last edit at 11/30/2018 02:24PM by shadowphile.

OK, here is my final report on repairing my machine.

-The new Duet came up fine, but I can reproduce the same problem bed-probing with the same high performance settings. That problem goes away if I lower the forces. I tried raising the motor current limit from 1.0 to 1.7 (extreme, just for this test) and surprisingly had the same behavior, which seems to indicate the motor may be skipping for some other reason. I'm not a stepper expert so will have to study this more.
-The old Duet talks fine over USB so I think the only thing that died is the ethernet module. Can I purchase that? I have no use for a second Duet board but a spare could be useful for replacement or mods.

Driver chip heating:
-I used a fine-gauge TC taped down with Kapton on one of the driver chips, then closed it up and ran a heavy job with the bed at 40c. The chip sit at 45c with no printing, and plateaued at 55.
-I don't have the room for a proper air-flow with a fan on the driver chips. However, something is better than nothing and my closed-off interior below the bed had no ventilation. I cut a big hole under the Duet with a thick 80mm fan firing straight up to the underside of the laterally-mounted Duet, and now the driver chips never exceed 42c under a regular but heavy job. Been wanting to get some active cooling in there for some time anyway.

My effector stage is fairly heavy so that probably explains why I can't jump around very fast. I designed for accuracy not speed and now I'm missing the speed .

I wish there was an easy way to test the limits of my machine. I ran fine for hours at 80 mm/s before something went wrong. I'm sure it also depends a LOT on the type of job.

lol you found the limits of your machine it appears.