My Duet 0.6 set fire to my print and x axis (yes) - Need help to understand what happened

Hey. Yes, my print literally caught fire. Getting the spectacular part out of the way first:



I smelled the smoke and a few second after I entered the room the fire alarm in the room also went off. The fire extinguisher killed the fire no problem. Despite having burned for some time it only killed the X axis and the print. Even the glass build plate is OK and the bed is entirely unmarked. It did give me a really big scare and it has made me reevaluate how to make a heated enclosure.


It was my fourth print or so after changing to Duet and I was still in the process of setting and testing.

I believe the chain of event is the following:

- The way I had secured the wiring harness to the X-carriage got it self loose so the cable started pulling directly on the heating cartridge

- Cartridge fell out and onto the print.

- After some time the heating cartridge is hot enough to set fire to an ABS print (my post experiment shows)

- After some time the belts burned and it stopped moving


Now, it is obviously my mistake that the cable harness got it self loose. I should not have entered it into unsupervised test (I was watching TV in the room next door) until such was OK. Once I understand this event completely I will make a post on the safety board about this learning.


BUT - Why did the DUET not save me?! The RAMPS which I have been using up-to now with Repetier software has an advanced system that detects if the temperature is not changing in 18s. If it is not changing then it turns everything off. This safety system is even given some sort of watchdog priority so if the Arduino freezes it kills it self.


The only thing I see that the Duet has is M570 which is pre-set to 120s. 120s is too long - But I tested it after the accident and M570 wont even work. It did not turn off the heating cartridge even after the 120s. At least not when issued from web panel.


I hope I am wrong and that there are something completely obvious I missed, but if I am right, then I really think the Duet needs an urgent safety overhaul as the Duet is so heavily promoted here among DIY printers.


The fire is actually some days old as I have spent the time investigating some of these things - And cleaning... Everything has this thin layer of soth that only hot soap-water gets off


1) Here is a test I made to see how long time it takes a heating cartridge to set fire to a print from cold.

[goo.gl]

In this case it took 90s. I am sure that had the cartridge been moving it would have been faster.



2) This is one that scares me the most - Here you see that the heater never turns it self off - I don't understand why? The Duet webpanel loses connection doing the "test". Is these safety features really not layered with highest priority? My heating cartridge is glowing red for 4 min until I turn it off.

[goo.gl]

The M570 hot end heating timeout certainly does work, because quite a lot of people find that during initial heating of the hot end, the heater cuts out before reaching the target temperature unless they increase the timeout.

In any 3D printer design, it's vital to make sure that the heater and thermistor cannot fall out of the heater block. Good hot end designs such as the E3D one have clamps to make sure they can't fall out. If you use a hot end design that doesn't secure the heater and thermistor to the heater block, then it's up to you fix that. The heater and thermistor wires should always be secured in place so that they can't droop and snag on anything.

In your case the problem happened after the initial heating phase, so there was no problem for the heater timeout to detect. Likewise I am fairly sure that the 18s timeout in Repetier is only active during the initial heating phase.

I'm glad you took the sensible precaution of having a suitable fire extinguisher at hand.

Edited 1 time(s). Last edit at 07/12/2016 03:45AM by dc42.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

You completely neglect to comment on the fact that repetier has a software that turns off after 18s because it is better designed.

It is looking at the heat change. Not absolute values.

You are hammering these forums telling everybody with ramps that they should upgrade to duet. You know many of these people are using the Chinese hotend.

Now, instead of accepting that there is something urgent to be upgraded you say this is a purely mechanical error.

I am very disappointed by this reply.

Finally. Watch my video. When started from the webpanel it does not (!) turn off the heater after 120s.

You have an inferior part of your firmware. Now I told you. Can you live with it, if you read someone dident have the precautions like me, and literally burn down their house in parts because you dident take the time to implement good safety?

Just to be crystal clear. In repetier the routine is always on. Whenever it is supposed to change temperature but it does not it counts 18s ( or other set value). If it does not start to follow in this time it turns off. I believe it is build into the controller

Are you quite certain that Repetier will turn off after 18s if the heating fails during a print, rather than at the start? It's not straightforward to work out that something is amiss once the target temperature has been reached and PID is operating, because the controller may turn the power up and down many times in 18 seconds, and the temperature changes lag the heating power. There are other algorithms that could be used, e.g. once the target temperature has been reached then it should remain within a few degrees of target, and any large departure (in particular any large drop in temperature) could be a sign that something isn't right.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

So with that 18s, it sounds like it only applies if the printer is given a command to change to a different temperature. In normal printing, once the temperature is set, it is not changed during the print so would that 18s ever come into play?

I think he's referring to the Thermal Runaway Protection that is in the Marlin firmware. It halts the machine if the temperature drops suddenly for say a bad thermistor or some other reason, although I don't know if it stops a print if the heater cartridge stops heating the hotend. It seemed to me to be more of a thermistor failure feature like the min heat and max heat stuff it also had in Marlin.

About safeties implemented in Firmware, you may have a look to this post:
[groups.google.com]

However, while it seems than Ryan had a look on the firmware codes, he did'nt go into specifics or details of implementation.

Edited 1 time(s). Last edit at 07/12/2016 03:28PM by PRZ.

---

Pierre

- Safety [reprap.org]
- Embedded help system for Duet and RepRap Firmware [forums.reprap.org]
- Enclosed delta printers Lily [rouzeau.net] and Lily Big [rouzeau.net]
- OpenScad delta printer simulator [github.com]
- 3D printing on my site [www.rouzeau.net]

Quote
dc42
Are you quite certain that Repetier will turn off after 18s if the heating fails during a print, rather than at the start? It's not straightforward to work out that something is amiss once the target temperature has been reached and PID is operating, because the controller may turn the power up and down many times in 18 seconds, and the temperature changes lag the heating power. There are other algorithms that could be used, e.g. once the target temperature has been reached then it should remain within a few degrees of target, and any large departure (in particular any large drop in temperature) could be a sign that something isn't right.

Hey, this reply is also for ElmoC. @ PDBeal - No, that is not what I am referring to.

First, I think this is a very serious topic. At first I was inclined just to forget about it and not tell anybody. It is after all embarrassing and I know a lot of people are reading this post and thinking "wow, I will never be that stupid to leave my wiring harness in a way the cartridge can fall out". I decided to tell because accidents like this must be learned from, so we can hopefully prevent the big accident ( a house burning down)

So lets learn something.

The Repetier software has two settings;

1)

// If the temp. is on hold target, it may not sway more then this degrees celsius, or we mark
// sensor as defect.
#define DECOUPLING_TEST_MAX_HOLD_VARIANCE 20

2)

/** Time in ms between a heater action and test of success. Must be more then time between turning heater on and first temp. rise!
* 0 will disable decoupling test */
#define EXT0_DECOUPLE_TEST_PERIOD 18000

It is the last one in question.

I made a test to see about this!

a) In this video you see the two heating cartridges side by side. One setup with RAMPS and one with DUET. I turn them both on.

VIDEO

REPETIER (RAMPS) turns off at exact 18s after heating is started. Duet never turns off! I think this is a problem in the webdue code. When started from the panel it dosen't even count the 120s!

In the case A the Ramps with Repetier would NOT have set fire to my print. Duet WOULD set fire to the print!

b) In this video I let both Duet and Repetier heat the heating block to 210 and keep it stable. Then I pull out the heating cartridge but leave the thermistor in. This is what I believe happened to my print. So this can be called a reconstruction.

VIDEO

Repetier (RAMPS) turns off the heater 41s after the extraction took place. Duet does not at any point turn off the heating cartridge.

Again - In this reconstruction of my event, Ramps with Repetier would NOT set fire to my print - Duet WOULD set fire to the print.


The above test is not even close to questionable. The Duet software have serious safety issues compared to what the Ramps offers. I made a big mistake but in safety systems single point of failure should always be avoided.

I sincerely hope you will reconsider your previous point of view (that this is purely mechanical and not a firmware issue) and review the Duet safety practices, there exist better practices and as shown, they can be implemented to work.

The guys from Marlin spend a lot of time last year to get a proper safety feature for the thermistor/hotend. There is a issue where they take care of any possible failure and ideas to catch them. This is one of the importance parts for a firmware, before correct steppings etc. Thermistors can fall out, break, short or whatever.

---

Triffid Hunter's Calibration Guide

--> X <-- Drill for new Monitor

Most important Gcode.

Hey DC42,

You posted over on the CoreXY forum that you will look into the software. That is really great to hear! I can't help you with the software but if you need any help testing your implementations then I will be happy to assist. I can setup more bench testing like the above videos.


There is a confusion that might have been gotten lost in this talk about already heated printers.

Please review my video (A) in the above. You see I am starting the heating from cold by the WebDuet interface and when doing that it does not turn off the heater after 120s. So I think the M570 S120 command is somehow not in effect when using the Webinterface? Is M570 wired directly to M104?

In that CoreXY thread you say: . It has three that relate at least in part to the heating: (a) the heat-up timeout,.

LarsK, thanks for running those tests. You got the video links the wrong way round, but I get the picture.

I didn't write the heat up timeout code, and I agree that is not not the best way of protecting the hot end in the event of a malfunction, because detection is slower than can be achieved using other techniques. I looked into why it didn't work in your test. What I found is that the timeout is activated when you make a tool active and when you put it on standby. It is not activated when you change the set temperature of an already-selected tool. This is clearly a bug. I will fix it in the next release, or else implement something better.

I have also added to my (long) firmware wish list implementation of a mechanism to detect heater cartridges and thermistors falling out during a print. Whether this gets done before I implement the feedforward temperature control or at the same time will depend on the priority that Duet users - and especially Duet WiFi users - assign to it. The other items mentioned by Wurstnase (thermistor breaking or getting short circuited) are already detected by the software watchdog and cause the heater to be shut down within a few milliseconds.

PS - I'll also take you up on your offer of running independent tests when I implement this.

Edited 1 time(s). Last edit at 07/13/2016 04:52PM by dc42.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

To have an idea, what can happen, what you could test. This is the github-issue from Marlin, I've talked about: [github.com]

---

Triffid Hunter's Calibration Guide

--> X <-- Drill for new Monitor

Most important Gcode.

Quote
Wurstnase
To have an idea, what can happen, what you could test. This is the github-issue from Marlin, I've talked about: [github.com]

Thanks, but it doesn't really help. All those tests relate to a disconnected thermistor - which is something that RRF detects (and shuts down the heater concerned) within a few milliseconds.

Is there a similar github issue relating to how Marlin behaves if the thermistor or heater falls out of the heater block?

Edited 1 time(s). Last edit at 07/14/2016 09:29AM by dc42.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Very good question. I'm really long out of developing Marlin.
Also I see now the current problems. Maybe you could test a moving window.

---

Triffid Hunter's Calibration Guide

--> X <-- Drill for new Monitor

Most important Gcode.

Thanks for posting your experiences Lars.

I'm gobsmacked that DC42 can do anything other than immediately post, saying that there is a serious safety issue in his firmware, and sort this out within a week.

The failure to do so speaks volumes.

When both Marlin and Repetier have safety features that will turn off the hotend if a temperature abnormality is detected, and the Duet firmware doesn't... well I'll stick with Repetier for the near future thanks. I value my house.

There are failure modes that software will not catch. There will ALWAYS be failure modes that software will not catch. Marlin didn't catch the intermittent partial short in my Mendel's heat bed PCB that melted down properly-secured Melzi heat bed terminals TWICE. Anyone using these machines unattended without overbuilding the machine like mad and implementing fully-independent secondary and tertiary safeguards is playing with fire (not to make light of the situation), using FOSS/H to do a commercial UL-listed product's job.

I would like to see feedforward with watchdogs prioritized for implementation as it would be the most robust solution to help with a number of legitimate potential safety issues and improve function and usability as well, but an arbitrary week deadline for a newly-identified (and vanishingly small for most users who have mechanically-secured heaters) risk, for a small FOSS developer, is ludicrous. That last post is some wildly inflammatory language directed at quite possibly the most-responsive FOSS dev I've ever seen.

Edited 1 time(s). Last edit at 07/20/2016 12:41AM by IMBoring25.

You've both got a point here, this is a serious safety issue that appears to be in the process of being addressed, but it does give pause for thought. I've never used duet or reprap firmware before but I've got one on order. I hope this issue is fixed soon, it is a concern. However I also take from this make sure that heater cartridge is secure.

Lessons to be learned all around regarding this issue!

---

Simon Khoury

Co-founder of [www.precisionpiezo.co.uk] Accurate, repeatable, versatile Z-Probes
Published:Inventions

Quote
nebbian
I'm gobsmacked that DC42 can do anything other than immediately post, saying that there is a serious safety issue in his firmware, and sort this out within a week.

The failure to do so speaks volumes.

I am not. You realize this is one guy working on this firmware (of which he's not being paid to develop and charges nothing for use of it) and you fail to realize even companies like Apple, Microsoft, IBM and the like discover critical security flaws in their software rarely have a solution to the fix in a week with a team of paid developers working on a fix. 9 times out of 10, the only critical security flaws that are fixed within a week are ones that are openly published and exploited by others, and the other critical flaws they know about are never published until they've fixed it.

Your also expecting a solution within a week for one guy working on this firmware in what I expect is his spare time for a single case 1 out of say 100 users that have run into this issue because of a poor mechanical design. You have to be joking, or have no real software / hardware / implementation experience.

And don't forget, dc42 did not write this firmware from scratch. He is working off a fork of the RepRap firmware. So any criticism about the lack of safety features should be directed towards the original authors, not dc42.

Quote
PDBeal

Quote
nebbian
I'm gobsmacked that DC42 can do anything other than immediately post, saying that there is a serious safety issue in his firmware, and sort this out within a week.

The failure to do so speaks volumes.

I am not. You realize this is one guy working on this firmware (of which he's not being paid to develop and charges nothing for use of it) and you fail to realize even companies like Apple, Microsoft, IBM and the like discover critical security flaws in their software rarely have a solution to the fix in a week with a team of paid developers working on a fix. 9 times out of 10, the only critical security flaws that are fixed within a week are ones that are openly published and exploited by others, and the other critical flaws they know about are never published until they've fixed it.

Your also expecting a solution within a week for one guy working on this firmware in what I expect is his spare time for a single case 1 out of say 100 users that have run into this issue because of a poor mechanical design. You have to be joking, or have no real software / hardware / implementation experience.

Just to make it clear, I do take this issue seriously. But it arose just before I went on holiday, and I am only just back. So no progress yet except a small amount of preparatory coding before I went away, and some planning.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Thanks for making this clear DC42. It's good to see that you take it seriously. I might have gotten the wrong impression from the initial posts, if so then I apologise.

I have spent some more time looking into heater protection mechanisms and doing a crude hazard analysis. It turns out that although it is quite easy to do something that works some of the time, it is difficult to implement a protection mechanism that works consistently on most or all 3D all printers during all phases of operation.

Here are some faults I identified that may result in a hazard:

1. Temperature sensor becoming disconnected (sensor may be thermistor, RTD or thermocouple).

2. Sensor becoming shorted, or shorted to ground.

3. Sensor falling out of heater block.

4. Heater falling out of heater block.

5. Sensor and heater both falling out of heater block (some people zip tie the sensor and heater wires together, so this isn't all that unlikely).

6. Short circuit or failed mosfet causing heater to run at full power.

7. Firmware hang causing heater to continue at set power, which could be full power if it was heating up.

Each of these might happen in the following phases:

A. Printer is idle and heater is supposed to be cold.

B. Heater is supposed to be heating up.

C. Heater is close to target temperature and is supposed to maintain that temperature.

D. Heater is supposed to be cooling down.

So we have 7 * 4 = 28 possible combinations of fault and phase. Of these:

Faults 1 A-D, 2 A-D and 7 A-D are easy to deal with, and RRF already handles them. Of the remaining 16:

3A, 4A and 5A can't be detected and present no immediate hazard.

3D, 4D and 5D are difficult to detect. The hazard is low because the heater is off or running at low power.

3B and 4B cause the sensor and heater to become decoupled. The rate of temperature rise during heating will be lower than normal, so it should be possible to detect, at least in the early stages of heating from cold. The problem comes when the heater is only just powerful enough to reach the target temperature, so that the temperature rise is extremely slow anyway during the later stages of heating. This is a common situation with the bed heater of budget 3D printer kits.

3C and 4C should be detectable, because PID will be unable to maintain the requested temperature if the separation of heater and sensor is substantial.

5B and 5C are difficult to detect when using PID unless they make the PID unstable, but will be detectable using feedforward temperature control.

6B is not detectable. 6A, 6C and 6D are. If the printer can control its own power e.g. using M80/M81, it could turn the power off. Otherwise the best it can do is sound an alarm - if the printer has any means of doing that.

In summary, we can detect the majority of hazard-causing combinations of fault and phase, but to do so we need to know what is normal as regards temperature rise during the various phases of heating. This is probably not to bad for the extruder, more difficult for the bed heater because of the greater range of heating rates, and very difficult for the chamber heater. Looks like we need a configuration or auto calibration mechanism to tell the printer what the expected heating rates are at various temperatures. We can only handle the shorted mosfet or equivalent fault if the printer can turn off its power supply.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Have you considered looking at acceleration of temperature? I would expect the acceleration to closely mirror heater power when the heater changes state.

Edited 1 time(s). Last edit at 07/22/2016 10:59AM by nebbian.

Also 6B is detectable, if the firmware detects the temperature rising sharply on first power on. (while heaters are off)

3B and 4B can be detectable if the temperature drops instead of rises (assuming this happens after some initial heat is produced). If this happens in normal operation, then you know that the heater is too underpowered to reach the target temperature, and you need to replace it.

If the temperature doesn't rise by say 5 degrees (pick a number) after 20 seconds of power (again, pick a number), then the two should be considered decoupled (test takes place just after heater is first turned on).

Edited 2 time(s). Last edit at 07/22/2016 11:10AM by nebbian.

Quote
nebbian
Also 6B is detectable, if the firmware detects the temperature rising sharply on first power on. (while heaters are off)

6B won't be detectable until after you rapidly overshoot your setpoint if your heating up.

At that point 6B has evolved into 6C, which is also the time that it becomes a potentially hazardous condition.

Quote
nebbian
Also 6B is detectable, if the firmware detects the temperature rising sharply on first power on. (while heaters are off)

That's 6A, not 6B.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Quote
nebbian
3B and 4B can be detectable if the temperature drops instead of rises (assuming this happens after some initial heat is produced). If this happens in normal operation, then you know that the heater is too underpowered to reach the target temperature, and you need to replace it.

If the temperature doesn't rise by say 5 degrees (pick a number) after 20 seconds of power (again, pick a number), then the two should be considered decoupled (test takes place just after heater is first turned on).

Detecting decoupled heater/thermistor when the heater is first powered on is only part of the problem. What if they become decoupled later on during the heating phase, for example during a tool change?

I have come to the conclusion that to do this properly, there need to be 3 configurable parameters for each of the 3 heater types (bed, extruder, chamber). I can provide default values that offer a reasonable amount of protection for most printers, but for some printers they may offer less than ideal protection, and for other printers the protection may trigger when nothing is wrong (e.g. in the common case of an under-powered bed heater being asked to reach a temperature that it can barely achieve).

I am still working on this and I am about to run tests on a new firmware release, but I remain convinced that nobody should leave a 3D printer unattended without taking some elementary safety precautions:

1. Smoke alarm in the room with the printer, if necessary coupled to other smoke alarrns to guarantee that it will be heard.

2. Suitable fire extinguisher (e.g. CO2) on hand.

3. Responsible person able to attend very quickly if the smoke alarm goes off.

4. Bed heater power chosen so that it can run at full power without getting dangerously hot. Failing that, a thermal cutout in series with either the bed heater or the power to the PSU.

And of course a sound hot end design so that the temperature sensor and heater cartridge cannot become decoupled form the heater block.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].