ESP8266 Duet web interface for any board

Quote
toxuin
That's great that I am not the only user of Esprinter anymore!
...
I'll take a look at errors and flaws you're describing and push an update.

Any time estimation? :p

It may be possible to achieve faster upload speed if we let the ESP8266 temporarily take control over the SD from the printer. Theoretically ESP8266 can do 40Mb/s on the SPI (~4MB/s). Some other bottleneck (wifi, SD card or CPU power) will probably limit the upload below that point. Here is the idea:

HARDWARE

• Connect SD card directly to the ESP8266, as in the SD WebServer example.
• The same SD signals also connect, through 1k resistors, to the printer (ramps/melzi/other).

ESPRINTER

• During normal operation ESP8266 should have all SD pins set to High-Z, allowing the printer to control the card via the resistors.
• On upload start:
  
  • If there is an SD print in progress then refuse upload.
  • Have the printer release the card (M22).
  • ESP8266 should enable SD pins allowing SPI operation, reset the card, and create a file for upload. The resistors will prevent the printer from driving the card signals at this stage.
• Upload data simply saves data to the file as it is received, instead of sending it to the printer via serial port.
• On upload end or cancel:
  
  • The uploaded file should be closed.
  • All SD signals from ESP8266 should be set to High-Z in order to release control of the SD card signals.
  • The printer should be issued an Initialize-SD command (M21) in order to regain SD control.
• All other operations (delete, mkdir, list) may still be handled via the printer's firmware, keeping changes at a minimum.

PRINTER-FIRMWARE

• Just remap serial and SD pins to something easily accessible (avoids PCB reworking).

On a Melzi board no soldering is required. Serial1, SPI, I2C and 4 analog inputs are exposed on an expansion connector, and any of these pins may also be used as a regular GPIO. The Arduino bootloader is locked to the Serial0 port and the USB converter, so no wifi-updates are possible. However, simply changing a #define in Marlin I moved the GCODE port to Serial1 (tested working with esp-link and NodeMcu). The SD's MISO, MOSI and SCK are the same of the SPI, and are already exposed. Another #define can be changed to move the CS pin to the expansion port, and then all required signals are easily accessible.

I was trying to implement using ESP8266SDWebServer as a starting point for SDcard operation, but SDs really don't like me, and I am too sleepy to insist right now. :-/

Edited 1 time(s). Last edit at 02/12/2016 12:29AM by lhartmann.

@lhartmann I love this idea! There are couple of things I'd like to mention:

• According to the RepRapFirmware source code there is no way to unmount an SD-card with G-code. However, there are methods to release and init the card, they're just not implemented as G-codes.
• Repetier firmware does support M22 (unmount) command. So I guess I will be (eventually) implementing your idea on Repetier first. Also, Melzi board (which I happen to have, too) will only run Repetier. Other boards like RADDS (which I also possess) have an SD-breakout too, so RepRapFirmware port of this feature would be awesome. It still needs M22 support.

Since lots of errors have been reported with Repetier (thanks to my amazing programming skills) I would like ESPrinter focus to be on RepRapFirmware first – as it has all the necessary JSON reply support. I will get ESPrinter to a stable state with RepRapFirmware and then will do my best to bring it over to Repetier.
If someone would like to give Repetier a proper JSON support - that'd be great. I tried to do that and submitted a patch which was accepted, but since so many errors were reported – it obviously needs a re-code.

I had lots of urgent unrelated stuff to do lately and to top it all I fried my printer board (soldering is another amazing skill of mine). This is why there are no time estimates. I am fixing my printer now and unless I finish that first, for now I do not have a testing machine.
In the meantime you can ask me things on Gitter (http://gitter.im/toxuin/ESPRINTER).

@dc42 Maestro, I would like to hear your comments on this: is this possible to implement a M22 in RepRapFirmware and is it even a good idea?.. Thoughts about two devices trying to talk to a single SD card gives me shivers.

Edited 1 time(s). Last edit at 02/12/2016 03:26AM by toxuin.

As a matter of fact I suggested M22 just cause I found it while looking for M21. :-)

I have never "safely removed" (M22) the SD from a printer before. I just pull it out, write new files, and plug it back on the printer. After hundreds of cycles it is still working fine. I believe that, as fas as the printer is not currently writing anything, we should be fine simply taking over the card without warning.

Alternatively we could first enable SD takeover then issue M21 (initialize). This will cause the printer to attempt access to the card and fail, as it can not control the SPI lines of the card anymore. This should make it clear to the printer that the card was "removed".

I think Dan Newman's port of RepRapFirmware for RADDS supports the SD card detect switch, and I've been meaning to port this support to my fork for the Duet. Once this is done, it would be a simple matter to implement M22.

However, if the ESP8266 uses SPI to talk to the card then the write speed is likely to be low: firstly because it is a 1-bit bus instead of the 4-bit bus used by the Duet; and secondly because the maximum speed that SD cards are specified to work at using SPI access is much lower (I forget how much) than the 42MHz normally used by the Duet.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

The slowest SD cards are expected to be able run SPI at 25MHz. Faster cards can go over 100MHz. [en.m.wikipedia.org]

This is still probably slower than the original duet, but we are not trying to make the duet board faster. This is the duet web interface for other boards, usually based on ATMEGA microcontrollers. This means we are trying to get around the sluggish 115.2kb/s assynchronous serial between ESP and ATMEGA, so even the slowest SD SPI is still 200x faster.

I'll probably get a chance to test this again tonight. I think my SPI clock migh have been too fast for the SD initialization sequence.

This is great! I would love to see a proof of concept.
SPI mode should give us about 100 kb/s transfer (?). This is not much, I know, but compared to 3 kb/s on Repetier now this is a BREAKTHROUGH. I am expecting it to be much more stable than spitting letters over serial, too.
It would require an extra connector, probably even motherboard-specific one, so probably SD takeover will be optional.
Duet is awesome and all, but it is kinda out of scope for the project: it already has "Duet web interface"! What I am trying to achieve is to get alike interface to other, even non-due boards. Duet uses 4-bit access and DMA (doesn't it?), ATMEGAs talk serial . That's like flying a jet plane compared to riding a bicycle: you wish your bike had a jet engine.

Hi all,

I've found this thread looking for a way to add remote capabilities to my printer. At the moment I am considering either web interface like this, or linux host with Repetier-Server or similar. Actually, both because I have two printers and one probably doesn't need linux onboard. I tried this project with latest Repetier-Firmware and found some issues. Since I have not seen any commits to this repository since December, I wish to ask about the status of this project. Is it alive and what was the last its state.

First, I tried to replicate that we have here. I had just one 4MB ESP8266 on NodeMcu, so added few lines to Arduino for ESP8266 to use 512K modules with 160K SPIFFS. Compressing the index.html to gzip it fits the flash. The next step was to enable the second serial in Repetier firmware - and here is what I found and need to discuss.

The dual serial port in Repetier firmware uses single queue. That means that anything printer returns is fed into both serials (main and aux). At the same time any characters from any of serials are queued into the same buffer. This means that:
- If one serial port is working (say, you're printing from USB or just have some host connected and polling), you have a lot of unexpected by the web app output on both ports. Say, "wait" messages confuse the web app as well as replies to other commands from other stream.
- If web app asks for something during other port use, it results in errors because both streams to printer are mixed char by char and neither is processed correctly.

So I guess what is the "correct" multi-port behavior of 3d printer interfaces. Should we have single or many queues? Should we remember the source of command and then respond only to asking stream, not to both? How it is done in a firmware that works with this version? How is it done in other multi-port firmwares?

I do not have a hardware to run RepRapSoftware firmware. But I am ok to add or port anything we need to Repetier-Firmware, including serial interface rewrites. All I need is to:
- understand the desired behavior (see above),
- be sure that Repetier may accept changes. The question is not about patches in general, but about, say, multi-port behavior change: I don't have own experience with "correctly" working printers, and any obvious solution may have unexpected side effects.

Any ideas?

And another one thing: there is an Unofficial ESP8266 SDK, as you know, that uses only small bit of official one. An example project has a web-server and TCP/IP stack that is able to go as fast as 1.2MB (megabytes, not bits) via WiFi, and up to 3Mb (megabits) or even much higher on a serial port. The example provides TCP2Serial feature running on such baud rates. I was going to port the web interface to it, so we can have, say, 1'000'000 baud on the serial port and wifi, and issue of writing to SD will not be as hard as it currently is. But I have no experience with web/js applications and have no time to do that for this project. That is another reason why I want to know first where we are on the web app level, and then to realize what we want to do with multi-port behavior on Repetier firmware, then implement anything we need about JSON responses, and finally maybe rework this project for fast ESP8266 SDK to really enjoy $2 web interface.

Can you help me with ideas?

Hello,
Project is not abandoned at all. I am finishing my bachelor's thesis currently and therefore had to take a pause. I will resume in aabout a month after I graduate.
RepRapFirmware uses two queues for commands, Repetier just mirrors everything, as far as I know.
About pull-requests to Repetier dev - they were surprisingly fast to accept (although author worked out some flaws out of my code) and seemed maximum helpfull in the process.
Native SDK seems like a lot of hassle, but if you are willing to do it - you're more than welcome. I can surely do the webapp part. My local repo still struggles to keep up with newer versions of duet web control: I haven't finish code migration to 0.7 when 0.8 came out, I've started over and now 0.10 is there...
Anyways, if you want - we can discuss details on Gitter.

RepRapFirmware uses a separate queue for each of the 3 main input streams (USB, serial and web). It remembers where each command came from and writes the response back to that channel. However, error messages are written to all channels.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

• Web app: I think we can just reuse the duet panel app as is? We can do it if we support all mandatory messages or, at least, emulate them in replies using web server code.
• Repetier and pull requests: I know, I have a very good experience with them as well. But I wonder about principal idea of separation of queues, not a minor fixes or extensions. Either they did not implement separate queues yet, or they have some thoughts against it doing a reply mirroring only. That was the question. Unfortunately, there were no commits in Repetier repo for 2 weeks, I guess they have a break or vacation.
• Native ESP SDK: yes, I know. That's why I said that there is another one, much, much faster and with many bugs fixed. It still isn't official and isn't supported since Espressif does not publish low level info. But it at least is not worse than Arduino, and while we can play with Arduino, the final version can use native SDK if we need some performance shifts (like faster uploads).
• Gitter might be ok, but to discuss details. At the moment I see the main culprint with queues implementation. That means we can't talk about "any board" until "any board" implements JSON replies AND separate queues.
• Queues: I think that should be the way to go: separate queues and source request memory. Still, it needs good thinking to not mix commands got from different channels which can break, say, printing or SD card upload. I have no idea how to prevent fatal changes from queue 2 while that channel knows nothing about what happens on queue 1... But at least now I understand why it does not work as is with Repetier-Firmware.

Thanks for responses.

I'd like to say a big thank you to toxuin for creating and publishing this project. Last August, I started the outline design of the Duet WiFi, with the aim of using a ESP8266 to provide WiFi connectivity to 3D printer electronics based on a modern 32-bit processor. The most important question was whether we would be able to achieve a reliable WiFi connection with the ESP8266, and a sufficiently fast file upload speed to the SD card. I was just starting to look at writing the web server code for the ESP8266 when toxuin started this thread. The code he published gave me a starting point for performing tests.

The performance tests I was able to do by hacking toxuin's code indicated that a sufficient upload speed should be available, by improving the coding of the underlying web server in the Arduino library, and by using a high speed SPI bus to communicate with the rest of the printer electronics instead of the UART.

The project progressed from that point, and there are now almost 50 pre-production Duet WiFi boards with beta testers and early adopters. We are getting good reports from them. You can read their feedback in the dedicated support forum for Duet WiFi users at [www.duet3d.com], which is readable to the public. The time taken to load DuetWebControl over a good connection is about 2 seconds, and the file upload speed to SD card over a good WiFi connection is up to 850kbytes/sec..

The code we are using on the ESP8266 is published at [github.com] and the patched Arduino ESP8266 core and libraries at [github.com]. We use a slightly modified version of DuetWebControl 1.11 which is at [github.com].

In the longer term I plan to do a complete rewrite of the Arduino ESP8266 http server, or possibly use the multi-threaded http server library that someone else has developed, to better serve multiple web clients and perhaps improve performance some more.

Once again, THANK YOU toxuin for making it easier for me to get this project off the ground.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Really nice to hear that! :-) 850kB/s is plenty fast, the largest print I ever did would have taken just 30s to upload.

I failed to find the schematic for the board. Was it released already?

Quote
lhartmann
Really nice to hear that! :-) 850kB/s is plenty fast, the largest print I ever did would have taken just 30s to upload.

I failed to find the schematic for the board. Was it released already?

The schematic will be released when the production boards start shipping.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Quote
dc42
The code we are using on the ESP8266 is published at [github.com] and the patched Arduino ESP8266 core and libraries at [github.com]. We use a slightly modified version of DuetWebControl 1.11 which is at [github.com].

Am I right assuming this project is ONLY for original Duet firmware, not for any boards?

Asking because I was going to do similar project for modified Repetier firmware, but found a lot of issues with supporting it from the printer end. Repetier firmware (and, actually, almost all firmwares except RepRapSoftware) requires major changes in handling many external channels (two serial ports, etc). The way it is implemented in Repetier (two inputs put chars to the same queue, any output goes to both serials) is lame and works only if you use either one of channels, not both (say, USB printing and wifi monitoring). So I dropped the idea of doing such port of firmware, mostly, because there are Linux boards with resources to run, say, Repetier server, for $10 and $3 costs WiFi dongle for them. While ESP8266 monitoring is really cool, it just doesn't work for ANY board by design. So I found no sense to continue.

The firmware I linked to is intended to work with RepRapFirmware running on the Duet WiFi hardware. This is essentially a development of the Duet, with the ESP8266 instead of wires Ethernet, more advanced stepper drivers (TMC2660), and a bunch of other improvements.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Thanks! I thought that Duet WiFi is an add-on board, but it is a completely different printer platform based on Duet.

Quote
osnwt
Thanks! I thought that Duet WiFi is an add-on board, but it is a completely different printer platform based on Duet.

That's right, see [www.duet3d.com] for the details.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Quote
dc42

Would it be possible to use your code for the ESP8266 and the modified DuetWebControl which you implemented in the DuetWifi to add WiFi capability to a Duet 0.8.5 ? (although a wifi to LAN bridge would be a far less difficult to implement).

If this is doable, how would we connect the ESP to the Duet board ?

Yes in theory, because that's how I prototyped it originally. I connected the ESP8266 to the SPI bus on the expansion connector. However, I ran it at a much lower speed (8MHz maximum AFAIR) because of the poor wiring compared to having the main processor and the ESP8266 on a single PCB with short traces between them. But as you say, a simpler solution is to use a bridge such as the TPLink WR-710N.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Are there detailed step-by-step instructions on how to install the required firmware and Web code on the ESP8266 to have this working?

The step-by-step instalation instructions in this thread have allowed me to install the Duet web interface in a spare ESP8266 and it seems to be running OK albeit without being connected to my printer.

Is there any chance that the required M408 code will be implemented in the latest Repetier firmware (1.0.0 dev)? And is the code present in toxuin Repetier fork fully working?

Thanks

Edited 1 time(s). Last edit at 10/04/2016 03:20AM by micheleamerica.

AFAIK, it's already implemented..
Look for JSON_OUTPUT ('Features' tab in web configurator).

I do not use the configurator, instead I manually update the Configuration.h file, and I do have the JSON_OUTPUT defined.

Looking at the code it does seem that toxuins code was merged in the 1.0.0 branch but the thing is that I keep getting "invalid gcode" if I send M408 to the printer. Is there anything else besides the configuration flag that may influence this gcode acceptance by the firmware?

Thanks in advance.

Well, I finally managed to find the issue.

The configuration file that comes with the DUE version of Repetier has FEATURE_JSON as the define for the ESP8266 Duet web interface. I added the JSON_OUTPUT and now the firmware is accepting the M408 gcode and spitting out a properly formated JSON string.

Now I only need to wire the NodeMCU to my RADDS board and see if things are working.

In any case thanks mandrav for your insights.

Let us know how it goes

So I have a RADDS board running dcnewman's port of dc42s RepRapFirmware. I was wondering what it would take to get this running on one of these:

[www.adafruit.com]

and interfacing with my RADDS board. They both have headers for SPI. The RADDS board is running 1.14 of RepRapFirmware.

Edited 1 time(s). Last edit at 11/15/2016 05:35PM by CthulhuLabs.