GCODE buffer multiline proposal

From RepRap
Jump to: navigation, search

Proposal for sending multiple lines of G-code

So far, this is a proposal, open for discussion.

Problem to solve

Each line of G-code sent from the host to the controller is answered with an ok before the next line can be sent without locking up communications. This makes operations very slow, as the usual USB-TTL converters and probably also the host's operating system drivers come with substantial latency, often 10 milliseconds. Accordingly, at most 5 lines can be sent per second, slowing communications down drastically from the theoretical data rate possible over 115200 baud.

The target is to send multiple lines of G-code without locking communications up, lowering the impact of this driver/USB delay.

What does not work

  • Using the XON / XOFF protocol
Using this protocol full data rate transfer can be achieved easily. However, also all the buffers in the host's operating system driver and the USB-TTL converter are filled. These buffers can be several thousand bytes, or several hundred lines of G-code.
Accordingly, if the host has demand for some non-queued command, like M114 or M112(!), the controller can react only after executing all the lines in the buffer. Because it takes that long until the command actually reaches the controller.
  • Storing serious amounts of data in the controller's serial queue.
For one, RAM is very precious on the ATmegas. Like just 4096 bytes on the commonly used ATmega644. And we have to store a movement queue for look-ahead and quite some state variables, too.
For two, this would require the controller to detect and execute out-of-order commands. Not trivial if you parse the G-code as each character comes in.

Proposal

With each ok or each rs, the controller firmware also sends the current number of free spots in it's movement queue. Like

ok Q:8

An ok without a Q: is treated like an ok Q:1.

With this information, the host can send more than one line of G-code without fearing to lock up communications. For example, if the queue has 8 spots free, the host can send 8 lines of G-code in one chunk and an additional M114. The M114 will be executed as soon as the previous 8 lines are parsed, not waiting for the bot to execute all the lines.

This is also backwards compatible for firmwares and likely hosts, as the later simply ignore the Q: notice.

Let's see how this works out in some situations:

After a reset

The controller either sends an ok Q:4 on it's own or the host can request one with a non-buffering command, like an emtpy line. The host will send, for example:

G1 X1.0
G1 X2.0
G1 X3.0

in one chunk and receive back, also in one chunk:

ok Q:3
ok Q:2
ok Q:1

The gain here is avoiding four times the driver/USB delay (two forth, two back), getting much closer to the data rate possible over the serial line.

The host sees everything is fine and, as all driver/USB queues are empty after that, immediate communications is still possible. The host can then send additional non-queued commands at regular intervals to find out when another spot becomes free:

M114   -->   ok Q:1
M114   -->   ok Q:1
M114   -->   ok Q:2
...

At this time the host can send additional G-code.

The host can also send additional data if the queue doesn't fill as much as expected. For example, when receiving after the above 3-line example:

ok Q:3
ok Q:2
ok Q:2

This indicates one spot was already freed as the data was parsed. If data can be sent really fast, no delay due to additional non-buffered commands happens.

An communications error

Assume, one of the lines is received ill and the answer from the host should be:

ok Q:3
rs 2814 Q:3
rs 2814 Q:3

As you see, the controller ignores the third line until line N2814 actually comes in again.

Actually, I'm not sure wether current firmwares act that way, but it should be possible to make them so. No luck with unnumbered lines, though, the line in error will be queued in the wrong order. --Traumflug 13:02, 22 December 2011 (UTC)
Queue overfilled

If the controller sends

ok Q:0

too much data was sent somehow. An error condition, but a fully recoverable one. The host simply stops sending stuff and sends empty lines at reasonable intervals until the queue is reported to have free spots again.

Fast data transfer when storing to an SD card file

If the controller can expect to receive data very quick in large amounts, like when just storing all the G-code to a file on the SD card, it simply reports a very large number of empty spots:

ok Q:9999

Then the host can send as much data as it thinks is reasonable.

Note: Without the additional usage of XON/XOFF, the controller actually has to stuff the data away as fast as it comes in. If this can't be guaranteed, reporting smaller numbers is advised.

Comments on this proposal

Please insert comments here:

How about copying the "net command" RFC2821/2 model

Some protocols replace the space with a hyphen to indicate the command is not complete.

An example would be (taken from original example)

G1-X1.0
G1-X2.0
G1 X3.0

and the response would be

ok-Q:3
ok-Q:2
ok Q:1

Note that email and other protocols using the "net command" model tend to use this primarily for multiline responses but considering we have a space this works well here for both tx and rx.

For multiple inline comments we assume the comment tag '//' is a continuation when followed by a hyphen.

i.e.

//- debug goes here
ok-Q:3
//- debug goes here
ok-Q:2
//- debug goes here
//- summary debug goes here
ok Q:1
I think you are confusing multi-line commands with sending multiple (queued) commands, they are different things Bobc 11:42, 5 May 2012 (UTC)
Good Plan

I like this idea a lot. It would seem that somewhere out there this has been done before. Is there not perhaps a standard for old world CNC drip-feed serial systems that may hint at a solution? It would be cool if a common standard was implemented that was in use and tested/specified already. The description above seems like it should work, having it backward compatible from both directions is a bonus.

My comments
  • Queue overfilled

Surely this is not an error case, but a normal case? We should assume that the host may send commands faster than they can be executed. If there are X slots free, the host sends X commands, after the last Q will be zero. So the case where the queue is filled (not over-filled!) is going to be quite common, and the host must poll for status using a dummy unbuffered command. If the host indeed sends a buffered command when the queue is full, then the firmware should probably just delay sending an "ok" until queue has free slots.

  • After reset

The Reprap host need only assume Q:1 at startup. It will get the actual queue size after first commands. If there is no "Q:x" then the firmware does not support queued commands.

  • Line Numbers becomes mandatory?

Unless the comms link is perfect, then resends can only work if line numbers are used.

Bobc 11:42, 5 May 2012 (UTC)

Further reading