Binary Format G_Code Files

Binary files.

In the past I have referred several times to my use of binary files with the PIC24. The original idea was to try and make the machine smoother by reducing segment pausing. Every time the print head stops to think where it has to go next you get a blob deposited on the object. Study some of the early printed items and you can clearly see this effect. Not to put to finer point on it:
segment pausing = poor print quality.

The PIC24 is quite a quick processor but early trials showed hesitation between lines of G-Code.
This is not too surprising when you consider what is going on. No matter what type of processor the more it has to do in a loop the slower it gets.

I know the following is stating the obvious but its worth doing a quick run down the activity for each line of code:

1)The G Code file is read into the buffer one character at a time up to the NL/CR marker
(multiple lines may be read in to give additional buffering)

2)With the line in the buffer we need to step through it and analyse the contents
Is it a G or M code, what axis data is mentioned, is Feed rate mentioned any delays etc.

3)With the line decoded the

This whole GCode thing is such a dead end. EEPROM print buffers like I use with Tommelise coupled with preprocessing of the XML GCode equivalent that I use into simple step instructions is ever so much more efficient. You are not asking your microcontroller to do a bunch of calculations on the fly. I use a HID USB port and half a meg of EEPROM to turn the trick and I never see a head hesitation. With an SD card you've got tons of room for big, bulky fully-processed stepper and extruder controller instructions. They don't make one much smaller than a gigabyte in size any more and it only costs $10, tops. Why not use it for what it is capable of doing for you and screw this GCode nonsense?

Edited 1 time(s). Last edit at 11/03/2008 09:14AM by Forrest Higgs.

Wow Forrest, you don't mince your words do you!
I had to come back on this one, G_Code is the way forward. Millions of CNC machines run on the stuff, its the default machine language. However you want to package it the REPRAP is a lightweight CNC machine with a new tool.

I did the notes on Binary format to help out with segment pausing on low end processors. If I understand you correctly you have preprocessed the G_Code into your own short form coordinates and packed them into an EEPROM, what's the difference apart from the format?

My PIC32 project runs through standard text files just like a PC, yes it is a processor hungry task but it I think it important to run with the standard G_Code text file, for me, the less I have to mess with the G_Code the better. It helps make the controller universal and easily transportable to any other CNC machine.

If you want to present your Tommelise as an education project then please promote G_Code, its a skill the students can take with them into industry, we have several CNC machines in our workshop, the last thing I want from a new employee is knowledge in an obscure control language, head crashes are expensive!

Package the code in whatever way you need to run with your machine, but teach G_Code, any post process on the standard file is a particular requirement of your machine and should be invisible to the user.

Tony

>
> Wow Forrest, you don't mince your words do you!
>
Nope, not hardly. I'm an acquired taste.

LOL! Your argument reminds me of a similar sermon I heard way back in the 1970's. The industry standard, such as it was, for systems programmers then was IBM JCL (job control language). It was a pig of a thing to build control modules with and to make matters near impossible had nothing resembling default values for anything. There was a whole priesthood of systems types, and I became one of them, after a fashion, simply because I didn't have the resources to hire that kind of nonsense done for me. There was a whole industry that sprang up around "teaching" people IBM JCL. One of IBM's big selling points was that if you bought some other system, the people who learned your "brand X" machine wouldn't be able to work on IBM machines, which were the overwhelming bulk of the industry. Bit of a circular argument, that one was. We're big, so you HAVE to buy from us so that your people will be interchangeable, and we'll get even bigger. The Japanese bought into this and their big electronics firms like Hitachi started offering mainframes cheaper than IBM's that ran IBM mainframe OS's and JCL. It was utterly mad in those days.

Within five years IBM on mainframes and JCL were pretty much history. I've never missed 'em.

As to "teaching" G-Code to kids, forget it. I want kids to be able to design and build things, not become cut-to-fit cogs for the CNC market. I saw that with JCL and IBM and didn't like it then. For sure I'm not going to go out of my way to prepetuate "standards" that are pointless.

There's an old project manager saying, viz,

"when you're up to your ass in alligators, it is hard to remember that your original intention is to drain the swamp"

I've found myself in the swamp so many times that in my declining years I have little patience when a project tries to develop subtexts to its original intent. In that regard G-Code is like Linux is like Arduino is like SDCC is like a lot of unnegotiable "standards" that have attempted to attach themselves to Reprap like so many limpets.

They just slow us down and, to quote Barak Obama, are "distractions" to what we need to be focussing on.

Edited 4 time(s). Last edit at 11/03/2008 12:50PM by Forrest Higgs.

As an end user with just enough programming knowledge to be dangerous as well as annoying, I'm not too concerned about whose format I use, I just want nice looking prints.

Good on you Forrest for watching out for us; everything you're saying makes sense. And Tony, I think your work speaks for itself - both approaches are looking pretty good.

For me, I actually started with a Pic development kit a few years ago, with an intent to hook up a new controller to my PC. While the kit was pretty useful, there wasn't a big development community around it, and I never did figure out how to properly get data from it back into my PC.

When I was bitten by the Reprap bug, I immediately bought a few Arduinos, and had them doing what I wanted within a day, mainly because of the existing user base. I was able to find some code that was close to what I wanted, and modify it so that it did what I wanted. Yeah, it would have been better to learn exactly how everything works, but I just wanted to hack some sensors and servos.

Right now, that's the big advantage with Gcodes for me - there's a lot of people out there already using it, so I can find tools that work. For instance, I'm doing a rather bone-headed technique to make multiple parts - I just re-home my machine x+20 mm over and cut and paste the gcodes in again. A plain old text editor works for that, and then I can use an open Gcode backplotter to see if I made any stupid mistakes that would cause a head collision. That's pretty useful to me. If we were running a custom binary format straight from the STL's, I'd be stuck editing STL's, which is a bit of a problem right now anyway.

Once we have a perfect toolchain, and everyone's using the same micros, then yeah, we'll probably have our own internal format and it'll all be good. Till then, a few external standards aren't the worst thing ever.

I have to say that right this second, on my machine, the Gcode software is the difference between making lumpy piles of plastic and making usable parts. So I vote for stuff that works!

Edited 1 time(s). Last edit at 11/03/2008 02:25PM by Wade.

Greetings all,

Text below is my opinion; feel free to agree, disagree, comment, or ignore:

I think going to G-code as an intermediate representation for "operations a darwin/repstrap should do" is an excellent idea! In fact, skeinforge and the g-code (subset) interpreter for arduino are (together) a *major* reason I got off the fence and started restrapping.

Whether additive or subtractive, a reprap is (to my mind, at least) a machine tool -- a tool used to make desired items. Since additive machining is relatively new, and not widely supported, it makes sense to me to adopt G-code as a lingua franca (and also a bridge) to the substantial amount of hardware and software out there that does use G-code. Since G-code is quite well supported in the machining industry, and even by open-source CAD-CAM systems (like the EMC project), this helps us. IMHO, this argument is substantial on its own, but there's more:

At present (and for the near future), there are a number of parts a reprap *cannot* make for itself (extruder nozzles, extruder barrels, extruder drive screws, insulator barrel....) Since these parts are *not* rep-rappable, and their design is also evolving (fairly rapidly), it makes sense to me to work with design tools that support making those parts -- via conventional (namely subtractive) machining on a mill or lathe. A G-code representation of "how to make part wxyz..." is a substantial part of doing this.

G-code = Dead End? -- I don't think so!
So far as I can tell, G-code is NOT likely to die out soon. I strongly suspect that it will still be in substantial use in ten years. In my last job, I worked with 3 machinists using both manual and CNC machines to make biomedical instrudment prototypes. Their CNC mill (Hass VMC) and CNC lathe (Okuma) both used G-code, and both were purchased fairly recently. (FYI, I did mostly electronics/software, but did some mechanical designs, which those guys were great at machining into reality. I've picked up some machining skills, but am still learning....)

Having (in reprap) the seeds of either a low-cost subtractive machine tool or (IMHO, even better!) a machine that can do both additive and subtractive machining is a way for reprap-like machines to be able to make more of their own parts, as well as more things people want. (Not everything people want/need is made of thermoplastic!) Admittedly, even a reprap-mini-mill-lathe-combo won't make everything (e.g. ball bearings, thread taps, NiChrome wire....), but it would make a considerably larger subset. And that, to me, is progress toward reprap's goals.

FYI, I'm currently fighting friction in the X&Y axes of a repstrap that will {I hope!} do both kinds of machining. I'm trying to get the subtractive part working first, so I can use that to "carve out" some of the extruder parts.

I think G-code is a good way to represent "machining process", until/unless the machining industry comes up with a replacement lingua-franca. So far, I haven't heard of a serious contender -- expecially one that is process-specific (go this fast; wait this long....) vs. yet another CAD format for describing a part's geometry. Having a means to create and use G-code enables reprappers to make use of existing machining hardware, software and services.

That's what I think; what about you?

-- Larry Pfeffer

Hi tofletcher,

Your binary format is a great idea!

It would indeed mostly or totally fix the segment pausing problem, which is standing in the way of high quality prints.

I suggest that you could add it to the skeinforge tool chain by making an export plugin and drop it into the export_plugins folder. An example export plugin, indeed the only export plugin, is gcode_small.py. You could convert that into binary_16_bytes.py or whatever.

Example python code for binary conversion using the struct module can be found in stl.py in import_plugins. The documentation for the struct module is at:
[www.python.org]

Cheers,
Enrique

THe pausing in Gcode systems could be solved my running the maths in the forground, queueing the output and running the stepping drives on an interrupt. Of course this is nowhere as simple but I always prefer using a neat software design over just adding more CPU hardware.

I went along to a local user group type thing last week (www.supermondays.org) and they were all keen on these new languages like Ruby. When I pointed out 'embeded systems' they said 'you mean like openmoko?' (open mobe phone) and how its all goinna be running PHP or Ruby (or sommat). Problem is it is battery powered, so if you use clever code you can make it smaller etc. BUT it then takes ages to get to market.

I wonder if reprap should dump arduino and go for somthing like the openmoko - it is open, has processor power and interfaces (good display, but the general I/O may be a problem)

I suspect that I amy have problems configuring the linux etc, and I realy like hacking about with PIC processors in assembly.

Enrique,

I never considered that this Binary format would be the sort of thing that could be included in the skeinforge tool chain. The short application I used to generate the binary file was written in VB.net, it takes standard G_Code and spits out a packed binary version. I don't know if I will be able to port it over to python any time soon but if there is general interest in the idea I will start looking into it.

First step is to learn Python!

Thanks for all your work on this project Enrique, I have not contacted you before but that is because I have not needed to, your code does a fantastic job. Any object printed on my machine has been generated by your G_Code, it does far more than I have yet discovered, so maybe other quality improvements can be had if I tweak a few more of the parameters.

Tony

Lampbus Wrote:
-------------------------------------------------------
> THe pausing in Gcode systems could be solved my
> running the maths in the forground, queueing the
> output and running the stepping drives on an
> interrupt.
>

Tried that on Tommelise 1.0. Mind, T1 was a gearmotor/encoder system, not a stepper system. Anyhow, there was still too much foreground work.

Me too, I tried it on the PIC24 and could not get it to show real improvment.
Tony

It's crazy to send floating point numbers to a micro with no floating point hardware that is also trying to do real time control. Any buffering scheme will fall down if you have lots of short segments, e.g. when making a tall cylinder, unless you buffer the whole object or pause between layers.

The only sensible route is to send binary commands to the micro. That is what I have done all along and I don't suffer from pauses. It also makes the software on the micro trivial and leaves the more complex stuff on the PC where you can write in a higher level language and have better debugging tools.

---

[www.hydraraptor.blogspot.com]

nophead Wrote:
-------------------------------------------------------
>
> The only sensible route is to send binary commands
> to the micro. That is what I have done all along
> and I don't suffer from pauses. It also makes the
> software on the micro trivial and leaves the more
> complex stuff on the PC where you can write in a
> higher level language and have better debugging
> tools.
>
That's exactly what I do with T2. Works like a charm! I buffer because I don't trust Windows as a real-time operating system, which I shouldn't. No pauses, whatsoever.

Hi tofletcher,

Since getting rid of the segment pausing is a high priority, I've made a binary_16_byte.py file. I don't know how you structured the floating point feedrate, so to start I wrote that as an integer.

While testing binary_16_byte, I found a bug in export, so also attached is the updated export file.

In general, I don't know exactly how you structured the flags and byte order, so you'll have to change and upgrade the python export plugin a lot.

I believe you'll find programming in python to be surprisingly easy. Just copy and change lines so as to grab the formatting & sometimes dreaded necessary whitespace

The python documentation is at:
[docs.python.org]

and the tutorial is at:
[docs.python.org]

Cheers,
Enrique

There are undoubtedly many ways to dish up the steps to the micro, both Chris and Forrest have post processed the coordinates into byte sized chunks to make it easy to work with.

Running the PIC24 with a text file doing all the number crunching detailed above, it just about managed, segmented circles were the only area where I had any real problem. Circles defined using I's and J's ran fine. If I wanted seamless motion on the machine under all circumstances, binary format was the only option.

The one and only downside(and its a small one) of the binary files is they are totally unintelligible when opened in a text reader. The way I was producing the binary output involves passing the text G_Code through a small prog. that changed the format. Then if I want to make a change in the G_Code I have to go back to the original text file and repeat the process.

With the current firmware on the PIC32 it can crunch its way through text files and not see any segment pausing. If you doubt this then check out the video posted in the Builders blog.(It runs much, much faster than this but not while printing PCL!)

For now this makes life that little bit easier. Going forward, there are a few things I would like to add to the firmware, some of them in the main loop so depending how this new code impinges on performance I can switch back to Binary as required. The code to run this format has been left in place, so all I need to do is tag a file with .bin and it will switch to

Would be good to check Forrest's and nophead's binary file format, and compare the 3 available approaches.

I tried to digg a bit more into this issue, but couldnt find more information about Forrest's or Nophead's software.

Is it available somewhere?

(Im building the motor controller boards right now, so Im a bit away of this whole issue, but within a month I will arrive to the same sort of problem. I hope.)

I don't have a binary file format at the moment. I send binary commands to my machine one at a time in Ethernet UDP packets at 100Mbits/s. I always intended to buffer the commands but I found it unnecessary as long as my PC is not doing anything too strenuous. I will get round to it one day.

There are just 10 commands, most of which get passed on to the head controller down an I2C bus at 100kbits. Each message starts with a 4 byte magic constant, a single byte command code and a single byte sequence number. Each new message increments the sequence number and HydraRaptor only executes new commands. If it sees a repeat command it returns the previous reply. The host waits for a reply to each command. If it does not get one with the correct sequence number within a timeout period it sends it again. That mechanism ensures it will recover if messages or replies are lost as UDP does not guarantee delivery.

The commands have different numbers of parameters which are all in binary and in the units the machine works in. I.e. stepper steps, thermistor ADC values, timer counts, etc. All the maths to convert from SI units is done on the PC, the micro controllers generally don't have to do anything harder than add or subtract, although they do both have hardware multiply.

The commands are: -

No op that just returns the machine status.
Go to xyz at specified timer rate with an optional table step time for acceleration and declaration.
Set vacuum cleaner on or off (for milling dust extraction).
Set milling spindle speed.
Set extruder temperature as an ADC value.
Get extruder temperature as an ADC value.
Run extruder motor at specified speed for specified number of shaft encoder steps, or forever.
Get the motor duty cycle as a binary fraction.
Get heater duty cycle as a binary fraction.
Set the fan on or off.

The reply consists of the entire machine state, e.g. the sequence number of the last command executed, a flag to say if the carbot is moving, the current motor positions and shaft encoder positions for X and Y, plus any return values from the last command.

Ethernet packets have a minimum size of 60 bytes and a maximum size of 1514 so no point in trying to save bytes.

---

[www.hydraraptor.blogspot.com]

it is a bit similar what i have done in the past:

[start][id][command][parameters][checksum][end]

End the reply was:
[start][id][command][parameters][status code][checksum][end]
[1byte][1byte][1byte][ 6bytes ][ 1byte][ 1byte][1byte]

It is like yours:
command - command
parameters - "magic number"
id - sequence number

Start and end code was required by a plc (and all my commands was fixed length: request 11bytes, reply 12 bytes).
I had checksum (what you dont), which was extremely helpful when I debugged. (cable problems, wrong connections, etc).

Thank you for writing your setup. I like it

I don't have start, end and checksum because they are all handled by the UDP / IP / Ethernet protocols. If you get a UDP packet it is guaranteed to be correct by a CRC in the Ethernet frame and checksums at the IP level and UDP level. But it may go missing or arrive out of sequence which is why I add a sequence number and retry logic. If you always wait for a reply before sending the next command the sequence number can be just 1 bit that toggles.

In my I2C protocol the first byte has one bit sequence number, 4 bit command and 3 bit length. That is followed by any parameters and then an 8 bit CRC. I2C handles start and stop codes.

Edited 1 time(s). Last edit at 11/11/2008 05:18PM by nophead.

---

[www.hydraraptor.blogspot.com]

> I don't have start, end and checksum because they are all handled by the
> UDP / IP / Ethernet protocols.

Checksum implemented in the application is useful because if two independent persons write software (micro-side, computer-side), it helps debugging and troubleshooting.
Really, its a must-have

Our setup was:
Touchscreen -> server1 -> database-server -> server2 -
-[TCP/IP]->gateway(simple python program)-[rs485]->plc

Its nice to have checksum implemented in protocol. But its helpful anyway, if it is implemented in the sofware.