Welcome! Log In Create A New Profile

Advanced

use of codec to speed up gcode operations

Posted by jamesdanielv 
use of codec to speed up gcode operations
February 25, 2011 08:38PM
I'm trying to get the communities pulse on what would generally be considered a codec.
first i apologize for the eyesore of text here.

The idea here is to use gcode , but before it goes to the arduino recode the data.

I have a codec in mind, but if the applications we use supported a method to include codecs, it would allow a great amount of creativity, innovation, simplicity and low cost. Ideally the investments in hardware and cpu technologies would not change. nor would the road map of each technology. What would happen is your pc/cellphone/pda becomes the main cpu powerhouse.


Have a codec compress and convert the data and simplify the math and motion from your host CPU, then on the arduino/ARM/atom next use a simple decode algorithm to convert data, error correct it and store it in a buffer.


Codec’s are often used to accommodate bottleneck issues, and performance issues on a CPU and data bus. This discussion has been brought up several times before in the past, but not often with the approach that the main host program essentially runs the same, a translator can convert the programs output, and the only thing that would be different is the decoder, the arduino.

Whenever something is different you run into several kinds of issues, and I was hoping for a discussion of what some of those would be.


Should this be a single solution or should it be a set of standards for codec’s for reprap? What kind of issues does the community see for implementation; would there be as many issues if many parts of setting up firmware were transparent to the user, such as configuration.h file was the same as it is now? So you just would drop your configuration into the firmware and burn it.



A couple of changes this 'theoretical for now' codec would/could enable.
--------------------------------------------------------------------------------------------
Compressed data streams with error correction built into the data. No more two way communication to rely on data and error correction. were moving on to telecommunication tech here. the codec would see and fix the error before placing it into the buffer.

Data would be converted into angles and velocity information.

Asynchronous communication about feedback and other issues. sent anyway every so often, use it or ignore it. wait for it don''t ask for it

Would have a compressed 1k data buffer to compensate for pc issues, so your pc is crunching other events, this thing will have an average of a 30 second buffer. (yes checks/prioritizes for stop/emergency conditions/resets immediately)

Would have a small buffer to repeat previous angles moves without sending all the data from host

no real time math required except for integer in kinetic acceleration buffer
Relies on small look up tables instead of math for even and fast responses.

Uses built in acceleration buffers, and allows for curve acceleration around at least right angles.
------------------------------------------------------------------

This type of setup would allow for the arduino to incorporate a few of the most time saving features and speeds only seen in advanced controller designs such as emc2. And it would mean the faster your home pc, the more you could do. an average print job will be up to 30 minute faster from having curve acceleration around corners, rather that stopping to run a right angle, a curve allows for acceleration into the next move before stopping. (count how many curves your job file has) would allow for x and y pulse rates in an angled move to exceed 50,000 step pulses per second (using digitalFastWrites)


I would expect that this would serial communicate at 19200, and eventually up to 57600, but data is compressed and discriminates in algorithm to have low data overhead in short moves and no longer requires waiting for an acknowledge message.


It does require interest from this community in methods for interpreting and encoding the data. Should it be built into whatever application you are using (saves >10% of host cpu power) or should it just be a codec method that the program chooses for you, for the near term this will be sending your data to an emulated port, and having the codec send data directly to the arduino?



Any suggestions, comments or feedback before I spend a great deal of time on this project? Any people interested in helping?

i would imagine my ultimate goal of this is buy in from reprap for there host application,replicator g, and rep snapper. I also want to create a standard way for plug ins such as codecs to be selected within those programs. This probably will take sometime to complete.
Re: use of codec to speed up gcode operations
February 25, 2011 10:00PM
Hi,

To me this sounds like a really good idea. But it's hard for me to judge its merits, because I don't know what the current RepRap communications are doing 'under the hood'. Would you be willing to explain how the current setup works?
Re: use of codec to speed up gcode operations
February 25, 2011 10:36PM
The current setup just sends gcode to the controller in plain text via serial, where the controller converts the gcode into movement.


Help improve the RepRap wiki!
Just click "Edit" in the top-right corner of the page and start typing.
Anyone can edit the wiki!
Re: use of codec to speed up gcode operations
February 26, 2011 12:04AM
Also the serial gcode requires an 'acknowledge after most commands. so a reply is "OK" and includes check-sum result. if an error occurs then the entire line of gcode is sent again. Normally this seems fine, but there is a delay penalty of .002s for usb chip to buffer and send response to serial interface of arduino, and then there is the processing and buffering of the message. At 57600 baud max amount of information sent per second. Average command is 20 or more characters for regular code, and 2-4 char for compressed code.

any errors that are in need of correction cut the commands available per second by a minimum of .002s x3 for acknowledge, retry command and acknowledge so a retry reduces command i/0 by 1/166 a second and then the time of resending data

.002sdelay response time minimum from usb internal delays for receive buffer

(20char 20bytes*8bits =160* 1/57600)=0.00278 s per command line
.002s delay for usb i/o delay
.002s +0.00278 =.00478s

~max speed 209 commands a second
-----------------------------------------------------------------------
New method for serial i/o independent data asynchronous. No worry about usb i/o delay. This is Transparent to arduino
Asynchronous communication speed with 2-4 byte error correction codes per command
(4char (4bytes) 8bits= 32* 1/57600)=0.000556s per command line

1s/0.000556=1798 commands sent a second

For the same bit rate you have about an 8 times increase in bandwidth. This is just for data sent thru serial cable. Does not included increased efficiencies in operation and processor performance, and compressed data usage. for example being able to move x and y aliased at the same time with step pulses greater than 50,000 per second



Serial data old way maximum 209 commands sent per second
New way minimum 1798 commands sent per second

Edited 1 time(s). Last edit at 03/02/2011 01:16AM by jamesdanielv.
Re: use of codec to speed up gcode operations
February 26, 2011 03:20PM
That sounds awesome. I experience significant slow downs when many short segments are sent, such as printing a high resolution circle. I imagine that this is because of the USB delay, since I am running at 115200 baud. I also have noticed slow downs when trying to do anything else with the computer, it looks like your idea would eliminate that as well.

Having the codec be separate from your host program would be nice, since people run several different softwares on several different OS's.


Help improve the RepRap wiki!
Just click "Edit" in the top-right corner of the page and start typing.
Anyone can edit the wiki!
Re: use of codec to speed up gcode operations
February 26, 2011 07:06PM
That sounds like a big improvement.

If the current command rate is 209 Hz, and the axis is moving at 30 cm/sec, that means that it travels a distance of 1.4 mm between recieved commands. At 1798 Hz, that's 0.17 mm between commands, which is definitely much better!

Go for it, and good luck!
Re: use of codec to speed up gcode operations
February 26, 2011 09:36PM
OK so it sounds we are sold on the communication improvement ability for short moves , so i should focus it to be at 57600 baud.

Also keep in mind that the processing is simplified to mostly look up table math and everything is preprocessed down to angles and velocity on your pc/pda/cell phone rather than the arduino. This means a lot more pulses per second for step/dir moves even with complicated moves such as x y z at the same time.

keep in mind that arcs often have more than one turn at a specific angle so the information would already be in buffer and just repeated. the codec is designed to really push small movements that are common such as arch, corners and circles. the speed limit will probably be the physical limit of the hardware and mass of the printer head.

With the response so far i will set up a page with the codec rules and theory shortly sometime within a month or so. At this point it is discussion and theory, it will be slowly applied to a firmware, depending upon interest. many thanks to those that replied.


the arduino programming would be completely different in structure, but the configuration.h and pin.h and pid table would be the same structure. (Calculations of temp would be offloaded from the arduino, the arduino would have a complex 32bit timer for integral, and 32bit software pwm for power output changes. Proportional and Derivative would be calculated on pc. error check of range would still be built into the arduino for safety reasons , but happen maybe 1/10th of a second)

are there any concerns if the configuration.h file and pin.h file are drop in replacements? what would be the best way to implement any changes that are different in the firmware for settings that are not currently there?

What method is the best to pursue? Codec in application, or the port emulator on pc with built in codec? any one have experience with port emulators for linux, and mac?

many thanks,

James
Re: use of codec to speed up gcode operations
April 04, 2011 12:14AM
here is a started wiki page for the above. Hopefully soon i will update it on how the forward error correction works.

[reprap.org]

James

Edited 2 time(s). Last edit at 04/04/2011 12:14AM by jamesdanielv.
Re: use of codec to speed up gcode operations
April 04, 2011 04:28PM
If the ack turnaround time is the limit, then you want to use the standard solution -- send commands in batches and ack the entire batch. That way, you could increase the throughput of commands sent between acks. This could be implemented in such a way that the firmware remains totally compatible with everything existing today, which would allow exisitng printer drivers to implement it when they desire to. You will, however, need more RAM buffering on the Arduino, which as I understand it is in short supply on the smaller chips. From what I read of the Gcode parsing code in the firmware, this may not even be too difficult to implement. I'd be happy to work with someone on modifying an experimental firmware to do this if someone is interested, but since I have no printer yet, I can't test anything, so I can't do it myself. We'd also need to find someone to modify a printer driver to see if the idea really is useful.

As for compression and error correction (separable concepts), both will require a lot more horsepower on the Arduino. Also, both make debugging the firmware harder since one cannot use simple programs to manually send commands to it. I don't see where either of these are useful right now, especially since they make debugging much harder.

Converting data into angles and velocity requires trig calculations to convert it back to X, Y distance over time, which is pretty much what the Gcode currently is. The host CPU is far more capable to make those calculations than the Arduino.


FYI, I think your reference to codecs and cellphones/PDAs isn't really relevant here. The codecs on those kinds of devices are implemented in silicon, and we're certainly not going to be doing that anytime soon smiling smiley. Were you maybe thinking of something more like a common plugin capability?
Re: use of codec to speed up gcode operations
April 04, 2011 05:27PM
jbayless Wrote:
-------------------------------------------------------
> That sounds like a big improvement.
>
> If the current command rate is 209 Hz, and the
> axis is moving at 30 cm/sec, that means that it
> travels a distance of 1.4 mm between recieved
> commands. At 1798 Hz, that's 0.17 mm between
> commands, which is definitely much better!

You probably meant 30mm/sec which at 209 Hz results in 0.14 mm. So I doubt there's any benefit of higher rates.

You will have a hard time to process even 209 g commands/s and do all the other stuff (step init, step, serial com, manage temps/heater) on the arduino. There's simply no way it can handle 1800Hz AND do forward error correction.

In fact I was also looking into FEC on the arduino. It could handle some simple codes like a Hamming code, but it will only correct single bit errors and almost double the data rate. It can't do more efficient codes like Reed Solomon at any meaningfull bandwith.

If too many short moves really become a problem probably the better solution would be to Implement circle/spline commands in the slicer/firmware.
Re: use of codec to speed up gcode operations
April 04, 2011 09:00PM
Not so. FEC can be incredibly simple. My method is sending each byte 3 times. Any time a byte is seen more than once it is accepted as data, within a certain pattern. There is 3x the overhead of data being sent, but then again no turnaround time waiting for acknowledge.

Any time a byte is seen more than once it is accepted as data, within a certain pattern. There also is a flush of data when a byte is accepted. This allows 2 bytes to be sent that are the same (6 bytes of data total in 2 bytes with possible 3 sets of matches, but data flush prevents 3 sets of matches. huh? will explain in detail later.). I have a work around for the same byte of data transmitting next to each other as well.
Such as

BBB (e is error ) data is B
EBB 2 of same so it is B
BEB 2 of same so it is B
BBE 2 of same so it is B
Granted you need to ensure a low error rate less than 3% of transmitting well within a USB serial device that is working correctly. A test will be build in that will allow the host to know the error rate. Possible solution is on the fly baud rate changes for high errors during initialization of printer run.

I will show more details possibly by next week. This does add a lot of data overhead, but it is still faster by several times waiting for the round about acknowledge from USB host.
Comparing single byte data is very fast on arduino it can take less than 10 commands many of then is just a xor of byte data. Like I said, I will post more later on.

Take the tonokip firmware and the teacup firmware. They can handle a tremendous amount of gcode data. It is about how you implement FEC and the methods used.


For the fast speeds of over 50,000 steps a second, it will require a lookup table approach as well as compression, decompression. It is possible, maybe no believable, but I'm working on it. Thanks for your reply.


There also is a flush of data when a byte is accepted. This allows 2 bytes to be sent that are the same.
Such as

BBB (e is error ) data is B
EBB 2 of same so it is B
BEB 2 of same so it is B
BBE 2 of same so it is B
Granted you need to ensure a low error rate less than 3% of transition well within a USB serial device that is working correctly. A test will be build in that will allow the host to know the error rate. Possible solution is on the fly baud rate changes for high errors during initialization of printer run.

I will show more details possibly by next week. This does add a lot of data overhead, but it is still faster by several times waiting for the round about acknowledge from USB host.
Comparing single byte data is very fast on arduino it can take less than 10 commands many of then is just a xor of byte data. Like I said, I will post more later on.

Take the tonokip firmware and the teacup firmware. They can handle a tremendous amount of gcode data. It is about how you implement FEC and the methods used.


For the fast speeds of over 50,000 steps a second, it will require a look up table approach as well as compression, decompression. It is possible, maybe not believable, but I'm working on it. Thanks for your reply. if there is an easier way to explain this method of FEC, let me know.

Edited 2 time(s). Last edit at 04/04/2011 10:34PM by jamesdanielv.
Re: use of codec to speed up gcode operations
April 05, 2011 12:54PM
Um.. I'm not a USB expert or anything, but isn't USB is a packetized protocol with CRCs in each packet? If so, you shouldn't be seeing any data errors induced in the USB transmission itself (or at least, not many).

Also, I don't think lower baudrates help when using USB. As I understand it, the actual rate of data transmission across the USB wire remains the same regardless of baudrate setting. Now, if you're running the serial data in a noisy environment, then yes, a lower baud rate might help, that is why lower baud rates exist after all. Hopefully you won't need to go all the way down to 300 baud like in the stone age smiling smiley
Re: use of codec to speed up gcode operations
April 06, 2011 01:49AM
This is a good question, that i think needs better explanation:
here is the page that makes the usb to serial chipsets [www.ftdichip.com]

An arduino is basically a board with 2 main chipsets on it. One is the ftdi chipset that processes serial information and the other is the 'brain' or the atmel processor 168,328,... and so on.

what the ftdi chipset is is a hardware usb device that supports a virtual comm port thru software. The information is sent usb into a chip, that does error correction and has data buffered, then the data is converted into serial information, and sent to the rx pin of the arduino. a signal is detected on the arduino and its i/o starts recording the data being sent.


Yes a direct USB has error correction, but the arduino uses a USB to serial converter, and then sends the serial data to the arduino within a 2 inch area space. Even though information is corrected, this is before data becomes true serial.

USB to serial converters are common place in microcontrollers, including pics, arduino, and arm.

Serial is not going away anytime soon. The reason is the complexity of implementing direct USB in drivers and in firmware. It is over many peoples heads, including mine.

New Arduino's now have a different serial chip that does allow programming it to be some other device other than serial, but that is quite complex and requires a very expensive Licensing fee (over $3000) . Many will just use the serial driver as it is royalty free.

Many microcontrollers that have simple uarts buffers use serial because it does not require a lot of memory, or a lot of speed. just the id buffer of usb uses 256bytes!

Here are three reasons for the need for error correction on the arduino

1) The USB to serial chipset,
2) The missing of data from timing issues related to the arduino itself.
3) Host glitch or timing issue related to host.

There are many reasons the reprap code has checksum reporting. The checksum reporting validates the data sent was correct from host.

the fact is errors exist, and error rate is well below 3% typically.
Re: use of codec to speed up gcode operations
April 08, 2011 08:48PM
Thanks for taking the time to elaborate. Now go forth and bring us FEC grinning smiley.

BTW, is there any chance you can make the firmware autonegotiate FEC/no-FEC with the host SW, preferably in a way that's backward compatible to old host SW? That way we wouldn't have to fork the FW, and we could prevent from having to have yet another host SW configuration parameter that needs to be understood by each individual printer owner before any prints can happen, or worse, introducing versionitis issues.
Re: use of codec to speed up gcode operations
April 10, 2011 04:47AM
are you saying to do something like send a command to arduino "FEC" and have it switch mode?

I don't know. It will be hard enough to implement the changes, document the instructions to set up the virtual comm port and write a version of the codec program.

I think clear documented instructions will be beneficial here.

for now i am just going to provide examples of how to do it, implement examples and a single proof of concept firmware that will be what people will use to compare and evaluate it. This project will take time, and it will take a lot of energy to convince people this is the way to go.

FEC also know as 'channel coding' is in everything time sensitive we do. it is in many forms,but your cpu cache uses it, your flash memory uses it. it is in virtually all communications devices.
Re: use of codec to speed up gcode operations
April 10, 2011 06:19PM
Markus Amsler Wrote:
-------------------------------------------------------
> You probably meant 30mm/sec which at 209 Hz
> results in 0.14 mm. So I doubt there's any benefit
> of higher rates.

No, I did mean 30 cm/sec, which is the speed that the ultimaker extrudes at. Without a doubt, RepRaps will be pushing past that speed before long.
Re: use of codec to speed up gcode operations
April 12, 2011 12:11AM
I like where this discussion is going. Still fairly green to the inner working of the reprap firmwares but I will throw in my 2 cents anyways.

$0.01)
Quote
patbob
As for compression and error correction ... ... both make debugging the firmware harder since one cannot use simple programs to manually send commands to it.

I would reword this to say: "Getting your idea accepted will require good script-able command line encoding/decoding tools." There is nothing more frustrating than trying to debug a system when the only devices that can connect to it have questionable firmware/implementations. This will be exacerbated by your attempts to reduce communication back to the host as the state of the system will become increasingly difficult to determine at any given point in time.

So I would suggest having some information flowing back to the host you, could do a tcp style ramp up except instead of increasing the speed you would decrease the overhead. In other words if you aren't seeing errors then you increase (double) the time between responses from the arduino/reprap. I think it is also important to keep a mechanism for the reprap to inform the host of it's state. Combine some minimal information going back to the host with tools like libreprap accessible for host developers while also having simpler command line implementations to aid debugging and you would probably have a winner.

$0.02)
Quote
patbob
BTW, is there any chance you can make the firmware autonegotiate FEC/no-FEC with the host SW, preferably in a way that's backward compatible to old host SW?
Again pretty green here but there does not appear to be a way to ask current firmware what it's capabilities are. This should start being implemented IMHO. Instead of negotiating and trying to load multiple codecs/interpreters onto the arduino why not have new firmwares broadcast codec they support when they boot up. Then the host can just pick the right plugin. I guess legacy firmware would be identified by "start", newer firmwares would broadcast something like "start jdcodecv01".

Adding a simple universal way to request the codec the firmware is running by sending unique bit pattern would also be handy. The question what bit pattern is least likely to interfere with codecs that could be implemented?
Re: use of codec to speed up gcode operations
April 12, 2011 01:49AM
I agree. So some time needs to be spent defining a protocol for codec’s as well as implementing this codec. Luckily there are references out there to look into.

I'll post again after updating the wiki page. Currently just implementing the method of the codec, but the protocol of announcing what codec it uses in firmware sounds like it needs to be there. Especially if I get enough buy in to have this feature put into the software’s directly. This codec is setup to mainly run as a port redirector on linux/mac or a virtual comm for windows. As far as the host software used, it is invisible like a codec should be.

As for data feedback, data is constantly sent back from the controller to the host. it just is not time dependant. (what I mean is it is there within 1/100-1/10 of a second, but you never know exactly when)

The information sent back to the host is currently:

Position coordinates in 16bit resolution xyz
Extruder temp
Bed temp
Error rate calculations – the current error rates detectable from .01% to errors > 1% (up to 3% of errors are correctable)

This information is updated by the codec, and then processed at the host as regular info.

The goal is to make data asynchronous, and independently controlled.

For example when sending data from arduino, if you use serial.write(byte) to write one single byte into the 1 byte tx buffer,you can immediately go on and do other tasks while the uart sends out the data. Having data sent asynchronously allows for data to be manipulated and sent back at fast speeds, while processor can handle tasks such as movements. also this means sending far less frequently than recieving data. the tasking of the processor will determing the exact rate of tx transmition.

The arduino has a 1 byte tx buffer, and a 128byte rx buffer,

Currently the way it works, is you have a string to send, pause everything, and have the processor wait until it can load the next character into the one byte hardware buffer. I know there are soft buffers and hardware.serial modifications that can be done, but those only work reliably if that is the only interrupt being executed.


I don’t know a reason to like the overhead of tcp/ip, or the use of 7 layers to process data. Someone on this forum maybe can change my mind on that.. I have implemented 5 stack on a pic chip, but was not really impressed with it. Especially over serial.

I know this is a lot more reading but here is tcp/ip and the model used. [en.wikipedia.org]
and here is the same thing explained in more detail from cisco [www.cisco.com]

Edited 1 time(s). Last edit at 04/12/2011 01:51AM by jamesdanielv.
Re: use of codec to speed up gcode operations
April 12, 2011 05:53AM
BTW
Position co-ordinates 16 bit resolution.
What does this imply for size of build?
I am contemplating building a mega sized mendel but still require very accurate positioning. How easy would it be to go to 32bit floating decimal for long term scaling?

I remember Bill Gates attitude to 32k Ram!

david
Re: use of codec to speed up gcode operations
April 12, 2011 06:24AM
It was 640k RAM and at the time it was enough...

Early scientists felt that the world will only need 3 to 5 computers. spinning smiley sticking its tongue out


Bob Morrison
Wörth am Rhein, Germany
"Luke, use the source!"
BLOG - PHOTOS - Thingiverse
Re: use of codec to speed up gcode operations
April 12, 2011 06:58AM
ok, we are getting something here.

Position coordinates in 32bit resolution xyze (4byte long)
Extruder temp
Bed temp
Error rate calculations – the current error rates detectable from .01% to errors > 1% (up to 3% of errors are correctable)
4 bytes for i/o changed states or status checks (things not invented yet, laser scanner feedback, for example)

This information is updated by the codec, and then processed at the host as regular info.

The goal is to make data asynchronous, and independently controlled.

I will post the method to solve the FEC for firmware in the next 3-7 days. keep in mind currently phase 1 is about error correction and speeding up data throughput to arduino. thanks for all your input.

Edited 1 time(s). Last edit at 04/12/2011 07:02AM by jamesdanielv.
Re: use of codec to speed up gcode operations
April 12, 2011 05:44PM
vizion Wrote:
-------------------------------------------------------
How
> easy would it be to go to 32bit floating decimal
> for long term scaling?

This is off topic, so you should really open a new thread to ask this question. I'll be happy to chime in about mantissa bits and all on such a thread, or drop me a private message.
Re: use of codec to speed up gcode operations
April 13, 2011 10:36AM
I thought it was on topic - if one is programming a codec then the math capabilities of that program are very relevant to the codec. James also has taken my point up & replied showing an intention to include 32bit capability for position co-ordinates for the codec.

So I respectfully disagree with any suggestion it is off topic for a discussion about a codec.

David
Re: use of codec to speed up gcode operations
April 13, 2011 10:41AM
Actually his private comment mentioned 320k but he said "I'll say 640k for public consumption and none can object!"
Re: use of codec to speed up gcode operations
April 13, 2011 03:14PM
The firmware will respond to the codec, by sending 32bit data back for positioning data.

just to be clear no data is sent back as floats . Positioning data is sent using long signed integers.

32 bit signed integer long is from -2,147,483,648 to 2,147,483,647.

This is overkill, but that's find for me.

you can read more about data types and how they work and store integer numbers here

[en.wikipedia.org]

you can read up here on how floats are stored and there accuracy level here.

[en.wikipedia.org]
Re: use of codec to speed up gcode operations
April 16, 2011 12:00PM
This thread smells like it belongs in alternative firmware to me.

I'd be interested in a codec that allows me to precompute moves using the per-axis acceleration algorithm I'm working on in floating point on the host, then dump all that data in integer form directly into the movebuffer on the atmega so it just needs to wrangle a couple of numbers every step.

Would be interesting to have some sort of format request handshake whereby the firmware can specify what format it wants its packet data in, including calculations on the host-side floating point data, eg; "give me int32 dx,dy,dz,de, uint32 f,dist,time where dx=Xnew-Xold, dy=Ynew-Yold,dz=Znew-Zold,de=Enew-Eold,f=F,dist=sqrt((dx*dx)+*dy*dy)+(dz*dz)) * 1000,time=f*16000000/dist/60"

Odemia Wrote:
--------------------------------------
> Again pretty green here but there does
> not appear to be a way to ask current
> firmware what it's capabilities are.

M115

And as for switching to binary/packet mode, a simple M-code would suffice, followed by a synchronisation handshake to ensure that the changeover completed successfully and that the firmware isn't lying about its capabilities. See http and starttls for existing thoroughly tested examples.


-----------------------------------------------
Wooden Mendel
Teacup Firmware
Re: use of codec to speed up gcode operations
April 18, 2011 11:15PM
I'm about to post the solution to the wiki for the codec for solving data that arrives in 3 sets of bytes. I have to still test this and am sure that some parts will not compile exactly as they are below. The reason to post it is for people to know about what is being done to solve the FEC.
The forward error correction only works if it has 3 bytes in buffer. I tried to make it as simple as possible for the arduino to process but think there is a simpler method out there. Also i think i need a little more code for error correction. So the code below is not final by any means. it is here for discussion.




this code buffers 3 bytes of single serial code, compares data A and C and compares B and A thru xor. Meaning if the results match they should be zero and not solve, having the else statement provide the capturing of the byte data.
if a match is found, and the result is correct the data then is stored, and the cache is cleared. Clearing the cache allows for the next byte to be the same byte value and still work the same.

Also i am trying to write a program to run a test with the reprap host and this codec for sending and receiving 10 megabytes of data. The program will be available for download after testing. any feedback is appreciated, and optimization tips here are greatly appreciated.

The test program will test for speed of completion, error rates recognized and errors corrected. I will use the reprap host code with the checksum correction to compare against this codec form. I expect the codec form to run and complete data transfer with all errors corrected and finish in ¼ the time. We will see, and everyone that wishes will be able to test the results themselves. this is part of phase 1 for the codec. this is the part that provides foward error correction. phase 2 will be about data compression


Here is the general code without variable initialization. this is for the arduino side.

{
the solution is basically a form of

C=B;
B=A;
A= serial.read ;// only 1 byte values accepted. need to read only 1 byte at a time thru


if (buff2){ // if second time is true then continue
if (buff3){ //3rd pass data has been captured and stored in abc and is now ready. if data does not match, buffer is not cleared.
if (B^C|A^C) { // do nothing if any values remain there is a mismatch also part of error detection routine}
if (B^C & A^C) { //if here then error with cb and ca. so error is most likely in C
if (A & B ) { store=B ; // hey ab match. c has error
A=B=C=0; //tests if ab are same and verifies that it is c that has error
buff2=false; buff3=false; //also clear buff counts}
else{// do nothing if any values remain there is a mismatch also part of error detection routine}

else { store=C ; // if false then 2 of 3 pieces of data match
A=B=C=0; //if match found also clear buffer
buff2=false; buff3=false; //also clear buff counts}


}// buff3 end
if (buff2){buff3=true;} //2st pass, set buff3 as true to go thru again
}//buff2 end
buff2=true;1st pass, set buff2 as true to go thru again

} //end of routine


I know this still needs a little work for example what if the third byte has an error and c is not correct, data would be rejected but not recovered. I am working on a solution for that. a solution now exists just needs to be proven.

Also I think the codec language should be python. It seems to be a fast language with port I/O and works for most people. I need to find out if it is possible to have more than one port open at a time with python, if so we are on our way, at least pending the results from the test program.

Python seems like a good choice, because it is open source, the code can be viewed without a compiler, and anyone is free to modify it as they please. I will need to provide a set of standards for implementing codec changes.

here is a page that has python for mobile devices, including iphone [www.awaretek.com]
python defiantly has multiport ability [pyserial.sourceforge.net]

Edited 4 time(s). Last edit at 04/20/2011 05:12AM by jamesdanielv.
Sorry, only registered users may post in this forum.

Click here to login