Re: Repository usage

Simon McAuliffe wrote:
> I'd just like to make a suggestion regarding the use of the subversion
> repository...
>
> I would personally like to see it only really used for source, not as a
> general repository for all sorts of binary files. That is because it
> tracks all versions of everything forever and if we keep putting in
> large binary files that essentially completely change on any revision,
> then the repository gets really large really quickly. That is not
> really the intended use of a system like subversion. Subversion stores
> only the changes between revisions but for certain file types,
> particularly compressed files or various compiled files, there is very
> little relationship between successive versions so you get full copies
> every time.
>
> So for example, PIC firmware binaries, STL files, photos or video clips
> are not so appropriate.
>
> Perhaps we can put some of those things just on the sourceforge
> downloads instead? Or somewhere else? They can still always be
> regenerated from what is in subversion, and they can be archived
> somewhere else too.
>
> What do others think?

We are all agreed that the repository is absolutely the best way to
handle the software development. And (almost implicitly, and without
much decision making) we use the same model for mechanical and
electronic development. This it is a huge success (and as far as I know
- which is not very far - it is also unique in hardware engineering).

The problem comes about because we have compilers for both Java and C,
and so the way that the repository tools work allows us to store small
amounts of source data for them, and even smaller changes, and then to
generate the big files we need from them locally using make (or equivalent).

But the CAD tools that we use have no similar process. You have to load
files in and save them in a different format interactively. This means
that we tend to save the output in order not to have to repeat that
interactive process all the time.

But those CAD tools (KiCad and AoI) are both open-source, so it ought to
be comparatively easy to add a batch compile option to them. I should
emphasize that I'm not suggesting that we devote our effort to this; I'm
just pointing out that it's both technically possible and desirable.

If this were to be done the problem that Simon correctly identified
would go away.

So. Are we actually causing a problem at the repository, in the sense
that we're taking so much space that we're getting complaints, or will
get complaints in the near future? It seems to me that disc capacity
and bandwidth are so fat these days (and getting fatter) that we're
probably not; well - not short-term, anyway. But if I'm wrong then we
should do something now.

But if I'm right, then I think we should make the compile-option
suggestion to the developers of AoI and KiCad, keep doing what we're
doing temporarily (because it's such a good engineering discipline), and
immediately switch to that option when it becomes available. (Note that
KiCad source files are text, incidentally.)

As a final thought, video compression looks for coherence not just in a
frame, but between frames in the sequence. It would be really neat to
use a similar lossless compression tool for binaries in a
version-control repository that treated each version as a frame in such
a sequence.

--

Best wishes

Adrian

Dr Adrian Bowyer
[staff.bath.ac.uk]
[reprap.org]
_______________________________________________
Developers mailing list
Developers@reprap.org
[reprap.org]

i think the major issue we're looking at here are the STL / AOI files. storing output from kicad is very small (<300k for various files) also, storing the firmware output from sdcc is very small.

the big hitters we're looking at here are STL files and these things are just monstrous (relatively) it would be very great if we could get a command line export to STL from AoI. that would allow us to script the whole process. if we can script it, then going from editing the AoI files => generating a release tarball gets a whole lot easier. the firmware can already be scripted. the kicad files can be left in there just fine. even the .pho (gerber), .ps, and .pdf files can be left in there as they arent all that big.

it would be nice to get kicad scripted, but its not really all that big of a deal. your average user isnt ever going to need to edit the boards. ideally these are a very constant / unchanging part of the project. most users will be getting the 'final' compiled part... an actual board.

i say we keep going as-is. we store the AoI files in subversion, store firmware source, all the kicad stuff, java sources, etc.

it should be pretty easy to setup a little script or process where we take the newest tarball, download it, replace it with whatever is newest (.STL, firmware, java exe) and then re-upload it to sourceforge as a binary release. as things get more scriptable, this process can become more automated.

it would also help to designate someone to be in charge of the releases. perhaps something like a release manager. just someone that knows how to build firmware, export to STL, and compile the java program. they get pinged when new stuff is ready for release and then they do their dance and release it. any volunteers? i think we're getting to the point where the work is piling up and it may be a good idea to recruit more help. the thing to keep in mind is that we need to recruit help with a goal in mind: release manager, firmware builder, java host manager, electronics overlord, etc. obviously anyone can contribute to anything they want to, but it would be nice to break up the duties a little bit so that there is some coherence and guidance to the project.