Forum is becoming UNUSABLE - something must be done!

For some time I have been putting up with slow page loads and occasional "502 Bad Gateway" errors. But the Bad Gateway errors have now reached the point that I am considering running my own bulletin board for the 3D printer firmware and hardware that I support.

The impression I have is that the back end servers that support the forum are often severely overloaded, and need to be beefed up. I don't know who funds those servers, and I am aware that whoever does so is probably dong a service to the 3D printer community without any recompense. But something needs to be done. As the 3D printing community grows, the situation will only get worse.

So I suspect some fund-raising is needed in order to fund an adequate platform or these forums. I have the following suggestions:

1. Identify those vendors who benefit from the publicity they get and the support their users get on these forums. Ask those vendors for a contribution. In return they might get acknowledgement as sponsors, and/or their own sub-forums for support of the products they sell. Such vendors might include Folgertech, RepRapPro, and (to a lesser extent I feel, because I sell low-value sensor boards rather than complete kits) myself.

2. Ask forum users for donations if they find the forums useful. Suggest a donation amount, perhaps GBP10, EUR12 or USD16 per annum.

3. If Phorum supports it, take advertising on the forum, with all adverts clearly identified as such. Most of us are well-used to seeing adverts on other web sites and on our mobile phones etc. and I for one would not find adverts obtrusive as long as they don't include animated graphics or flash.

Does anyone have any alternative ideas on how to keep these forums working and responsive?

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Quote
dc42
So I suspect some fund-raising is needed in order to fund an adequate platform or these forums.

The problem is not a lack of funds. The topic of a new server occasionally comes up on the reprap admin mailing list. The last time this was discussed (more than six months ago) Adrian Bowyer mentioned that the ad bar on the wiki brings in enough funds to pay for a new server. I do not know if this is still the case, but I expect it is.

The problem is that there is a lack of leadership and a lack of people with the right skills who are willing to take on the project. The one man who could solve this problem with a little bit of leadership is Adrian, but he steadfastly refuses to make any decisions regarding this matter. There are other volunteers on the admin mailing list who have the necessary skills (I am not one of them) but for various reasons they have been unwilling to take the lead on this, even when offered payment for their services.

The volunteers who keep reprap.org limping along can barely agree on simple day-to-day matters. I have zero confidence that they can migrate the site to a new server without some kind of strong leadership from Adrian. Your best bet at solving this is to appeal directly to Adrian and try to get him to participate.

Phorum is written in PHP and MySQL, in other words the common LAMP web stack. I find it difficult to believe that you can't find anyone with those common skills willing to do a server migration.

The Phorum software itself is starting to show its age, and judging by this thread on their support forum it has been pretty feature-stagnant since 2007:

http://www.phorum.org/phorum5/read.php?14,144313

I imagine that there's no clean migration path from Phorum to any of the newer message board packages, and losing all of the message archives (that the newbies don't search anyway) would be a huge negative to changing software.....

I'm glad to hear that funding is not a problem, for now at least. I hope someone can be found to manage the server migration. I don't have the necessary skills to help, even if I did have the time.

---

Large delta printer [miscsolutions.wordpress.com], E3D tool changer, Robotdigg SCARA printer, Crane Quad and Ormerod

Disclosure: I design Duet electronics and work on RepRapFirmware, [duet3d.com].

Quote
vreihen
I find it difficult to believe that you can't find anyone with those common skills willing to do a server migration.

There are perhaps ten people who actively volunteer their time as admins for reprap.org. I do not know the exact number because we basically never talk to each other. Of these ten people, perhaps two or three have the skills needed to perform a server migration. Of those two or three people, zero are willing to take the initiative and actually do it. OK, so how about we hire someone instead of relying on volunteers? The only person with the authority to hire someone "with those common skills willing to do a server migration" is Adrian Bowyer, and he has shown no interest in taking the initiative to do this. So is it really still that difficult to believe?

Quote
dc42
I hope someone can be found to manage the server migration.

The topic came up six months ago and promptly fizzled out in a puff of indecision. The reprap admin list has been silent ever since. So don't expect this to happen anytime soon.

... the same happened since 2007 several times now

After problems with the hoster and hardware, we've managed to switch two times to other servers, but after initial setup the interest level of the 'server-admins' didn't stay for long

I'm one of the oldest admins now (after all the other around Sebastien vanished), but didn't have the time, skills or motivation to do this on my own ... so my suggestion too -- try to motivate Adrian as the official owner of RepRap.org

---

Viktor
--------
Aufruf zum Projekt "Müll-freie Meere" - [reprap.org] -- Deutsche Facebook-Gruppe - [www.facebook.com]

Call for the project "garbage-free seas" - [reprap.org]

Here's a crazy idea, since the site is more or less locked into Phorum due to the archives. Instead of buying a server, how about paying the Phorum developers to host it?

http://www.mysnip-solutions.de/en/hosting/phorum.html

An overview of the features we provide:

1) You are getting a fully separate Phorum instance with your own database and your own webspace.

2) Always enough power so that Phorum will always run with optimal performance. Different hosting optimizations like a php-cache are in use to enhance Phorums performance.

3) Upgrades are being applied by us at every new release. Import hotfixes are being applied for critical issues.

4) Standard plugins can be installed by request.

5) Daily data backups, up to 7 days back.

6) SSL-support (defined by package)

... I've sent an email to Adrian ... maybe he has some ideas too?

---

Viktor
--------
Aufruf zum Projekt "Müll-freie Meere" - [reprap.org] -- Deutsche Facebook-Gruppe - [www.facebook.com]

Call for the project "garbage-free seas" - [reprap.org]

Quote
vreihen
how about paying the Phorum developers to host it?

This sounds like a fine idea, but it does not get around the critical problem we have. In order for this idea (or any idea) to work

1. Someone needs to make the decision that we are going to do it.
2. Someone with knowledge and access would have to, at the very least, act as a point of contact when migrating the forums from their present server to their new home (wherever it is).

Without Adrian stepping in, we are incapable of making the decision. Furthermore, there is literally no one among the volunteer admins who is willing and able to act as that point of contact. (If it turns out anyone is willing and able to do this then feel free to chime in.)

There is the additional problem of the wiki. Perhaps the wiki could stay where it is and only the forum would migrate... but I don't know. (This is exactly the kind of nuts-and-bolts question that we are unable to determine and unable to take action on without some kind of executive guidance.)

Quote
VDX
... I've sent an email to Adrian ... maybe he has some ideas too?

Thanks!

Thank you also for your "long horizon" perspective - knowing that this has happened several times before makes me less worried about the future...

While I currently don't have the time to actually help out on an admin level (this might change, but I can't guarantee any free time atm till November), I can give people some idea of what might be involved in doing a move. Some of the details will depend on certain other tools that are in use.

I have some experience with Phorum (I set up a test install to play with it and test various Phorum modules - I suggested a few for here but none of the admin staff took it up), I've dealt with Apache for years, very rarely with nginx, and I've got a fair bit of experience with Mediawiki.

Phorum: If it's just forums.reprap.org, then redirecting the host is easy. The rest is most likely just copying the files and DB to the new server. May have to change the config to use the new credentials for the database, but not much to change.

Mediawiki: The wiki is on reprap.org itself, which could be left as is. That said, moving is much the same sort of process as moving the forums.

Note on hosts: forums.reprap.org is the same host currently as reprap.org - there is no reason we can't separate them, which of course will split the load. I suspect whoever set it up thought of this so it'd be easy to do.

Rights needed to move stuff: DB access (to do a full DB dump), File access (to make a copy of the files), DNS access (to change where forums.reprap.org points).

BTW: I'd suggest simply moving forums.reprap.org as it's by far the easiest piece to move, and should cause the least impact.

I thought the issue was an upgrade, not a relocation . . . simply migrate to the new system, readdress it, and replace the old one . . . this ought to be pretty trivial excersise in data migration, especially if not crippled by WhinDoze . . . no need to change anything in DNS . . .

Migrate intact, and *then* consider application upgrades - typically too many variables if all is done at the same time.

- Tim

Edited 1 time(s). Last edit at 08/25/2015 02:39PM by tadawson.

I've been a PHP developer since 1996, I'm a certified Parallels Linux Virtuoso System Administrator. I've been a Database administrator for the better part of 20 years and I have been in software development for most of my adult life.

With the credentials out of the way, although I'm not currently on the admin list if I can contribute in any way, please let me know. Depending on the current server and platform architecture there might be some juice to be squeezed. I only get the bad gateway errors once in a while.. not frequently. Phorum might even provide a means to exclude "old" messages from search results thereby speeding up the entire system. Indexing would be an issue with search results and the database server (is it an all in one box?) could probably benefit from some tuning.

Quote
thetazzbot
is it an all in one box?

There is a little information on the server in this thread.

Quote
thetazzbot
if I can contribute in any way, please let me know

If you contact Traumflug he might set you up with access.

tadawson: Err, if you're moving the data to a new server with a different IP, then you're going to be changing DNS.

If you think you can get just a single IP routed anywhere in the world to a new server, then you're in for a shock - that's not how things work.

The only way we'd be able to keep the IP is ASSUMING that we stay with the same company, and assuming they build a new VM for us in the same area of the network. Otherwise even for them, it's much easier to just give us a new server with a new IP.

As for the reason of splitting off forums to its own server: Limits the amount of work done, and also limits the impact of any change "going wrong" taking out the whole site. There's also probably other stuff on that machine that we're not seeing (as we don't have any admin access to truly "know" what is on there), and moving it without knowing what it is will surely break it.

One issue: During the DNS change-over, there will be a time when requests go to both sites (some parts of the world will not have updated their DNS records), so during that time you need to lock out requests on the old server. This applies for ANY DNS migration, not just splitting forums off on it's own. The length of time is about 2-3 times the length of the TTL (assuming that TTL has had 2-3 times the previous TTL to propagate to the world)

WRT DNS: The current TTL is 900 seconds (or 15 minutes, which seems low and increases the number of DNS requests). If a migration goes ahead, well before the migration time change the TTL on the affected DNS records down to 300 (5 minutes). This would reduce any cross-over time to no more than 15 minutes. Once it's settled and we know we're not switching back, the TTL could be put back to a higher value (eg: 1800 at a start, possibly higher).

FWIW: The "502 Bad Gateway" thing is an nginx-ism - as in it's due to the way nginx works handling PHP queries, and means PHP is not accepting requests from the webserver.

No, you move the data to the new server, decommision the old one, and bring the new one up on the same address . . . severely common procedure, and avoids dns dinking . . . one less thing to screw up, and leaves the old machine powered off (but intact) in case a fallback is needed.

And, if you bothered to read, I said 'if this is an upgrade and not a relocation', so the rest of your comment missed the mark as well . . . I do this stuff regularly as part of my profession . . . everything I said is valid, and verified to work *IF* that is the case. I just can't figure out why anyone would think that a hardware upgrade implied a relocation . . . they have exactly *NOTHING* to do with each other . . . And I have had to migrate domains between providers (IP changes) with the stuff sitting in the same place enough times to avoid it like the clap if possible . . .

And silly me . . . Since a 'server' is clearly not a 'vm', when someone says 'update the server', that means a standalone piece of hardware . . . and 'our server' means owned and physically managed by the owner, not JoeBob's Web Rapists charging a fortune for a leaseof an obsolete box, and physically upgradable. If that's not the case, folks should use clear, correct terminology when describing the environment . . .

- Tim

Edited 2 time(s). Last edit at 08/29/2015 01:15AM by tadawson.

One of the things vreihen suggested was to get the Phorum people to host it. That would imply a move, rather than an upgrade, which would mean that DNS would have to change.

The issue is getting people who have the time to admin the setup. The current people are time-poor, going by everything mentioned in this thread, which does make the idea of having the Phorum people host it attractive, as then the existing admin team can spend more of their available time working on other things.

Quote
Cefiar
One of the things vreihen suggested was to get the Phorum people to host it. That would imply a move, rather than an upgrade, which would mean that DNS would have to change.

The issue is getting people who have the time to admin the setup. The current people are time-poor, going by everything mentioned in this thread, which does make the idea of having the Phorum people host it attractive, as then the existing admin team can spend more of their available time working on other things.

Given their volunteer resources, I'm wondering if they have been keeping up with security updates? Has it ever been upgraded to MySQL 5.x? Are there any plugins/functions to help with the spam problem? This is why I suggested hosting it with the people who best know how to manage this software, especially when there isn't even a definitive answer on whether it is running on bare metal or in a VM.....

Matt is not quite right when he says that I, "steadfastly refuse to make any decisions regarding this matter". I have repeatedly said that RepRap has some money that could be used to solve the problem, and that - if the community and admins put up a single, coherent, agreed, budgeted proposal - then I will pay out funds to make it happen. If needs be that could be a combination of a one-off sum plus a recurrent expenditure.

I presume that people won't mind if I briefly say here what I won't do:

1. Be a sysadmin myself on a daily basis (I have put one or two things into this project, and so I think that I have earned the
time to do other things; I hope that this is not seen as selfish);
2. Implement the agreed solution myself;
3. Regularly referee discussions and disagreements; and
4. Impose what seems to me to be the best solution.

And what I would like, or be prepared, to do:

1. To have sysadmin priveliges for emergencies;
2. Occasionally make a decision when requested because discussions have reached a real impasse;
3. Outline the characteristics that I think a solution should exhibit in general, as long as this is not seen as prescriptive.

I shall write that outline in the next post.

---

best wishes

Adrian

[reprap.org]
[reprapltd.com]

Here is a list of the characteristics that I think a solution should have:

1. Continuity - the existing content should be preserved and then new content will be added to it.
2. Continuity - the existing users should automatically have accounts on the new system (maybe with a password reset...).
3. The new system should first appear (as reprap-new.org) with ephemeral content as a sandbox for users to try out.
3. Published agreed deadlines. Thus on date D the old system would go read-only; on D+3 (o.w.e.) the new system would be up as reprap.org with all the old content and be read-only; on D+5 the new system would become writable - that sort of thing.
4. When the new system becomes reprap.org we keep the old read-only system for quite a while as reprap-old.org.
5. Sysadmins should maintain a duty rota so that each of them does not get overloaded and so that there's always a competent person nominated for each day to fix things.
6. Sysadmins should get paid.
7. The people who do the migration (who may not all be the regular sysadmins) should get paid.
8. Anything else that you have thought of and that I have missed...

---

best wishes

Adrian

[reprap.org]
[reprapltd.com]

It's also very frustrating, when you write a long elaborated answer, then hit the "post message" button and the post disappears with a "page not available" message.
I'm not sure if this is related to the phorum SW or my G3-provider, just wanted to let off some steam.
-Olaf

... for longer posts I'm writing them in a text editor for spellcecking and to have a 'backup' in case of a forums error ... this hit me several times in other forums too

---

Viktor
--------
Aufruf zum Projekt "Müll-freie Meere" - [reprap.org] -- Deutsche Facebook-Gruppe - [www.facebook.com]

Call for the project "garbage-free seas" - [reprap.org]

Quote
VDX
... for longer posts I'm writing them in a text editor for spellcecking and to have a 'backup' in case of a forums error ... this hit me several times in other forums too

It didn't work!

Quote
David J

Quote
VDX
... for longer posts I'm writing them in a text editor for spellcecking and to have a 'backup' in case of a forums error ... this hit me several times in other forums too

It didn't work!

... yes, but this was no 'long' post, so no spell-checking running

---

Viktor
--------
Aufruf zum Projekt "Müll-freie Meere" - [reprap.org] -- Deutsche Facebook-Gruppe - [www.facebook.com]

Call for the project "garbage-free seas" - [reprap.org]

Adrian,
I'm glad you have stressed continuity.
Since that is a high priority I would recommend upgrading one component at a time: start with just the hardware. Upgrade the phorum software later as desired. Then no issues with passwords, posts missing, bad links, etc.

Quote
Adrian Bowyer
8. Anything else that you have thought of and that I have missed...

The simplest upgrade: is it feasible to simply upgrade the data drives in the current database server to SSD, if that is the bottleneck?

---

My printer: Raptosaur - Large Format Delta - [www.paulwanamaker.wordpress.com]
Can you answer questions about Calibration, Printing issues, Mechanics? Write it up and improve the Wiki!

Quote
Adrian Bowyer
Here is a list of the characteristics that I think a solution should have:

1. Continuity - the existing content should be preserved and then new content will be added to it.
2. Continuity - the existing users should automatically have accounts on the new system (maybe with a password reset...).
3. The new system should first appear (as reprap-new.org) with ephemeral content as a sandbox for users to try out.
3. Published agreed deadlines. Thus on date D the old system would go read-only; on D+3 (o.w.e.) the new system would be up as reprap.org with all the old content and be read-only; on D+5 the new system would become writable - that sort of thing.
4. When the new system becomes reprap.org we keep the old read-only system for quite a while as reprap-old.org.
5. Sysadmins should maintain a duty rota so that each of them does not get overloaded and so that there's always a competent person nominated for each day to fix things.
6. Sysadmins should get paid.
7. The people who do the migration (who may not all be the regular sysadmins) should get paid.
8. Anything else that you have thought of and that I have missed...

I think some people here are jumping to a random solution and very few have stopped to assess the problem. We need to define very clearly what the problems are and then come up with a solution after all the problems are discovered.

Adrian is talking about building a whole new system and turning the current one into an archive. Why? Is there something wrong with the current system that requires the entire thing to be rebuilt?

If you guys do decide to rebuild the whole site, I would recommend using subdomains for development and archiving of the old site. For example, "dev.reprap.org" could be the location of the new website while it is still in development. Once the current site is phased out, it would move to a subdomain like "archive.reprap.org." This keeps everything simple, straightforward, and doesn't require multiple domains be registered (save $$$ yearly).

Eventually what I would like to see is a single sign-on system to centralize account creation. This would keep accounts on the forum and wiki tied together with the same username and password. This adds convenience and also helps with administration. Unfortunately, I don't have extensive experience with implementing a single sign-on system for websites. However, I don't imagine it would be that difficult, even if it had to be custom written.

I'd be willing to help with whatever is needed, for free. I get paid enough through my current job and would love to give back to the Reprap community that made building my own 3D printer possible. If you guys are interested in my help at all, just let me know.

For one final piece of advice, unless everyone decides to build a whole new site, I would recommend rolling out any changes in stages. Doing too much at once is a sure-fire way to run into problems that are difficult to diagnose.


==== Follow-up ====
I did a little bit more research and thinking about the issue, here is what I've found so far:

• The forum here runs Phorum 5, which does not appear to be under active development anymore. Eventually, I think it would be wise to move to a different forum software, such as PhpBB. I looked around a bit and there appears to be a migration path from Phorum 5 to PhpBB, so it should be possible to migrate the forum software without losing data.
• The server may get periodically overloaded, causing nginx to lose connection with the backend PHP FastCGI handler. Monitoring server resources over a week should give us a very good idea of how well the server is handling the load over time. After getting a solid understanding of resource utilization, then we can make the call of whether or not a server upgrade is needed.
• The website does not have any kind of single sign-on system in place, so separate accounts are needed for the wiki and forum. phpBB, with the use of a plugin, supports "bridging" or "single sign-on" with mediawiki. This would utilize the phpBB user database instead of the wiki maintaining its own.
• SPAM is a huge problem and is currently remedied by the mods/admins creating accounts for the wiki, which is inconvenient for both users and admins. Switching to single sign-on with a reCaptcha and email verification for new PhpBB accounts should help solve the issue, with minimal inconvenience to users.
• The forum has WAY too many sections. I think it would eventually be wise to rethink the different sections and see if we can condense them down as much as possible. I would imagine that having so many different sections just adds confusion and lowers the user experience for new users.

If we eventually decide to go with a solution similar to the one outlined above, I definitely believe it should be done in several stages to let the dust settle between each change. I also agree with Adrian's stress on keeping continuity. I think that is super important and would love to have a solution that retains all posts, users, and even has a solution to redirect from old forum URLs to the same post in the new forum, if we decide to change away from Phorum.

I guess I should also quickly outline my experience. For work, I am the lead system administrator for a Plesk Automation web hosting environment running on clustered Hyper-V servers. I have a strong passion for Linux and am familiar with both Debian and Red Hat variants. I have experience with both Apache and Nginx. I also have experience with PHP and MySQL development.

Edited 2 time(s). Last edit at 04/06/2016 01:59PM by b4ndit.

For monitoring, there's Munin running: [reprap.org]

While Phorum is indeed no longer under active development, it still works fairly well. Last time I looked at PhpBB it was a really huge piece of software with all kinds of features imaginable, needed or not. Very likely it increases server load a lot. Also not sure how users like such a loaded user interface.

Doing incremental changes is indeed an excellent idea, here I agree 100%.

---

Generation 7 Electronics	Teacup Firmware	RepRap DIY

Quote
Traumflug
For monitoring, there's Munin running: [reprap.org]

While Phorum is indeed no longer under active development, it still works fairly well. Last time I looked at PhpBB it was a really huge piece of software with all kinds of features imaginable, needed or not. Very likely it increases server load a lot. Also not sure how users like such a loaded user interface.

Doing incremental changes is indeed an excellent idea, here I agree 100%.

PhpBB is definitely feature-rich, which isn't necessarily a bad thing. I'm not sure how much it would effect server load over Phorum, but I would bet that the resource usage would be more dependent on how well the software is written. Either way, it's something I could easily test by installing phpBB and Phorum to two VMs and running a benchmark against them.

Phorum not being under active development is definitely worrying, though, because a vulnerability could be found in the current latest version and there would be no choice but to self-patch or switch forum software. I suggested phpBB because it's highly rated, user friendly, and there is an upgrade path from Phorum 5, but I'm definitely open to suggestions.

Looking at Munin shows that the server frequently gets load average spikes. I'd be curious to dig into that to see if we can figure out a root cause of why that is happening.

Do you happen to know the specs of the server? For example, the OS, CPU (# of cores, GHz), and RAM? It'll make it easier for me to interpret the data on Munin.


==== Update ====

I've been looking through all the information on Munin trying to find a correlation between the load average spikes and resource usage. I think I found what I was looking for. The load average spikes dramatically when IO spikes on the /dev/xvda3 disk.

I'm not sure what this disk has on it, but it may be worth looking into either better caching or upgrading that disk (or maybe even distributing the load somehow). Thoughts?





Edited 2 time(s). Last edit at 04/06/2016 11:42PM by b4ndit.

As for phpBB,

The problem with phpBB is that while it works, it's not really better for administration than Phorum.

Many times in the past, Captcha support has broken (automatically allowing people to register) or methods for bypassing it have been found rendering it useless. Add that to the regular case of the Captcha systems being broken (whether through brute-force, or by handing off to cheap humans for processing) and spammers will still get through.

At that point, you really need decent admin tools to clean things up, and phpBB IMO is not up to the job. The amount of admin work needed to run a popular phpBB forum is quite high, much more than Phorum.

I'm not saying phpBB can't be made to work, but it does require a fair bit of admin legwork, which imposes heavily on the time any admins have.

I'm also not saying don't move away from Phorum. Just choose your options wisely with a bit of forethought rather than just "Well, they use it, it must be good!" thinking.

Quote
Cefiar
As for phpBB,

The problem with phpBB is that while it works, it's not really better for administration than Phorum.

Many times in the past, Captcha support has broken (automatically allowing people to register) or methods for bypassing it have been found rendering it useless. Add that to the regular case of the Captcha systems being broken (whether through brute-force, or by handing off to cheap humans for processing) and spammers will still get through.

At that point, you really need decent admin tools to clean things up, and phpBB IMO is not up to the job. The amount of admin work needed to run a popular phpBB forum is quite high, much more than Phorum.

I'm not saying phpBB can't be made to work, but it does require a fair bit of admin legwork, which imposes heavily on the time any admins have.

I'm also not saying don't move away from Phorum. Just choose your options wisely with a bit of forethought rather than just "Well, they use it, it must be good!" thinking.

Thanks for your input Cefiar!

I like both phpBB and Phorum and each has their pros and cons. The main reason why we would end up switching away from Phorum would be if it stops being updated by the maintainers. Unfortunately, they are already very slow at putting out updates with a couple years between the last minor version and the latest. If a vulnerability comes out in Phorum and they don't fix it, then we'll be forced to self-patch or migrate to something else. For right now, though, we're sticking with Phorum.

For people that stumble upon this post in the future, there is more happening behind-the-scenes. I want you guys to know that this thread hasn't fallen on deaf ears.