Snippets from the reprap-admin list Administration of the reprap.org domain and its server is pretty well hidden so far. But it exists! There's a invitation-only reprap-admin mailing list and with the recent dropouts, this list became pretty active quickly. To give the RepRap community a better feeling about what happens, I'll post here a snippet from this list. With agreement from the list members, of course. QuoteOn 12 April 2013 14:47, bill2or3 wrote: I did a round of server tuning, here are the changes I made: - added robots.txt files to the wiki and forums, to tell msnbot, bingbot and YandexBot not to request > 2 pages per minute. Those 3 bots combined were accounting for *thirty percent* of the hits to the wiki, but refer only a handful of visitors. This has made a huge difference. - one-month 'Expires:' headers for the icon images on forums.reprap.org. This will let those icons persist in client browser caches, across browser restarts. Saves hits to the server, speeds up load times for return visitors. - made the forum 'update viewcount' queries LOW_PRIORITY, so that actual useful SELECT queries will happen first. - set apache ssl session cache to the minimum size, 8k. It was set to 512k before, entirely unused. I think we can disable mod_ssl entirely, which should free up a few megs. - tiny fix to ReCaptcha.php, to avoid about 9 megs of errors messages per day hitting /var/log/messages The Munin cpu and network graphs are pretty flat since these changes. Before, they were very inconsistent, occasionally spiking up by %400 or just missing big chunks where the server was too busy to collect the graph data. I'm guessing that what was happening was that bingbot came to visit, from several different IP's at once. All those bingbot crawlers were hitting forums, making lots of mostly identical requests. All those queries eventually hit the 'messages' table, causing the sql table to lock, and grinding everything to a crawl. The graphs show that apache hits/apache traffic/ethernet traffic all dropped by about 1/3, and CPU seems to have a lot more breathing room. If nobody has any objections, I'd like to let it run for a few days, just to be sure everything is stable, and then enable mod_gzip and disable mod_ssl, in Apache. Depending how it looks after that, perhaps investigate using APC php caching, and/or parsing php code via fastcgi instead of with mod_php. Overall, I'm pretty happy with these results, and expect to quit hearing "is it down?" asked in IRC. cheers, Bill https://reprap.org/forum/read.php?33,199173,199173#msg-199173 Sat, 12 Sep 2026 09:55:47 -0400 Phorum 5.2.23 https://reprap.org/forum/read.php?33,199173,385849#msg-385849 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,385849#msg-385849
And If I see Bill's work correctly, he's currently in progress with abandoning Apache entirely in favour of Nginx. Progress so far about 80%, files; pictures and wiki PHP are served by Nginx and php-fpm already.

One funny tidbit: I forwarded the above enhancement to Phorums bug tracker, but they refused to accept it. It's considered to be a feature, because there might be people which want to serve a different JavaScript for each forum section. Which confirms my opinion that software developers have a strong tendency to make things complicated :)]]> Traumflug Administration, Announcements, Policy Mon, 21 Jul 2014 06:07:49 -0400 https://reprap.org/forum/read.php?33,199173,385747#msg-385747 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,385747#msg-385747
Q: Are we using mod_expires at all, and if not, could we consider setting it up?

We can then improve the amount of client side caching that is done on things like images, javascript, css, etc, by forcing their cache expiry times to be longer than they currently are.

This should reduce the amount of fetched traffic everywhere.

eg: [www.inmotionhosting.com]]]> Cefiar Administration, Announcements, Policy Sun, 20 Jul 2014 21:49:37 -0400 https://reprap.org/forum/read.php?33,199173,384774#msg-384774 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,384774#msg-384774
Quote
bill2or3
I've edited the forum file at /data/www/forums/htdocs/include/phorum_get_url.php to set it so the javascript.php requests don't have the forum id added.

before:
GET /javascript.php?225
after:
GET /javascript.php

The javascript doesn't vary by forum, so having that forum-id on the request was causing the file to be re-requested when a user switched between forums, rather than just using the javascript file it'd already downloaded.
[...]
A spot-check of yesterdays access_log shows that it transfered 3121 megs of javascript.php, but it'd be 1387 megs if there were no duplicate javascript.php traffic.
]]> Traumflug Administration, Announcements, Policy Fri, 18 Jul 2014 06:21:44 -0400 https://reprap.org/forum/read.php?33,199173,381566#msg-381566 Re: Snippets from^H^H^H^H for the reprap-admin list https://reprap.org/forum/read.php?33,199173,381566#msg-381566 reprap.org] and [www.mediawiki.org]]]> Traumflug Administration, Announcements, Policy Fri, 11 Jul 2014 14:46:44 -0400 https://reprap.org/forum/read.php?33,199173,373182#msg-373182 Wiki user cleanup https://reprap.org/forum/read.php?33,199173,373182#msg-373182 www.mediawiki.org] That's about 84% of the previously 40'544 ones. Unused accounts are ones with zero edits and older than 100 days.

I hope this script didn't catch more accounts than expected. In case you're human and your account got lost, please recreate it and add at least one edit, e.g. by creating a user page for yourself. In case you had edits and the account is gone anyways, please report here.

What's left is 6'657 users, 2'200 of which are probably still spammers, but not for longer than 100 days. Active user count is 149. Users, which have edited something in the last 30 days.]]> Traumflug Administration, Announcements, Policy Sun, 22 Jun 2014 13:01:57 -0400 https://reprap.org/forum/read.php?33,199173,254674#msg-254674 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,254674#msg-254674
Quote
Traumflug
Blogging time :-)

Notified by a lot of error messages in /var/log/forum_error_log I found images embedded into forum posts didn't work the way they should. Forum admins tended to disable the embed_images module, which doesn't match with some other part of the configuration.

The problem was, /usr/bin/convert insists on additionally writing a file when told to send the output to stdout. That's clearly a bug. As it didn't have privileges to write (what a luck!), it failed.

To fix this, I canged /data/www/forums/htdocs/includes/api/images.php to write this file to /dev/null. This workaround might break when 'convert' removes its bug. Nevertheless, it's the first time I see properly thumbnailed image boxes in our forum.

Time needed to find the right place and the right type of this one-line fix: about 6 hours. :-)

As always, I made a copy of images.php before patching. A person doing a wiki or forum upgrade may please watch out for files with a date appended (or "2013" in it's name), they're originals of patched files.

You see?


]]> Traumflug Administration, Announcements, Policy Sun, 13 Oct 2013 22:49:56 -0400 https://reprap.org/forum/read.php?33,199173,242892#msg-242892 spam-post permissions adjustments. https://reprap.org/forum/read.php?33,199173,242892#msg-242892
cheers,
Bill]]> bill2or3 Administration, Announcements, Policy Thu, 05 Sep 2013 13:06:00 -0400 https://reprap.org/forum/read.php?33,199173,239176#msg-239176 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,239176#msg-239176 >
> I have (mostly) installed the latest Mediawiki version 1.21.1, at
> [reprap.org]. It's using a separate database &
> db user, so should have zero overlap with the production Wiki.
>
> It seems to run fine, I've updated a few of the extensions, and a
> few still need updating. Also, the themes need an update & copy,
> and the image files need to be copied over. Right now it's just
> symlinking the image directory, and has uploads disabled.]]> Traumflug Administration, Announcements, Policy Sat, 24 Aug 2013 09:00:27 -0400 https://reprap.org/forum/read.php?33,199173,222937#msg-222937 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,222937#msg-222937
Seems I have to force some specific actions from the server-level admins :S]]> VDX Administration, Announcements, Policy Mon, 01 Jul 2013 04:14:32 -0400 https://reprap.org/forum/read.php?33,199173,222900#msg-222900 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,222900#msg-222900
Note: I did report it, so it may be gone by the time you see this.]]> Cefiar Administration, Announcements, Policy Sun, 30 Jun 2013 21:19:24 -0400 https://reprap.org/forum/read.php?33,199173,222455#msg-222455 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,222455#msg-222455 >> Hi Admins,
>>
>> ... regarding the forum -- in the last time the spammers get much more
>> nasty with registering or
>> spamming even without a valid account.
[...]

Am 29.06.2013 02:39, schrieb Adrian Bowyer:
> We certainly shouldn't allow posting without an account. I've disabled
> the reply option for public users, which should cut the problem down.
>
> Best wishes]]> Traumflug Administration, Announcements, Policy Sat, 29 Jun 2013 07:14:49 -0400 https://reprap.org/forum/read.php?33,199173,214958#msg-214958 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,214958#msg-214958 www.iblocklist.com] perhaps? I know some of those are paid subscription lists, but there are others that aren't, such as "forumspam", "dshield" and the like, which are useful while allowing you to test the setup without the need to pay for list subscription first. I'd also expect that there would be some manual blocks regardless, as I'm sure there are a few places that just target us specifically (mainly just bad clients and web spiders).

Note: I'm not expecting that these would be updated every day or something, but say once a month, or manually by an admin (eg: "Force an update").]]> Cefiar Administration, Announcements, Policy Mon, 03 Jun 2013 18:29:40 -0400 https://reprap.org/forum/read.php?33,199173,214468#msg-214468 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,214468#msg-214468
Two months ago I removed a a few hundred blocks already as sort of a test-balloon. The raise of daily spam was, if there was a raise at all, not noticeable. Obviously, spammers have plenty of addresses to use, so blocking a few of them hurts only the legitimate users which happen to be in the same IP address pool of their internet service provider.

Another experience is, only few spammers try to re-use their accounts for multiple edits. Accordingly I regularly block only those accounts where I see repeated re-appearances. And always with a 6 month expiry.]]> Traumflug Administration, Announcements, Policy Sun, 02 Jun 2013 09:45:25 -0400 https://reprap.org/forum/read.php?33,199173,202213#msg-202213 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,202213#msg-202213
In any case, they gave up shortly after I removed their access to the file they kept requesting.

cheers,
Bill]]> bill2or3 Administration, Announcements, Policy Tue, 23 Apr 2013 11:46:02 -0400 https://reprap.org/forum/read.php?33,199173,202106#msg-202106 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,202106#msg-202106 Traumflug Administration, Announcements, Policy Tue, 23 Apr 2013 05:20:25 -0400 https://reprap.org/forum/read.php?33,199173,202068#msg-202068 Re: Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,202068#msg-202068
good job done :)-D

Can you 'put your finger' on location and time the majority of this 'unwanted' requests occure?

But I think, the databases are so big now and totally overwhelmed with high frequent queries, so we should look for better hardware/software anyhow ...]]> VDX Administration, Announcements, Policy Tue, 23 Apr 2013 02:20:44 -0400 https://reprap.org/forum/read.php?33,199173,202023#msg-202023 Re: Snippets from the reprap-admin list - part 2. https://reprap.org/forum/read.php?33,199173,202023#msg-202023
Stopping these hits reduced the outbound wiki bandwidth by somewhere around %25.

The reprap.org server is looking happier each week.

Bill]]> bill2or3 Administration, Announcements, Policy Mon, 22 Apr 2013 22:21:57 -0400 https://reprap.org/forum/read.php?33,199173,199173#msg-199173 Snippets from the reprap-admin list https://reprap.org/forum/read.php?33,199173,199173#msg-199173
Quote

On 12 April 2013 14:47, bill2or3 wrote:


I did a round of server tuning, here are the changes I made:

- added robots.txt files to the wiki and forums, to tell msnbot, bingbot
and YandexBot not to request > 2 pages per minute. Those 3 bots combined
were accounting for *thirty percent* of the hits to the wiki, but
refer only a handful of visitors. This has made a huge difference.

- one-month 'Expires:' headers for the icon images on forums.reprap.org.
This will let those icons persist in client browser caches, across browser
restarts. Saves hits to the server, speeds up load times for return visitors.

- made the forum 'update viewcount' queries LOW_PRIORITY, so that actual
useful SELECT queries will happen first.

- set apache ssl session cache to the minimum size, 8k. It was set to
512k before, entirely unused. I think we can disable mod_ssl entirely,
which should free up a few megs.

- tiny fix to ReCaptcha.php, to avoid about 9 megs of errors messages per day
hitting /var/log/messages

The Munin cpu and network graphs are pretty flat since these changes.
Before, they were very inconsistent, occasionally spiking up by %400 or
just missing big chunks where the server was too busy to collect the graph
data.

I'm guessing that what was happening was that bingbot came to visit,
from several different IP's at once. All those bingbot crawlers were hitting
forums, making lots of mostly identical requests. All those queries eventually
hit the 'messages' table, causing the sql table to lock, and grinding
everything to a crawl.

The graphs show that apache hits/apache traffic/ethernet traffic all dropped
by about 1/3, and CPU seems to have a lot more breathing room.

If nobody has any objections, I'd like to let it run for a few days, just to
be sure everything is stable, and then enable mod_gzip and disable mod_ssl,
in Apache.

Depending how it looks after that, perhaps investigate using APC php caching,
and/or parsing php code via fastcgi instead of with mod_php.

Overall, I'm pretty happy with these results, and expect to quit hearing
"is it down?" asked in IRC.

cheers,
Bill
]]> Traumflug Administration, Announcements, Policy Fri, 12 Apr 2013 08:37:44 -0400