admincraft chat regarding ovh raid issues

WayGroovy: So... the new core protect upgrade doesn't spam the console? So I have zero clues as to how long this update is taking?
WayGroovy: Well, I can tell something is happening, because /plugins/CoreProtect/database.db is increasing in file size. Nothing is logging to the console though.
WayGroovy: I don't like how it says, in the startup, "don't restart until this is done" but gives zero indication of how done it is.
WayGroovy: My bad. There was an update. 4% complete.
chiisana: lol, that's gonna take some time
WayGroovy: 10 minutes per %, so approx 16+ hours.
WayGroovy: With no restarts. Gosh I hope this doesn't fail over.
chiisana: should be generally okay
WayGroovy: 7% complete
WayGroovy: what does 8820520k cached in TOP mean?
WayGroovy: is that 8.8 GB of data cached to write?
WayGroovy: 11%. Users on all of my servers reporting heavy lag, but not very descriptive. Server with transfer in progress shows 0-1% cpu and 20% of 3072 MB ram usage. Most users log off within seconds.
WayGroovy: Users aren't just logging off. Looks like they may be getting kicked.
WayGroovy: Not sure which plugin is doing the kicking, or if it is a bukkit/spigot thing.
dzadnik: sounds like network issues
WayGroovy: Traffic is low: http://i.imgur.com/5BOBN2K.png , looking deeper
WayGroovy: vnstat doesn't show much
WayGroovy: Speed test shows 6.84 Mbps down and 15.32 Mbps up
WayGroovy: subsequent tests much faster, possibly due to caching somewhere
WayGroovy: Any ideas on what I can test to see if this is a network issue, and not related to the coreprotect upgrade, which coincidentally started simultaneiously to the issue?
WayGroovy: http://pastebin.com/wdqbvNav another server running on the machine, but not in sleep mode. CP2 update started at 1:29 system clock time.
WayGroovy: time dd if=/dev/zero of=/tmp/test-hd bs=1M count=1000

gives out

1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 18.1464 seconds, 57.8 MB/s

real    0m20.200s
user    0m0.000s
sys     0m0.876s
WayGroovy: Any ideas on what I can do? Right now I've got 4 minecraft servers aparrantly out of comission due to the coreprotect update in progress. I've tried 'renice +1 {pid}' to give the server running the update a lower priorirty, but that has had no apparant effect.
WayGroovy: Does the coreprotect update occur in the primary minecraft process in linux?
dzadnik: no
WayGroovy: If I bump the process id in nice for the main java running the server, would that have any effect? right now it looks like everything in top is running at 0, except cpuset, khelper, netns, and kintegrityd, which are at -20
dzadnik: doubt it
chiisana: cache is file system cache
chiisana: linux kernel looks at files you use regularly and caches them in ram for you
chiisana: if your application requires ram, it will make those resources available automatically w/o lag
chiisana: renice'ing things only matter when your cpu is full
chiisana: if your cpu is not full, it will just use what it can
dzadnik: fill dat CPU up!
chiisana: Groovy, how frequently are the updated perecentage message appearing for you?
chiisana: These:
Upgrading... X% complete.
WayGroovy: minutes and minutes apart
WayGroovy: about 20+?
chiisana: just scanning through the code right now and trying to see how it works, does it report every percent?
WayGroovy: yep,
WayGroovy: I'm not exactly sure what the holdup is. I imagine it's drive access possibly.
WayGroovy: I'm very curious how an upgrade on a VPS would affect the other VPS on the shared machine.
chiisana: can you please do `iostat` without the quotes for me in your ssh as root?
chiisana: you're running things in VM? what's the drive setup on the host looking like?  RAID?
WayGroovy: I'm not in VM, but I am running raid 1 i believe
chiisana: what do you mean for the VPS part then?
WayGroovy: I'm curious if it had been in a VPS envrionment if the other vps's would be afffected
chiisana: ah
chiisana: it really depends...  I'm not sure what is the hold up right now, too
WayGroovy: Nor I
chiisana: what's the iostat command output looking like?
WayGroovy: http://pastebin.com/utyvBaN5
WayGroovy: note: that is currently outside my realm of understanding. I read spaghetti better than that.
chiisana: well, the two column we care about for the time being are Blk_read/s   Blk_wrtn/s
chiisana: blocks read per second and blocks written per second
chiisana: what we are seeing are two drives, each with 3 partitions, making 2 raid paritions some how
WayGroovy: Interesting. I have no idea why that is that way.
WayGroovy: Filesystem            Size  Used Avail Use% Mounted on
/dev/md1               10G  3.7G  5.8G  40% /
/dev/md2              1.8T   69G  1.7T   4% /home
tmpfs                 7.8G   24K  7.8G   1% /dev/shm
chiisana: probably one partition for boot and another for actual content, the third not in use one would be swap
chiisana: yep, looks about same
WayGroovy: I've tried shutting down the other mc servers, it had no visible affect
chiisana: no, it wouldn't
chiisana: from the looks of it, I'm not sure if the delay is coming from resources
chiisana: I stopped working on code for a bit before Dan started the converter, so I don't really know it that well
chiisana: I'm seeing a few places where we could potentially increase performance, but I really don't think it would be this drastic
chiisana: something else must be holding things up.... but I don't know what.
chiisana: CPU is definitely not the bottle neck
WayGroovy: I feel bad, because 2 of the mc servers on my machine aren't my servers. I've sublet them, one to a coworker, one to an internet friend.
chiisana: so shutting down other servers won't help at all
chiisana: are other servers affected, too?
WayGroovy: yep
chiisana: what are the problems on that side?
chiisana: any error log messages?
WayGroovy: similar, read timed out, disconnected due to flying
WayGroovy: but all network health applications I've run seem... wonderful
chiisana: well
WayGroovy: I even ran a rsync last night to my home machine, no issue, similar to usual speeds.
chiisana: my guess from looking at the code and everything I can see points at potential disk io blocking
chiisana: but, the weird thing is this
chiisana: your %iowait is only at 2.22, while %idle is at 85.69
chiisana: that means your CPU is waiting for disk roughly 2.22% of the time, but not doing anything 85.69% of the time
chiisana: it really could be doing more stuff and wait more... but I don't know whats going on and why it isn't
WayGroovy: Hmm. I'm curious, but I'm clueless as to what it could be. I'm mostly surprised at how it's doing it.
WayGroovy: I have noticed that the coredata folder size is shrinking over time, as the coreprotect db file grows
WayGroovy: I'm assuming that's natural with the conversion process
chiisana: yes, the idea is we progressively shave the old data into the database
chiisana: that's how we can pick up even if the server crashes/shuts down during upgrade
chiisana: so we know where to continue
WayGroovy: Ah. Well that part is nice.
WayGroovy: subsequent iostats are similar. negligible variation
chiisana: are you using MySQL or SQLite?
WayGroovy: flatfile to sqlite
chiisana: hm... I wonder if that's whats causing the delay...
WayGroovy: I've got a meeting to go to, need to grab my coat.
chiisana: alright
chiisana: I'll see if I can do something about it, but it would mean interupting the update and losing the changes happened up to that point :/
WayGroovy: Well, no players have been online for more than 2 minutes since starting,
WayGroovy: I think cumulative online time in the past... lemme check mcstats
WayGroovy: http://mcdigr.com/dashboard/public/europa.waygroovys.com
chiisana: ouch, your TPS took a huge hit :/
WayGroovy: yeah.
WayGroovy: i hadn't thought to look there yet.
WayGroovy: averaging around 14-15, low as low as 3
chiisana: hm... the thing I thought was hammering database looks like it is cached already...
WayGroovy: blocks broken or placed since starting per mcdigr is 0/0, so no problem restarting.
chiisana: alright, I'll try to poke around and see if I can find anything
WayGroovy: k. i'll be by in an hour or two. thanks for looking.
chiisana: don't thank me yet... thank after I fix it :P  have a good meeting :P
WayGroovy: (it's outside. in the cold :(
WayGroovy: postponted. 15-inf minutes.
WayGroovy: 64% complete
WayGroovy: oh shit... how is rsync going to handle this sqlite file?
WayGroovy: Fuckit. I'll take the conversion offline and install prism.
WayGroovy: 24 hours downtime, x 4 servers, (* 8 players), = approx 52736 hours of playtime down due to the update process.
WayGroovy: Maths. Not even once. 192 hours.
chiisana: I suspect rsync wont like sqlite db too much if it is being written into at the same time...
chiisana: that's not a core protect thing but more of a filesystem thing...
chiisana: as for going to prism... :x
chiisana: I was busy making data to test upgrade thing :x
chiisana: only managed to fill about 450mb of data so far :x
WayGroovy: Understandable. I was getting pinged on far too many communication fronts
WayGroovy: I suppose I could send you a historical version of my coredata
chiisana: how big is your coredata folder?
WayGroovy: 3 GB
WayGroovy: close to 4
chiisana: root@builddit:/home/minecraft/CoreProtect/plugins/CoreData# du -hs
448M .
WayGroovy: it's much smaller now
chiisana: that's my size so far... even with a brush I branded my data generation brush
WayGroovy: lols
chiisana: "//br sphere 5%TNT,5%redstone_torch,5%iron_block 5"
WayGroovy: throw some sand into that for fun
chiisana: flying around using WorldEdit to generate spheres of death, lol
WayGroovy: pretty fun indeed.
chiisana: seriously though... it lags me due to graphics lag :/
WayGroovy: I wish I had a clue about php
chiisana: O.o
chiisana: actually... 600mb of data should be enough... that's ~1/6 of what you've got so I should see 3 hours conversion, assuming linear scaling?
WayGroovy: assuming that. I suppose.
chiisana: I'll just pull the latest fro bukkitdev and see what happens...
WayGroovy: dan replied to my post, says that he can do 12GB in a few hours, so my machine must be pathetic in comparison.
chiisana: well, FWIW, I'm trying it on a 4 core 4GB ram VPS
WayGroovy: Fair enough.
chiisana: so whatever result I get is going to be probably acceptable as a baseline
chiisana: I'm using spigot, but I don't think that'd make any difference in the conversion process?
chiisana: ah... whatever, I'll just use bukkit RB
chiisana: then it would be more definitive
WayGroovy: I was using spigot as well, but sure, that'll give a good baseline
chiisana: 18:20:16 [INFO] [CoreProtect] Upgrading... 1% complete.
18:20:20 [INFO] [CoreProtect] Upgrading... 2% complete.
18:20:25 [INFO] [CoreProtect] Upgrading... 3% complete.
18:20:28 [INFO] [CoreProtect] Upgrading... 4% complete.
18:20:31 [INFO] [CoreProtect] Upgrading... 5% complete.
chiisana: root@builddit:/home/minecraft/CoreProtect.1.x/plugins/CoreData# du -hs
722M .
WayGroovy: obv leaps and bounds faster.
chiisana: root@builddit:/home/minecraft# iostat
Linux 2.6.32-5-amd64 (builddit)  02/28/2013  _x86_64_ (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.19    0.00    0.03    0.00    0.00   99.77

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
vda               0.15         0.30         6.67     249036    5623184
dm-0              0.84         0.29         6.67     245466    5623168
dm-1              0.00         0.00         0.00       1032          0
[2/28/2013 9:21:44 PM] chiisana: iostat also shows much less stuff
[2/28/2013 9:22:24 PM] chiisana: so perhaps something else was doing huge amounts of read and write on your server
[2/28/2013 9:22:57 PM] WayGroovy: avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.82    0.02    1.25    2.32    0.00   85.60

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              61.63      4641.20       520.42 27225812091 3052838408
sda1              1.12         0.31        18.14    1843355  106439928
sda2             60.50      4640.87       502.25 27223906936 2946230952
sda3              0.00         0.01         0.03      61160     167528
sdb              61.59      4641.24       520.39 27226038547 3052670880
sdb1              1.12         0.31        18.14    1807753  106439928
sdb2             60.46      4640.93       502.25 27224229322 2946230952
sdb3              0.00         0.00         0.00        832          0
md2              62.35         9.91       495.12   58107194 2904414464
md1               1.90         0.61        14.99    3572786   87940432
WayGroovy: perhaps so,
WayGroovy: that's my current conditions
WayGroovy: with it not currently doing anything
chiisana: are you stll converting?
chiisana: O.o
WayGroovy: no, just running mc server
WayGroovy: s
WayGroovy: dafaq
chiisana: how many online?
WayGroovy: 4
WayGroovy: oh, players?
chiisana: dafaq?
WayGroovy: my poor disks!
WayGroovy: how to determine source....
WayGroovy: klill processes randomly?
chiisana: no, don't do that
WayGroovy: :D
chiisana: look for iotop in your package system
WayGroovy: reboot machine?
WayGroovy: iotop. k
WayGroovy: blasted missing dependency
WayGroovy: python-ctypes
WayGroovy: nope, that's not it
chiisana: the only thing I can think of is if the software raid is doing some kind of background scrubbing... but even then, it _really_ shouldn't be doing it to the point where it is affecting performance that much
chiisana: doesn't your package manager help you with dependencies?
WayGroovy: Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package iotop.noarch 0:0.4.3-4.el5 set to be updated
--> Processing Dependency: kernel >= 2.6.18-199 for package: iotop
--> Finished Dependency Resolution
iotop-0.4.3-4.el5.noarch from base has depsolving problems
  --> Missing Dependency: kernel >= 2.6.18-199 is needed by package iotop-0.4.3-4.el5.noarch (base)
Error: Missing Dependency: kernel >= 2.6.18-199 is needed by package iotop-0.4.3-4.el5.noarch (base)
 You could try using --skip-broken to work around the problem
 You could try running: package-cleanup --problems
                        package-cleanup --dupes
                        rpm -Va --nofiles --nodigest
The program package-cleanup is found in the yum-utils package.
-bash-3.2#
chiisana: wtf, it depends on a kernel version?
WayGroovy: \o/
WayGroovy: \_o_/
WayGroovy: No package kernel available.
chiisana: I have no idea how to work with rpm :/
chiisana: 18:28:22 [INFO] [CoreProtect] Upgrading... 100% complete.
18:28:22 [INFO] [CoreProtect] Logging new data. Please wait...
18:28:23 [INFO] --------------------
18:28:23 [INFO] [CoreProtect] Upgrade successfully completed.
18:28:23 [INFO] --------------------
>
WayGroovy: I know how to so little
dzadnik: ugh
dzadnik: coreprotect 2.1 to-do list is so long already
dzadnik: going to take a break for a week or so first before beginning any work on it :P
dzadnik: also, most of the things in the prism 1.4 release, CoreProtect can already do :P
WayGroovy: No ideas on my issue, other than my hardware isn't up to snuff,?
dzadnik: and nope
 Edited 9:34:20 PM] dzadnik: try using smartctl maybe
dzadnik: see if your drive is failing
chiisana: My best guess is something is eating through your disk IO... but I have no idea what
dzadnik: shouldn't be minecraft
dzadnik: it uses way less disk i/o than people think ;P
WayGroovy: /dev/sda passed, checking /dev/sdb
WayGroovy: sdb passed
chiisana: rpm -q kernel  ?
WayGroovy: package kernel is not installed

da...shit
chiisana: not necessarily wrong
WayGroovy: yum list grep kernel
chiisana: I don't know RPM, and that's just a command I found online that might reveal what kernel version is installed according to rpm
WayGroovy:  2.6.18-53.1.21.el5
WayGroovy: kernel-headers.x86_64                      2.6.18-53.1.21.el5          installed
chiisana: so you're using 2.6.18-53, but iostat wants 2.6.18-199
chiisana: have you updated your system recently?
WayGroovy:
No Packages marked for Update
chiisana: hm... :3
chiisana: wget http://guichaz.free.fr/iotop/files/iotop-0.5-1.noarch.rpm
rpm -i iotop-0.5-1.noarch.rpm
chiisana: I wonder if you can install the rpm on iotop's homepage
chiisana: not sure about that second command... I vageuly recall -i for install, but if it is wrong, man rpm
chiisana: supper time, afk for now
chiisana: hopefully we can some how get iotop in there so we can check out what's eating your disk write
chiisana: your md1 and md2 aren't seeing the read, so the high read is probably background scrubbing for software raid
chiisana: but one of your md# is seeing a lot of write still... so something is creating a lot of data and writing to disk a lot
chiisana: just shy of 500blk write/sec, and assuming conventional ish 512kb blocks, that's almost 250mb/s
WayGroovy: food sounds good, i'm getting frustrated.
chiisana: rpm from website didn't help either?
WayGroovy: no, it's not
chiisana: (worry)
WayGroovy: need to update python, looking for that now, had an alarm, had to leave keyboard for 20 min or so
chiisana: try this old package:
chiisana: wget http://repo.or.cz/w/iotop.git/snapshot/416f4c9419a7b5ecf3f1b3dcf366217b072ae79e.zip
chiisana: unzip 416<tab>
chiisana: cd iotop/bin
chiisana: ./iotop
WayGroovy: No module named iotop.ui
To run an uninstalled copy of iotop,
launch iotop.py in the top directory
chiisana: O.o
chiisana: ok,
cd ..
./iotop.py
WayGroovy: no love
chiisana: what's the error this time? O.o
WayGroovy:
-bash-3.2# ./iotop.py
-bash: ./iotop.py: No such file or directory
chiisana: are you in the iotop dir?
WayGroovy: yes, but i also tried from /
chiisana: should go to the folder named iotop after you extract
chiisana: do a `ls` there for me please
chiisana: lemme see what's included in that package for you
WayGroovy: AHAHHAHA
WayGroovy: there it be
chiisana: :P
WayGroovy: oh hell
WayGroovy: so many columns, but mostly 0.00 B/s or %
chiisana: anything under Disk Write?
chiisana: (that's not 0
WayGroovy: http://pastebin.com/HsYVWrED
WayGroovy: so many 0s
chiisana: wtf all 0
chiisana: total disk write is virtually nothing as well
chiisana: what does iostat say now?
WayGroovy:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.81    0.02    1.25    2.34    0.00   85.58

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              61.64      4638.13       520.72 27227255203 3056759024
sda1              1.12         0.31        18.16    1844891  106603088
sda2             60.50      4637.81       502.53 27225348512 2949987888
sda3              0.00         0.01         0.03      61160     168048
sdb              61.60      4638.15       520.69 27227387355 3056590976
sdb1              1.12         0.31        18.16    1809217  106603088
sdb2             60.47      4637.85       502.53 27225576666 2949987888
sdb3              0.00         0.00         0.00        832          0
md2              62.43        10.37       495.40   60896058 2908157032
md1               1.90         0.61        15.01    3575738   88088776
WayGroovy: kkwnwbllooooow
chiisana: ok, now I am officially confused....
chiisana: iotop says nothing is writing, but iostat say you've got something writing
chiisana: what do you get when you do:
cat /proc/mdstat
WayGroovy: md1               1.90         0.61        15.01    3575738   88088776

-bash-3.2# ./iotop.py
Total DISK READ: 0.00 B/s | Total DISK WRITE: 30.76 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
28002 be/4 visualke    0.00 B/s   11.54 K/s  0.00 %  0.00 % java -Xincgc -Xmx60M -Xms60M -XX:MaxPermSize=80M -jar Minecraft_RKit.jar kev:Babies
19867 be/4 europa      0.00 B/s    0.00 B/s  0.00 %  0.00 % java -Dfile.encoding=utf-8 -Djline.terminal~t.jar nogui -d yyyy-MM-dd HH:mm:ss -nojline
11877 be/4 visualke    0.00 B/s    0.00 B/s  0.00 %  0.00 % java -Xms1536M -Xmx1536M -Djline.terminal=j~r /home/visualkev/cbs/craftbukkit.jar nogui
11878 be/4 visualke    0.00 B/s    0.00 B/s  0.00 %  0.00 % java -Xms1536M -Xmx1536M -Djline.terminal=j~r /home/visualkev/cbs/craftbukkit.jar nogui
22528 be/4 visualke    0.00 B/s    0.00 B/s  0.00 %  0.00 % java -Xincgc -Xmx60M -Xms60M -XX:MaxPermSize=80M -jar Minecraft_RKit.jar kev:Babies
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init [3]
WayGroovy: ugly, but I caught some non zeros
WayGroovy: so iotop is functioning, somewhat
chiisana: there's only md1 in your /proc/mdstat ?
chiisana: what the heck is going on with your md2 then.... can please also check:
mdadm --detail /dev/md2
WayGroovy: http://pastebin.com/WjVPJksn
chiisana: looks okay
WayGroovy: -bash-3.2# who -b
         system boot  2012-12-23 07:55
chiisana: that's 2 months of uptime, nice and solid :)
chiisana: compared to my crappy 9 days :P
19:39:22 up 9 days, 19:19,  2 users,  load average: 0.44, 0.10, 0.03
WayGroovy: I'd hate to break that, but if I must,
chiisana: well, I don't know what's going on
WayGroovy: Me neither.
chiisana: I don't think rebooting would solve anything....
WayGroovy: me neither...
chiisana: since, if you didn't start anything, then whatever is doing it is doing it automagically
chiisana: and if you did start something that you need, you'll likely start it again anyways :P
chiisana: can you do: iostat -x
WayGroovy:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.81    0.02    1.25    2.34    0.00   85.58

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.42    39.49 36.04 25.59  4637.54   520.66    83.70     0.03   12.39   2.67  16.46
sda1              0.01     1.16  0.01  1.11     0.31    18.16    16.50     0.10   88.27  55.36   6.20
sda2              0.41    38.33 36.03 24.47  4637.21   502.47    84.96     0.66   10.98   2.35  14.21
sda3              0.00     0.00  0.00  0.00     0.01     0.03    13.95     0.00   37.67  24.80   0.01
sdb               0.43    39.50 36.02 25.57  4637.56   520.63    83.75     0.58    9.38   2.69  16.57
sdb1              0.00     1.16  0.01  1.11     0.31    18.16    16.51     0.10   86.00  54.38   6.08
sdb2              0.43    38.34 36.00 24.46  4637.25   502.47    85.01     0.48    7.96   2.37  14.31
sdb3              0.00     0.00  0.00  0.00     0.00     0.00    48.94     0.00    8.47   8.00   0.00
md2               0.00     0.00  0.49 61.92    10.37   495.35     8.10     0.00    0.00   0.00   0.00
md1               0.00     0.00  0.02  1.88     0.61    15.01     8.21     0.00    0.00   0.00   0.00
chiisana: the -x should give us breakdown of how long the average wait time are for the inidividual drives
WayGroovy: ugly, let me pastebin
WayGroovy: http://pastebin.com/mQQy6zuF
chiisana: I'm not familiar with all the details, but what I do notice is your sdX1, which makes up your md1, used for boot, and main OS, is pretty slow
chiisana: not a whole lot is done on it, so it shouldn't matter, though
WayGroovy: iotop/setup.py install worked
chiisana: your sdX2, which makes up your md2, home directory, most probably where your minecraft server + worlds etc. are, is fairly fast... but have a lot of read and write happening on the sdX2 level, but not on md2 level
chiisana: well, a lot of read, not happening on md2 level
chiisana: but the writes are happening on md2 level
chiisana: my best guess tells me your raid is doing background scrubbing with the read process, which is a very good thing to do... background scrubbing is when your drive is supposedively not doing anything, it will try to read every single byte from your drive to make sure there are no bad sector etc.
chiisana: so when a drive fail happens, you don't risk losing data because other drive(s) in the raid array is bad
chiisana: however, this doesn't explain the writes we're seeing in iostat (but not in iotop)
WayGroovy: hmm.
chiisana: I'm afriad I'd have to admit defeat here as well :x
WayGroovy: I will cry to my pillow tonight.
chiisana: I'll do more research on this... but in the mean time, your guess as to what's going on is probably as good as my...
WayGroovy: 198.27.64.58/~europa/404.swf
chiisana: lol
WayGroovy: 1261 to a partition!
md: md2: data-check done.
grsec: From 66.249.86.225: signal 11 sent to /opt/avg/av/bin/avgscand[avgscand:28373] uid/euid:0/0 gid/egid:0/0, parent /opt/avg/av/bin/avgtcpd[avgtcpd:28224] uid/euid:0/0 gid/egid:0/0 by /opt/avg/av/bin/avgscand[avgscand:28381] uid/euid:0/0 gid/egid:0/0, parent /opt/avg/av/bin/avgtcpd[avgtcpd:28224] uid/euid:0/0 gid/egid:0/0
grsec: From 66.249.86.225: denied hardlink of /home/europa/Super_Spice_Bros_64.swf (owned by 0.0) to /home/europa/public_html/Super for /usr/libexec/openssh/sftp-server[sftp-server:25761] uid/euid:500/500 gid/egid:500/500, parent /usr/sbin/sshd[sshd:25760] uid/euid:500/500 gid/egid:500/500
The scan_unevictable_pages sysctl/node-interface has been disabled for lack of a legitimate use case.  If you have one, please send an email to linux-mm@kvack.org.
WayGroovy: other 5 lines
chiisana: ?
WayGroovy: No idae
WayGroovy: http://serverfault.com/questions/236399/how-can-i-find-which-process-is-causing-this-io-read-tried-iotop-already trying to follow along with this
chiisana: I just remembered these two threads:
chiisana: http://ww.reddit.com/r/admincraft/comments/16f99f/trying_to_get_to_the_root_of_my_performance_issues/
http://ww.reddit.com/r/admincraft/comments/1451ep/help_odd_tick_rate_no_lag_ramdisk/

Seems like OVH have had a history with RAID being weird
WayGroovy: very interesting. I've always had no issue, even with upwards of 40 people on the main server,
WayGroovy: BUT, you don't typically go investigating io r/w when there isn't an issue.
chiisana: and typically you don't really have a problem on your server... only during excess disk io such as upgrading core protect do you start to notice the problem...
WayGroovy: bash-3.2# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   45820 MB in  2.00 seconds = 22934.12 MB/sec
 Timing buffered disk reads:  326 MB in  3.01 seconds = 108.42 MB/sec
chiisana: root@builddit:/home/minecraft# hdparm -Tt /dev/vda

/dev/vda:
 Timing cached reads:   11938 MB in  2.00 seconds = 5975.69 MB/sec
 Timing buffered disk reads: 748 MB in  3.01 seconds = 248.82 MB/sec
chiisana: looks like cached read is insanely awesome, butbuffered disk read is not as good
WayGroovy: bash-3.2# mount
/dev/md1 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw)
/dev/md2 on /home type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
chiisana: that looks about right
WayGroovy: frizzing googling and not comprehending 100% but bfuddling through
WayGroovy:  http://browser.primatelabs.com/geekbench2/view/1705220
WayGroovy: shit. none of that info is disk io
chiisana: lol
WayGroovy: oh god, how did i wind up http://www.bringinthecats.com/
WayGroovy: warning
WayGroovy: seizure warning
chiisana: http://browser.primatelabs.com/geekbench2/view/1705243  <-- your server rapes my VM
WayGroovy: Damn
WayGroovy: daaa FUCK
chiisana: so it really is just that disk io that's holding you back
WayGroovy: running iozone now
WayGroovy: no idea
WayGroovy: http://www.thegeekstuff.com/2011/05/iozone-examples/
dzadnik: lol
dzadnik: CoreProtect 2 versus CoreProtect 1
dzadnik: http://i.imgur.com/6MmJA2q.png
WayGroovy: lols
WayGroovy: Quite impressive.
WayGroovy: your own data?
dzadnik: no
chiisana: I blame dan and his weird anonymous thread thing and random sleeping :P
dzadnik: converting 10.7GB of data took me 3 hours and 31 minutes
WayGroovy: >:{ flip all the fucking tables
dzadnik: :P
dzadnik: also
dzadnik: ugh
dzadnik: was so hopefull about using http://coreix.net/ as my new UK DC
dzadnik: but after dealing with their support team
dzadnik: well
dzadnik: their support team isn't as nice as their website design, I'll just say.
chiisana: ovh/hetzner :P
chiisana: except, groovy's having raid problems with ovh as with a few others on /r/admincraft...
WayGroovy: I'm not entirely sure that my numbers indicate a problem exactly...
WayGroovy: i don't know enough.. damn.
chiisana: well, there is a.... notable amount of slow down when it comes to large amount of disk read + write...
chiisana: and tests for other parts show your server is pretty awesome...
WayGroovy: https://docs.google.com/spreadsheet/ccc?key=0AsUG-_IEV-_OdEVoWEd3SVh2TlRMRm5LdnFGWVdhUmc&usp=sharing
WayGroovy: fucking numbers, what do they mean
chiisana: Command line used: iozone -a -b iozoneout.xls
 Output is in Kbytes/sec
WayGroovy: so, low numbers would be bad?
chiisana: yes, I guess
WayGroovy: I need some basis of comparison
WayGroovy: http://blog.serverfault.com/2010/07/06/777852755/
chiisana: running it on my test VPS already
WayGroovy: https://code.google.com/p/iozone-results-comparator/wiki/Overview
chiisana: https://docs.google.com/a/terrandin.com/spreadsheet/ccc?key=0AnVSLrpE2qnedDVZSE02VmZ1T1ZfdDB5a2I0T2ExRHc&usp=sharing
chiisana: yours looks better tbh
WayGroovy: shared to any with link?
WayGroovy: or just waygroovy@?
chiisana: should be any, no idea :D
chiisana: @gmial?
WayGroovy: hm. won't open for me, says i need to req access
chiisana: mail
chiisana: try again?
WayGroovy: I'm in (uuuuunnnnhh)
WayGroovy: I think the takeaway from all of this is I don't have a freaking clue, but I know how to google and run various diagonistics, but don't know how to properly interperet them.
chiisana: well, it is telling me your server should theoretically perform way better than my
chiisana: which makes sense, i5 (2011+) vs E5450 (2009)
chiisana: DDR3 vs DDR2
WayGroovy: sure, but... why coreprotect, why did you take that arrow in the knee?
chiisana: that is a mystery in life :/
WayGroovy: could it have something to do with the number of open files?
WayGroovy: cat /proc/sys/fs/file-max > 1613010
chiisana: root@builddit:~# cat /proc/sys/fs/file-max
405036
WayGroovy: various rage noises
dzadnik: aroooo
chiisana: so, um...
dzadnik: slurp
WayGroovy: slobber