Your server is running very slow.<\/p>\n
Top looks Okay. Loads are not insane. Ram is high, but not out of bounds. sar shows high i\/o wait times. Swap usage is not an issue. MySql process show hundreds of sleeping processes. Restarting MySql or Apache clears them, but then they start right back up.<\/p>\n
WHAT? This make no sense!<\/p>\n
\ntail \/var\/www\/vhost\/domain.com\/statistics\/log\/access_log\n<\/pre>\n\u2026<\/p>\n
[07\/Dec\/2013:17:08:17 -0700] \u201cGET \/local\/image_product480000_1\/mlomeupenvtb2012tb201212tb201212044071d032736e44d9b3e5b914d378f9e2jpg.jpg HTTP\/1.0\u2033 200 16322 \u201c-\u201d \u201c-\u201d
\n[07\/Dec\/2013:17:08:17 -0700] \u201cGET \/local\/image_product480000_1\/pics2dsstaticcomprodimg165178300jpg.jpg HTTP\/1.0\u2033 200 12690 \u201c-\u201d \u201c-\u201d
\n[07\/Dec\/2013:17:08:17 -0700] \u201cGET \/local\/image_product480000_1\/slimagesmacyscomisimageMCYproducts4optimized515264fpxtif.jpg HTTP\/1.0\u2033 200 10497 \u201c-\u201d \u201c-\u201d
\n[07\/Dec\/2013:17:08:17 -0700] \u201cGET \/local\/image_product480000_1\/plefuxcom6120111219A0361000WNipadiphonebatteriesexternal5000mah3751965bigjpg.jpg HTTP\/1.0\u2033 200 9638 \u201c-\u201d \u201c-\u201d
\n[07\/Dec\/2013:17:08:17 -0700] \u201cGET \/local\/image_product480000_1\/taylorgiftscomimagesp43126500jpg.jpg HTTP\/1.0\u2033 200 59977 \u201c-\u201d \u201c-\u201d<\/p>\nNotice how these connections are coming from the server itself instead of from an external IP.<\/p>\n
Now look at who is connecting to the server:<\/p>\n
\nnetstat -nat | grep :80 | gawk '{ print $5; }' | gawk -F: '{ print $1 }' | sort | uniq -c | sort -n\n<\/pre>\n2 66.249.73.222
\n3 157.55.32.143
\n3 199.30.20.68
\n3 199.30.20.76
\n4 131.253.24.85
\n4 199.30.20.106
\n4 23.67.252.11
\n4 65.55.55.229
\n5 174.125.28.4
\n12 23.67.252.59
\n325 64.150.184.165<\/p>\nAgain, all coming from the server. The solution to the problem was discovered in \/tmp<\/p>\n
\nls -la \/tmp\n<\/pre>\ntotal 44532
\ndrwxrwxrwx 4 root root 3522560 Dec 7 17:12 .
\ndrwxr-xr-x 24 root root 4096 Dec 6 13:03 ..
\ndrwx\u2013x\u2013x 2 apache apache 4096 Feb 29 2012 .bash
\n-rw-r\u2013r\u2013 1 apache apache 37281 Oct 13 10:21 .dsf
\n-rw-r\u2013r\u2013 1 apache apache 37287 Oct 13 17:46 .dsf.1
\n-rw-r\u2013r\u2013 1 apache apache 37287 Oct 13 17:46 .dsf.2
\n-rw-r\u2013r\u2013 1 apache apache 37287 Oct 13 17:46 .dsf.3
\n-rw-r\u2013r\u2013 1 apache apache 37287 Oct 13 17:46 .dsf.4
\n-rw-r\u2013r\u2013 1 apache apache 37287 Oct 13 17:46 .dsf.5
\n-rw-r\u2013r\u2013 1 apache apache 37287 Oct 13 17:46 .dsf.6
\n-rw-r\u2013r\u2013 1 apache apache 37281 Oct 13 18:18 .dsf.7
\n-rw-r\u2013r\u2013 1 apache apache 37281 Oct 13 18:18 .dsf.8<\/p>\nnow,<\/p>\n
\nls -la \/tmp\/.bash\n<\/pre>\ntotal 27392
\ndrwx\u2013x\u2013x 2 apache apache 4096 Feb 29 2012 .
\ndrwxrwxrwx 4 root root 3522560 Dec 7 17:14 ..
\n-rwx\u2013x\u2013x 1 apache apache 146 Nov 12 2012 1
\n-rwxr-xr-x 1 apache apache 323 Jan 13 2011 autorun
\n-rwx\u2013x\u2013x 1 apache apache 8922 Jan 23 2006 b
\n-rwx\u2013x\u2013x 1 apache apache 19557 May 9 2005 b2
\n-rwxr-xr-x 1 apache apache 11445 Jan 5 2011 bang
\n-rwxr-xr-x 1 apache apache 12321980 Feb 29 2012 bangnew
\n-rwxr-xr-x 1 apache apache 11824732 Jan 23 2011 bangold
\n-rw-r\u2013r\u2013 1 apache apache 44 Aug 3 03:28 cron.d
\n-rwx\u2013x\u2013x 1 apache apache 14679 Nov 2 2005 f4
\n-rwxr-xr-x 1 apache apache 15988 Sep 7 2002 juno
\n-rw-r\u2013r\u2013 1 apache apache 11 Aug 3 03:28 mech.dir
\n-rwx\u2013x\u2013x 1 apache apache 566 Jan 20 2013 mech.set
\n-rwxr-xr-x 1 apache apache 27 Jan 11 2011 run
\n-rwx\u2013x\u2013x 1 apache apache 152108 Jan 11 2011 sshd:
\n-rwxr-xr-x 1 apache apache 17 Nov 5 2008 start
\n-rwxr-xr-x 1 apache apache 8231 Feb 29 2012 std
\n-rwxr-xr-x 1 apache apache 13399 Aug 6 2000 stealth
\n-rwx\u2013x\u2013x 1 apache apache 8790 Jan 23 2006 stream
\n-rwxr-xr-x 1 apache apache 17690 Feb 6 1996 synk
\n-rwxr-xr-x 1 apache apache 6442 Jun 23 2011 talk
\n-rwxr\u2013r\u2013 1 apache apache 166 Aug 3 03:28 update
\n-rwx\u2013x\u2013x 1 apache apache 14841 Jul 22 2005 v
\n-rwxr-xr-x 1 apache apache 14911 Mar 6 2002 v2<\/p>\nEnd Result<\/p>\n
End result: This server ahs been root compromised. The only solution is to reinstall and slave drive the existing compromised drive.<\/p>\n","protected":false},"excerpt":{"rendered":"
Your server is running very slow. Top looks Okay. Loads are not insane. Ram is high, but not out of bounds. sar shows high i\/o wait times. Swap usage is not an issue. MySql process show hundreds of sleeping processes. Restarting MySql or Apache clears them, but then they start right back up. WHAT? This … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[135,136],"class_list":["post-62","post","type-post","status-publish","format-standard","hentry","category-apache","tag-apache-access_log-server-connecting-to-itself","tag-connecting-with-its-own-ip"],"_links":{"self":[{"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/posts\/62","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/comments?post=62"}],"version-history":[{"count":0,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/posts\/62\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/media?parent=62"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/categories?post=62"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.qbytes.cloud\/index.php\/wp-json\/wp\/v2\/tags?post=62"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}