So far we've tried finding the files where iptables rules might be to no avail. Then I remembered centos7 doesn't use LSB scripts and uses systemd. So I tried looking in the few places I understood to look for that. I've even tried just renaming /{path}/usr/sbin/iptables-restore to iptables-restore-bkp to see if I could just stop iptables from loading the rules.

chamunks: to be honest, not sure how it works in el7, but some distros run legacy etc/rc.d scripts on the top of systemd
at least SLE/suse does

chamunks: it looks like the rules should be in /etc/sysconfig/iptables. I don't use the native service myself

chamunks: when we are in that, I love the ridiculous logic of systemd components ... for instance in `systemctl list-unit-files' one would expect the paths in the output pointing to units files

http://hastebin.com/asabisupow.avrasm
ntz you're assuming I can execute binaries in the hosts main OS
annoyingly enough I'm not that lucky.
Anyways the hastebin there is showing how theres nothing in there even alluding to iptables.

yeah you can't run systemctl without systemd running

chamunks: if you have another, running instance of el7 you can use the systemctl disable command in there to see what it does.

*there being /mnt/etc/systemd/basic.target.wants/iptables.service
el7 I'm guessing is Centos7

yes..

Is iptables even in use? Not firewalld?

Just the first time I've seen it referred to as el7

el7 is RHEL7 and any derivatives or clones such as CentOS 7

chroot is your friend.

Also isn't that likely to be enabled by default so not in the /etc/systemd tree?

chamunks, try /etc/sysconfig/firewalld

yes, chroot may work, I'm not sure if systemctl disable would then work without having systemd running or not, it might.

pj, the recovery environment runs systemd unless he is using a non centos disk to boot.

I have no physical access to the machine

chamunks, you can chroot under ssh

its remoting in via SSH into the datacenter.

chamunks: that's fine

it wont disconnect you.

no I just meant sorry I was replying to someone else.

you would want to bind mount /proc, /sys and /dev then you can chroot.

sling00 I've not really had much experience using chroot so I'll have to google that.

Sling: he didn't actually say if the "Recovery OS" is CentOS or not, I was assuming it wasn't.

also, firewalld sits in /etc/systemd/system/basic.target.wants
so add your prefix before it and bat it out of the park :p
if its there.

I'm not entirely sure what the recovery OS is.
Some OVH proprietary blend I'm assuming.
sling00 http://hastebin.com/asabisupow.avrasm sadly its not.

chamunks, easiest way to fix it would be to do do what pj said with the mount binds, and then chroot the install and fix the iptables settings
http://hastebin.com/ahacanaqek.hs

So seems that I can while chroot'd into /mnt/ then ran the following. http://hastebin.com/cececiyuho.1c pj not entirely sure where I should be mounting those specific directories.

exit the chroot
do that

oh brilliant

then you will have a semi normal chroot.
you still wont have access to systemd stuff
easiest way to fix it is to try to fix iptables in the chroot, save it, and see if it works on reboot

such highlight

Greetings, for sudoers should the following command not allow the the noted user account to switch to or run anything as the noted user "eseuser ALL=(mule) ALL"

$ sudo service memcached status
memcached dead but subsys locked

Followed through the chrooting with the binds, I'm getting an error trying to mess with the rules.

chamunks, http://hastebin.com/ifotocalaq.avrasm ignore the ipv6 stuff if you are not using it.
also i think the "start fresh" stuff should happen first
borrowed that stuff from a ubuntuforums post ^

i need to test an external smtp server. what command can i use via cli to do so?

subvhome: telnet

Okay that seems to appear as though it worked.
Well it accepted the commands with no errors.

chamunks, iptables --list

appears empty.
I'm imagining thats likely correct.
Is there a command for saving these?

use iptables-save instead, --list doesn't show everything
doesn't centos7 use firewalld ?

Sling, alot of people convert..because alot of howtos tell them too...or they are stubborn.

convert from firewalld to regular iptables?
howto's--

Stopping memcached: [FAILED]

Sling, yes.
in the same way that alot of how to's have people disable selinux in lieu of just adding the proper rule.

Sigh, rebooting still with no luck.
http://i.imgur.com/91Q0Yxh.png ping ping ping ping
sling00 / pj no luck.
Well its been an experience and learning how to chroot like this was vastly helpful I've had other scenarios where I would have killed for execution inside the recovery OS

chamunks, weird, did you do the iptables --save ?
er
iptables-save

I did iptables-save
sling00 yeah I ran that before rebooting into the main OS.

I would try again in the recovery environment and look in /etc/sysconfig/iptables and make sure nothing is wonky at the end of it.
by try again i mean go back, chroot, reset the rules, and also try saving it with service iptables save or /usr/libexec/iptables/iptables.init save

well iptables-save just sends all rules to stdout, it doesn't 'save' anything

I'll give it one more go. Like all of this stuff is backed up so the host OS is semi disposable its just annoying that we've not been able to solve the problem. Sometimes its better to learn how to fix something than do a windows style resolution to every problem. (blow it off and reinstall)

chamunks, agreed, it'd be alot easier if the recovery OS was centos7 or something that uses systemd that way you could just chroot and then systemctl disable firewalld && systemctl disable iptables

with firewalld just do 'firewall-cmd --permanent --add-service=servicename' (and optionally a specific zone) and you're done
no need to fiddle with iptables itself

chamunks, post output of ls /etc/systemd/system/basic.target.wants/
before leaving the chroot
so we can end the debate once and for all whether the machine is using firewalld or iptables, i assumed from earlier when you said neither firewalld.service or iptables.service existed that something wonky had happened, but may have missed something.

kk
[root@rescue /]# ls /etc/systemd/system/basic.target.wants/ microcode.service

doesnt look like either one is enabled ....wonky

well yeah he is in rescue mode
oh wait, basic.target
nvm me :)

chamunks, you are in the chroot right?

yerp

yep

I am in the chroot