kafka_zk_hdf_takeover


        
            
                Markdown 2.79 KB
                                    
                        | None                    
                
                                        |
    0    0                            

            
                
                                    raw
                    download
                    clone
                    embed
                    print
                
                                    report
                
                
                            
        
        
            ###################

##################
update resolve to search for shortnames on socialgist.local
search socialgist.local
update /etc/hostname to socialgist.local
update /etc/hosts to proper socialgist.local (or remove completely from hosts file if using DNS)
add to server.properties on old brokers -
inter.broker.protocol.version=0.8.0

log.message.format.version=0.8.0
restart all brokers to have proper name and new 0.8.0 configs
Ambari Install
wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

yum update

yum install -y ambari-server

wget http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.1.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.1.1.0-35.tar.gz

yum install -y ambari-agent ambari-server
silent installer with defaults
ambari-server setup -s 
install MPACK for Ambari
ambari-server install-mpack \

--mpack=./hdf-ambari-mpack-3.1.1.0-35.tar.gz \

--purge

--verbose
edit ambari agent to connect to proper server   hostname=$ambari_server
start ambari-agent everywhere
ambari-agent start
start the server
ambari-server start
go to ambari UI and install ZK and Kafka... On respective new server...
change zk port off of 2181 and kafka off of 9092 initally if they are using them. 
stop zk, and kafka.
add custom broker configs in ambari (as a default)
inter.broker.protocol.version=0.8.0

log.message.format.version=0.8.0
make a host group per broker your moving so you can set its id to the old id.
broker.id=THE OLD BROKER ID

Update the advertised listener to be the hostname you want!!! This is important if we are migrating DNS too!! 
add jmx port in kafka-env.sh
JMX
if [ "x$ISKAFKASERVER" == "xtrue" ]; then

export JMX_PORT=9999

fi
Special Configs for Kafka
num.network.threads=8

socket.send.buffer.bytes=8388608

log.segment.bytes=536870912

num.replica.fetchers=4
change permissions on zk data dirs, and kafka data dirs for ambari users to access 'kafka' , 'zookeeper'
if you dont install ambari first, zookeeper user and kafka users may not exist!
chmod -R 771 /tmp/zookeeper

chmod -R 771 /tmp/kafka-logs

chown -R zookeeper:kafka /tmp/zookeeper

chown -R kafka:kafka /tmp/kafka-logs
for zookeeper
update hostgroup in ambari for new zk, port and data dir as old zk settings

add custom zoo.cfg for old zkservers
shutdown the target ZK

start new takeover zk
update os limits for file

soft    nofile            10240
hard    nofile            10240

update network device domain
/etc/sysconfig/network-scripts/ifcfg-em1
for each kafka server create a ambari-host group and assign its proper broker id
broker.id=$N

stop old kafka

start takeover kafka
install Zabbix Agents
(ask aaron)