180519 15:01:06 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
…SST in progress, setting sleep higher………….. ERROR!
[root@padasantri2 ~]# cat /var/lib/mysql/padasantri2.err
180519 15:01:06 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
180519 15:01:06 mysqld_safe WSREP: Running position recovery with –log_error=’/var/lib/mysql/wsrep_recovery.c9wKQp’ –pid-file=’/var/lib/mysql/padasantri2-recover.pid’
180519 15:01:06 [Note] /usr/sbin/mysqld (mysqld 5.5.60-MariaDB-wsrep) starting as process 7998 …
180519 15:01:08 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
180519 15:01:08 [Note] WSREP: wsrep_start_position var submitted: ‘00000000-0000-0000-0000-000000000000:-1’
180519 15:01:08 [Note] /usr/sbin/mysqld (mysqld 5.5.60-MariaDB-wsrep) starting as process 8043 …
180519 15:01:08 [Note] WSREP: Read nil XID from storage engines, skipping position init
180519 15:01:08 [Note] WSREP: wsrep_load(): loading provider library ‘/usr/lib64/galera/libgalera_smm.so’
180519 15:01:08 [Note] WSREP: wsrep_load(): Galera 25.3.23(r3789) by Codership Oy loaded successfully.
180519 15:01:08 [Note] WSREP: CRC-32C: using hardware acceleration.
180519 15:01:08 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
180519 15:01:08 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.111.99; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc
180519 15:01:08 [Note] WSREP: GCache history reset: ffc830b7-5b29-11e8-babc-ce6c6b02ea18:0 -> 00000000-0000-0000-0000-000000000000:-1
180519 15:01:08 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
180519 15:01:08 [Note] WSREP: wsrep_sst_grab()
180519 15:01:08 [Note] WSREP: Start replication
180519 15:01:08 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
180519 15:01:08 [Note] WSREP: protonet asio version 0
180519 15:01:08 [Note] WSREP: Using CRC-32C for message checksums.
180519 15:01:08 [Note] WSREP: backend: asio
180519 15:01:08 [Note] WSREP: gcomm thread scheduling priority set to other:0
180519 15:01:08 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
180519 15:01:08 [Note] WSREP: restore pc from disk failed
180519 15:01:08 [Note] WSREP: GMCast version 0
180519 15:01:08 [Note] WSREP: (c9d9b8a9, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567
180519 15:01:08 [Note] WSREP: (c9d9b8a9, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1
180519 15:01:08 [Note] WSREP: EVS version 0
180519 15:01:08 [Note] WSREP: gcomm: connecting to group ‘clusterpadasantri’, peer ‘192.168.111.98:,192.168.111.99:’
180519 15:01:08 [Note] WSREP: (c9d9b8a9, ‘tcp://0.0.0.0:4567’) connection established to c9d9b8a9 tcp://192.168.111.99:4567
180519 15:01:08 [Warning] WSREP: (c9d9b8a9, ‘tcp://0.0.0.0:4567’) address ‘tcp://192.168.111.99:4567’ points to own listening address, blacklisting
180519 15:01:08 [Note] WSREP: (c9d9b8a9, ‘tcp://0.0.0.0:4567’) connection established to 3fceffdc tcp://192.168.111.98:4567
180519 15:01:08 [Note] WSREP: (c9d9b8a9, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers:
180519 15:01:09 [Note] WSREP: declaring 3fceffdc at tcp://192.168.111.98:4567 stable
180519 15:01:09 [Note] WSREP: Node 3fceffdc state prim
180519 15:01:09 [Note] WSREP: view(view_id(PRIM,3fceffdc,8) memb {
3fceffdc,0
c9d9b8a9,0
} joined {
} left {
} partitioned {
})
180519 15:01:09 [Note] WSREP: save pc into disk
180519 15:01:09 [Note] WSREP: gcomm: connected
180519 15:01:09 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
180519 15:01:09 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
180519 15:01:09 [Note] WSREP: Opened channel ‘clusterpadasantri’
180519 15:01:09 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
180519 15:01:09 [Note] WSREP: Waiting for SST to complete.
180519 15:01:09 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
180519 15:01:09 [Note] WSREP: STATE EXCHANGE: sent state msg: 033bdb87-5b3d-11e8-ac58-33bdb90508ca
180519 15:01:09 [Note] WSREP: STATE EXCHANGE: got state msg: 033bdb87-5b3d-11e8-ac58-33bdb90508ca from 0 (padasantri1)
180519 15:01:09 [Note] WSREP: STATE EXCHANGE: got state msg: 033bdb87-5b3d-11e8-ac58-33bdb90508ca from 1 (padasantri2)
180519 15:01:09 [Note] WSREP: Quorum results:
version = 4,
component = PRIMARY,
conf_id = 7,
members = 1/2 (joined/total),
act_id = 0,
last_appl. = -1,
protocols = 0/8/3 (gcs/repl/appl),
group UUID = ffc830b7-5b29-11e8-babc-ce6c6b02ea18
180519 15:01:09 [Note] WSREP: Flow-control interval: [23, 23]
180519 15:01:09 [Note] WSREP: Trying to continue unpaused monitor
180519 15:01:09 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
180519 15:01:09 [Note] WSREP: State transfer required:
Group state: ffc830b7-5b29-11e8-babc-ce6c6b02ea18:0
Local state: 00000000-0000-0000-0000-000000000000:-1
180519 15:01:09 [Note] WSREP: New cluster view: global state: ffc830b7-5b29-11e8-babc-ce6c6b02ea18:0, view# 8: Primary, number of nodes: 2, my index: 1, protocol version 3
180519 15:01:09 [Warning] WSREP: Gap in state sequence. Need state transfer.
180519 15:01:09 [Note] WSREP: Running: ‘wsrep_sst_rsync –role ‘joiner’ –address ‘192.168.111.99’ –datadir ‘/var/lib/mysql/’ –defaults-file ‘/etc/my.cnf’ –parent ‘8043”
180519 15:01:12 [Note] WSREP: Prepared SST request: rsync|192.168.111.99:4444/rsync_sst
180519 15:01:12 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
180519 15:01:12 [Note] WSREP: REPL Protocols: 8 (3, 2)
180519 15:01:12 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
180519 15:01:12 [Note] WSREP: Service thread queue flushed.
180519 15:01:12 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (ffc830b7-5b29-11e8-babc-ce6c6b02ea18): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
180519 15:01:12 [Note] WSREP: Member 1.0 (padasantri2) requested state transfer from ‘*any*’. Selected 0.0 (padasantri1)(SYNCED) as donor.
180519 15:01:12 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
180519 15:01:12 [Note] WSREP: Requesting state transfer: success, donor: 0
180519 15:01:12 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> ffc830b7-5b29-11e8-babc-ce6c6b02ea18:0
180519 15:01:12 [Note] WSREP: (c9d9b8a9, ‘tcp://0.0.0.0:4567’) connection to peer c9d9b8a9 with addr tcp://192.168.111.99:4567 timed out, no messages seen in PT3S
180519 15:01:12 [Note] WSREP: (c9d9b8a9, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
180519 15:03:19 [Warning] WSREP: 0.0 (padasantri1): State transfer to 1.0 (padasantri2) failed: -255 (Unknown error 255)
180519 15:03:19 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():737: Will never receive state. Need to abort.
180519 15:03:19 [Note] WSREP: gcomm: terminating thread
180519 15:03:19 [Note] WSREP: gcomm: joining thread
180519 15:03:19 [Note] WSREP: gcomm: closing backend
180519 15:03:23 [Note] WSREP: (c9d9b8a9, ‘tcp://0.0.0.0:4567’) connection to peer 3fceffdc with addr tcp://192.168.111.98:4567 timed out, no messages seen in PT3S
180519 15:03:23 [Note] WSREP: (c9d9b8a9, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://192.168.111.98:4567
180519 15:03:24 [Note] WSREP: (c9d9b8a9, ‘tcp://0.0.0.0:4567’) reconnecting to 3fceffdc (tcp://192.168.111.98:4567), attempt 0
180519 15:03:25 [Note] WSREP: evs::proto(c9d9b8a9, LEAVING, view_id(REG,3fceffdc,8)) suspecting node: 3fceffdc
180519 15:03:25 [Note] WSREP: evs::proto(c9d9b8a9, LEAVING, view_id(REG,3fceffdc,8)) suspected node without join message, declaring inactive
180519 15:03:25 [Note] WSREP: view(view_id(NON_PRIM,3fceffdc,8) memb {
c9d9b8a9,0
} joined {
} left {
} partitioned {
3fceffdc,0
})
180519 15:03:25 [Note] WSREP: view((empty))
180519 15:03:25 [Note] WSREP: gcomm: closed
180519 15:03:25 [Note] WSREP: /usr/sbin/mysqld: Terminated.
180519 15:03:25 mysqld_safe mysqld from pid file /var/lib/mysql/padasantri2.pid ended
WSREP_SST: [ERROR] Parent mysqld process (PID:8043) terminated unexpectedly. (20180519 15:03:25.692)
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 8088 (20180519 15:03:25.693)
WSREP_SST: [INFO] Joiner cleanup done. (20180519 15:03:26.196)

how to solve it?

180519 12:59:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
180519 12:59:08 mysqld_safe WSREP: Running position recovery with –log_error=’/var/lib/mysql/wsrep_recovery.S81kcU’ –pid-file=’/var/lib/mysql/padasantri2-recover.pid’
180519 12:59:08 [Note] /usr/sbin/mysqld (mysqld 5.5.60-MariaDB-wsrep) starting as process 12812 …
180519 12:59:10 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
180519 12:59:10 [Note] WSREP: wsrep_start_position var submitted: ‘00000000-0000-0000-0000-000000000000:-1’
180519 12:59:10 [Note] /usr/sbin/mysqld (mysqld 5.5.60-MariaDB-wsrep) starting as process 12857 …
180519 12:59:10 [Note] WSREP: Read nil XID from storage engines, skipping position init
180519 12:59:10 [Note] WSREP: wsrep_load(): loading provider library ‘/usr/lib64/galera/libgalera_smm.so’
180519 12:59:10 [Note] WSREP: wsrep_load(): Galera 25.3.23(r3789) by Codership Oy loaded successfully.
180519 12:59:10 [Note] WSREP: CRC-32C: using hardware acceleration.
180519 12:59:10 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
180519 12:59:11 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.111.99; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc
180519 12:59:11 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 00000000-0000-0000-0000-000000000000:-1
180519 12:59:11 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
180519 12:59:11 [Note] WSREP: wsrep_sst_grab()
180519 12:59:11 [Note] WSREP: Start replication
180519 12:59:11 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
180519 12:59:11 [Note] WSREP: protonet asio version 0
180519 12:59:11 [Note] WSREP: Using CRC-32C for message checksums.
180519 12:59:11 [Note] WSREP: backend: asio
180519 12:59:11 [Note] WSREP: gcomm thread scheduling priority set to other:0
180519 12:59:11 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
180519 12:59:11 [Note] WSREP: restore pc from disk failed
180519 12:59:11 [Note] WSREP: GMCast version 0
180519 12:59:11 [Note] WSREP: (c0241b7c, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567
180519 12:59:11 [Note] WSREP: (c0241b7c, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1
180519 12:59:11 [Note] WSREP: EVS version 0
180519 12:59:11 [Note] WSREP: gcomm: connecting to group ‘clusterpadasantri’, peer ‘192.168.111.98:,192.168.111.99:’
180519 12:59:11 [Note] WSREP: (c0241b7c, ‘tcp://0.0.0.0:4567’) connection established to c0241b7c tcp://192.168.111.99:4567
180519 12:59:11 [Warning] WSREP: (c0241b7c, ‘tcp://0.0.0.0:4567’) address ‘tcp://192.168.111.99:4567’ points to own listening address, blacklisting
180519 12:59:14 [Note] WSREP: (c0241b7c, ‘tcp://0.0.0.0:4567’) connection to peer c0241b7c with addr tcp://192.168.111.99:4567 timed out, no messages seen in PT3S
180519 12:59:14 [Warning] WSREP: no nodes coming from prim view, prim not possible
180519 12:59:14 [Note] WSREP: view(view_id(NON_PRIM,c0241b7c,1) memb {
c0241b7c,0
} joined {
} left {
} partitioned {
})
180519 12:59:14 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50074S), skipping check
180519 12:59:44 [Note] WSREP: view((empty))
180519 12:59:44 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
at gcomm/src/pc.cpp:connect():158
180519 12:59:44 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
180519 12:59:44 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1458: Failed to open channel ‘clusterpadasantri’ at ‘gcomm://192.168.111.98,192.168.111.99’: -110 (Connection timed out)
180519 12:59:44 [ERROR] WSREP: gcs connect failed: Connection timed out
180519 12:59:44 [ERROR] WSREP: wsrep::connect(gcomm://192.168.111.98,192.168.111.99) failed: 7
180519 12:59:44 [ERROR] Aborting

180519 12:59:44 [Note] WSREP: Service disconnected.
180519 12:59:45 [Note] WSREP: Some threads may fail to exit.
180519 12:59:45 [Note] /usr/sbin/mysqld: Shutdown complete

180519 12:59:45 mysqld_safe mysqld from pid file /var/lib/mysql/padasantri2.pid ended

how solve this error?

– setenforce 0

I am getting same “Connection timed out” errors, while my configurations are below

####################
[server]
#general_log_file=/var/log/mysql/mysqld.log
#general_log=1
log_warning=2
log-error=/var/log/mysql/error.log
#log-bin=/var/log/mysql-bin.log
#log-slow-queries=/var/log/mysql/mysql-slow-queries.log
####################
[mysqld]
datadir=/var/lib/mysql
####################
[mariadb]
wsrep_cluster_address=gcomm://hostname1,hostname2
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_node_address = hostname1
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
wsrep_cluster_name=corp-zabbix-server
wsrep_debug=on
wsrep_sst_auth=mariadb-user:cerner
wsrep_sst_method=mysqldump
wsrep_on=ON
####################

but my side everything looks fine. its in still first node, I haven’t started in second node.

i.e if I restart all the servers then how nodes will decide that which will be bootstrep node.

140618 10:11:50 [ERROR] WSREP: Permission denied
140618 10:11:50 [ERROR] WSREP: failed to open gcomm backend connection: 13: error while trying to listen ‘tcp://0.0.0.0:4567?socket.non_blocking=1’, asio error ‘Permission denied’: 13 (Permission denied)
at gcomm/src/asio_tcp.cpp:listen():814
140618 10:11:50 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():202: Failed to open backend connection: -13 (Permission denied)
140618 10:11:50 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1291: Failed to open channel ‘cluster1’ at ‘gcomm://XXX.XXX.XXX.9’: -13 (Permission denied)
140618 10:11:50 [ERROR] WSREP: gcs connect failed: Permission denied
140618 10:11:50 [ERROR] WSREP: wsrep::connect() failed: 7
140618 10:11:50 [ERROR] Aborting

There is no firewall on both servers. I would be grateful for any ideas.