API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Jan 26th, 2018
156
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 14.96 KB | None | 0 0
raw download clone embed print report
  1. [2018-01-26 18:03:05,270] INFO Starting task postgres-events-connector-0 (org.apache.kafka.connect.runtime.distributed.Distri
  2. butedHerder:851)
  3. [2018-01-26 18:03:05,270] INFO Creating task postgres-events-connector-0 (org.apache.kafka.connect.runtime.Worker:361)
  4. [2018-01-26 18:03:05,270] INFO ConnectorConfig values:
  5. connector.class = io.debezium.connector.postgresql.PostgresConnector
  6. key.converter = null
  7. name = postgres-events-connector
  8. tasks.max = 1
  9. transforms = null
  10. value.converter = null
  11. (org.apache.kafka.connect.runtime.ConnectorConfig:223)
  12. [2018-01-26 18:03:05,270] INFO EnrichedConnectorConfig values:
  13. connector.class = io.debezium.connector.postgresql.PostgresConnector
  14. key.converter = null
  15. name = postgres-events-connector
  16. tasks.max = 1
  17. transforms = null
  18. value.converter = null
  19. (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:223)
  20. [2018-01-26 18:03:05,270] INFO TaskConfig values:
  21. task.class = class io.debezium.connector.postgresql.PostgresConnectorTask
  22. (org.apache.kafka.connect.runtime.TaskConfig:223)
  23. [2018-01-26 18:03:05,270] INFO Instantiated task postgres-events-connector-0 with version 0.7.0 of type io.debezium.connector.postgresql.PostgresConnectorTask (org.apache.kafka.connect.runtime.Worker:376)
  24. [2018-01-26 18:03:05,271] INFO ProducerConfig values:
  25. acks = all
  26. batch.size = 16384
  27. bootstrap.servers = [billing-kafka001.drt.ewr.qa5.net:9092]
  28. buffer.memory = 33554432
  29. client.id =
  30. compression.type = none
  31. confluent.batch.expiry.ms = 30000
  32. connections.max.idle.ms = 540000
  33. enable.idempotence = false
  34. interceptor.classes = null
  35. key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
  36. linger.ms = 0
  37. max.block.ms = 9223372036854775807
  38. max.in.flight.requests.per.connection = 1
  39. max.request.size = 1048576
  40. metadata.max.age.ms = 300000
  41. metric.reporters = []
  42. metrics.num.samples = 2
  43. metrics.recording.level = INFO
  44. metrics.sample.window.ms = 30000
  45. partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
  46. receive.buffer.bytes = 32768
  47. reconnect.backoff.max.ms = 1000
  48. reconnect.backoff.ms = 50
  49. request.timeout.ms = 2147483647
  50. retries = 2147483647
  51. retry.backoff.ms = 100
  52. sasl.jaas.config = null
  53. sasl.kerberos.kinit.cmd = /usr/bin/kinit
  54. sasl.kerberos.min.time.before.relogin = 60000
  55. sasl.kerberos.service.name = null
  56. sasl.kerberos.ticket.renew.jitter = 0.05
  57. sasl.kerberos.ticket.renew.window.factor = 0.8
  58. sasl.mechanism = GSSAPI
  59. security.protocol = SSL
  60. send.buffer.bytes = 131072
  61. ssl.cipher.suites = null
  62. ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
  63. ssl.endpoint.identification.algorithm = null
  64. ssl.key.password = [hidden]
  65. ssl.keymanager.algorithm = SunX509
  66. ssl.keystore.location = /etc/pki/tls/private/drt.ewr.qa5.net.jks
  67. ssl.keystore.password = [hidden]
  68. ssl.keystore.type = JKS
  69. ssl.protocol = TLS
  70. ssl.provider = null
  71. ssl.secure.random.implementation = null
  72. ssl.trustmanager.algorithm = PKIX
  73. ssl.truststore.location = /etc/pki/tls/certs/trust.jks
  74. ssl.truststore.password = [hidden]
  75. ssl.truststore.type = JKS
  76. transaction.timeout.ms = 60000
  77. transactional.id = null
  78. value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
  79. (org.apache.kafka.clients.producer.ProducerConfig:223)
  80. [2018-01-26 18:03:05,277] INFO Kafka version : 1.0.0-cp1 (org.apache.kafka.common.utils.AppInfoParser:109)
  81. [2018-01-26 18:03:05,277] INFO Kafka commitId : bedb2a8697fecd0d (org.apache.kafka.common.utils.AppInfoParser:110)
  82. [2018-01-26 18:03:05,277] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:847)
  83. [2018-01-26 18:03:05,529] INFO user 'billing-service' connected to database 'staging-billing' on PostgreSQL 9.6.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11), 64-bit with roles:
  84. role 'replication' [superuser: false, replication: true, inherit: true, create role: false, create db: false, can log in: true]
  85. role 'sensu' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: true]
  86. role 'postgres' [superuser: true, replication: true, inherit: true, create role: true, create db: true, can log in: true]
  87. role 'pg_signal_backend' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: false]
  88. role 'collectd' [superuser: true, replication: false, inherit: true, create role: false, create db: false, can log in: true]
  89. role 'billing-service' [superuser: true, replication: false, inherit: true, create role: false, create db: false, can log in: true] (io.debezium.connector.postgresql.PostgresConnectorTask:81)
  90. [2018-01-26 18:03:05,529] INFO Found previous offset source_info[server='staging-billing', lsn=0/12072861, txId=1904, useconds=1516988965250212000, snapshot=false] (io.debezium.connector.postgresql.PostgresConnectorTask:95)
  91. [2018-01-26 18:03:05,529] INFO Taking a new snapshot as per configuration (io.debezium.connector.postgresql.PostgresConnectorTask:107)
  92. [2018-01-26 18:03:05,557] INFO WorkerSourceTask{id=postgres-events-connector-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:158)
  93. [2018-01-26 18:03:05,569] INFO Step 0: disabling autocommit (io.debezium.connector.postgresql.RecordsSnapshotProducer:134)
  94. [2018-01-26 18:03:05,570] INFO Step 1: starting transaction and refreshing the DB schemas for database 'staging-billing' and user 'billing-service' (io.debezium.connector.postgresql.RecordsSnapshotProducer:138)
  95. [2018-01-26 18:03:05,709] WARN The Kafka Connect schema name 'staging-billing.public.land_registry_price_paid_uk.Value' is not a valid Avro schema name, so replacing with 'staging_billing.public.land_registry_price_paid_uk.Value' (io.debezium.connector.postgresql.PostgresSchema:167)
  96. [2018-01-26 18:03:05,709] WARN The Kafka Connect schema name 'staging-billing.public.land_registry_price_paid_uk.Key' is not a valid Avro schema name, so replacing with 'staging_billing.public.land_registry_price_paid_uk.Key' (io.debezium.connector.postgresql.PostgresSchema:167)
  97. [2018-01-26 18:03:05,710] INFO Step 2: locking each of the database tables, waiting a maximum of '10.0' seconds for each lock (io.debezium.connector.postgresql.RecordsSnapshotProducer:149)
  98. [2018-01-26 18:03:05,740] INFO read xlogStart at '0/1207E1B0' from transaction '1907' (io.debezium.connector.postgresql.RecordsSnapshotProducer:169)
  99. [2018-01-26 18:03:05,740] INFO Step 3: reading and exporting the contents of each table (io.debezium.connector.postgresql.RecordsSnapshotProducer:175)
  100. [2018-01-26 18:03:05,740] INFO exporting data from table 'public.land_registry_price_paid_uk' (io.debezium.connector.postgresql.RecordsSnapshotProducer:183)
  101. [2018-01-26 18:03:05,744] WARN The Kafka Connect schema name 'staging-billing.public.land_registry_price_paid_uk.Envelope' is not a valid Avro schema name, so replacing with 'staging_billing.public.land_registry_price_paid_uk.Envelope' (io.debezium.connector.postgresql.PostgresSchema:167)
  102. [2018-01-26 18:03:05,749] INFO finished exporting '15' records for 'public.land_registry_price_paid_uk'; total duration '00:00:00.009' (io.debezium.connector.postgresql.RecordsSnapshotProducer:189)
  103. [2018-01-26 18:03:05,749] INFO Step 4: committing transaction '1907' (io.debezium.connector.postgresql.RecordsSnapshotProducer:198)
  104. [2018-01-26 18:03:05,755] INFO Step 5: sending the last snapshot record (io.debezium.connector.postgresql.RecordsSnapshotProducer:202)
  105. [2018-01-26 18:03:05,755] INFO Snapshot completed in '00:00:00.198' (io.debezium.connector.postgresql.RecordsSnapshotProducer:215)
  106. [2018-01-26 18:03:05,755] INFO Snapshot finished, continuing streaming changes from 0/1207E1B0 (io.debezium.connector.postgresql.RecordsSnapshotProducer:98)
  107. [2018-01-26 18:03:05,820] INFO REPLICA IDENTITY for 'public.land_registry_price_paid_uk' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns (io.debezium.connector.postgresql.PostgresSchema:106)
  108. [2018-01-26 18:04:05,277] INFO WorkerSourceTask{id=postgres-events-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:306)
  109. [2018-01-26 18:04:05,278] INFO WorkerSourceTask{id=postgres-events-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:323)
  110. [2018-01-26 18:04:05,281] INFO WorkerSourceTask{id=postgres-events-connector-0} Finished commitOffsets successfully in 3 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:405)
  111. [2018-01-26 18:04:31,930] INFO 127.0.0.1 - - [26/Jan/2018:18:04:31 +0000] "GET /connectors/postgres-events-connector/status HTTP/1.1" 200 187 3 (org.apache.kafka.connect.runtime.rest.RestServer:60)
  112. [2018-01-26 18:05:05,281] INFO WorkerSourceTask{id=postgres-events-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:306)
  113. [2018-01-26 18:05:05,282] INFO WorkerSourceTask{id=postgres-events-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:323)
  114. [2018-01-26 18:06:05,282] INFO WorkerSourceTask{id=postgres-events-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:306)
  115. [2018-01-26 18:06:05,282] INFO WorkerSourceTask{id=postgres-events-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:323)
  116.  
  117. {
  118. "name": "postgres-events-connector",
  119. "config": {
  120. "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
  121. "database.hostname": "billing-postgres001.drt.ewr.qa5.net",
  122. "plugin.name": "wal2json",
  123. "database.sslmode": "require",
  124. "database.port": "5432",
  125. "database.user": "billing-service",
  126. "database.password": "<passwd>",
  127. "database.dbname": "staging-billing",
  128. "database.server.name": "staging-billing",
  129. "slot.name": "subscriptionbilling001",
  130. "snapshot.mode": "always"
  131. }
  132. }
  133.  
  134. [root@subscription-billing001 schema-registry]# curl localhost:8083/connectors/postgres-events-connector/status | jq
  135. % Total % Received % Xferd Average Speed Time Time Time Current
  136. Dload Upload Total Spent Left Speed
  137. 100 187 100 187 0 0 26157 0 --:--:-- --:--:-- --:--:-- 26714
  138. {
  139. "name": "postgres-events-connector",
  140. "connector": {
  141. "state": "RUNNING",
  142. "worker_id": "10.192.80.251:8083"
  143. },
  144. "tasks": [
  145. {
  146. "state": "RUNNING",
  147. "id": 0,
  148. "worker_id": "10.192.80.251:8083"
  149. }
  150. ],
  151. "type": "source"
  152. }
  153.  
  154. CRUD ops on DB:
  155.  
  156. staging-billing=# COPY land_registry_price_paid_uk FROM '/var/lib/pgsql/pp-monthly-update-new-version-1-line.csv' with (format csv, encoding 'win1252', header false, null '', quote '"', force_null (postcode, saon, paon, street, locality, city, district, key_column));
  157. COPY 15
  158. staging-billing=# delete from land_registry_price_paid_uk; DELETE 15 staging-billing=# COPY land_registry_price_paid_uk FROM '/var/lib/pgsql/pp-monthly-update-new-version-1-line.csv' with (format csv, encoding 'win1252', header false, null '', quote '"', force_null (postcode, saon, paon, street, locality, city, district, key_column));
  159. COPY 15
  160. staging-billing=# delete from land_registry_price_paid_uk; DELETE 15 staging-billing=# COPY land_registry_price_paid_uk FROM '/var/lib/pgsql/pp-monthly-update-new-version-1-line.csv' with (format csv, encoding 'win1252', header false, null '', quote '"', force_null (postcode, saon, paon, street, locality, city, district, key_column));
  161. COPY 15
  162. staging-billing=#
  163.  
  164. pp-monthly-update-new-version-1-line.csv:
  165. "{5CF9903B-7D0C-7F13-E053-6C04A8C074B1}","397500","2003-05-16 00:00","BS22 7YA","D","N","F","COURT FARM BARN","","","BOURTON","WESTON-SUPER-MARE","NORTH SOMERSET","NORTH SOMERSET","A","A","71001"
  166. "{5CF9903B-8F66-7F13-E053-6C04A8C074B1}","235000","2003-09-10 00:00","PE9 3AE","D","N","F","HOLLY HOUSE","","BARNACK ROAD","BAINTON","STAMFORD","CITY OF PETERBOROUGH","CITY OF PETERBOROUGH","A","A","72001"
  167. "{5CF9903B-9AC1-7F13-E053-6C04A8C074B1}","150000","2003-11-28 00:00","WA16 7RH","F","N","F","THE OAKS","FLAT 14","FAULKNERS LANE","MOBBERLEY","KNUTSFORD","CHESHIRE EAST","CHESHIRE EAST","A","A","7300"
  168. "{5CF9903B-9F97-7F13-E053-6C04A8C074B1}","95000","2003-03-05 00:00","PL13 1AT","T","N","F","TYRINA","","LOWER CHAPEL STREET","","LOOE","CORNWALL","CORNWALL","A","A","74002"
  169. "{5CF9903B-A762-7F13-E053-6C04A8C074B1}","112950","2003-09-19 00:00","CF5 1HF","T","N","F","25","","CLIVE ROAD","CANTON","CARDIFF","CARDIFF","CARDIFF","A","A","75004"
  170. "{5CF9903B-D640-7F13-E053-6C04A8C074B1}","155000","2003-08-27 00:00","CM0 7LY","D","N","F","RAMSEY WICK","","MAIN ROAD","ST LAWRENCE","SOUTHMINSTER","MALDON","ESSEX","A","A","7600"
  171. "{5CF9903B-EE62-7F13-E053-6C04A8C074B1}","147000","2003-11-17 00:00","WD5 0BF","F","N","L","14","","HARLECH ROAD","","ABBOTS LANGLEY","THREE RIVERS","HERTFORDSHIRE","A","A","7700"
  172. "{5CF9903C-0D03-7F13-E053-6C04A8C074B1}","198000","2003-06-26 00:00","CT6 6EE","D","N","F","17","","HERNEVILLE GARDENS","","HERNE BAY","CANTERBURY","KENT","A","A","7800"
  173. "{5CF9903C-2270-7F13-E053-6C04A8C074B1}","346000","2003-10-01 00:00","LE2 2PD","D","N","F","35","","MEADOWCOURT ROAD","","LEICESTER","OADBY AND WIGSTON","LEICESTERSHIRE","A","A","7900"
  174. "{5CF9903C-3A79-7F13-E053-6C04A8C074B1}","163000","2003-10-13 00:00","TD15 1QY","T","N","F","4A","","MAIN STREET","SPITTAL","BERWICK-UPON-TWEED","NORTHUMBERLAND","NORTHUMBERLAND","A","A","8000"
  175. "{5CF9903B-7D0C-7F13-E053-6C04A8C074B1}","397500","2003-05-16 00:00","BS22 7YA","D","N","F","COURT FARM BARN","","","BOURTON","WESTON-SUPER-MARE","NORTH SOMERSET","NORTH SOMERSET","A","A","7100"
  176. "{5CF9903B-8F66-7F13-E053-6C04A8C074B1}","235000","2003-09-10 00:00","PE9 3AE","D","N","F","HOLLY HOUSE","","BARNACK ROAD","BAINTON","STAMFORD","CITY OF PETERBOROUGH","CITY OF PETERBOROUGH","A","A","720012"
  177. "{5CF9903B-9AC1-7F13-E053-6C04A8C074B1}","150000","2003-11-28 00:00","WA16 7RH","F","N","F","THE OAKS","FLAT 14","FAULKNERS LANE","MOBBERLEY","KNUTSFORD","CHESHIRE EAST","CHESHIRE EAST","A","A","73000"
  178. "{5CF9903B-9F97-7F13-E053-6C04A8C074B1}","95000","2003-03-05 00:00","PL13 1AT","T","N","F","TYRINA","","LOWER CHAPEL STREET","","LOOE","CORNWALL","CORNWALL","A","A","74003"
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!