Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- my
- name
- is
- Wyatt
- Perl
- I'm
- a
- product
- engineer
- at
- Giant
- and
- you
- can
- find
- links
- to
- the
- slides
- and
- everything
- I'm
- going
- to
- talk
- about
- today
- at
- my
- website
- at
- JSC
- ich
- komm
- slash
- nerd
- summit
- so
- I
- want
- to
- start
- by
- telling
- a
- children's
- story
- actually
- think
- that
- children's
- stories
- are
- pretty
- useful
- I
- learned
- quite
- a
- bit
- whenever
- I'm
- reading
- them
- to
- my
- kids
- for
- example
- I
- was
- reading
- llama
- llama
- red
- pajama
- and
- that
- taught
- me
- how
- to
- do
- callbacks
- or
- Chicken
- Little
- actually
- taught
- me
- about
- cascading
- failure
- and
- so
- I
- took
- Chicken
- Little
- and
- I
- definitely
- took
- liberties
- with
- it
- so
- I
- have
- an
- adaptation
- of
- Chicken
- Little
- and
- I'm
- calling
- it
- chicken
- micro
- so
- let's
- let's
- begin
- one
- day
- a
- chicken
- micro
- was
- making
- a
- request
- to
- the
- corn
- server
- when
- boom
- the
- request
- timed
- out
- goodness
- gracious
- the
- server's
- must
- be
- down
- I
- must
- go
- and
- tell
- the
- system
- operator
- so
- chicken
- micro
- went
- along
- until
- she
- came
- to
- Boxey
- Loxy
- where
- are
- you
- going
- in
- such
- a
- hurry
- asked
- Foxy
- Loxy
- I'm
- going
- to
- tell
- the
- operator
- that
- the
- servers
- are
- down
- I
- made
- a
- request
- and
- never
- got
- a
- response
- oh
- dear
- I'll
- go
- with
- you
- so
- chicken
- micro
- and
- boxy
- Loxy
- went
- along
- until
- they
- came
- to
- robot
- Rob
- where
- are
- you
- going
- in
- such
- a
- hurry
- asked
- robot
- Rob
- we're
- going
- to
- tell
- the
- operator
- that
- the
- servers
- are
- down
- chicken
- micro
- made
- a
- request
- to
- never
- got
- a
- response
- I'll
- go
- with
- you
- so
- chicken
- micro
- Foxy
- Loxy
- and
- robot
- Rob
- made
- it
- to
- the
- system
- operator
- I
- made
- a
- request
- to
- the
- corn
- server
- and
- the
- request
- timed
- out
- the
- servers
- are
- down
- Oh
- chicken
- micro
- the
- servers
- are
- not
- down
- but
- the
- server
- the
- load
- balancer
- sent
- you
- to
- was
- under
- heavy
- load
- please
- try
- your
- request
- again
- so
- we
- can
- learn
- some
- lessons
- from
- chicken
- micro
- for
- example
- if
- your
- server
- is
- overloaded
- respond
- with
- a
- 503
- and
- ideally
- your
- load
- balancer
- or
- whatever
- will
- take
- you
- out
- of
- rotation
- but
- even
- if
- we
- did
- have
- a
- load
- balancer
- in
- the
- mix
- how
- would
- chicken
- micro
- know
- that
- the
- next
- request
- would
- even
- succeed
- because
- chicken
- micro
- doesn't
- know
- the
- health
- of
- the
- servers
- behind
- the
- load
- balancer
- and
- so
- this
- brings
- me
- to
- a
- point
- where
- as
- you're
- moving
- from
- micro
- services
- and
- you're
- trying
- to
- re
- are
- c't
- X
- or
- monolith
- sometimes
- you
- you
- leak
- over
- your
- monolithic
- architecture
- and
- this
- can
- bring
- about
- some
- of
- the
- same
- issues
- you
- had
- with
- your
- monolith
- but
- before
- I
- get
- too
- far
- ahead
- of
- myself
- I
- definitely
- need
- to
- clarify
- what
- I
- mean
- by
- micro
- service
- because
- there's
- lots
- of
- definitions
- people
- have
- varying
- thoughts
- on
- it
- but
- when
- I
- describe
- a
- micro
- service
- I'm
- talking
- about
- a
- small
- service
- and
- ideally
- it's
- the
- decomposed
- from
- a
- monolith
- and
- I
- think
- most
- importantly
- each
- service
- needs
- to
- be
- able
- to
- be
- isolated
- and
- independently
- deployable
- and
- lastly
- you
- should
- be
- stateless
- so
- that
- it's
- easy
- to
- scale
- out
- your
- services
- and
- from
- these
- characteristics
- we
- can
- get
- a
- lot
- of
- benefits
- because
- it's
- able
- to
- be
- independently
- deployed
- you
- are
- able
- to
- embrace
- failure
- so
- if
- your
- other
- dependencies
- are
- down
- it
- won't
- impact
- you
- and
- from
- this
- you
- can
- actually
- move
- very
- quickly
- you
- can
- deliver
- lots
- of
- business
- value
- if
- something
- doesn't
- work
- out
- you
- can
- just
- tear
- down
- that
- service
- and
- try
- again
- because
- everything
- is
- so
- tiny
- ideally
- and
- able
- to
- be
- independently
- deployable
- but
- there
- are
- definitely
- some
- anti
- patterns
- that
- we
- run
- into
- especially
- as
- we
- move
- from
- the
- monolith
- to
- the
- micro
- services
- so
- there's
- a
- very
- helpful
- book
- by
- Mark
- Richards
- called
- micro
- services
- anti
- patterns
- and
- pitfalls
- I
- encourage
- you
- to
- go
- check
- it
- out
- at
- storm
- o'reilly
- and
- it's
- actually
- free
- so
- even
- more
- reason
- but
- I
- want
- to
- focus
- on
- a
- couple
- anti
- patterns
- today
- so
- the
- balancer
- between
- micro-services
- and
- then
- how
- startup
- order
- can
- matter
- so
- if
- we
- look
- at
- the
- situation
- with
- chicken
- micro
- where
- the
- load
- balancer
- was
- between
- the
- micro-services
- chicken
- micro
- had
- a
- timeout
- and
- how
- would
- chicken
- micro
- know
- when
- to
- make
- the
- next
- request
- do
- you
- just
- retry
- immediately
- well
- how
- do
- you
- know
- that
- there
- are
- other
- servers
- that
- are
- even
- healthy
- behind
- that
- load
- balancer
- so
- you
- have
- that
- disconnect
- when
- you
- have
- a
- load
- balancer
- situated
- between
- your
- micro
- services
- and
- even
- if
- you
- had
- a
- 503
- response
- telling
- you
- like
- retry
- in
- a
- minute
- how
- would
- you
- know
- that
- that
- would
- succeed
- or
- maybe
- you
- could
- have
- just
- reached
- right
- immediately
- because
- there
- could
- be
- other
- servers
- that
- are
- healthy
- and
- lastly
- the
- startup
- order
- matters
- is
- another
- anti-pattern
- I
- think
- because
- everything
- should
- be
- independently
- deployable
- you
- really
- shouldn't
- be
- tied
- to
- assuming
- that
- your
- database
- is
- up
- for
- example
- so
- in
- this
- this
- code
- snippet
- we're
- trying
- to
- make
- a
- connection
- to
- my
- sequel
- and
- we're
- binding
- herself
- to
- a
- DB
- host
- through
- the
- environment
- variable
- another
- issue
- with
- this
- too
- is
- if
- that
- host
- changes
- you're
- gonna
- have
- to
- restart
- the
- entire
- process
- because
- it's
- an
- environment
- variable
- and
- then
- if
- the
- database
- is
- not
- up
- you're
- not
- really
- handling
- into
- that
- so
- you
- have
- to
- think
- about
- things
- differently
- you
- have
- to
- really
- are
- assume
- that
- not
- everything
- is
- going
- to
- be
- working
- whenever
- your
- application
- starts
- or
- when
- your
- micro
- service
- starts
- goodness
- gracious
- are
- there
- any
- patterns
- that
- can
- help
- me
- well
- fortunately
- there
- are
- so
- the
- one
- I
- want
- to
- focus
- on
- is
- the
- autopilot
- pattern
- it
- is
- not
- geared
- specifically
- for
- micro
- services
- but
- it
- actually
- lends
- itself
- well
- to
- helping
- with
- the
- issues
- that
- I
- highlighted
- so
- the
- main
- goals
- of
- the
- autopilot
- pattern
- or
- that
- you're
- able
- to
- build
- applications
- that
- can
- be
- deployed
- and
- scaled
- very
- easily
- easily
- your
- application
- and
- the
- workflow
- that
- you
- used
- to
- create
- that
- application
- should
- work
- the
- same
- on
- your
- development
- laptop
- as
- any
- it
- should
- work
- the
- same
- in
- any
- cloud
- or
- data
- center
- and
- you
- shouldn't
- be
- tied
- to
- a
- specific
- cloud
- or
- infrastructure
- and
- so
- we
- do
- have
- a
- website
- explaining
- more
- about
- the
- autopilot
- pattern
- just
- autopilot
- patterned
- at
- i/o
- I
- encourage
- you
- to
- check
- that
- out
- we
- also
- have
- quite
- a
- few
- example
- applications
- so
- we
- have
- a
- autopilot
- pattern
- github
- organization
- with
- examples
- of
- MongoDB
- my
- sequel
- influx
- and
- then
- we
- even
- have
- full
- blueprint
- applications
- like
- WordPress
- that
- are
- using
- this
- pattern
- and
- typically
- all
- of
- those
- examples
- that
- we
- have
- are
- using
- docker
- and
- containers
- and
- to
- do
- that
- we
- have
- a
- tool
- a
- free
- and
- open
- source
- tool
- called
- container
- pilot
- which
- I'll
- talk
- about
- in
- a
- little
- bit
- but
- because
- we
- don't
- have
- that
- load
- balancer
- between
- the
- services
- we
- do
- need
- a
- way
- to
- discover
- you
- know
- what
- services
- are
- where
- and
- if
- they're
- healthy
- or
- not
- and
- so
- to
- do
- that
- we
- use
- console
- and
- we
- also
- because
- we're
- using
- console
- don't
- have
- to
- expose
- any
- health
- checks
- outside
- of
- the
- container
- so
- this
- makes
- it
- very
- easy
- to
- to
- container
- eyes
- and
- container
- pilot
- eyes
- I
- guess
- the
- the
- databases
- so
- you
- can
- have
- your
- health
- check
- query
- your
- database
- and
- make
- sure
- everything
- is
- working
- correctly
- and
- report
- back
- to
- a
- console
- that
- you're
- you're
- healthy
- so
- to
- kind
- of
- explain
- more
- about
- what
- this
- actually
- means
- in
- practice
- I
- have
- a
- node
- example
- so
- I'm
- using
- --happy
- which
- is
- a
- wonderful
- module
- I
- hope
- you
- all
- check
- it
- out
- if
- you
- haven't
- already
- using
- piloted
- which
- I
- wrote
- specifically
- to
- work
- with
- this
- pattern
- and
- then
- rec
- for
- making
- requests
- so
- this
- is
- the
- the
- diagram
- of
- the
- application
- I'm
- going
- to
- show
- you
- you
- don't
- need
- to
- understand
- all
- of
- it
- but
- just
- the
- general
- idea
- is
- that
- we
- have
- IOT
- sensor
- data
- coming
- in
- and
- it's
- pushed
- to
- Nats
- which
- is
- a
- message
- queuing
- system
- and
- then
- we
- have
- three
- workers
- pulling
- that
- sensor
- data
- off
- humidity
- in
- motion
- and
- temperature
- and
- that
- pushes
- it
- to
- the
- serializer
- which
- is
- then
- saved
- to
- influx
- DB
- and
- on
- the
- front
- end
- we
- have
- the
- front
- end
- which
- exposes
- some
- grass
- so
- let
- me
- show
- you
- what
- this
- looks
- like
- so
- right
- now
- I
- have
- it
- running
- it's
- it
- it's
- all
- using
- docker
- so
- I
- started
- it
- with
- docker
- compose
- so
- this
- is
- a
- dashboard
- of
- consul
- and
- you
- can
- see
- all
- the
- services
- and
- they're
- green
- they're
- all
- healthy
- and
- this
- is
- the
- front
- end
- with
- as
- you
- can
- see
- random
- random
- data
- since
- I
- don't
- actually
- have
- the
- smartthings
- hub
- here
- but
- the
- nice
- thing
- is
- I
- can
- stop
- any
- of
- those
- services
- and
- everything
- that
- depended
- on
- it
- will
- be
- notified
- immediately
- so
- I
- can
- stop
- the
- serializer
- so
- that'll
- go
- down
- and
- then
- container
- pilot
- will
- realize
- that
- that
- service
- is
- not
- healthy
- and
- stop
- sending
- a
- heartbeat
- to
- console
- so
- if
- I
- go
- back
- to
- console
- refresh
- and
- see
- the
- serializer
- is
- no
- longer
- healthy
- and
- the
- workers
- actually
- are
- able
- to
- respond
- to
- that
- and
- not
- even
- try
- to
- make
- a
- request
- because
- they
- know
- there's
- nothing
- that
- they
- can
- make
- a
- request
- to
- so
- you
- don't
- have
- that
- failure
- and
- similarly
- the
- front-end
- knows
- that
- the
- serializer
- is
- down
- so
- it
- stops
- making
- requests
- back
- to
- that
- but
- if
- you
- bring
- it
- back
- up
- everything
- starts
- working
- again
- so
- everything
- well
- the
- the
- time
- series
- data
- is
- not
- the
- easiest
- to
- work
- with
- there
- so
- everything
- starts
- working
- again
- and
- responding
- to
- that
- change
- in
- your
- environment
- so
- you
- have
- a
- lot
- of
- self
- healing
- qualities
- built
- into
- this
- so
- let's
- go
- back
- so
- I
- wanted
- to
- highlight
- some
- of
- the
- code
- snippets
- that
- make
- this
- possible
- this
- is
- from
- the
- serializer
- and
- I'm
- using
- the
- piloted
- module
- which
- just
- keeps
- a
- local
- cache
- of
- whatever
- services
- the
- serializer
- cares
- about
- in
- this
- case
- so
- give
- me
- a
- healthy
- address
- for
- in
- Flex
- DB
- and
- then
- you
- can
- see
- if
- I
- don't
- have
- one
- I'm
- just
- going
- to
- use
- an
- in-memory
- database
- but
- if
- I
- do
- then
- I'll
- create
- a
- connection
- so
- that's
- how
- you
- achieve
- that
- isolation
- so
- being
- able
- to
- handle
- the
- case
- where
- you
- don't
- have
- any
- healthy
- dependencies
- and
- to
- respond
- to
- the
- change
- events
- so
- if
- the
- in
- Flex
- DB
- comes
- up
- again
- then
- we'll
- send
- a
- signal
- to
- the
- process
- and
- piloted
- will
- handle
- that
- and
- give
- you
- an
- event
- to
- reload
- and
- reconnect
- to
- the
- database
- or
- not
- at
- all
- and
- the
- way
- we
- achieve
- this
- is
- with
- a
- lot
- of
- help
- from
- container
- pilot
- which
- runs
- inside
- the
- container
- and
- it'll
- register
- each
- of
- the
- services
- for
- you
- with
- console
- it
- takes
- care
- of
- providing
- some
- lifecycle
- hooks
- and
- gives
- you
- out-of-the-box
- telemetry
- reporting
- with
- Prometheus
- and
- we
- started
- it
- as
- an
- init
- process
- because
- it's
- running
- as
- pit
- one
- so
- it'll
- actually
- take
- care
- of
- reaping
- any
- child
- processes
- or
- child
- zombie
- processes
- and
- do
- your
- health
- checks
- for
- you
- register
- your
- service
- as
- I
- said
- and
- and
- watch
- and
- notify
- you
- if
- anything
- changes
- and
- you
- can
- find
- the
- source
- code
- for
- it
- it's
- written
- and
- go
- but
- it's
- on
- our
- joint
- org
- under
- your
- container
- pilot
- and
- so
- you
- control
- container
- pilot
- with
- a
- very
- simple
- configuration
- file
- we're
- using
- JSON
- 5
- now
- it
- also
- supports
- JSON
- if
- that's
- what
- you're
- into
- but
- you
- configure
- it
- with
- what
- you
- care
- about
- so
- the
- watches
- this
- is
- again
- the
- serializer
- so
- I'm
- watching
- for
- any
- changes
- and
- in
- Flex
- DB
- so
- if
- more
- instances
- come
- up
- or
- they
- go
- down
- tell
- me
- about
- it
- and
- we
- can
- figure
- that
- through
- the
- job
- which
- will
- send
- a
- sick
- up
- signal
- to
- your
- node
- process
- it
- actually
- doesn't
- restart
- the
- process
- because
- we're
- handling
- that
- signal
- and
- we're
- refreshing
- our
- local
- cache
- of
- the
- healthy
- addresses
- and
- to
- configure
- your
- service
- that
- you
- want
- registered
- you
- just
- need
- to
- tell
- container
- pilot
- what
- port
- you're
- gonna
- listen
- on
- and
- give
- it
- a
- health
- check
- so
- in
- this
- case
- I
- just
- have
- a
- curl
- command
- that's
- just
- hitting
- localhost
- and
- that'll
- keep
- sending
- a
- heartbeat
- to
- console
- to
- let
- it
- know
- that
- you're
- still
- healthy
- or
- not
- and
- so
- because
- of
- this
- we're
- actually
- able
- to
- do
- some
- of
- the
- patterns
- that
- you
- would
- have
- would
- also
- see
- in
- micro-services
- land
- like
- circuit
- breakers
- in
- so
- a
- while
- back
- we
- actually
- built
- into
- happy
- this
- load
- feature
- so
- it'll
- respond
- with
- a
- 503
- if
- your
- process
- is
- under
- too
- much
- load
- which
- you
- define
- what
- that
- means
- can
- be
- memory
- or
- in
- this
- case
- the
- event
- loop
- delay
- getting
- too
- bogged
- down
- and
- so
- in
- this
- case
- if
- your
- process
- gets
- too
- bogged
- down
- you're
- not
- healthy
- anymore
- and
- then
- all
- of
- the
- other
- micro
- services
- that
- depended
- on
- you
- are
- notified
- and
- will
- stop
- calling
- you
- but
- we
- do
- need
- load
- balancing
- somewhere
- since
- you're
- likely
- to
- have
- more
- than
- one
- service
- that
- you're
- calling
- write
- more
- instances
- more
- than
- one
- instance
- of
- that
- service
- that
- you're
- calling
- so
- piloted
- out
- of
- the
- box
- we
- just
- do
- round-robin
- load
- balancing
- but
- you
- can
- decide
- however
- you
- want
- to
- load
- balance
- your
- requests
- on
- the
- client
- it
- could
- be
- random
- it
- could
- be
- round-robin
- it's
- really
- up
- to
- you
- so
- I
- want
- to
- also
- highlight
- how
- cool
- this
- is
- with
- scaling
- so
- I
- have
- Nats
- running
- here
- and
- I
- just
- have
- one
- instance
- of
- it
- and
- Nats
- operates
- by
- clustering
- itself
- together
- like
- you
- would
- typically
- have
- to
- configure
- what
- you
- know
- instances
- 2.22
- cluster
- but
- because
- we're
- using
- console
- to
- discover
- other
- Nats
- instances
- we
- can
- actually
- cluster
- automagically
- I'll
- say
- so
- I'll
- just
- use
- the
- scale
- command
- from
- dr.
- Campos
- and
- we
- can
- add
- more
- instances
- this
- of
- the
- serializer
- we
- can
- also
- add
- more
- instances
- of
- Nats
- and
- little
- docker
- will
- spin
- up
- these
- new
- instances
- for
- us
- and
- as
- they
- come
- online
- they'll
- be
- registered
- with
- console
- so
- if
- we
- go
- back
- to
- console
- we
- can
- start
- to
- see
- more
- instances
- coming
- up
- so
- now
- we
- have
- three
- instances
- of
- Nats
- running
- and
- Nats
- is
- also
- we
- have
- a
- container
- pilot
- solution
- for
- it
- so
- it'll
- auto
- cluster
- itself
- so
- you
- don't
- have
- to
- do
- any
- configuration
- it
- just
- knows
- from
- talking
- to
- console
- that
- there
- are
- other
- Nats
- instances
- once
- this
- refreshes
- so
- it
- clustered
- itself
- to
- the
- other
- other
- instances
- there
- and
- then
- again
- you
- could
- scale
- it
- up
- more
- or
- bring
- it
- down
- it's
- really
- whatever
- you
- want
- let's
- see
- so
- even
- though
- I
- was
- kind
- of
- disparaging
- load
- balancers
- between
- your
- services
- they
- still
- definitely
- have
- a
- purpose
- more
- at
- the
- edge
- more
- as
- a
- API
- gateway
- so
- in
- this
- case
- I
- was
- using
- traffic
- and
- Express
- gateway
- there
- are
- lots
- of
- solutions
- out
- there
- so
- definitely
- keep
- that
- in
- mind
- don't
- entirely
- expose
- your
- micro
- services
- directly
- to
- your
- customers
- so
- the
- last
- point
- I
- want
- to
- make
- is
- about
- telemetry
- reporting
- so
- we
- support
- Prometheus
- out
- of
- the
- box
- and
- you
- can
- report
- on
- any
- metric
- you
- care
- about
- again
- I
- like
- the
- event
- loop
- delay
- so
- in
- this
- example
- I'm
- just
- configuring
- container
- pilot
- to
- report
- on
- the
- event
- loop
- delay
- for
- me
- and
- then
- Prometheus
- will
- recognize
- that
- and
- and
- console
- and
- go
- out
- and
- start
- pulling
- in
- the
- data
- from
- every
- instance
- as
- it
- comes
- up
- so
- this
- is
- what
- you
- would
- see
- in
- Prometheus
- to
- see
- the
- event
- loop
- delay
- so
- if
- you
- want
- to
- learn
- more
- we
- have
- auto
- pilot
- pattern
- do
- I
- have
- links
- again
- if
- you
- go
- to
- a
- node
- summit
- off
- of
- my
- website
- and
- then
- there's
- some
- other
- links
- there
- for
- you
- as
- well
- so
- thank
- you
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement