2021-06-01 12:20:20 +00:00
# Copyright (C) 2012-2021 Zammad Foundation, http://zammad-foundation.org/
2021-03-10 16:25:26 +00:00
VCR_IGNORE_MATCHING_HOSTS = %w[ elasticsearch selenium zammad.org zammad.com znuny.com google.com login.microsoftonline.com github.com ] . freeze
2021-05-12 11:37:44 +00:00
VCR_IGNORE_MATCHING_REGEXPS = [ %r{ ^192 \ .168 \ . \ d+ \ . \ d+$ } ] . freeze
2020-09-10 11:20:01 +00:00
2018-09-04 09:49:44 +00:00
VCR . configure do | config |
config . cassette_library_dir = 'test/data/vcr_cassettes'
config . hook_into :webmock
2018-12-03 14:10:36 +00:00
config . allow_http_connections_when_no_cassette = false
config . ignore_localhost = true
config . ignore_request do | request |
uri = URI ( request . uri )
2020-09-10 11:20:01 +00:00
next true if VCR_IGNORE_MATCHING_HOSTS . any? { | elem | uri . host . include? elem }
next true if VCR_IGNORE_MATCHING_REGEXPS . any? { | elem | uri . host . match? elem }
2018-12-03 14:10:36 +00:00
end
2018-12-13 09:06:44 +00:00
config . register_request_matcher ( :oauth_headers ) do | r1 , r2 |
2021-05-12 11:37:44 +00:00
without_onetime_oauth_params = - > ( params ) { params . gsub ( %r{ oauth_(nonce|signature|timestamp)="[^"]+", } , '' ) }
2018-12-13 09:06:44 +00:00
r1 . headers . except ( 'Authorization' ) == r2 . headers . except ( 'Authorization' ) &&
r1 . headers [ 'Authorization' ] & . map ( & without_onetime_oauth_params ) ==
r2 . headers [ 'Authorization' ] & . map ( & without_onetime_oauth_params )
end
2018-09-04 09:49:44 +00:00
end
Refactoring: Automatic RSpec VCR cassette name helper
This commit was prepared to support upcoming additions to the test suite
(specifically, better coverage for existing Twitter functionality).
These upcoming changes will depend heavily on VCR.[0]
(VCR is a Ruby gem that makes it easier to write and run tests
that call out to external services over HTTP
by "recording" HTTP transactions to a YAML file
and "replaying" them later.)
VCR is widely-used (4600 GitHub stars), but its API is a little clumsy--
You have to manually specify the name of a "cassette" file every time:
it 'does something' do
VCR.use_cassette('path/to/cassette') do
...
end
end
This commit adds an RSpec metadata config option
as a shorthand for the syntax above:
it 'does something', :use_vcr do
...
end
This config option automatically generates a cassette filename
based on the description of the example it's applied to.
=== Analysis of alternative approaches
Ideally, these auto-generated cassette filenames should be unique:
if filenames collide, multiple examples will share the same cassette.
A first attempt generated names based on `example.full_description`,
but that led to errors:
Errno::ENAMETOOLONG:
File name too long @ rb_sysopen - /opt/zammad/test/data/vcr_cassettes/models/ticket/article/ticket_article_callbacks_observers_async_transactions_-_auto-setting_of_outgoing_twitter_article_attributes_via_bg_jobs_when_the_original_channel_specified_in_ticket_preferences_was_deleted_but_a_new_one_with_the_same_screen_name_exists_sets_appropriate_status_attributes_on_the_new_channel.yml
Another idea was to use MD5 digests of the above,
but in fact both of these approaches share another problem:
even minor changes to the description could break tests
(unless the committer remembers to rename the cassette file to match):
an altered description means VCR will record a new cassette file
instead of replaying from the original.
(Normally, this would only slow down the test instead of breaking it,
but sometimes we modify tests and cassettes after recording them
to hide sensitive data like API keys or login credentials.)
The approach taken by this commit was to use partial descriptions,
combining the parent `describe`/`context` label with the `it` label.
This does not guarantee uniqueness--
even in the present refactoring, it produced a filename collision--
but it's a good middle ground.
[0]: https://relishapp.com/vcr/vcr/docs
2019-11-12 08:17:21 +00:00
Maintenance: Enhance VCR helper with better failure messages
1ebddff95 added an RSpec metadata flag to simplify the use of VCR:
describe 'super cool feature', :use_vcr do
it 'totally works' { ... }
end
Under the hood, this option automatically generates filenames
for the VCR cassettes it uses on each example it's applied to.
These filenames are based on the example descriptions;
for the sample block above, the resulting filename would be
super_cool_feature_totally_works.yml
This introduces the risk of test regressions
that could be extremely confusing and hard to debug,
simply because someone changed the example description.
This commit injects custom error messages into RSpec
to elucidate this problem and avoid needless debugging.
=== Design challenges
This error message injection uses Ruby's Module#prepend
to monkey-patch methods defined in RSpec,
meaning that the changes are coupled to RSpec's implementation.
In short, if the implementation changes,
this custom error messaging could break.
Specifically, there are two different modes of failure in RSpec,
and the custom error was thus injected in two corresponding places:
* `.notify_failure` for normal test failure
(i.e., when an expectation is not met); and
* `.handle_matcher` for exceptions raised during a test.
2019-11-18 15:27:21 +00:00
module RSpec
VCR_ADVISORY = << ~ MSG . freeze
If this test is failing unexpectedly , the VCR cassette may be to blame .
This can happen when changing ` describe ` / ` context ` labels on some specs ;
see commit message 1 ebddff95 for details .
Check ` git status ` to see if a new VCR cassette has been generated .
If so , rename the old cassette to replace the new one and try again .
MSG
module Support
module VCRHelper
def self . inject_advisory ( example )
# block argument is an #<RSpec::Expectations::ExpectationNotMetError>
2020-02-19 11:21:52 +00:00
define_method ( :notify_failure ) do | e , options = { } |
super ( e . exception ( VCR_ADVISORY + e . message ) , options )
Maintenance: Enhance VCR helper with better failure messages
1ebddff95 added an RSpec metadata flag to simplify the use of VCR:
describe 'super cool feature', :use_vcr do
it 'totally works' { ... }
end
Under the hood, this option automatically generates filenames
for the VCR cassettes it uses on each example it's applied to.
These filenames are based on the example descriptions;
for the sample block above, the resulting filename would be
super_cool_feature_totally_works.yml
This introduces the risk of test regressions
that could be extremely confusing and hard to debug,
simply because someone changed the example description.
This commit injects custom error messages into RSpec
to elucidate this problem and avoid needless debugging.
=== Design challenges
This error message injection uses Ruby's Module#prepend
to monkey-patch methods defined in RSpec,
meaning that the changes are coupled to RSpec's implementation.
In short, if the implementation changes,
this custom error messaging could break.
Specifically, there are two different modes of failure in RSpec,
and the custom error was thus injected in two corresponding places:
* `.notify_failure` for normal test failure
(i.e., when an expectation is not met); and
* `.handle_matcher` for exceptions raised during a test.
2019-11-18 15:27:21 +00:00
end
example . run
ensure
remove_method ( :notify_failure )
end
end
singleton_class . send ( :prepend , VCRHelper )
end
module Expectations
module VCRHelper
def self . inject_advisory ( example )
define_method ( :handle_matcher ) do | * args |
super ( * args )
rescue = > e
raise e . exception ( VCR_ADVISORY + e . message )
end
example . run
ensure
remove_method ( :handle_matcher )
end
end
PositiveExpectationHandler . singleton_class . send ( :prepend , VCRHelper )
NegativeExpectationHandler . singleton_class . send ( :prepend , VCRHelper )
end
end
RSpec . configure do | config |
Testing: Prevent synchronization issues with VCR
=== Background: What is VCR?
VCR caches HTTP network traffic for tests
that call out to third-party services (e.g., the Twitter REST API).
It "records" to a YAML file the first time a test is run,
and we save those files to the repo so they can be "replayed" later.
=== The problem & the naive approach
The ways Zammad uses third-party services can be time-sensitive.
For instance, Zammad assumes that when it fetches new tweets,
it's getting an up-to-date response from the Twitter API,
and ignores all tweets 15+ days old.
Of course, if we're using VCR, that's not the case--
in the test, Zammad receives a listing of tweets that's frozen in time.
Thus, 15 days after the test was first run,
it won't import any tweets at all.
Initially, the fix was to add the following to the spec file:
before { travel_to '2020-02-06 13:37 +0100' }
This hardcodes the date when the tests were first written,
and it has significant drawbacks:
1. It leads to hard-to-diagnose bugs when adding new test cases.
Your new test case will not have a VCR cassette associated with it;
the first time you run it, it will travel way back in time
and try to interact with the Twitter REST API.
The Twitter REST API requires an OAuth signature
based on a current timestamp,
and will reject the one generated during the test:
Twitter::Error::Unauthorized: Timestamp out of bounds.
Because RSpec, VCR, and Channel#fetch all suppress error output in
different ways, this error message takes some work to find.
2. If you succeed in adding new test cases,
they won't match the timestamp above.
=== A better solution
This commit adds a new option to the :use_vcr metadata tag:
it 'does something', use_vcr: :time_sensitive
Examples tagged in this way will automatically travel back to exactly
when a VCR cassette was recorded prior to using it.
=== Discussion
You may notice that the VCR auto-record logic was moved
from a helper module to a simple block.
The helper module was intended to improve organization and clean up the
config code, but due to context/binding issues, it was not possible to
call `travel_to` from inside the helper module.
2020-03-06 10:44:59 +00:00
config . around ( :each , use_vcr : true ) do | example |
vcr_options = Array ( example . metadata [ :use_vcr ] )
2021-07-01 11:23:52 +00:00
spec_path = Pathname . new ( example . file_path ) . realpath
cassette_path = spec_path . relative_path_from ( Rails . root . join ( 'spec' ) ) . sub ( %r{ _spec \ .rb$ } , '' )
cassette_name = " #{ example . metadata [ :example_group ] [ :full_description ] } / #{ example . description } " . gsub ( %r{ [^0-9A-Za-z \ -]+ } , '_' ) . downcase
# handle file name limit of 255 chars
if cassette_name . length > 253
hexdigest_cassette_name = Digest :: SHA256 . hexdigest ( cassette_name )
shortened_casset_name = " #{ cassette_name . first ( 30 ) } - #{ cassette_name . last ( 30 ) } - #{ hexdigest_cassette_name } "
Rails . logger . info " Detected too long VCR filename ' #{ cassette_name } ' ( #{ cassette_name . length } ) and therefore converted it to ' #{ shortened_casset_name } ' "
cassette_name = shortened_casset_name
end
Testing: Prevent synchronization issues with VCR
=== Background: What is VCR?
VCR caches HTTP network traffic for tests
that call out to third-party services (e.g., the Twitter REST API).
It "records" to a YAML file the first time a test is run,
and we save those files to the repo so they can be "replayed" later.
=== The problem & the naive approach
The ways Zammad uses third-party services can be time-sensitive.
For instance, Zammad assumes that when it fetches new tweets,
it's getting an up-to-date response from the Twitter API,
and ignores all tweets 15+ days old.
Of course, if we're using VCR, that's not the case--
in the test, Zammad receives a listing of tweets that's frozen in time.
Thus, 15 days after the test was first run,
it won't import any tweets at all.
Initially, the fix was to add the following to the spec file:
before { travel_to '2020-02-06 13:37 +0100' }
This hardcodes the date when the tests were first written,
and it has significant drawbacks:
1. It leads to hard-to-diagnose bugs when adding new test cases.
Your new test case will not have a VCR cassette associated with it;
the first time you run it, it will travel way back in time
and try to interact with the Twitter REST API.
The Twitter REST API requires an OAuth signature
based on a current timestamp,
and will reject the one generated during the test:
Twitter::Error::Unauthorized: Timestamp out of bounds.
Because RSpec, VCR, and Channel#fetch all suppress error output in
different ways, this error message takes some work to find.
2. If you succeed in adding new test cases,
they won't match the timestamp above.
=== A better solution
This commit adds a new option to the :use_vcr metadata tag:
it 'does something', use_vcr: :time_sensitive
Examples tagged in this way will automatically travel back to exactly
when a VCR cassette was recorded prior to using it.
=== Discussion
You may notice that the VCR auto-record logic was moved
from a helper module to a simple block.
The helper module was intended to improve organization and clean up the
config code, but due to context/binding issues, it was not possible to
call `travel_to` from inside the helper module.
2020-03-06 10:44:59 +00:00
request_profile = [
:method ,
:uri ,
vcr_options . include? ( :with_oauth_headers ) ? :oauth_headers : nil
] . compact
VCR . use_cassette ( cassette_path . join ( cassette_name ) , match_requests_on : request_profile ) do | cassette |
if vcr_options . include? ( :time_sensitive ) && ! cassette . recording?
travel_to ( cassette . http_interactions . interactions . first . recorded_at )
end
example . run
end
end
2020-11-24 16:20:57 +00:00
config . around ( :each , use_vcr : true ) do | example |
RSpec :: Support :: VCRHelper . inject_advisory ( example )
end
config . around ( :each , use_vcr : true ) do | example |
RSpec :: Expectations :: VCRHelper . inject_advisory ( example )
end
Maintenance: Enhance VCR helper with better failure messages
1ebddff95 added an RSpec metadata flag to simplify the use of VCR:
describe 'super cool feature', :use_vcr do
it 'totally works' { ... }
end
Under the hood, this option automatically generates filenames
for the VCR cassettes it uses on each example it's applied to.
These filenames are based on the example descriptions;
for the sample block above, the resulting filename would be
super_cool_feature_totally_works.yml
This introduces the risk of test regressions
that could be extremely confusing and hard to debug,
simply because someone changed the example description.
This commit injects custom error messages into RSpec
to elucidate this problem and avoid needless debugging.
=== Design challenges
This error message injection uses Ruby's Module#prepend
to monkey-patch methods defined in RSpec,
meaning that the changes are coupled to RSpec's implementation.
In short, if the implementation changes,
this custom error messaging could break.
Specifically, there are two different modes of failure in RSpec,
and the custom error was thus injected in two corresponding places:
* `.notify_failure` for normal test failure
(i.e., when an expectation is not met); and
* `.handle_matcher` for exceptions raised during a test.
2019-11-18 15:27:21 +00:00
end