Cloudfare stopping downloads

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Brame
    Junior Member
    • Dec 2022
    • 7

    #1

    Cloudfare stopping downloads

    Hi, I have a python script to bulk download historical data, however today I just keep hitting a Cloudflare "Please wait" page meaning that 99% of my downloads fail and I'm left with a file that looks like the below. Is there an issue?

    Note I can't paste the word "script" without getting a 403 so had to edit it to say s.c.r.i.p.t below

    Code:
    <!DOCTYPE html>
    <html lang="en" ng-app="myApp" class="no-js">
    <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <title>Exchange Historical Data</title>
    <meta name="des.c.r.i.p.tion" content="">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="app/style.css">
    <s.c.r.i.p.t>
        var dataLayer = [];
        dataLayer.push({
          'event': 'ga_pageLoad',                                    //denotes the signal to gtm that a page has loaded.
          'ga_target_property': 'UA-96801787-1',                 //denotes the appropriate ga tracking id for this product and device type.
          'acc_id': null,//[account ID],                                    //denotes the numeric account id for the user at page load.
          'vertical': 'ecomm',                                    //denotes the channel/vertical of the loaded page. Values will include 'sports', 'games', 'promotions', 'ecomm', etc.
          'product': 'historicprices',                        //denotes the platform/product of the loaded page. Values will include sportsbook smx, sportsbook sbw, etc.
          'locale': null,// [locale],                        //denotes the language of the user at time of page load, as set by the language selector. Values will only be in iso 639-1 format.
          'reg_status': null,// [reg status],                    //denotes the registration status of the user at time of page load. Values will only be 'registered' or 'unregistered'. THIS IS NOT A FUNCTION OF LOGIN STATUS.
          'login_status': null,//[login status],                         //denotes the login status of the user at time of page load. Values will only be 'logged in' or 'logged out'.
          'jurisdiction': null,//[jurisdiction],                         //denotes the jurisdicton from where the user is operating in and subject to. Values will include 'uk/irl', 'international', 'spain', 'italy, etc.
          'orientation': 'desktop',                            //denotes the orientation of the screen at time of page load. Values will only be 'portrait', 'landscape' or 'desktop'.
          'app_id': 'historicdata',                                      //denotes the alphanumeric app id of the app, app key or cougar key at time of page load.
          'country': null,//[country],                                     //denotes the user's country as dictated by geo ip lookup.
          'bucket_id': null,//[bucket_id],                                //used when using mantis buckets for throttling different versions.
          'test_group': null,//[test_group],                              //used for split testing.
          'test_name': null,//[test_name],                                //used for split testing.
          'efid': null,//[efid],                                         //marketing parameter. Variables are stored as cookies on promotions pages on betfair.com and populate dimensions in gtm schema.
          'rfr': null,//[rfr],                            //marketing parameter. Variables are stored as cookies on promotions pages on betfair.com and populate dimensions in gtm schema.
          'pid': null,//[pid],                            //marketing parameter. Variables are stored as cookies on promotions pages on betfair.com and populate dimensions in gtm schema.
          'ttp': null,//[ttp],                            //marketing parameter. Variables are stored as cookies on promotions pages on betfair.com and populate dimensions in gtm schema.
          'bid': null,//[bid],                            //marketing parameter. Variables are stored as cookies on promotions pages on betfair.com and populate dimensions in gtm schema.
          'pid': null,//[pid],                            //marketing parameter. Variables are stored as cookies on promotions pages on betfair.com and populate dimensions in gtm schema.
          'promo_code': null,//[promo_code],                             //marketing parameter. Value is set as cookie at registration and populates the gtm schema.
          'mpch': null,//[mpch],                            //marketing parameter. Variables are stored as cookies on promotions pages on betfair.com and populate dimensions in gtm schema.
          'pla': null,//[pla],                            //marketing parameter. Variables are stored as cookies on promotions pages on betfair.com and populate dimensions in gtm schema.
          'clkid': null,//[clkid],                        //marketing parameter. Variables are stored as cookies on promotions pages on betfair.com and populate dimensions in gtm schema.
          'sid': null,//[sid],                            //marketing parameter. Variables are stored as cookies on promotions pages on betfair.com and populate dimensions in gtm schema.
          'StickyTags': null,//[StickyTags],                    //marketing parameter. Variables are stored as cookies on promotions pages on betfair.com and populate dimensions in gtm schema.
          'TrackingTags': null,// [trackingTags],                    //marketing parameter. Variables are stored as cookies on promotions pages on betfair.com and populate dimensions in gtm schema.
          'device_model': "chrome",                             //denotes the marketing name of device as according to device detection service (i.e. iphone 6s, galaxy s7 edge etc.)
          'brand': "bf"                                         //denotes the bf or pp brand.
        });
      </s.c.r.i.p.t>
    
    <s.c.r.i.p.t>
        (function (w, d, s, l, i) {
          w[l] = w[l] || []; w[l].push({
            'gtm.start':
              new Date().getTime(), event: 'gtm.js'
          }); var f = d.getElementsByTagName(s)[0],
            j = d.createElement(s), dl = l != 'dataLayer' ? '&l=' + l : ''; j.async = true; j.src =
              'https://www.googletagmanager.com/gtm.js?id=' + i + dl; f.parentNode.insertBefore(j, f);
        })(window, document, 's.c.r.i.p.t', 'dataLayer', 'GTM-NN99G4H');
      </s.c.r.i.p.t>
    
    </head>
    <body>
    
    <nos.c.r.i.p.t>
        <iframe src="https://www.googletagmanager.com/ns.html?id=GTM-NN99G4H" height="0" width="0" style="display:none;visibility:hidden"></iframe>
      </nos.c.r.i.p.t>
    
    <xdw-header></xdw-header>
    <div class="main">
    <div ng-view class="main-container"></div>
    </div>
    <div class="clear-fix"></div>
    <xdw-footer></xdw-footer>
    <p style="text-align:center"> Build 2018-10-03</p>
    <s.c.r.i.p.t src="app/app.js"></s.c.r.i.p.t>
    <s.c.r.i.p.t src="app/partials.js"></s.c.r.i.p.t>
    <s.c.r.i.p.t defer src="https://static.cloudflareinsights.com/beacon.min.js/vaafb692b2aea4879b33c060e79fe94621666317369993" integrity="sha512-0ahDYl866UMhKuYcW078ScMalXqtFJggm7TmlUtp0UlD4eQk0Ixfnm5ykXKvGJNFjLMoortdseTfsRT8oCfgGA==" data-cf-beacon='{"rayId":"7a09631538818885","token":"d048f65d27954a24aa6b1d7d2ddcb256","version":"2023.2.0","si":100}' crossorigin="anonymous"></s.c.r.i.p.t>
    </body>
    </html>​
    ​
  • BetfairDeveloperProgram
    Administrator
    • Oct 2008
    • 680

    #2
    Hi Brame

    If this issue is still occurring, please can you provide details of the specific request your making to the Historical Data API?

    Kind Regards

    Neil

    Comment

    • Brame
      Junior Member
      • Dec 2022
      • 7

      #3
      Yeah it's still happening, I'm getting it trying to download specific files, but I'm also getting it just hitting this URL:

      https://historicdata.betfair.com/api...oadListOfFiles

      I'm also getting the following cloudfare page:
      Code:
      historicdata.betfair.com needs to review the security of your connection before proceeding.

      Comment

      • BetfairDeveloperProgram
        Administrator
        • Oct 2008
        • 680

        #4
        Hi Brame

        We recently implemented rate limiting on this service and believe this is what's causing the issue you are experiencing.

        This rate limiting will take affect if 100 requests in less than 10 seconds.

        Therefore, please can you check that you aren't exceeding that limit and if so, make changes to ensure that you don't breach this limit?

        Kind Regards

        Neil

        Comment

        • Brame
          Junior Member
          • Dec 2022
          • 7

          #5
          Unfortunately that doesn't seem to be the issue. I just tried updating my script to be single threaded and even adding a 5 second delay between downloading each file returned, I'm still getting the original error

          Comment

          • pillbox
            Junior Member
            • Oct 2014
            • 7

            #6
            I'm experiencing the same problems.

            Comment

            • BetfairDeveloperProgram
              Administrator
              • Oct 2008
              • 680

              #7
              Hi pillbox

              Thanks for letting us know. We are investigating further.

              Please can you provide a specific example including the timestamp?

              Kind Regards

              Neil​

              Comment

              • Brame
                Junior Member
                • Dec 2022
                • 7

                #8
                I just tried this morning to get some timestamps for you, but I'm just getting
                Code:
                ngErrorRedirect
                repeatedly now

                Comment

                • BetfairDeveloperProgram
                  Administrator
                  • Oct 2008
                  • 680

                  #9
                  Thanks Brame, please can you provide an example request that returns this error. Please can you also check if the same download fails when attempting via the historical data website front end?

                  Neil

                  Comment

                  • Brame
                    Junior Member
                    • Dec 2022
                    • 7

                    #10
                    Hi Neil - please see below a full dump of the request and response. I have removed the ssoid, but double checked it using https://docs.developer.betfair.com/v...nt-operations/ and it was correct.

                    UPDATE - this is working from the download page, but it's a little temperamental. sometimes the UI just crashes when trying to select the Market Type or File Type and it won't let me select them so I have to refresh and try again.

                    Code:
                    < POST /api/DownloadListOfFiles HTTP/1.1
                    < Host: historicdata.betfair.com
                    < User-Agent: python-requests/2.28.1
                    < Accept-Encoding: gzip, deflate
                    < Accept: */*
                    < Connection: keep-alive
                    < content-type: application/json
                    < ssoid: REMOVED
                    < Content-Length: 479
                    <
                    < {
                                    "fromDay" : 24,
                                    "fromMonth": 1,
                                    "fromYear" : 2023,
                                    "toDay": 24,
                                    "toMonth" : 1,
                                    "toYear" : 2023,
                                    "sport":"Soccer",
                                    "plan":"Basic Plan",
                                    "eventId": null,
                                    "eventName": null,
                                    "marketTypesCollection": ["MATCH_ODDS"],
                                    "countriesCollection" : [],
                                    "fileTypeCollection" : ["M"]}
                    
                    > HTTP/1.1 302 Found
                    > Date: Mon, 06 Mar 2023 11:30:34 GMT
                    > Content-Type: text/html; charset=utf-8
                    > Transfer-Encoding: chunked
                    > Connection: keep-alive
                    > Cache-Control: private
                    > Location: https://historicdata.betfair.com/home/ngErrorRedirect
                    > Vary: Cookie
                    > Access-Control-Allow-Origin: *
                    > Access-Control-Allow-Methods: *
                    > Access-Control-Allow-Headers: Accept, Authorization,Content-Type
                    > CF-Cache-Status: DYNAMIC
                    > Server-Timing: cf-q-config;dur=6.0000020312145e-06
                    > Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=81g59i3j50cQ8Ki5y7lc4cUptEDpa3199X79x%2BU6JGLiru2LM5TqfaQyBt8gaBYzPaqQOTD1%2BikQ0OM%2BKhCc6synK4LLd%2FflyILpoICfiwDOejnq2dRfHpAYsDdwFW37Pn58v0v%2BuEQvoQ%3D%3D"}],"group":"cf-nel","max_age":604800}
                    > NEL: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}
                    > Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
                    > X-Content-Type-Options: nosniff
                    > Server: cloudflare
                    > CF-RAY: 7a3a4476cc297330-LHR
                    > alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
                    >
                    
                    <html>
                       <head>
                          <title>
                             Object moved
                          </title>
                       </head>
                       <body>
                          <h2>Object moved to <a href="https://historicdata.betfair.com/home/ngErrorRedirect">here</a>.</h2>
                          <s.c.r.i.p.t defer src="https://static.cloudflareinsights.com/beacon.min.js/vaafb692b2aea4879b33c060e79fe94621666317369993" integrity="sha512-0ahDYl866UMhKuYcW078ScMalXqtFJggm7TmlUtp0UlD4eQk0Ixfnm5ykXKvGJNFjLMoortdseTfsRT8oCfgGA==" data-cf-beacon='{"rayId":"7a3a4476cc297330","token":"d048f65d27954a24aa6b1d7d2ddcb256","version":"2023.2.0","si":100}' crossorigin="anonymous"></s.c.r.i.p.t>
                       </body>
                    </html>​
                    
                    < GET /home/ngErrorRedirect HTTP/1.1
                    < Host: historicdata.betfair.com
                    < User-Agent: python-requests/2.28.1
                    < Accept-Encoding: gzip, deflate
                    < Accept: */*
                    < Connection: keep-alive
                    < ssoid: REMOVED
                    <
                    
                    > HTTP/1.1 200 OK
                    > Date: Mon, 06 Mar 2023 11:30:34 GMT
                    > Content-Type: text/html; charset=utf-8
                    > Transfer-Encoding: chunked
                    > Connection: keep-alive
                    > Cache-Control: private
                    > Vary: Cookie,Accept-Encoding
                    > Access-Control-Allow-Origin: *
                    > Access-Control-Allow-Methods: *
                    > Access-Control-Allow-Headers: Accept, Authorization,Content-Type
                    > CF-Cache-Status: DYNAMIC
                    > Server-Timing: cf-q-config;dur=6.0000020312145e-06
                    > Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=ddC531JlQGhPZrn7TF0bF2jtcpC6vqjimSvfGoOakqsIrRAMbe8dbknNPHOuEQ6G59Oai5HKJdnTbOtUZZevSbGN748zrX2q%2BrVMr4pduAL3RlkoH48VdZqxLOOeDkTYIUAx4iBguqlf%2Bg%3D%3D"}],"group":"cf-nel","max_age":604800}
                    > NEL: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}
                    > Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
                    > X-Content-Type-Options: nosniff
                    > Server: cloudflare
                    > CF-RAY: 7a3a44779d247330-LHR
                    > Content-Encoding: gzip
                    > alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
                    >  
                    
                    <!DOCTYPE html>
                    <html>
                       <head>
                          <meta name="viewport" content="width=device-width" />
                          <title>ngErrorRedirect</title>
                       </head>
                       <body>
                          <div>
                             Error
                          </div>
                          <s.c.r.i.p.t defer src="https://static.cloudflareinsights.com/beacon.min.js/vaafb692b2aea4879b33c060e79fe94621666317369993" integrity="sha512-0ahDYl866UMhKuYcW078ScMalXqtFJggm7TmlUtp0UlD4eQk0Ixfnm5ykXKvGJNFjLMoortdseTfsRT8oCfgGA==" data-cf-beacon='{"rayId":"7a3a44779d247330","token":"d048f65d27954a24aa6b1d7d2ddcb256","version":"2023.2.0","si":100}' crossorigin="anonymous"></s.c.r.i.p.t>
                       </body>
                    </html>​​
                    Last edited by Brame; 06-03-2023, 12:43 PM.

                    Comment

                    • BetfairDeveloperProgram
                      Administrator
                      • Oct 2008
                      • 680

                      #11
                      Thanks Brame - we are investigating and I'll let you know as soon as I have further info.

                      Neil

                      Comment

                      • Brame
                        Junior Member
                        • Dec 2022
                        • 7

                        #12
                        Is there any update on this?

                        Comment

                        • BetfairDeveloperProgram
                          Administrator
                          • Oct 2008
                          • 680

                          #13
                          Hi Brame,

                          Sorry for the delay.

                          The issue is being investigated by our dev team (ref #UKIINC-4769) If you can provide further examples we can add these to the open incident as they'll help with the ongoing investigation.

                          Also if you can confirm the frequency failure rate as a % of failed vs successful requests this will also be useful.

                          Kind Regards

                          Neil​

                          Comment

                          • Brame
                            Junior Member
                            • Dec 2022
                            • 7

                            #14
                            Sorry I've been away for a few days but just come back and retried this and it's still failing.

                            I just ran a test downloading 6,325 files, 3,132 failed - so 49.52% failed...

                            I have thousands of examples, in the same format as above as a full requests and responses but is there a better way to share them?

                            Comment

                            • BetfairDeveloperProgram
                              Administrator
                              • Oct 2008
                              • 680

                              #15
                              Hi Brame,

                              Thanks for the feedback, please can you send examples of failed requests to Developer Support via https://support.developer.betfair.co...s/requests/new or by email to bdp@betfair.com

                              It would be good to understand if all the failed requests are of the type as shown the earlier reports or if you are also still seeing the originally reported HTML error too

                              Thanks

                              Neil

                              Comment

                              Working...
                              X