2026-10-10 19:18:46 [scrapy.utils.log] INFO: Scrapy 2.19.0 started (bot: audit_engine)
2026-10-10 19:18:46 [scrapy.utils.log] INFO: Versions:
{'lxml': '6.1.3',
 'libxml2': '2.14.6',
 'cssselect': '1.5.0',
 'parsel': '1.12.1',
 'w3lib': '2.5.0',
 'Twisted': '26.4.0',
 'Python': '3.13.5 (main, Aug 10 2026, 12:06:59) [GCC 14.2.0]',
 'pyOpenSSL': '26.4.0 (OpenSSL 4.0.3 29 Sep 2026)',
 'cryptography': '50.0.2',
 'Platform': 'Linux-6.8.0-138-generic-x86_64-with-glibc2.41'}
2026-10-10 19:18:46 [scrapy.crawler] DEBUG: Using AsyncCrawlerProcess
2026-10-10 19:18:46 [asyncio] DEBUG: Using selector: EpollSelector
2026-10-10 19:18:46 [scrapy.addons] INFO: Enabled addons:
[]
2026-10-10 19:18:46 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2026-10-10 19:18:46 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2026-10-10 19:18:46 [scrapy.extensions.telnet] INFO: Telnet Password: eae681e1d6fdc7ad
2026-10-10 19:18:47 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.logcount.LogCount',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.throttle.AutoThrottle',
 'scrapy.extensions.remote_control.RemoteControl']
2026-10-10 19:18:47 [scrapy.crawler] INFO: Overridden settings:
{'AUTOTHROTTLE_ENABLED': True,
 'AUTOTHROTTLE_MAX_DELAY': 10.0,
 'AUTOTHROTTLE_START_DELAY': 1.0,
 'BOT_NAME': 'audit_engine',
 'CONCURRENT_REQUESTS_PER_DOMAIN': 2,
 'COOKIES_ENABLED': False,
 'DOWNLOAD_DELAY': 1.0,
 'LOG_FILE': '/var/lib/scrapyd/logs/scraping_test_project/auditor_bot/2026-10-10T19_18_43.log',
 'NEWSPIDER_MODULE': 'audit_engine.spiders',
 'SPIDER_MODULES': ['audit_engine.spiders']}
2026-10-10 19:18:48 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2026-10-10 19:18:48 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.start.StartSpiderMiddleware',
 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware',
 'scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware']
2026-10-10 19:18:48 [scrapy.middleware] INFO: Enabled item pipelines:
['audit_engine.pipelines.AuditMariaDBPipeline']
2026-10-10 19:18:48 [py.warnings] WARNING: /usr/local/lib/python3.13/dist-packages/scrapy/pipelines/__init__.py:60: ScrapyDeprecationWarning: AuditMariaDBPipeline.open_spider() requires a spider argument, this is deprecated and the argument will not be passed in future Scrapy versions. If you need to access the spider instance you can save the crawler instance passed to from_crawler() and use its spider attribute.
  self._check_mw_method_spider_arg(mw.open_spider)

2026-10-10 19:18:48 [py.warnings] WARNING: /usr/local/lib/python3.13/dist-packages/scrapy/pipelines/__init__.py:63: ScrapyDeprecationWarning: AuditMariaDBPipeline.close_spider() requires a spider argument, this is deprecated and the argument will not be passed in future Scrapy versions. If you need to access the spider instance you can save the crawler instance passed to from_crawler() and use its spider attribute.
  self._check_mw_method_spider_arg(mw.close_spider)

2026-10-10 19:18:48 [py.warnings] WARNING: /usr/local/lib/python3.13/dist-packages/scrapy/pipelines/__init__.py:66: ScrapyDeprecationWarning: AuditMariaDBPipeline.process_item() requires a spider argument, this is deprecated and the argument will not be passed in future Scrapy versions. If you need to access the spider instance you can save the crawler instance passed to from_crawler() and use its spider attribute.
  self._check_mw_method_spider_arg(mw.process_item)

2026-10-10 19:18:48 [scrapy.core.engine] INFO: Spider opened
2026-10-10 19:18:48 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2026-10-10 19:18:48 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2026-10-10 19:18:48 [scrapy.extensions.remote_control] INFO: Remote control HTTP server listening on port 43975 (job 32-54aae9f7ebf145e3962526373332163c)
2026-10-10 19:18:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com> (referer: None)
2026-10-10 19:18:50 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:18:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-2.html> (referer: https://books.toscrape.com)
2026-10-10 19:18:50 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:18:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-3.html> (referer: https://books.toscrape.com/catalogue/page-2.html)
2026-10-10 19:18:51 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:18:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-4.html> (referer: https://books.toscrape.com/catalogue/page-3.html)
2026-10-10 19:18:52 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:18:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-5.html> (referer: https://books.toscrape.com/catalogue/page-4.html)
2026-10-10 19:18:53 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:18:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-6.html> (referer: https://books.toscrape.com/catalogue/page-5.html)
2026-10-10 19:18:54 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:18:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-7.html> (referer: https://books.toscrape.com/catalogue/page-6.html)
2026-10-10 19:18:55 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:18:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-8.html> (referer: https://books.toscrape.com/catalogue/page-7.html)
2026-10-10 19:18:57 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:18:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-9.html> (referer: https://books.toscrape.com/catalogue/page-8.html)
2026-10-10 19:18:58 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:18:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-10.html> (referer: https://books.toscrape.com/catalogue/page-9.html)
2026-10-10 19:18:59 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:18:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-11.html> (referer: https://books.toscrape.com/catalogue/page-10.html)
2026-10-10 19:19:01 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-12.html> (referer: https://books.toscrape.com/catalogue/page-11.html)
2026-10-10 19:19:02 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-13.html> (referer: https://books.toscrape.com/catalogue/page-12.html)
2026-10-10 19:19:03 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-14.html> (referer: https://books.toscrape.com/catalogue/page-13.html)
2026-10-10 19:19:04 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-15.html> (referer: https://books.toscrape.com/catalogue/page-14.html)
2026-10-10 19:19:05 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-16.html> (referer: https://books.toscrape.com/catalogue/page-15.html)
2026-10-10 19:19:07 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-17.html> (referer: https://books.toscrape.com/catalogue/page-16.html)
2026-10-10 19:19:08 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-18.html> (referer: https://books.toscrape.com/catalogue/page-17.html)
2026-10-10 19:19:10 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-19.html> (referer: https://books.toscrape.com/catalogue/page-18.html)
2026-10-10 19:19:11 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-20.html> (referer: https://books.toscrape.com/catalogue/page-19.html)
2026-10-10 19:19:13 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-21.html> (referer: https://books.toscrape.com/catalogue/page-20.html)
2026-10-10 19:19:14 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-22.html> (referer: https://books.toscrape.com/catalogue/page-21.html)
2026-10-10 19:19:15 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-23.html> (referer: https://books.toscrape.com/catalogue/page-22.html)
2026-10-10 19:19:17 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-24.html> (referer: https://books.toscrape.com/catalogue/page-23.html)
2026-10-10 19:19:18 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-25.html> (referer: https://books.toscrape.com/catalogue/page-24.html)
2026-10-10 19:19:20 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-26.html> (referer: https://books.toscrape.com/catalogue/page-25.html)
2026-10-10 19:19:21 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-27.html> (referer: https://books.toscrape.com/catalogue/page-26.html)
2026-10-10 19:19:22 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-28.html> (referer: https://books.toscrape.com/catalogue/page-27.html)
2026-10-10 19:19:23 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-29.html> (referer: https://books.toscrape.com/catalogue/page-28.html)
2026-10-10 19:19:25 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-30.html> (referer: https://books.toscrape.com/catalogue/page-29.html)
2026-10-10 19:19:26 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-31.html> (referer: https://books.toscrape.com/catalogue/page-30.html)
2026-10-10 19:19:27 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-32.html> (referer: https://books.toscrape.com/catalogue/page-31.html)
2026-10-10 19:19:28 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-33.html> (referer: https://books.toscrape.com/catalogue/page-32.html)
2026-10-10 19:19:30 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-34.html> (referer: https://books.toscrape.com/catalogue/page-33.html)
2026-10-10 19:19:31 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-35.html> (referer: https://books.toscrape.com/catalogue/page-34.html)
2026-10-10 19:19:32 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-36.html> (referer: https://books.toscrape.com/catalogue/page-35.html)
2026-10-10 19:19:34 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-37.html> (referer: https://books.toscrape.com/catalogue/page-36.html)
2026-10-10 19:19:35 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-38.html> (referer: https://books.toscrape.com/catalogue/page-37.html)
2026-10-10 19:19:36 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-39.html> (referer: https://books.toscrape.com/catalogue/page-38.html)
2026-10-10 19:19:37 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-40.html> (referer: https://books.toscrape.com/catalogue/page-39.html)
2026-10-10 19:19:38 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-41.html> (referer: https://books.toscrape.com/catalogue/page-40.html)
2026-10-10 19:19:39 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-42.html> (referer: https://books.toscrape.com/catalogue/page-41.html)
2026-10-10 19:19:40 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-43.html> (referer: https://books.toscrape.com/catalogue/page-42.html)
2026-10-10 19:19:41 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-44.html> (referer: https://books.toscrape.com/catalogue/page-43.html)
2026-10-10 19:19:42 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-45.html> (referer: https://books.toscrape.com/catalogue/page-44.html)
2026-10-10 19:19:43 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-46.html> (referer: https://books.toscrape.com/catalogue/page-45.html)
2026-10-10 19:19:44 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-47.html> (referer: https://books.toscrape.com/catalogue/page-46.html)
2026-10-10 19:19:45 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-48.html> (referer: https://books.toscrape.com/catalogue/page-47.html)
2026-10-10 19:19:47 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-49.html> (referer: https://books.toscrape.com/catalogue/page-48.html)
2026-10-10 19:19:48 [charset_normalizer] DEBUG: Encoding detection: Unable to determine any suitable charset.
2026-10-10 19:19:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://books.toscrape.com/catalogue/page-50.html> (referer: https://books.toscrape.com/catalogue/page-49.html)
2026-10-10 19:19:48 [scrapy.core.engine] INFO: Closing spider (finished)
2026-10-10 19:19:48 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (0 items) in: file:///var/lib/scrapyd/items/scraping_test_project/auditor_bot/2026-10-10T19_18_43.jl
2026-10-10 19:19:48 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 15429,
 'downloader/request_count': 50,
 'downloader/request_method_count/GET': 50,
 'downloader/response_bytes': 277193,
 'downloader/response_count': 50,
 'downloader/response_status_count/200': 50,
 'elapsed_time_seconds': 59.93490502703935,
 'feedexport/success_count/FileFeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2026, 10, 10, 19, 19, 48, 111464, tzinfo=datetime.timezone.utc),
 'httpcompression/response_bytes': 2542961,
 'httpcompression/response_count': 50,
 'items_per_minute': 0.0,
 'log_count/DEBUG': 99,
 'log_count/INFO': 4,
 'memusage/max': 87355392,
 'memusage/startup': 87355392,
 'request_depth_max': 49,
 'response_received_count': 50,
 'responses_per_minute': 50.847457627118644,
 'scheduler/dequeued': 50,
 'scheduler/dequeued/memory': 50,
 'scheduler/enqueued': 50,
 'scheduler/enqueued/memory': 50,
 'start_time': datetime.datetime(2026, 10, 10, 19, 18, 48, 176477, tzinfo=datetime.timezone.utc)}
2026-10-10 19:19:48 [scrapy.core.engine] INFO: Spider closed (finished)