2026-10-10 19:14:31 [scrapy.utils.log] INFO: Scrapy 2.19.0 started (bot: audit_engine)
2026-10-10 19:14:31 [scrapy.utils.log] INFO: Versions:
{'lxml': '6.1.3',
 'libxml2': '2.14.6',
 'cssselect': '1.5.0',
 'parsel': '1.12.1',
 'w3lib': '2.5.0',
 'Twisted': '26.4.0',
 'Python': '3.13.5 (main, Aug 10 2026, 12:06:59) [GCC 14.2.0]',
 'pyOpenSSL': '26.4.0 (OpenSSL 4.0.3 29 Sep 2026)',
 'cryptography': '50.0.2',
 'Platform': 'Linux-6.8.0-138-generic-x86_64-with-glibc2.41'}
2026-10-10 19:14:31 [scrapy.crawler] DEBUG: Using AsyncCrawlerProcess
2026-10-10 19:14:31 [asyncio] DEBUG: Using selector: EpollSelector
2026-10-10 19:14:31 [scrapy.addons] INFO: Enabled addons:
[]
2026-10-10 19:14:31 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2026-10-10 19:14:31 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2026-10-10 19:14:31 [scrapy.extensions.telnet] INFO: Telnet Password: e1a696517598ca0d
2026-10-10 19:14:31 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.logcount.LogCount',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.throttle.AutoThrottle',
 'scrapy.extensions.remote_control.RemoteControl']
2026-10-10 19:14:31 [scrapy.crawler] INFO: Overridden settings:
{'AUTOTHROTTLE_ENABLED': True,
 'AUTOTHROTTLE_MAX_DELAY': 10.0,
 'AUTOTHROTTLE_START_DELAY': 1.0,
 'BOT_NAME': 'audit_engine',
 'CONCURRENT_REQUESTS_PER_DOMAIN': 2,
 'COOKIES_ENABLED': False,
 'DOWNLOAD_DELAY': 1.0,
 'LOG_FILE': '/var/lib/scrapyd/logs/crawltest/auditor_bot/2026-10-10T19_14_27.log',
 'NEWSPIDER_MODULE': 'audit_engine.spiders',
 'SPIDER_MODULES': ['audit_engine.spiders']}
2026-10-10 19:14:32 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2026-10-10 19:14:32 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.start.StartSpiderMiddleware',
 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware',
 'scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware']
2026-10-10 19:14:32 [scrapy.middleware] INFO: Enabled item pipelines:
['audit_engine.pipelines.AuditMariaDBPipeline']
2026-10-10 19:14:32 [py.warnings] WARNING: /usr/local/lib/python3.13/dist-packages/scrapy/pipelines/__init__.py:60: ScrapyDeprecationWarning: AuditMariaDBPipeline.open_spider() requires a spider argument, this is deprecated and the argument will not be passed in future Scrapy versions. If you need to access the spider instance you can save the crawler instance passed to from_crawler() and use its spider attribute.
  self._check_mw_method_spider_arg(mw.open_spider)

2026-10-10 19:14:32 [py.warnings] WARNING: /usr/local/lib/python3.13/dist-packages/scrapy/pipelines/__init__.py:63: ScrapyDeprecationWarning: AuditMariaDBPipeline.close_spider() requires a spider argument, this is deprecated and the argument will not be passed in future Scrapy versions. If you need to access the spider instance you can save the crawler instance passed to from_crawler() and use its spider attribute.
  self._check_mw_method_spider_arg(mw.close_spider)

2026-10-10 19:14:32 [py.warnings] WARNING: /usr/local/lib/python3.13/dist-packages/scrapy/pipelines/__init__.py:66: ScrapyDeprecationWarning: AuditMariaDBPipeline.process_item() requires a spider argument, this is deprecated and the argument will not be passed in future Scrapy versions. If you need to access the spider instance you can save the crawler instance passed to from_crawler() and use its spider attribute.
  self._check_mw_method_spider_arg(mw.process_item)

2026-10-10 19:14:32 [scrapy.core.engine] INFO: Spider opened
2026-10-10 19:14:32 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2026-10-10 19:14:32 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2026-10-10 19:14:32 [scrapy.extensions.remote_control] INFO: Remote control HTTP server listening on port 38329 (job 28-7044ef3f65f2433c983fcf06ed85a905)
2026-10-10 19:14:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://toscrape.com> (referer: None)
2026-10-10 19:14:32 [scrapy.core.engine] INFO: Closing spider (finished)
2026-10-10 19:14:32 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (0 items) in: file:///var/lib/scrapyd/items/crawltest/auditor_bot/2026-10-10T19_14_27.jl
2026-10-10 19:14:32 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 223,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 1213,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 0.6640594438649714,
 'feedexport/success_count/FileFeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2026, 10, 10, 19, 14, 32, 842789, tzinfo=datetime.timezone.utc),
 'httpcompression/response_bytes': 3939,
 'httpcompression/response_count': 1,
 'items_per_minute': None,
 'log_count/DEBUG': 1,
 'log_count/INFO': 4,
 'memusage/max': 88391680,
 'memusage/startup': 88391680,
 'response_received_count': 1,
 'responses_per_minute': None,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2026, 10, 10, 19, 14, 32, 178725, tzinfo=datetime.timezone.utc)}
2026-10-10 19:14:32 [scrapy.core.engine] INFO: Spider closed (finished)